Never had the pleasure of working the Charlottes, but I've heard they are no fun.
Newfoundland is no picnic, mainly becasue of the weather, but I'd have to say that the eastern Arctic and northern Labrador would have to qualify as the worse (and best) places places I've flown.
There is extremely rugged terrain, with relief that exceeds most of the hills in the west. The east side of Baffin around Auyuittuq park had cliffs that fall close to 7000' to the ocean. The Torngats, Kiglapaits and Kaumajets in Labrador are equally spectacular. Of course, the weather can be brutal beyond your wildest imagination.
But what I think makes these places the most difficult, even more so than those areas with more severe topography, is the incredible remoteness, lack of vegetation, and - when I was up there - lack of communication. I don't know how many times I'd be scooting along, picking my way through the weather, and realize that the nearest helicopter to come get me if I had a problem was 900 miles away, and had to come through the same wx to find me. 5680 was useless, and unless you were near a NWS site, Arctic Radio was not an option. The nearest full-on SAR capability was in Trenton ON.
With no trees and no shelter, even a failure to start for a dead battery could be a disaster if the weather got bad and nobody could get to you. Pretty serious stuff.