The difference is that you can always put on more clothing to deal with the cold. We had no power after Sandy for 4 days. Temps outside were in the 30s. I kept a pot of water boiling on the gas stove. That was good enough to keep us in the high 50s/low 60s indoors. With an extra layer of clothing, it was tolerable.
I remember as a kid going to school when the heat was out, even on days it was in the teens. The kids all just kept their coats on. On the other hand, with high temperatures there's very little you can do. Once all your clothing is removed, that's as good as it gets. As you noticed today, sweating has no cooling effect whatsoever when the humidity is high. In fact, the sweat makes you even more uncomfortable.