If you have a power meter, heart rate monitor, or you simply pay attention to how fast you ride, note how your performance drops off when you ride in the heat. It isn't just you; studies (such as this one) suggest that the heat may cause your power output to drop over 5%. To be precise, in a study done on the Australian National squad, their 30 minute power output at 73 degrees (345 watts) declined 6.5% (to 323 watts) when the heat climbed to 90 degrees. Note that these tests were done indoors, and did not account for wind chill.
The harder you ride, the more heat your body produces. If, for example, you're putting out a 400 watt effort, you generate a total of 2,000 watts--1,600 of which are wasted as heat.
Your body's ability to get rid of heat is certainly a limiting factor in your performance in the same way that your VO2 max, muscular strength and lactate threshold are (Nielsen, et al, 1997).
Just as there are ways you can minimize the effect of air resistence itself (through aerodynamic equipment and body positioning), there are ways you can minimize the effect of hot air:
(1) Drink extra fluid. Athletes can lose over 3 liters of water in an hour. That's more than you carry in your bottle holders. Drink up.
(2) Precooling. You've seen Garmin's cooling vest worn during their warmups. Several studies suggest that cooling the body before exertion in heat is effective.
(3) Head cooling. Several studies suggest the specific benefits of cooling your head while exercising.
As stated in my previous post, the Aspen Cooling Collar, worn during two 20 minute TT tests, did seem to keep me cooler. Without more extensive testing, I can't say definitively that the collar made me faster. I can reiterate that when I wore the collar I felt better than I normally do when putting in a hard workout in 95 degree heat. And for 15$, the collar is one of those small investments that probably offers a real, if unsexy, benefit.