Four factors affect your physical comfort, and high performance homes address each of them:
1. Temperature. This is one that gets most of the attention. Your home probably has heating and cooling systems that adjust the temperature of the house to bring it into the fairly narrow range that we humans find comfortable, typically about 70 to 80 degrees Fahrenheit. Most heating and cooling systems have a thermostat.
2. Humidity. In the southeastern US, this is a biggie because it's a humid climate. When the air is too humid, our bodies can't use evaporative cooling as effectively because the perspiration doesn't evaporate so easily. Air conditioners, then, have two jobs: Lowering the temperature and reducing the humidity in the air. Few homes have humidistats, though, and most air conditioners aren't sized appropriately for the moisture loads.
3. Air Movement. We all know the feeling of standing in front of the air conditioner vent and feeling the cold air rush over our bodies. We also know that a seat under the ceiling fan makes us feel cooler in the summer. In the winter, we don't want air movement across our skin because we're trying to keep our heat. With appropriate use of fans (along with humidity control), we can raise the temperature of the thermostat in the cooling season and still be comfortable. Stopping the drafts in winter keeps us warm.
4. Mean Radiant Temperature. Least understood of the four factors, radiant temperature has to do with how hot or cold all the surrounding surfaces are. In a bonus room with uninsulated kneewalls that have a blazing hot attic on the other side, it's going to be difficult to be cool enough no matter what the air temperature in that room is because those walls are blasting us with radiant heat. Likewise standing near uninsulated walls or uncovered single-pane windows in winter will make us feel cold even if the furnace is blasting because our body is radiating much more heat to those cold surfaces than they're radiating back.
To achieve comfort, you have to address all of these issues. You also have to remember that the house is a system, and that the building envelope and heating and cooling systems must be designed and installed appropriately and properly.