Yes, to a certain degree it undermines the point of having good insulation (as in, really good - when the thermal mass of the air starts to matter). Modern "nearly zero-heating" houses have long heat exchangers in the vents to avoid the waste.