It's nice they make the SWR meters so small and self contained now! Still, inserting it makes the antenna longer. This would most impact the highest frequency band readings.
The way the transmitters are designed, they roll back power when they see bad loads (usually - can't count on it though).
In any case, HT antennas are notoriously poor. In some cases, only the loss in the antenna makes the SWR "good." But usually they're good enough. If they aren't in a particular application, you can build a "jungle antenna" for the band in question, or make twin lead "j-pole" some of which work well on 2m and 3/4m.