You might try, if you hate math, to do the adjustment numbers for the armholes and such, to select a smaller size than you would make for your measurements.

Socks are knitted tighter than 'recommended' gauge for yarn, because socks get a lot of terrible wear. Sweaters get worn, but nothing like getting rubbed a couple hundred times a wear against the heel of a shoe while sweating, so that's not really going to be an issue. :D