There are several things you can check. First, we don't have en_US language files for VirtueMart, By default, VirtueMart is provided with en_GB language files. All other translations, including Spanish are provided and maintained by volunteers. Unfortunately American VM users are too lazy to invest time to keep language files adapted for American English up to date. Most of them seem to be happy with using the en_GB files plus maybe some language overrides.
If you did configure Joomla with en_US, VirtueMart probably doesn't find any corresponding en_US language files.
You can enable the 'Debug Language' tool in Joomla configuration, to find out which language files are loaded when you look at your VM category and product pages.
Another check you can do is to go to 'Extensions' - 'Module Manager' - switch from 'site' to 'administrator' in the filter dropdown list and publish 'Multilingual Status'. Thereafter you can click on 'Multilanguage Status' in the status bar. If everything is green on the status page, your Joomla multilanguage setup should be correct and you can unpublish the 'Multilingual Status' module again. If something is red, you should solve that problem first.
In my own bilingual VM shop I'm using the provided en_GB language files and have the Joomla system language code plugin set to 'en' only (without 'US'). And I don't use the hreflang tag either. That way you can avoid that Google filters search results for the potential customers geo-location. In your case, Google might not list your site if somebody from the UK is searching for bermuda shorts.