You should create three menus, one for each language and make sure that the correct language tag is assigned to each menu item you add to a menu. Make shure you add the same categories in all menus and then associate the menu items to the other languages in the Joomla menu manager.
The Joomla language switcher depends on menu items an their IDs. Switching languages in VirtueMart frontend views is identical to doing so in Joomla. When you are looking at 'Product X' in 'Category A' in Finnish and switch to English, it looks for a menu item 'Category A' in English. If it doesn't find that, the Joomla language switcher will fall back to the next higher level for which it finds a menu item in English, which might be your homepage. If you have a lot of products, you would have to create a lot of (hidden) menu items, if you want to associate products. I wouldn't recommend that.