This is sequence you should follow, when you need to update your virtuemart.
For VM2.0.4 and higher users:
Just install always directly the latest version over your old version.
For vm1.0 users:
Update to vm1.1 http://dev.virtuemart.net/projects/virtuemart/wiki/Migrate_Joomla!_10_%28incl_VirtueMart%29_to_Joomla!_15
There is no experience yet doing it directly to j1.7.
For vm1.1 users:
Use the steps described here http://dev.virtuemart.net/projects/virtuemart/wiki/Upgrading_VM1_to_VM2_with_provided_packages and/or look here http://forum.virtuemart.net/index.php?topic=95236.0
For RC2 users below M version:
update first to the M version (1.9.8M) http://forum.virtuemart.net/index.php?topic=93008.0
For RC2M users:
Update to RC3 http://forum.virtuemart.net/index.php?topic=94183.0
For people with the final,
just use the 2.0.4 in the extension installer of joomla.
You can maybe directly update any RC2 version directly with the final. I would give it a try with a backup in the hand. But when you get problems, then you should follows this sequence. But do not use a 2.0.x for this. The 2.0.x are only updating the 2.0.x series.
Of course, this is only important, when you want to save the data, in the other case just use the latest installer and use afterwards "Reset all tables ..", more explanation here http://dev.virtuemart.net/projects/virtuemart/wiki/Tools
a complete guide by user from j1.5 vm1.1 to j2.5 vm2.02 http://forum.virtuemart.net/index.php?topic=98485.0