Hi,
To upgrade from phpshop 1.2 pl3 :
1) install the components
2) before clicking upgrade from phpShop, you have to do an upgrade of your database (if you read the page before click on button you can read that, i did the same mistake

)
-> go to your joomla home, then > administrator > components > com_virtuemart > admin > sql
Then open the file which is 'upgrade from phpsop 1.2 pl3 to VM xxx'
select all the queries and launch it with phpMyAdmin (or another Mysql client). Becareful, replace the table prefix 'jos' with the ones you use (generally 'mos')
3) Now you can click on the upgrade button from the backend.
4) uninstall your phpshop modules
5) install the virtuemart modules
6) That's all, except if you use a specific menu to access your product categorie because all the categories ID has been changed. So don't forget to change them in your specific menu.
Hope it helps you
Ben