Hi,
wow, your have a lot of products, it's depending how many user are surfing on the same time and how big you WebHosting Server is what can you try.
I think you have already turned on caching. Then you can also try to tweak some settings from php and mySQL to get more performance.
For mySQL you can try to run this one tuning-primer.sh it will give you hints how to config you mySQL DB you can find some info on google about it. Itīs helping some times really good.
And you can try this module
http://codingmall.com/de/component/content/article/6-joomla/65-speed-up-virtuemartGood luck!