Author Topic: How many products can VM practically support?  (Read 1693 times)

behemoth

  • Jr. Member
  • **
  • Posts: 81
How many products can VM practically support?
« on: March 18, 2019, 16:39:20 pm »
I've seen the impressive bookstore with 8miliion products in (http://forum.virtuemart.net/index.php?topic=135859.msg473594#msg473594), but from the description there's a huge amount of custom development to enable this.

How many products would you expect VM to be able to handle out of the box?

We've currently got about 1.5 million products but this is going to keep growing and I'm struggling to provide a useable site at a decent speed. What should I consider to improve things? Is there an ideal category / structure? are many smaller categories better / worse than fewer large ones? We've been trying to implement BreakDesigns Custom Filters and applied a number of custom fields to each product. We've most recently tried adding all the products to a single category so that all products can be accessed through the filter. This doesn't appear to have been a success.

any feedback, advice, suggestions gratefully received. We're currently running 3.4.2 under Joomla 3.8.12 but I'm after general opinions.

Studio 42

  • Contributing Developer
  • Sr. Member
  • *
  • Posts: 4341
  • Joomla & Virtuemart developper
    • Studio 42 - Virtuemart & Joomla extentions
  • VirtueMart Version: 2.6 & 3
Re: How many products can VM practically support?
« Reply #1 on: March 19, 2019, 02:24:40 am »
I like break design, but if you want fast filtering, it's not the right solution.
I have added tags in my "shortcode" plugin, it's very faster because it only use integer(index) to get values.
See demo here https://pro.st42.fr/android and scroll down bottom right you see a "filter" button(this module is on beta test only).
See doc https://shop.st42.fr/en/shorcodes-english-doc#searchable
About category structure, it's not the biggest problem. most time the problem is about ordering and index. Using for eg. ordering by product name is slow (or short description even worse).
Customfields can slowdown the shop too, it's why you should disable it in category if possible and use other tricks to display additional informations in the category view or/and product modules.
Abusing of product > childs tree too deep can have same effect
Do not use the option "ignore" about product featured ... in the config, you end up with mysql slow queries.
Adding permanent module cache and optimise the codes....
But at end you need to do some test, because you have some other things to check eg. Joomla system and content plugins.

behemoth

  • Jr. Member
  • **
  • Posts: 81
Re: How many products can VM practically support?
« Reply #2 on: March 19, 2019, 10:19:30 am »
Hi Studio 42,

Thanks for your detailed response. I hope I can request clarification on some of your points:

Quote
I have added tags in my "shortcode" plugin, it's very faster because it only use integer(index) to get values.
   Am I right in thinking this works mostly like the existing custom fields? Is it compatible with existing custom fields? We use a number of parent / child products multivariant combinations.

Quote
See demo / doc
   These filters look really powerful. Do you offer a trial? When do you expect it to come out of beta?

Quote
Customfields can slowdown the shop too, it's why you should disable it in category if possible
   I'm sorry, I don't understand this; I thought customfields were only applied to products. Can you show me where to disable them for categories?

Quote
Abusing of product > childs tree too deep can have same effect
   Can you explain what you mean by 'abusing' them? Of our products, where there's a product variation (eg a large one and a small one), we use the product / child multivariants, so there would be a single, unpurchaseable parent product with two purchaseable children products. Is this a problem?

Quote
Do not use the option "ignore" about product featured ... in the config, you end up with mysql slow queries.
   Which option are you referring to here? Is this the issue with the 'Omit' setting under Shopfront settings?

Quote
Adding permanent module cache and optimise the codes....
   Can you recommend a cache? I used JotCache in the past, but the site was so big that it never got completely cached so I presumed it wasn't worth it.

Quote
other things to check eg. Joomla system and content plugins.
   How do you mean? It's a very basic Joomla install. Only a handful of articles to manage things like delievery and returns policies, terms & conditions etc.

I'm really excited to try to implement some of these things. I look forward to your reply!

balai

  • 3rd party VirtueMart Developer
  • Full Member
  • *
  • Posts: 1415
Re: How many products can VM practically support?
« Reply #3 on: March 19, 2019, 11:51:07 am »
To my opinion, VM and Joomla is not the best option to handle a million products shop.
If you want to do that you should be prepared for some modifications and addition of extra layers.

I don't think that trying a different custom field or a filtering system can boost your shop spectacularly.
Speaking from personal experience. I work almost 10 years in optimizing shops.

You will possibly need to cluster your database. Running such a database in 1 machine is possibly not very good option.
Examining that from performance and stability context.

PHP implemented as php-fpm. I would prefer ngninx over apache

Also you will probably need some caching layers e.g. OpCache for php precompiling and Varnish or Reddis for HTTP caching.

ElasticSearch or Solr can be used for fast (outsourced) search and filtering.

VM itself may need some query optimization, to get rid unused joins.

Beyond that you will need a very strong server or even a better a cloud solution.

After all, you will need to constantly monitor that to find additional bottlenecks. A tool like New Relic can help a lot.

Studio 42

  • Contributing Developer
  • Sr. Member
  • *
  • Posts: 4341
  • Joomla & Virtuemart developper
    • Studio 42 - Virtuemart & Joomla extentions
  • VirtueMart Version: 2.6 & 3
Re: How many products can VM practically support?
« Reply #4 on: March 19, 2019, 12:28:59 pm »

Quote
I have added tags in my "shortcode" plugin, it's very faster because it only use integer(index) to get values.
   Am I right in thinking this works mostly like the existing custom fields? Is it compatible with existing custom fields? We use a number of parent / child products multivariant combinations.

My tag system is not working as other customfields, it's why it's faster. But you have a tool inside the plugin to import the tags for products so you can for eg export your current customfields and reimport this as tags

Quote
See demo / doc
   These filters look really powerful. Do you offer a trial? When do you expect it to come out of beta?
It's a beta because the module need to be adapt in some case because it load using ajax

Quote
Customfields can slowdown the shop too, it's why you should disable it in category if possible
   Can you explain what you mean by 'abusing' them? Of our products, where there's a product variation (eg a large one and a small one), we use the product / child multivariants, so there would be a single, unpurchaseable parent product with two purchaseable children products. Is this a problem?
It can be slow if you use deep levels(4 or 5 for eg)

Quote
Do not use the option "ignore" about product featured ... in the config, you end up with mysql slow queries.
   Which option are you referring to here? Is this the issue with the 'Omit' setting under Shopfront settings?

if you dont enable "omit", this can do slow queries if you render multiple list(featured,latest ....)
Quote
Adding permanent module cache and optimise the codes....
   Can you recommend a cache? I used JotCache in the past, but the site was so big that it never got completely cached so I presumed it wasn't worth it.
You have some module that use own cache system and some lightspeed server have lsCache that can speed up better some modules. In php 7+ you have  already another cache by default that can speedup your website by default.

Quote
other things to check eg. Joomla system and content plugins.
   How do you mean? It's a very basic Joomla install. Only a handful of articles to manage things like delievery and returns policies, terms & conditions etc.
It was only some other suggestions

About balai "
To my opinion, VM and Joomla is not the best option to handle a million products shop."
Other solutions do not run faster(magento, prestashop ...) if you have really a big shop you need a good server, thats all

balai

  • 3rd party VirtueMart Developer
  • Full Member
  • *
  • Posts: 1415
Re: How many products can VM practically support?
« Reply #5 on: March 20, 2019, 10:42:06 am »
Maybe this goes a bit off topic but i would like to answer some of your points, since they regard my work.

Quote
I like break design, but if you want fast filtering, it's not the right solution.
Custom Filters can work with any kind of custom field, including plugins. So your plugin can be integrated as well.
Moreover Custom Fields For All (which is compatible with Custom Filters), work by assigning the value ids to the products and not the value itself.
So what you describe as a "very faster" solution is already there for years.

Quote
About balai "To my opinion, VM and Joomla is not the best option to handle a million products shop."
Other solutions do not run faster(magento, prestashop ...) if you have really a big shop you need a good server, thats all
Magento is much much heavier than Virtuemart, but has all the layers and integrations to be extremely scalable and fast.
https://devdocs.magento.com/guides/v2.3/cloud/architecture/pro-architecture.html
Well you can say that you can add the same layers to VM as well. Yes this can be done, but will require much work and a good budget, while Magento gives all that stack from the zero point (NOT for free).



Studio 42

  • Contributing Developer
  • Sr. Member
  • *
  • Posts: 4341
  • Joomla & Virtuemart developper
    • Studio 42 - Virtuemart & Joomla extentions
  • VirtueMart Version: 2.6 & 3
Re: How many products can VM practically support?
« Reply #6 on: March 20, 2019, 11:21:11 am »
balai, your answer about server is exactly what i say.
For a big shop you need a good server and certainly a better for magento.
My concern was to explain how you can get the best form Virtuemart, not en alterative solution.
It's as if some have a car he want to optimise and you say, change car because the other have the options by default.

behemoth

  • Jr. Member
  • **
  • Posts: 81
Re: How many products can VM practically support?
« Reply #7 on: March 25, 2019, 10:28:11 am »
We've got a pretty decent cloud server: 12 Core x 2.60 Ghz with 192 GB RAM. It's the other optimisations I need to work on.

Studio 42

  • Contributing Developer
  • Sr. Member
  • *
  • Posts: 4341
  • Joomla & Virtuemart developper
    • Studio 42 - Virtuemart & Joomla extentions
  • VirtueMart Version: 2.6 & 3
Re: How many products can VM practically support?
« Reply #8 on: March 25, 2019, 11:20:57 am »
about "omit" in config. It should not be selected, i answered the opposite in the message before.

behemoth

  • Jr. Member
  • **
  • Posts: 81
Re: How many products can VM practically support?
« Reply #9 on: March 26, 2019, 10:26:10 am »
To confirm, "Omit" should be set to "No" for all six options?

Studio 42

  • Contributing Developer
  • Sr. Member
  • *
  • Posts: 4341
  • Joomla & Virtuemart developper
    • Studio 42 - Virtuemart & Joomla extentions
  • VirtueMart Version: 2.6 & 3
Re: How many products can VM practically support?
« Reply #10 on: March 26, 2019, 14:47:50 pm »
To confirm, "Omit" should be set to "No" for all six options?
YES
If you dont check it and have 6 product per type x 6 options, you get a similar query
where virtuemart_product_id != 1 and virtuemart_product_id != 2 and virtuemart_product_id != 3 and virtuemart_product_id != 4 ...
this 30 times.
I already said to valerie isaken to change query to virtuemart_product_id not in ('1','2','3' ...)
The query is not quicker but not detected as slow query in some servers.

Milbo

  • Virtuemart Projectleader
  • Administrator
  • Super Hero
  • *
  • Posts: 10070
  • VM3.2 Cached and Optimized
    • VM3 Extensions
  • VirtueMart Version: VirtueMart 3 on joomla 3
Re: How many products can VM practically support?
« Reply #11 on: March 27, 2019, 15:58:43 pm »
Hi Studio 42,

Thanks for your detailed response. I hope I can request clarification on some of your points:

Quote
I have added tags in my "shortcode" plugin, it's very faster because it only use integer(index) to get values.
   Am I right in thinking this works mostly like the existing custom fields? Is it compatible with existing custom fields? We use a number of parent / child products multivariant combinations.

Vm offers by default something similiar, you may also take a look what the core can already do nativly
I should fix your bug, please support the VirtueMart project and become a member
______________________________________
Extensions approved by the core team: http://extensions.virtuemart.net/

Milbo

  • Virtuemart Projectleader
  • Administrator
  • Super Hero
  • *
  • Posts: 10070
  • VM3.2 Cached and Optimized
    • VM3 Extensions
  • VirtueMart Version: VirtueMart 3 on joomla 3
Re: How many products can VM practically support?
« Reply #12 on: March 27, 2019, 16:05:58 pm »
"he query is not quicker but not detected as slow query in some servers."
The sql for the omit feature got enhanced ( a lot faster than before), also the featured products random problem is now fast.
Concerning child products. It is not a problem to create a multivariant with more than 100 child products, when you see the product in the category browse view, the child products are NOT loaded! It is also not a problem to have deeper levels, but it makes usually not a lot sense to have more than 3 derivates of a parent.

When you display 20 child products of an unpublished parent product (pattern) and this products have for example no own price set, they are a lot faster loaded, than just using 20 normal products.

child products can make your store easier to maintain AND faster. But child products can also slow down your shop. It depends on how good you use them.
I should fix your bug, please support the VirtueMart project and become a member
______________________________________
Extensions approved by the core team: http://extensions.virtuemart.net/

behemoth

  • Jr. Member
  • **
  • Posts: 81
Re: How many products can VM practically support?
« Reply #13 on: April 03, 2019, 17:08:28 pm »
Quote
Vm offers by default something similiar, you may also take a look what the core can already do nativly
What do you mean?

behemoth

  • Jr. Member
  • **
  • Posts: 81
Re: How many products can VM practically support?
« Reply #14 on: April 05, 2019, 09:58:59 am »
Look for professional help with this now:

http://forum.virtuemart.net/index.php?topic=142343.0