News:

You may pay someone to create your store, or you visit our seminar and become a professional yourself with the silver certification

Main Menu

How to add 300,000+ products easily?

Started by daviat, January 17, 2012, 18:11:39 PM

Previous topic - Next topic

daviat

Does anyone know the best and fastest way to add 300,000 products in virtuemart. I have a client whose site requires that but being a one man shop I am finding it impossible. Any ideas?

thanks,
Daniel

darkgreenlantern2814

The easiest way that I've found to do that is to find the tables in your database associated with the products and either make a CSV spreadsheet and enter them manually in there (it's much easier than through Virtuemart) and then importing it to your database via CSV Improved or phpMyAdmin.

Let me know if I need to explain either.
Joomla! v1.5.23
Virtuemart v1.1.9 stable


FidelGonzales

I haven't used CSV Improved on 2.0 yet but have found that performing the import in 2,000 to 4,000 product batches is safer and easier, especially if you have a lot of data loaded into each product, then I would certainly limit the amount of products per batch import. Test and backup before each import, ensuring you have an intuitive way to quickly identify the backups should anything go wrong.

For this project, I would recommend nothing less than a dedicated server. You want speed and security. Hopefully the client realizes the enormous undertaking you're to embark upon and aren't going cheap on the hosting through some shared service. I wouldn't be surprised to have multiple people assisting with the development of the product database. Either way, the biggest strain is going to be upon the MySQL database attempting to serve a product and category database of that magnitude, a database size I have not heard of in Virtuemart. The max I have reached is somewhere around 10,000 products, for which the search engine traffic alone was an issue for the startup, not to mention the nominal marketing initiatives. Definately limit the use and even disable some of the product/category listing modules, at least during development, since some tend to call all the products within a category and could well cause timeout errors.

Furthermore, you will want to ensure you have the products categorized in a legible manner to the user and the search engines. Syncing these products with the category structure prior to import can be tricky, which is why I will often create a single product by which I can manually plant in certain categories and then perform exports to easily and accurately copy and paste the product category path for several hundred or thousand products at a time. This makes it simple for me, since products may exist in several if not hundreds of categories, considering I create my own manufacturer categories and sub-categories within those. The VM manufacturer sort is too limited when it contains a several thousand products. The search engine has difficulty finding anything beyond page one, and with that many products, there is virtually 10-20 other pages of content the search engine cannot see.

Good luck.

daviat

I am a one man shop so this job doesn't seem realistic for me. I quoted 1450.00 for a site with cart, then he sprung the 300,000 product insertion. It would take me ages just to enter products. Sites like this require a team and I doubt he has the cash to pay for multiple programmers. Thanks for all the advice. :) Much appreciated.

archos

Your custumer has the money to have 300 000 items to sell (let's asume it has only one of each) + warehouse to store them,(or maybe they are stamps),and no money to hire more than a guy to realize the shop ?

Yesterday I've done a test with phpmyadmdim to fill a small store. I had 3 tables to fill related to product, price and language) I export and "re-import" as CSV files. No problem, just take care to generate the unique primary key.
If the 300000 items are still on an excel sheet with all the necessary data, developping a Macro could do the trick.(In that case I would use the product reference as image name, but if you have to rename the 300000 pictures accordingly it's also a big job)

lipes

300.000 products?! WOW! o_O
I have at this momment +- 8000 products in my fathers little store DB, with VM1.1.9 and J1.5 .. i hope find a solution (in the future) to migrate all products, costumers, images and so on to VM2 and J1.7.3 ... we cannot afford some exterbak process or contract any programmer / developer.. i've to do this things step by step firstly in my localhost..
I also hope that this not takes my lifetime "forever".. i dont win nothing with this (not my job.. i'm yet a student) because i'm only trying to help my mother and dad from the biggest crises over my country... i'll need to find the fields on DB and pass the SQL to excel and inserting them in the new DB..  and then hope it works..
VM V. online: J2.5.14 | VM 2.0.20a | SQL 5.1.70 | PHP 5.3.25

stinga

We have 229290 products without a problem...
I have a perl script I use called load_prd that just populates the nessary tables.
Took a few hours to code up, was not that hard.
Stinga.
614869 products in 747 categories with 15749 products in 1 category.
                                             Document Complete   Fully Loaded
                Load Time First Byte Start Render   Time      Requests      Time      Requests
First View     2.470s     0.635s     1.276s          2.470s       31            2.470s      31
Repeat View  1.064s     0.561s     1.100s          1.064s       4             1.221s       4

franzpeter

Stinga,

thats right. It is possible but not advisable to use CSVI or some other product import tools, which rely on Joomla or Virtuemart API. That is just too slow. It is better to do that directly into the database without using the API, a component or something like that. It is a well known problem. For example I use a component, which should import ICECAT pictures and descriptions to the products. It takes days to do the first import if you have for example 10 000+ products in the catalogue. It is not a specific Joomla or Virtuemart issue, that happens with all CMS (Magento too). A independent script is the best solution. My script (VM 1.1.9) does it in just a few minutes, what would take days with a regular component.

Kupernikuz

We just feeded 400.000+ in a shop before filtering. After filtering vi feed the shop with about 200.000 products every 2 weeks because of price updates, information updates and because the law says that we have to do it that way. And in short there comes even more.

We have build our own little JAVA script, that runns local on a PC, where we have stored the data ASCII files, and that script feeds the database tables with informations needed for Virtuemart to run, and it works. The client still don't go online because that we need some more modifications, but when I can post a link to the shop.

So, Daniel, it depends on if You have your data in SQL format, or in ASCII files or any kind of files, from where the relevant data can be read and inserted in the tables that Virtuemart need.
It is a kind of personalizing your shop :-)

If you still work on that shop, I can suggest you to contact a programmer, wich has good expiriences with SQL and know how to take out the data from files.
I can tell, that when you use a php script for filtering data out and inserting from files, it takes a LOT more time, as when using a JAVA program. So go after a programmer that can code it for you in JAVA or Python or something else that is powerfull and don't run serverside on a webserver.. Now, thats Just my expirience :-)


RolandD

@stinga & @franzpeter: I agree with your statements that a dedicated script is the best solution at hand but if we all could code, we wouldn't need generic solutions :) The downside of your own script is that you will have to maintain it and it is only usable for the specific import you create. A generic solution can be used by a larger group with sometimes less effort. Further generic components often offer more options than just a product import. In the end everyone should decide what they see fit for their shop.

@franzpeter: What I do find interesting is that your ICEcat script takes only a few minutes to import thousands of ICEcat images where I would expect that the download of these images alone would exceed that time. Care to explain?
Regards,

RolandD

CSVI
http://www.csvimproved.com/

PRO

Wtf is icecat? I see it in my csvimproved mysql tables

franzpeter

@Roland,

Roland, maybe I did write it a little bit ambiguous. I did want to say that my daily import script for importing distributor articles (20 000 - 50 000), which does not use any Joomla or VM functions or models, just needs a few minutes for the daily update import. If I import everything from new on (categories, images and of course products) it may take, because of the picture files, around 1,5 or 2 hours. After first import, it goes very fast. I use a commercial Joomla component to import IceCat data (not CSVI). If I even switch off all pictures and say it should only import the descriptions, it works fast until it reaches about 1100 products, after that it imports maybe 1 additional product in three minutes a.s.o. so it may take days to update the other 19 000 or 25 000 products (even without pictures). CSVI is faster, that is true, but by far not as fast as a php script, which does not rely on any joomla or VM models or functions.

franzpeter

@BanquetTables.pro

IceCat provides product data (descriptions, images, gallery images, technical data sheets for a lot of products). There are others like CNET to just mention another player. It is an excellent resource if you sell computers and parts, electronic components a.s.o..

PRO

Quote from: franzpeter on February 17, 2012, 11:12:06 AM
@BanquetTables.pro

IceCat provides product data (descriptions, images, gallery images, technical data sheets for a lot of products). There are others like CNET to just mention another player. It is an excellent resource if you sell computers and parts, electronic components a.s.o..

ok, thanks