Cannont migrate 12k products to VM 2.18

[PRO]

Guys I'm more than willing to let you guys take a look, I have been struggling with this for months and my client is really running out of patience.

Just an FYI the day before yesterday I actually uninstalled VM and deleted all the tables and tried it again with the same exact outcome. I have rebuilt the site 4 times with Jupgrade and all the same.

Thanks for your time.
Joe

test this on a clean Joomla install and Vmart and see what happens

[Joe Bourque]

I have installed a clean joomla install with Virtuemart and still no luck. I have about 2,000 images and 12,000 products. If anyone can give any advice or assistance with this issue please let me know.

Thanks
Joe

[Joseph Kwan]

Try comment out line 1057 of the file administrator/components/com_virtuemart/helpers/migrator.php. See if this works better.
Anyway, 12000 products may be too heavy for the migrator script. You probably will encounter problems down the road unless you can increase the memories and script time.

[Joe Bourque]

I ran it the 1st time and 970 1001 products imported then I tried again a few times and it's only migrating 1 at a time



        Info: port Products take 1000 vm1 entries for migration
        Info: Coulndt find category for product, maybe just not in a category
        Info: Migration: 1 products processed

[Joseph Kwan]

Apparently, the migrator runs out of memory.
Ah... I forgot to mention you have to choose migrate products only. That will leave more memories for product migration. Did you try that?

[Joe Bourque]

Yes I'm migrating the products only. I have allocated 124mb of memory I can adjust it to 512M

The other strange part is I have 117 categories and the some of the products are not showing the category saying category not found although I see all the categories.

Joe

[Joe Bourque]

So I bumped the memory up to 512 still no luck


Start with a new migration process and setup log maxScriptTime 159 maxMemoryLimit 498
Worked on general migration for 229.350474119 seconds



        Info: port Products take 1000 vm1 entries for migration
        Info: Coulndt find category for product, maybe just not in a category
        Info: Migration: 1 products processed

I looks like the migration does not continue I went into my sql db and deleted the data in virtuemart_migration_oldtonew_ids then rerun the migration script it imports 1000 more products which are all duplicates.

Please advise if anyone knows how to fix this

Joe

[Joseph Kwan]

Did you have debug ON in VM configuration? There may be more error messages coming up if debug is ON.

[Joe Bourque]

Yes I did nothing else I can see

[PRO]

Joe Bourque,
what info do you need to be migrated?

the whole thing?

users? orders? products , cats?

[rjcroasdale]

Hellowz ! ^_^ !

Hi PRO, Hi Milbo, Hai Joe & Mr Kwan (ty for your work on 1.2b)

Having a similar problem here - product migration just stopped @2411 or something - would not migrate any further products.

Thanks Kwan, your comment out //$maxItems = 5; seemed to get things moving, but quite slowly - before commenting out maxitems the product migration was absolutely stuck migrating 0 products on each migration, sql process was using CPU though.

Now, with //$maxItems = 5; its processing products again, I'm going to up the max exec and mem limit again and hoping this will help process more products.

P.s 22, 000 products

Best Regards
RJ

**EDIT** 22,130 products migrated

[yakkes]

Hi all,

I also was stuck on importing +26000 products. Kwan's solution to comment out $maxItems solved partially the problem. I had increased memory_size to 512MB and timeout to 600 but even then mysqld also was running wild.

Finally traced down the excessive cpu use to the LEFT JOIN SQL just below $maxItems in migrator.php which was deadslow.
The problem was the datatype of #__vm_product_mf_xref.product_id which had varchar(32) type. Changed this to INT(11) and from then on the LEFT JOIN was running smoothly.

Looking back it seems VARCHAR(32) was used in v1.1.2 and never altered to INT(11) since.

[svarmuza]

Hello,
i have the same problems in migrating... i use J 2.5.14 and VM 2.0.22a...

...first i tried to do this directly on server but because of low execution_time and memory_limit i couldn't migrate too many products and images (i have 12 000 products and 30 000 images)

... now im trying to migrate localy (nstaled xamp) and increased memory limit to 2000mb and execution_time to 12000s (2 hrs : )... i managed to migrate all images ok... but products again stoped at cca 7 000... and when i run migration start again... it migrates 0

... i gues i will just have to increase more execution_time (maybe to 5 hrs) in php.ini of server...and just let it process for hours (it seems if data migration is interupted, and it is not migrated in 1 try... later doesnt want to continue right...)

I will keep you informed about this...

Greetings,
Sasa

[Maxim Pishnyak]

What migration method/manual do you use?

[svarmuza]

Hello...
I found interesting situation and cant figure it out... ( i have 12 000 products to migrate, but to debug it, first i used 16 to see what is wrong)

1. I enabled debug function in VM... and imported 16 products to my old VM 1.9 table vm_product
2. i migrated products and from 16 only 9 migrated ok... but in table _virtuemart_products, there are all 16 products migrated... but in my _virtuemart_products_hr_hr table there are only 9 (and only these show in administration)
....After migration I got this debug message (this is message just for one... rest are the same)

Code Select Expand

vmdebug Check returned false VmTableData #__virtuemart_products
vmdebug $product["virtuemart_product_id"] or $product["product_id"] is EMPTY? Var1:

Array
(
    [product_id] => 13
    [vendor_id] => 1
    [product_parent_id] => 0
    [product_sku] => CERNIT 200
    [product_s_desc] => Blue
    [product_desc] =>
    [product_thumb_image] => resized/Cernit_Number_On_4aa8b2d661609_150x150.jpg
    [product_full_image] => Cernit_Number_On_4aa8b2d68c16c.jpg
    [product_publish] => Y
    [product_weight] => 62.0000
    [product_weight_uom] => G
    [product_length] => 0.0000
    [product_width] => 0.0000
    [product_height] => 0.0000
    [product_lwh_uom] => CM
    [product_url] =>
    [product_in_stock] => 24
    [product_available_date] => 2009-08-30 00:00:00
    [product_availability] =>
    [product_special] => N
    [product_discount_id] => 0
    [ship_code_id] =>
    [cdate] => 1251699144
    [mdate] => 1353275956
    [product_name] => Cernit Number One
    [product_sales] => 1
    [attribute] =>
    [custom_attribute] =>
    [product_tax_id] => 4
    [product_unit] => komad
    [product_packaging] => 0
    [child_options] => N,N,N,N,N,Y,20%,10%,
    [quantity_options] => none,0,0,1
    [child_option_ids] =>
    [product_order_levels] => 0,0
    [manufacturer_id] => 1
    [virtuemart_vendor_id] => 1
    [categories] => Array
        (
        )

    [published] => 1
    [mprices] => Array
        (
            [product_price_id] => Array
                (
                    [0] => 0
                )

            [product_id] => Array
                (
                    [0] => 13
                )

            [product_price] => Array
                (
                    [0] => 10.56098
                )

            [product_currency] => Array
                (
                    [0] => 160
                )

            [price_quantity_start] => Array
                (
                    [0] => 0
                )

            [price_quantity_end] => Array
                (
                    [0] => 0
                )

            [product_price_publish_up] => Array
                (
                    [0] => 0
                )

            [product_price_publish_down] => Array
                (
                    [0] => 0
                )

            [created_on] => Array
                (
                    [0] => 2009-08-31 06:12:24
                )

            [modified_on] => Array
                (
                    [0] => 2012-11-18 21:59:16
                )

        )

    [created_on] => 2009-08-31 06:12:24
    [modified_on] => 2012-11-18 21:59:16
    [virtuemart_media_id] => 2030
    [virtuemart_product_id] =>
)
...here at the bottom   [virtuemart_product_id] =>   there is no product id number, but i checked in database and in _virtuemart_products... there are virtemart_product_id exist for all products...

ANOTHER FUNNY THING:
At second attempt to test it... i used only those 7 products that havent't migrated at first try.... and here 6 products migrated ok, and one with the debug error mentioned above...

VM debug error that is apearing is connected with ...administrator/components/com_virtuemart/helpers/migrator.php file....   lines around 1236

Code Select Expand

if($this->_keepOldProductIds){
$product['virtuemart_product_id'] = $product['product_id'];
}

$product['virtuemart_product_id'] = $productModel->store($product);

if(!empty($product['product_id']) and !empty($product['virtuemart_product_id'])){
$alreadyKnownIds[$product['product_id']] = $product['virtuemart_product_id'];
} else {
vmdebug('$product["virtuemart_product_id"] or $product["product_id"] is EMPTY?',$product);
}

Im not a programmer, so I have no clue why this is happening...
If someone could help me with this, i would be very happy.
Thanks,
Sasa