Migration from 1.1.9 to 2.0.16d / 2.0.18a - Killing my data

[xpozay]

I am really struggling to get my upgrade to work.  I have followed the various recommendations include http://forum.virtuemart.net/index.php?topic=95236.0 and also done it manually (one step at a time).  Done it in J1.5, followed by the migration before moving to 2.5.  Done the migration in J2.5.  Thus far manually has been the most successful although there are far too many differences in the tables to make it feasible to use the data.  For example these are the results of a migration using this approach
http://forum.virtuemart.net/index.php?topic=95236.0  with J1.5.23, VM 1.1.9.

vm_orders = 14537
vm_order_history = 28822
vm_order_item = 23689
vm_order_user_info = 14537
vm_user_info = 10244

virtuemart_order_userinfos = 29074
virtuemart_order_items = 47378
virtuemart_order_histories = 57636
virtuemart_order_orders = 29074
virtuemart_userinfos = 10221
virtuemart_vmusers = 51701

Note: My jos_users is 51701

You will notice than none of the before and after tables have the same quantities - not even near.  In fact, if you look at the details of say user_info vs userinfos they are so far different. 

Aside from the above, when I migrate all at once, is seems like the number of orders actually doubles.  Eg  VM 1 Order Number  15179, is copied across but also has a duplicate 151792013-01-20-18-25-16_  (note the first 5 digits is the same).  The rest of the fields are the same.

The only errors displayed during the migration is

vmError: Table userinfos check failed: Unknown address_type
vmError: Migration storeAddress BT

If I run debug this generates the above error message

Array
(
    [id] => 31665
    [name] => Fnnnnnnnnnnnnnnnn
    [username] => ununununununununu
    [email] => xxxxxxxxxxxx@gmail.com
    [password] => yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
    [usertype] => Registered
    [block] => 0
    [sendEmail] => 0
    [gid] => 18
    [registerDate] => 2011-12-17 07:06:49
    [lastvisitDate] => 2011-12-17 07:07:04
    [activation] =>
    [params] =>

    [user_info_id] =>
    [user_id] =>
    [address_type] =>
    [address_type_name] =>
    [company] =>
    [title] =>
    [last_name] =>
    [first_name] =>
    [middle_name] =>
    [phone_1] =>
    [phone_2] =>
    [fax] =>
    [address_1] =>
    [address_2] =>
    [city] =>
    [state] =>
    [country] =>
    [zip] =>
    [user_email] =>
    [extra_field_1] =>
    [extra_field_2] =>
    [extra_field_3] =>
    [extra_field_4] =>
    [extra_field_5] =>
    [cdate] =>
    [mdate] =>
    [perms] =>
    [bank_account_nr] =>
    [bank_name] =>
    [bank_sort_code] =>
    [bank_iban] =>
    [bank_account_holder] =>
    [bank_account_type] =>
    [vm_childsschool] =>
    [vm_condo] =>
    [vm_childsname] =>
    [vm_previousmark] =>
    [vm_level] =>
    [vm_ic_number] =>
    [vendor_id] =>
    [shopper_group_id] =>
    [customer_number] =>
    [group_id] =>
    [group_name] =>
    [group_level] =>
    [virtuemart_user_id] => 31665
    [virtuemart_country_id] => 0
    [virtuemart_state_id] => 0
    [user_is_vendor] => 0
    [virtuemart_vendor_id] => 0
)

The above user id, 31665 has never placed an order, so it is logical that there is no BT and this record should be ignored and no message even provided (thus not reaching the maximum 1000 messages).

I don't have any addresses with ST, they all have BT and all records have "BT" in the field vm_address_type.  I do notice that address_type_name is mixed 'something'; "-default"; and "Null".  Any problem with this?

Also some other information regarding my data:
- I have many user records with no usertype, the above says "Registered", but there are many others about 12000 out of the 51000 with no usertype.  This field is being corrected using spupgrade when the users are migrated. 
- State & City are not used in my VM1.  Will it make a difference during migration if these are marked as mandatory?  Do I need to match the user fields before migration?
- All products details are simple, no child products, on variants, no attributes.

My products, categories, manufactures, vendors all come across OK.  Users always produces the above errors and when I run the migrator manually, I never get to orders because users gives me a problem.  Can I migrate orders without migrating users?

As said, I am really stuck on this one.  I have been at this for almost two weeks and getting to the point where I will not put old orders online and force my users to enter addresses again.  This is clearly not a good option and will cause significant turmoil from my users.  Is there a manual process I can take to transfer records?

[illovo]

vmError: Table userinfos check failed: Unknown address_type
vmError: Migration storeAddress BT

I just can confirm. I tried several migrations 1.1.9 to 2.0.18a with several migration settings. Checked all entries of table jos_vm_user_info if there are empty fields concerning addresses.
No way.

[xpozay]

@illovo.  Sorry what do you by no  way?

I have progressed on this a little and getting closer to a result that I require.

1) I do it step by step instead of all at once
2) I have 1 live server and 3 staging servers.
3) Staging
- 1st staging is my back of my live- J1.5.23, VM 1.1.19 - I never touch my live.
- 2nd staging is running J2.5 & VM 2.0.18a
- 3rd staging is used to run J2.5 & VM 2.0.16d,

4) Copy my old VM data from 1st staging to 2nd staging
- In my case before I migrate users I run a batch routine on my jos_users to ensure all usertypes = Registered and Group = 18, except for my admin. I use phpmyadmin to issue a mysql command.  I have not done enough testing to ensure that this problem with my jos_user is not negatively affecting my VM migration.
- In my case I add back in (dummy record with userid and email) any missing users that were deleted from Joomla but had orders in VM earlier.  I used phpmyadmin to insert new records with the appropriate userid. IF I do not do this the new virtuemart_userinfo will be short of records in comparison to the old vm_userinfo. IHMO this is a bug in the migrator as it should not be making decisions to delete records but just moving data across.
- Run the migrator to step by step move over general, users, products

4) Copy old VM data from 1st staging to 3rd staging and copy over new VM data from 2nd staging to 3rd staging
- Run migrator for orders
- Run migrator for set shop owner

- I am not sure why I need to use 2nd and 3rd staging yet.  But so far whenever I try to migrate users on VM 2.0.14 or VM 2.0.18 it does not migrate my orders properly but VM 2.0.16d is OK.  This may be particular to my environment, I am not sure yet.
- Also, I have customised the migrator on 2.0.16d to migrate the modified dates not just the creation dates.  IMHO this is a bug in the migrator as it should not set the modified dates to "the migrator run date"

--------------------------------------------------------------------------------------------------------------
What is outstanding in orders
- Order currency is not copied across / set
- Shipping method and Payment method are not copied across  (not the methods but the data in the orders)
- Shipping cost is not copied across
- The above three are important for running reports, etc. 

I am not sure if the above three is not done in the code or it if it is in my environment.