Full Review of a Large Store VM1 to VM2 Migration

[edango]

Good morning,

First off, I'll apologize for the length of this post, but I thought the community might be interested in the experience of someone who had done a huge migration from VM1 to VM2 of a large store which basically uses every aspect of Virtuemart.  I definitely want to thank the Virtuemart developers for all of their hard work, I know its a huge undertaking and takes countless hours.  I've used Virtuemart and Joomla since almost the beginning (anybody remember Mambo??) and it's honestly been one of the key areas for me to succeed as a freelance developer to the point now where I run a small development firm.  While I don't post very often, I definitely scour the forum weekly to read the latest and am always thankful for the helpful community in answering issues and bugs. 

I haven't seen anyone post a detailed review of this and some of the issues I'm addressing may already be resolved, if that's the case, my apologies, but I did think areas of this could be helpful for people to consider and be aware of.  My migration was to VM2.0.8

I believe one of the major areas that many people don't realize is that transferring a site from VM1 to VM2 is not an update, is a full on migration and for a larger site, you need to be a programmer/developer, unless you plan to spend hours updating your products manually.  I'm highlighting specific issues we hit (that were either unexpected or where we knew we had to write workarounds for).  Because of the sheer size of the shop, we couldn't do this manually, so when I mention "convertors" below, it was scripts we wrote to help the SQL migration of data from the old VM1 format to the VM2 format.  Plus, we realized many features or functionality were heavily modified or didn't even exist in VM2.  I can understand the perspectives of the developers for the limitations of some of the features, however, the ability to include previous functionality that is being used, in this migration, was crucial.  So we had to either re-write the frontend or develop specific migration tools for areas to cover those features.  (NOTE: All of our hacks/implementations were not affected after we did an upgrade from 2.0.7 to 2.0.8 - so they weren't core specific hacks). 

We decided to write individual "convertor scripts" to cover alot of the below migration addons, mostly because we ended up doing more than half a dozen migrations before we had the final result we wanted and it simplified the SQL process.  If anyone is interested, feel free to contact me or I can upload a few of them for people to access and experiment with?  I tried to just give an overview of the below points and not go into great detail (figured this review/post was long enough as it is) but I'm happy to expand if anyone has any questions. 

Original Specs - 1300+ product VM1 shop that used the following:
- Multiple attributes with pricing
- Child products with pricing
- VAT taxed based system (UK site)
- Specific SEF system in place because of site's age
- Dimensions/Weight specific shipping system in place

Overall, we were able to do a "test run" of about 50-100 products without too many issues, the problems we first encountered were the VM products with any type of additional information beyond the name, price and description.

1) Convertor for multi-buy pricing
- We found out the multi-buy discount system in VM1 didn't even exist in VM2 without a paid solution.  We implemented this and then had to write a separate migrator script to recognize the product values / discount percentages.  Since over half of the products we had included multibuy discounts, to encourage more buying, a manual solution for this simply wasn't an option.

2) Convertor for attribute pricing / display
- This was one of the biggest surprises for us, that the migration suite built into VM didn't include multiple attribute support and more importantly, didn't include pricing.  We had to write another convertor script for this to accommodate the pricing addons we utilized.

3) Convertor for tax values (VAT related) and discount types
- While it was more a simple convertor, we did have multiple tax values implemented to accommodate for the VAT values that were originally present (for those who are not familiar, UK's VAT system is extremely strict).  The original migration of the tax and discount values auto-created a separate "type" for each difference in a product (i.e. if one product had a 10% discount at 19.99 and another had a 10% discount at 29.99 it create two separate discount types in the system).  This ultimately created over 400+ tax/discount entries in VM2 and wasn't manageable.  So we developed another conversion tool to accommodate the adjustments of tax values to the new format present in VM2

4) Adjustment for the thumbnail/product image sync
- We did notice issues where the current migration suite would timeout during thumbnail/image sync to the products from VM1 to VM2, simply because of the sheer number (easily 6000+ as most products had at least 2 images and the thumbnails).  There wasn't a major adjustment here, mostly to the memory/timeout to accommodate and process it longer (I know you can run this multiple times but it still didn't encompass or properly sync all of the images after half a dozen tries, hence why we proceeded with a modification for it). 

5) Convertor to include dimensions/weight of products accurately
- This was another, simple convertor script to recognize and change the SQL from VM1 to be reported accurately in VM2 (including the appropriate weight type)

6) Hack to eliminate the "browse categories" above products/categories
- I know this is detailed elsewhere on the forum, but for future VM dev, it'd be nice to have an option to turn off the categories displayed above search/category results, that way the user can see the products quickly without having to scroll.

7) Hack to override the new VM SEF rewrite rules in JoomSEF to mirror the old ones
- This was a time consuming task for us, as we had to modify the mod_rewrite rules within .htaccess to accommodate how the SEF's system changed in VM2.  Specifically, on the VM1/J1.5 site, we were using JoomSEF to handle our SEF, but their addon to VM2 defaulted to options that differed greatly from VM1/J1.5.  We made adjustments to JoomSEF on the VM2/J2.5 side but also had to hard-code some workarounds into .htaccess to keep the SEF structure identical (and to mimic it for future URL creation for products/new categories).  Our client had spent thousands on marketing and could not afford to adjust the SEF structure at all (plus long-standing backlinks were in place).

8 ) Fixed pagination error for categories
- We did notice some caching related issues (probably more of a conflict with JoomSEF than VM2) but had to develop a workaround so if a user changed the number of results to view and then browsed to page two or three, went back to page one or two, that the displayed results were accurate. 

9) Hack to sync the VM1's order numbers to continue the progression/order in VM2
- I read several discussions about this, specifically the comparison between the Order ID and the Order Number.  While I can see and appreciate the security involved with making the Order Number more difficult, there was already a system in place with my client's website, physical store and warehouse, where the order numbering sequence couldn't change (i.e. it would have a ripple effect).  We therefore were able to write another workaround to mimic/mirror the Order Number's last 5 digits to continue in this sequence/order.  That way, the orders converted previously from VM1 were the correct number (if someone had to look them up previously) but the numbering sequence was still technically the same (i.e. older order was 24567 and new VM2 order was 02b1024568 for example).  This allowed us to keep using the security measures that were implemented, without confusing our client's warehouse staff (NOTE: on the export adjustment we made with CSVI VM template, we auto-eliminated the first 5 digits of the order number so the warehouse staff only saw the last 5)

10) Convertor for foreign character issues in VM product display
- Honestly, this could be more of a template conflict but I wanted to note we had to run several SQL replacements and adjustments, as we had foreign character issues in names, descriptions and even pricing briefly (based on the punctuation).

We also developed several additional modifications that I thought might be of interest:

1) Modifications to the price display
- We made adjustments to include a larger price display, line-through the original price and differing font colors/styles

2) "Deal of the Day" module creation
- Created a module that would display a specific product designated (by selection the category/product and/or entering the product ID).   Basically a simplified version of the top ten / featured / latest but isolates to one product.

3) Modifications to the checkout, category, search and product pages
- We made many modifications to the above pages, including the checkout display of pricing, happy to discuss these further if anyone is interested

4) "Ask a Question" displays the question/answer on the product page (with pagination after x questions)
- This was actually a mod we did on VM1/J1.5 that we brought over to VM2, which basically converts the existing QA system to display previous QA that was asked, it's great for SEO + eliminates support over time, as customers can read previous questions asked (might be a ticklist item to add to VM in the future?)

5) "Add to Cart" modification for products with attributes on the category/search pages
- I'm pretty sure this discussed previously, but we did implement a hack to display add to cart options on category/search results where products had attributes or multiple selections as we wanted to streamline the buying process as fast as possible.

We're currently in discussion with our client to implement the following methods as well:

1) Ability to edit orders in the backend
- I can understand the "theory" behind turning off this functionality in VM2, as it could be argued that an order should be edited after submission.  However, my client takes numerous phone calls/support from their customers and frequently (i.e. several times a day because of the volume they do) they have to either add/delete products to an order that was already submitted and then adjustment the pricing for the customer.  This streamlines the process so that the customer doesn't have to checkout and is crucial.  I really feel like this feature should be re-implemented into later versions, however, we will probably be developing a hack for it in the coming weeks.

2) Modifications to carry over alpha user points to merge with the new VM2 system
- We've had numerous problems merging the latest AUP J2.5 version in VM2, plus the upgrade from VM1 to include the assigned point system that was in place before is crucial.  It's basically a work in progress at the moment, but I wanted to mention it as an issue, in case others were wondering if it happens.

Final Review:
Overall, VM2 is a solid upgrade and fixes alot of issues we had from VM1.  However, the migration suite missed keypoints for a larger store and people should really understand that its more of a move to a completely NEW storefront and system instead of a full upgrade (I know this is mentioned elsewhere, heavily, but I wanted to note it). 

The major reason we upgraded was because of security concerns and the future lack of support of J1.5.  Especially because of the volume and traffic on my client's site, he wanted to overhaul his design as well and insure there were no other issues present.  If it hadn't been for this, we probably would have stayed with VM1 until early 2013.  However, the end result for us was a heavily modified VM2 that's ordering, product catalog and frontend was quite similar to VM1 and had full support for all of our client's required features.  This process took us over six weeks and in the excess of 150+ hours, though we expect with our current migration tools, we could probably do other migrations in the 25-50 hour range.  We'll be test-driving this in the coming weeks on another client's smaller store, so it will be a good indication of the versatility of the additional migrator tools we developed.

I do plan to recommend to my other clients to upgrade before the end of the year, I feel the security concerns, coupled with the new features available in J2.5 (and its related extensions) are necessary for continued growth.  I do hope the VM2 migration suite for the general public expands and I'm happy to include more notes and/or attachments that have the above migration tools I mentioned, if anyone is interested?  (I was going to submit some to the VM developer team, but was slightly unsure of the process to do this?).  The tools still require SQL/PHP knowledge though, just as a forewarning.   If anyone needs further help, feel free to PM.

In closing, my client is happy with the final result and the improved VM2 interface, but we definitely hit more speedbumps on this migration than expected (And, to note, way more than the J1.5 to J2.5 core upgrade).  However, the sheer size of the products, articles and database (over 150,000 SEF urls for example) made this process much more difficult than the migration for smaller stores.  I'm not trying to dishearten anyone from using VM2, I think it has some great new features and a slick interface, however, be prepared before attempting to move your store.  I'd definitely recommend for novice people to consider hiring a developer or, if the size of your shop is small, to do the process manually.  I hope this review is helpful as I had not seen any detailed posts about the migration of a large store from VM1 to VM2.  Cheers!

[callumgilhooly]

Hi Edango,

I'm currently struggling through a migration of a large store (60k products plus) and was wondering if you were going to either compile these scripts into a fully operational migration tool and sell it (I'd be willing to pay) or if you could just release the scripts as they are?

I have run the in-built migration tool a number of times and get a different result each time (products missing / category hierarchy not right). I've also tried mapping across directly but that causes failures as well. (My SQL skill level isn't that high  )

Any help I can get is most appreciated

Thanks

James