VirtueMart Forum

VirtueMart 2 + 3 + 4 => Product creation => Topic started by: kderg on September 28, 2013, 07:55:38 AM

Title: Media synchronization stopps after 11k images
Post by: kderg on September 28, 2013, 07:55:38 AM
Hello there,

i didn't found a solution by searching - so lets have it a try. System (now - local for testing (PHP: 5.4.19  VM: 2.0.22c ))

From an existing shop i copied 23000 images, sorted in 1 level folders  (1500 folders like .../product/artistname1/imagename1, .../product/artistname2/imagename15, ...).

If i use media synchronization after running several times to proceed the migration process - it will show the message "0 media for product in folder images/stories/virtuemart/product/ synchronized"(translated).
Everytime it reaches  more than 11000 entries (11180, 11371,...) the synchronization stops. No error message - it just stops.
If i raise script-time or memory - only the steps to get to that count are bigger.

I hope someone has an idea - thanks.

Edit: I'm worried because - if i would write some script to add all the images to media, probably the synchronization will in future not work, too. So that the one who creates new products (no programmer) cant use this (but its necessary by this amount of files)
Title: Re: Media synchronization stopps after 11k images
Post by: Maxim Pishnyak on September 29, 2013, 21:26:22 PM
You could bypass in built VM media synch with the help of CSVI.

Also how is look when synch is stopping? Could you check logs for possible errors maybe?
Title: Re: Media synchronization stopps after 11k images
Post by: kderg on September 30, 2013, 11:22:05 AM
There are no php or joomla log entries.

The look is normal (like before the invisible border of 11k images) with following message:
Found prior migration process, resume migration maxScriptTime 95 maxMemoryLimit 242
0 Media for product in folder images/stories/virtuemart/product/ synchronized (translated)

To read thousands of products into the database, CSVI could be useful. But i wrote a script to do that directly from another shop - but before, i want to check if care and maintance of products could be done without problems by a not technical user with VM. So i hope to find out if VM is not able to handle this amount of images and someone has allready the experience with that behaviour.
Title: Re: Media synchronization stopps after 11k images
Post by: jjk on October 01, 2013, 09:41:02 AM
Quote from: kderg on September 28, 2013, 07:55:38 AM
If i use media synchronization after running several times to proceed the migration process - it will show the message "0 media for product in folder images/stories/virtuemart/product/ synchronized"(translated).
If the VM info displays "0 media for product in folder images/stories/virtuemart/product/ synchronized", I would assume that the synchronization script finished normal, but didn't find any new images.
Quote
Everytime it reaches  more than 11000 entries (11180, 11371,...) the synchronization stops. No error message - it just stops.
Just for clarification - are the first 11000 entries stored in the database table xxxx_virtuemart_medias?


Title: Re: Media synchronization stopps after 11k images
Post by: kderg on October 01, 2013, 10:01:57 AM
QuoteIf the VM info displays "0 media for product in folder images/stories/virtuemart/product/ synchronized", I would assume that the synchronization script finished normal, but didn't find any new images.
It seems to be so, but the problem ist - there are more images. The structure is the same - no difference between the read in images and the forgotten.

QuoteJust for clarification - are the first 11000 entries stored in the database table xxxx_virtuemart_medias?
Thats right - they are correct stored and a not technical user could use them.
Title: Re: Media synchronization stopps after 11k images
Post by: jjk on October 01, 2013, 10:33:57 AM
If your website is on a live server, this hint given by forum member franzpeter might apply to your case:
"If the server execution time is lower than the defined script execution time, the server will stop that script."
Title: Re: Media synchronization stopps after 11k images
Post by: kderg on October 01, 2013, 11:09:36 AM
Thanks forthe hint - i made a copy to a local xampp for testing. I didn't found something like a server execution time to set up - so i shortened the php max execution time to 60. but it only changes the script time to try to resume the migration process. So the message is again:
QuoteFound prior migration process, resume migration maxScriptTime 47 maxMemoryLimit 242
0 Media für product im Verzeichnis images/stories/virtuemart/product/ synchronisiert
0 Media für category im Verzeichnis images/stories/virtuemart/category/ synchronisiert
0 Media für manufacturer im Verzeichnis images/stories/virtuemart/artist/ synchronisiert
0 Media für vendor im Verzeichnis images/stories/virtuemart/vendor/ synchronisiert
0 Media für forSale im Verzeichnis /var/www/web0/files/vmsec/ synchronisiert
Insgesamt 0 Dateien synchronisiert
I already removed all folders so all images are directly in the product folder. The synch is faster - in only two steps at 10418 - but than the message i quoted comes again.
Title: Re: Media synchronization stopps after 11k images
Post by: jjk on October 01, 2013, 12:50:22 PM
Can you upload a screenshot showing the complete blue VM infos (like in your post above) which are displayed after executing 'Synchronize media to VirtueMart' and another screenshot showing the paths entered in your 'Media Files Settings' in VM Configuration > Templates tab?

Do your missing images have a different mime type (jpg, gif, png) than the first 11k? Also note, that on linux systems (case sensitive) 'jpg' is not the same as 'JPG' and the path '/Images' is not the same as '/images'.
Title: Re: Media synchronization stopps after 11k images
Post by: kderg on October 01, 2013, 16:02:11 PM
ok - its nearly solved - but on the hard way - i write it here for others which have the same problem or for the developers to fix it.
Because if i raise the max time
--> the synch-process got only a part of this time longer
--> had to be time waisting programming
--> i dived into the migrator and the media-model of VM and made some time-diagnostics:
maxscripttime (before set):120
maxscripttime (after set(0.8)):95
before get stored Media: 0.0085031986236572
before exeSortSearchList: 0.0002140998840332
after exeSortSearchList: 0.036107063293457
after explode data: 0.036119937896729
timebeforerun:0.0031430721282959
timeperobejcts:39.016096830368
timeper1.operation:5.0992753505707
timeper2.operation:3.9622149467468
timeper3.operation:0.026073932647705
timeafterrun: 39.048449993134
after createMediaByIds: 39.084590911865
after get stored Media: 39.09311413765
after run through stored Media (before port): 40.086740016937
now before _portbyMedia(product): 40.08677315712
now knownNames: 40.160742044449
stored Medias: 10418
known Names: 10402
ct:0-BREAK LVL3: 40.161013126373 >= 38
ct:0-BREAK LVL2: 40.161020994186 >= 38
ct:0-BREAK LVL1: 40.161028146744 >= 38
Files in dir: 0
now before _portbyMedia(category): 40.162118196487
now knownNames: 40.261563062668
stored Medias: 10418
known Names: 12
ct:0-BREAK LVL3: 40.261991977692 >= 38
ct:0-BREAK LVL2: 40.262000083923 >= 38
ct:0-BREAK LVL1: 40.262006044388 >= 38
Files in dir: 0
now before _portbyMedia(manufacturer): 40.262067079544
now knownNames: 40.347643136978
stored Medias: 10418
known Names: 3
ct:0-BREAK LVL3: 40.348098039627 >= 38
ct:0-BREAK LVL2: 40.348106145859 >= 38
ct:0-BREAK LVL1: 40.348112106323 >= 38
Files in dir: 0
now before _portbyMedia(vendor): 40.348170995712
now knownNames: 40.431425094604
stored Medias: 10418
known Names: 1
ct:0-BREAK LVL3: 40.431828022003 >= 38
ct:0-BREAK LVL2: 40.431836128235 >= 38
ct:0-BREAK LVL1: 40.431842088699 >= 38
Files in dir: 0
now before _portbyMedia(forSale): 40.431897163391
now knownNames: 40.514025211334
stored Medias: 10418
known Names: 0
finished: Insgesamt 0 Dateien synchronisiert

--> now it was clear, that the search through files only could start under 38s - if you have 120s script-time (0.4*0.8*scriptime)
--> so by searching in the media model the time eater was an if condition (timeperobejcts is long (39s) - but the operations in the if case only took 8s)
--> and i had luck: the array testing in createMediaByIds could get solved much better:

//array_key_exists is very much more expensive than isset - but it has an advantage - if the value of the array the key points to is NULL - it gives true back --> this matters only if the object at the key is used
if (!array_key_exists ($id, $_medias)) {
//becouse $_medias[$id] is not used becouse it gets in every case set by
$_medias[$id] = VmMediaHandler::createMedia($data,$file_type,$mime);
//we can replace the key check through the much faster
if (!isset($_medias[$id])) {

now it runs smoothly through the array.
Same problem in the migrator method _portMediaByType - there is an array-string check done by in_array($relUrlName, $knownNames) -_> not necessary because the string could not be NULL in a useful case so change in the method _portMediaByType the following 4 lines

$foldersInDir = array();
//add after this or two lines before - but after the loop that filles knownNames:
$knownNames2 = array();
                foreach(array_values($knownNames) as $kN)
                {$knownNames2[$kN] = 1;}
//this is an extra work - but much faster in all than the in_array check - now change that in_array:
if(!in_array($relUrlName, $knownNames)){
//to
if(!isset($knownNames2[$relUrlName])){

Now you are able to read much more images in less time into the database:
maxscripttime (before set):120
maxscripttime (before after set):95
before get stored Media: 0.0096321105957031
before exeSortSearchList: 0.00019311904907227
after exeSortSearchList: 0.035093069076538
after explode data: 0.035104036331177
timebeforerun:0.0022590160369873
timeperobejcts:6.9510846138
timeper1.operation:3.4394881725311
timeper2.operation:3.4311983585358
timeper3.operation:0.023700952529907
timeafterrun: 6.9792010784149
after createMediaByIds: 7.0143239498138
after get stored Media: 7.0239770412445
after run through stored Media (before port): 8.0393559932709
now before _portbyMedia(product): 8.0393991470337
now knownNames: 8.1249430179596
stored Medias: 10418
known Names: 10402
now knownNames2: 8.1320929527283
Files in dir: 11564
finished: Die Synchronisation ist noch nicht fertig, bitte erneut ausführen

after all images i got now the time
maxscripttime (before set):120
maxscripttime (before after set):95
before get stored Media: 0.0059220790863037
before exeSortSearchList: 0.00013494491577148
after exeSortSearchList: 0.053668022155762
after explode data: 0.053678035736084
timebeforerun:0.0026390552520752
timeperobejcts:14.623208045959
timeper1.operation:7.2240476608276
timeper2.operation:7.228611946106
timeper3.operation:0.049824953079224
timeafterrun: 14.678614854813
after createMediaByIds: 14.732311964035
after get stored Media: 14.738251209259
after run through stored Media (before port): 16.786662101746
now before _portbyMedia(product): 16.786694049835
now knownNames: 16.969238996506
stored Medias: 21244
known Names: 21228
now knownNames2: 16.985204219818
Files in dir: 761
now before _portbyMedia(category): 55.267039060593
now knownNames: 55.46275305748
stored Medias: 21244
known Names: 12
now knownNames2: 55.462796211243
ct:0-BREAK LVL3: 55.463193178177 >= 38
ct:0-BREAK LVL2: 55.46320104599 >= 38
ct:0-BREAK LVL1: 55.463207006454 >= 38
Files in dir: 0
now before _portbyMedia(manufacturer): 55.463271141052
now knownNames: 55.652594089508
stored Medias: 21244
known Names: 3
now knownNames2: 55.652621030807
ct:0-BREAK LVL3: 55.653075218201 >= 38
ct:0-BREAK LVL2: 55.653084039688 >= 38
ct:0-BREAK LVL1: 55.653090000153 >= 38
Files in dir: 0
now before _portbyMedia(vendor): 55.65314912796
now knownNames: 55.835015058517
stored Medias: 21244
known Names: 1
now knownNames2: 55.83504319191
ct:0-BREAK LVL3: 55.835482120514 >= 38
ct:0-BREAK LVL2: 55.835489034653 >= 38
ct:0-BREAK LVL1: 55.835494995117 >= 38
Files in dir: 0
now before _portbyMedia(forSale): 55.835554122925
now knownNames: 56.046247005463
stored Medias: 21244
known Names: 0
now knownNames2: 56.046269178391
finished: Insgesamt 761 Dateien synchronisiert


I hope this is reasonable for other - please message me if you take this into VM. Than i could sleep quite if updates come along...
Title: Re: Media synchronization stopps after 11k images
Post by: Milbo on October 01, 2013, 18:26:09 PM
A bit too late, can you create a svn patch out of it? I cant find this, I think you worked on a bit outdated version maybe? The explanation is adequate.
Title: Re: Media synchronization stopps after 11k images
Post by: kderg on October 01, 2013, 19:09:33 PM
What do you mean with to late?
Version is - like i mentioned above - 2.0.22c. I saw now the version 2.0.24 is out --> but there it is the same...
Im working with Git - here the complete functions with added mini-documentation of the 2 changes parts (the second is - if you want to take it an improved to the suggestion i made in the last post):
1.: administrator\components\com_virtuemart\helpers\migrator.php
private function _portMediaByType($url, $type){

$knownNames = array();
//create array of filenames for easier handling
foreach($this->storedMedias as $media){
if($media->file_type == $type){
//Somehow we must use here the right char encoding, so that it works below
// in line 320
//CHANGED FOR PERFORMANCE
$knownNames[$media->file_url] = 1;
//END-CHANGED FOR PERFORMANCE
}
}

$filesInDir = array();
$foldersInDir = array();

if($type!='forSale'){

$path = str_replace('/', DS, $url);
$foldersInDir = array(JPATH_ROOT . DS . $path);
} else {
$foldersInDir = array($url);
}

if (!is_dir($foldersInDir[0])) {
vmError($type.' Path/Url is not set correct :'.$foldersInDir[0]);
return 0;
}

while(!empty($foldersInDir)){
foreach($foldersInDir as $dir){
$subfoldersInDir = null;
$subfoldersInDir = array();
if($type!='forSale'){
$relUrl = str_replace(DS, '/', substr($dir, strlen(JPATH_ROOT . DS)));
} else {
// vmdebug('$dir',$dir);
$relUrl = $dir;
}
if($handle = opendir($dir)){
while(false !== ($file = readdir($handle))){

//$file != "." && $file != ".." replaced by strpos
if(!empty($file) && strpos($file,'.')!==0  && $file != 'index.html'){

$filetype = filetype($dir . DS . $file);
$relUrlName = '';
$relUrlName = $relUrl.$file;
// vmdebug('my relative url ',$relUrlName);

//We port all type of media, regardless the extension
if($filetype == 'file'){
//CHANGED FOR PERFORMANCE
if(!isset($knownNames[$relUrlName])){//!in_array($relUrlName, $knownNames)){
//END-CHANGED FOR PERFORMANCE
$filesInDir[] = array('filename' => $file, 'url' => $relUrl);
}
}else {
if($filetype == 'dir' && $file != 'resized' && $file != 'invoices'){
$subfoldersInDir[] = $dir.$file.DS;
// vmdebug('my sub folder ',$dir.$file);
}
}
}

if((microtime(true)-$this->starttime) >= ($this->maxScriptTime*0.4)){
break;
}
}
}
$foldersInDir = $subfoldersInDir;
if((microtime(true)-$this->starttime) >= ($this->maxScriptTime*0.4)){
break;
}
}
if((microtime(true)-$this->starttime) >= ($this->maxScriptTime*0.4)){
break;
}
}

$i = 0;
foreach($filesInDir as $file){

$data = null;
$data = array('file_title' => $file['filename'],
    'virtuemart_vendor_id' => 1,
    'file_description' => $file['filename'],
    'file_meta' => $file['filename'],
    'file_url' => $file['url'] . $file['filename'],
    'media_published' => 1
);

if($type == 'product') $data['file_is_product_image'] = 1;
if($type == 'forSale') $data['file_is_forSale'] = 1;

$this->mediaModel->setId(0);
$success = $this->mediaModel->store($data, $type);
$errors = $this->mediaModel->getErrors();
foreach($errors as $error){
$this->_app->enqueueMessage('Migrator ' . $error);
}
$this->mediaModel->resetErrors();
if($success) $i++;
if((microtime(true)-$this->starttime) >= ($this->maxScriptTime)){
vmError('Attention script time too short, no time left to store the media, please rise script execution time');
break;
}
}

return $i;
}


2.: administrator\components\com_virtuemart\models\media.php
function createMediaByIds($virtuemart_media_ids,$type='',$mime='',$limit =0){

if (!class_exists('VmMediaHandler')) require(JPATH_VM_ADMINISTRATOR.DS.'helpers'.DS.'mediahandler.php');

$app = JFactory::getApplication();
$lang =& JFactory::getLanguage();
$medias = array();

static $_medias = array();

if(!empty($virtuemart_media_ids)){
if(!is_array($virtuemart_media_ids)) $virtuemart_media_ids = explode(',',$virtuemart_media_ids);

//Lets delete empty ids
//$virtuemart_media_ids = array_diff($virtuemart_media_ids,array('0',''));

$data = $this->getTable('medias');
foreach($virtuemart_media_ids as $k => $virtuemart_media_id){
if($limit!==0 and $k==$limit and !empty($medias)) break; // never break if $limit = 0
if(is_object($virtuemart_media_id)){
$id = $virtuemart_media_id->virtuemart_media_id;
} else {
$id = $virtuemart_media_id;
}
if(!empty($id)){
//CHANGED FOR PERFORMANCE
if (!isset($_medias[$id])){//!array_key_exists ($id, $_medias)) {
//END-CHANGED FOR PERFORMANCE
$data->load((int)$id);
if($app->isSite()){
if($data->published==0){
$_medias[$id] = $this->createVoidMedia($type,$mime);
continue;
}
}
$file_type = empty($data->file_type)? $type:$data->file_type;
$mime = empty($data->file_mimetype)? $mime:$data->file_mimetype;
if($app->isSite()){
$selectedLangue = explode(",", $data->file_lang);
//vmdebug('selectedLangue',$selectedLangue);
if(in_array($lang->getTag(), $selectedLangue) || $data->file_lang == '') {
$_medias[$id] = VmMediaHandler::createMedia($data,$file_type,$mime);
if(is_object($virtuemart_media_id) && !empty($virtuemart_media_id->product_name)) $_medias[$id]->product_name = $virtuemart_media_id->product_name;
}
} else {
$_medias[$id] = VmMediaHandler::createMedia($data,$file_type,$mime);
if(is_object($virtuemart_media_id) && !empty($virtuemart_media_id->product_name)) $_medias[$id]->product_name = $virtuemart_media_id->product_name;
}
}
if (!empty($_medias[$id])) {
$medias[] = $_medias[$id];
}
}
}
}

if(empty($medias)){
$medias[] = $this->createVoidMedia($type,$mime);
}

return $medias;
}
Title: Re: Media synchronization stopps after 11k images
Post by: e-trader on November 07, 2013, 16:13:11 PM
Thanks so much for these improvements, this has helped with migrating one of our sites with 19k images. Previously it was not possible to finish it, now they are synchronizing.
Title: Re: Media synchronization stopps after 11k images
Post by: Parsimonious on December 14, 2013, 22:22:32 PM
Thank you! Thank you! Thank you!

I've been running our store on Virtuemart 2 for over a year now and all of a sudden last week I ran into the same problem you described. Everything would look normal, but 0 media would sync. My Joomla  & Virtuemart installations are both up to date so I had no idea what to do. Today I found your post and I took your changes and pasted them into VM 2.0.26 ran my synch and it worked. I am extremely relieved as our store continues to grow and I have hundreds of new images I need to add. I just wish they'd fix it in the core updates so I don't have to remember to re-modify the files every time.
Title: Re: Media synchronization stopps after 11k images
Post by: Wedal on September 30, 2015, 05:45:00 AM
This solution relevant for VM3.0.9. Thank you, kderg!

Milbo, please consider that solution and, if possible, add it to the code VM3.

Array_key_exists and in_array functions are executed in a loop iterating through all media. If media items > 10k, then
we have more than 100M iterations.
Title: Re: Media synchronization stopps after 11k images
Post by: Milbo on September 30, 2015, 22:51:22 PM
Thank you for digging that out. Is added. I will committ it soon.
Title: Re: Media synchronization stopps after 11k images
Post by: Archsakas on April 03, 2016, 23:23:46 PM
It did not solve my problem and made me some more problems with images. Now all images information was resetted. "Unique Filename", "Displayed image subtitle" and "Image Alt-Text" all have changed to default file name..
And if I upload new image in subfolder in images/stories/virtuemart/product it does not sinchronize in virtuemart media manager. It works fine if I upload in product folder.