BUG verursacht Ranking Probleme! Absulut nicht Suchmaschienen Tauglich.
Wir haben aufgrund der Probleme keinen anderen Ausweg als das Shop per robots.txt zu blockieren!!
Jedes Produkt kann durch Änderung der Kategorie Alias in der URL in einer anderen Kategorie angezeigt werden
Ich erkläre das Problem mal anhand Ihres Demo Shops
Hier das Apple Iphone 4 in der Kategorie Mobile Phones
http://demo.virtuemart.net/index.php/2012-01-13-09-33-20/virtuemart-default-layout/mobile-phones/apple-iphone-4-detail
Durch die Änderung der Kategorie z.B. auf Computer
http://demo.virtuemart.net/index.php/2012-01-13-09-33-20/virtuemart-default-layout/computer/apple-iphone-4-detail
Fazit: Wir haben z.B. Knapp 2500 Produkte im Shop mit 80 Kategorien.
Da alle Crawler diese in allen Kategorien anzeigen lassen = 2500 x 80 = 2000000 Pages im Index!!!! Pages mit doppeltem Content !!!
...............................................................
Probleme entstehen auch mit der Sitemap, da Crawler oder Sitemap Generatoren mit dem Cralwer System was ebenfalls sämtliche Suchmaschienen verwenden ins endlose Laufen
Einzigst Funktionsfähiges Prinzip bietet XMAP, da diese auf Virtuemart zugreifft und die Produkte in der richtigen Kategorie in der Sitemap generiert. LEIDER suchen die SUCHMASCHIENEN auch ausserhalb der Sitemap. UND XMP packt sowieso nur sehr wenig Produkt bis es aussteigt!!
................................................................
Ein weiteres Großes Problem ist das beliebige Alias für Kategorien in der URL angegeben werden können und diese auch ebenfalls gecrawlt werden können.
z.B. Funktioniert auch Apple Iphone 4 in der Kategorie Virtuemart-Mobilphonecenter
http://demo.virtuemart.net/index.php/2012-01-13-09-33-20/virtuemart-default-layout/virtuemart-mobilphonecenter/apple-iphone-4-detail
http://forum.virtuemart.net/index.php?topic=79800.0
Thiz izz a german bug ;D
* VM is already complicated in English *
I think you need to look for a canonical url solution to solve this problem. I have programmed a little snippet for myself to solve this. But blocking your url, will only cause loss of traffic. If you can configure it canonical, then google and other search engines will combine the traffic of both urls and show only one in the index.
I'm still improving my code, but for now I have enough with this. It echos a canonical tag ther where I want it. Pretty basic for the moment.
<?php
//arrays with the page url as key and the target url as value
// page url is the page where the canonical tag should appear
// and the target url is the preffered url
$canonicals=array(
"http://www.pillini.be/en/frequently-asked-questions" => "http://www.pillini.be/en/buying-shoes-and-bags-online/frequently-asked-questions"
//other urls are deleted for overview reasons
);
$page_url=JURI::current();
if (array_key_exists($page_url, $canonicals)) {
echo "<link rel=\"canonical\" href=\"".$canonicals[$page_url]."\"/>";
}
?>
in the future I want to adapt this so that the last part of the sef url is analysed. And if there are two or more url's that exist with the same url ending, a canonical tag is echoed. But I will need the database for this.
No it is not a german bug and we have already canonical URLs.
So the question is what is wrong with them and also it is very important to know the version, because we changed the behaviour for vm2.0.18a
I did try that out using VM 2.0.18a. It is true that with the alias number I can go to the product from every category. But:
If I enter for example:
http://www.mydomain.de/2013-01-06-08-37-22/asus-eeepad-bundle-tf600tg-1b016r-detail.html
it is possible to show the product but VM creates the correct canonical link like:
<link href="http://www.mydomain.de/notebook/tablet-pc/asus-eeepad-bundle-tf600tg-1b016r-detail.html" rel="canonical" />
for example.
But indeed, it seems to be a bug in the router file. Every product can be made visible in every category and even on the start page by simply adding the last part of the url i.e. your product.html for example and that works with every language. Instead of showing an error page, because it is the wrong way to call the product, VM 2.0.18a allows to call the product with the wrong url, but adds the correct canonical url. I agree with fl, that is absolutely bad for crawlers, because they see a lot of useless links which go to the canonical url. Big shops with a lot of categories and maybe 50 000 products can easily have >500 000 possible links.
I'm not sure about this, because the canonical url in VM2 is new to me. But in my opinion it is pretty wrong. I've chosen a product that is shown in two categories.
page 1
the page url is this: http://www.pillini.be/en/webshop/womens-shoes/boot-streep-zwart-9985-detail
canonical is <link rel="canonical" href="http://www.pillini.be/en/webshop/womens-shoes/boot-streep-zwart-9985-detail">
page url = canonical url
page 2 of the same product but in different category
the page url is: http://www.pillini.be/en/webshop/womens-shoes/short-boots/boot-streep-zwart-9985-detail
canonical is <link rel="canonical" href="http://www.pillini.be/en/webshop/womens-shoes/short-boots/boot-streep-zwart-9985-detail">
again...
page url = canonical url
Isn't it supposed to be?
url page 1 -> with canonical url directing to page 2 (because the url is longer)
url page 2 -> no canonical needed
Or am I missing the point here?
My experience to far is that Google doesn't complain about duplicate content (read Google's explanation of what they consider duplicate content) in shops, because Google knows that a product in a webshop often can be reached via several different urls.
Original citation from Google: "Examples of non-malicious duplicate content could include: "...Store items shown or linked via multiple distinct URLs..."
In the example given in the above forum post, the position of your product page SERPs will depend to a certain extent on how good the match of the user's serch term is with your url. If the user searches i.e. for 'Womens short boots', the ranking of the second url would be much higher than the first url - which would be good for you :-)
Crawling my life site Google does a pretty good job displaying only ONE out of the available 'duplicates/different urls' (Google citation: "...we'll identify what we think is the best version"). After the site has been online for a while, I noticed i.e. that Google indexes all child products which share the description with their parent product, but usually displays only the parent product. Overall, Google has indexed approximately 2000 urls on my site and filtered out almost 280000 (looks vastly exaggerated to me, since the site has approximately 600 products only x 2 languages), but obviously this doesn't hurt the ranking at all, since I've got plenty of No1 positions. If Google thinks that your urls are malicious, they will tell you, provided you registered at their webmaster tools.
PS: I currently don't use any sitemap generator for my shop, because imho none of them produces good sitemaps from the shop. XMAP does work, if you make a change in one of it's files (uninstalled it meanwhile and currently I don't remember what I changed to make it work)
@jjk,
but I hope that you agree that this is a bug. It cannot be in a good ecommerce system, that you can reach the items by using the last part of the url (the item itself) from everywhere what has to do with start page, any category, the categories pages a.s.o., even if VM 2 puts the correct canonical url inside the source code. Take a look at shop ware or prestashop demos or open cart to see how it should work. Adding canonical url to cure a logical bug is not the right way, it simply should not happen. The question with VM 2 and the bug is: What are product categories for? Actually they have barely any purpose.
If Google did filter out about 280 000 url, Google did a nice job but there are other web crawlers too and: It is not the job of Google to correct existing bugs in VM 2.
Just my opinion!
Quote from: franzpeter on January 16, 2013, 10:42:38 AM
@jjk, ...but I hope that you agree that this is a bug.
If you mean that it is possible to replace your VM category name with "my-fantasy-catagory-name" in the url of your browser and VM still displays the product, I agree (but I'm not worried about that funny feature :-)
@jjk,
if you declare a bug as feature I agree with you. With a little bit of marketing idle talk we could even say:
VM 2 has an amazing feature, it finds every product anywhere by just entering the last part of the url.
Quote from: Milbo on January 13, 2013, 14:16:34 PM
http://forum.virtuemart.net/index.php?topic=79800.0
Betrifft alle Versionen
Google Ranking faktoren kennt keiner genau. Doch bei Shops über 100 Kategorien dürfen trotz alle dem schnell Probleme mit doppeltem Inhalt auftachen. Fakt ist, das Prinzip ist von grund heraus Falsch. das ist ein VM Core problem.
Einen kleinen schritt zu verbesserung hat WD gemacht http://www.wd-profi.de/virtuemart-erweiterungen/174-virtuemart-seo-
leider müsste diese lösung noch ausgebaut werden, da dort ebenfalls das problem beseht.
Wie gesagt XMAP ist das einzigeste Sitemap programm das dafür ausgelegt ist. Und XMAP Steigt bei ungefähr über 4000 arikel aus und ist nicht mehr in der lage eine Sitemap zu generieren. Andere Programme dir nach dem Crawler prinzip arbeiten. Wie z.B. http://www.xml-sitemaps.com/standalone-google-sitemap-generator.html oder Google selbst generieren ins endlose. Fakt ist. Es ist uns leider nicht mehr möglich eine sitemap zu generieren und wir haben bereits jetzt schon übner 60.000 sites im Google index.
WTF. Und das schlimmste niemanden vom Virtuemart interessiert das Problem und ist darüber bemüht dieses zu beheben. Nicht ein mal ein Ticket wurde erstellt. Thx.
Quote from: fl on January 19, 2013, 14:32:54 PM
Einen kleinen schritt zu verbesserung hat WD gemacht
That was old VirtueMart 1.x stuff and definitely is not compatible with VM2.
Concerning xmap:
xmap might work if you make two adjustments in the code:
In the file components\com_xmap\views\xml\view.html.php around line 44-45 change the max_execution_time
I've got mine set to 10 minutes (you will problably need need more), which is:
@ini_set('max_execution_time',600);
Also I've used a slightly modified version of the file ...\plugins\xmap\com_virtuemart\com_virtuemart.php
(file is attached below)
However, since that generated sitemap is not what I want, I'm not using it. (Google has indexed all my products anyway)[attachment cleanup by admin]
@fl,
genauso ist es. Das einzige, was bisher hilft (allerdings bei vielen Kategorien und bei vielen Produkten unmöglich): redirect per htacess. Im Grunde genommen müsste man sämtliche SEF Fehler per redirect auffangen. Für Produkt xyz in der Subkategorie product/subcategory würde man schon 2 redirects benötigen: einmal um zu verhindern, dass das Produkt über die Startseite abrufbar ist, dann bzgl. Kategorie Produkt. Beide müssten von www.domain.com/xyz.html bzw. www.domain.com/product/xyz.html auf www.domain.com/product/subcategory/xyz.html umgeleitet werden. Dies stelle man sich allein schon mal bei 200 Produkten in 4 Kategorien vor!
Yes, that is true. The only way - for now - would be to write a lot of redirects to the htaccess file. We would need to trap all SEF url problems. For product xyz in a subcategory 'subcategory' with the main category 'product' we already need 2 redirects. To prevent calling the product from the start page without a category url and to prevent to call a product from the main category page without the subcategory. With only 200 products in 4 categories, it may take a lot of time to write redirects for all those misleading links.
And it is not sufficient to add a canonical url! I see a lot of those things in webmaster tools, telling something about double title tags, produced by the strange way VM 2 allows to call a product from everywhere with nearly every url.
Indeed, it is a no go for a shopping cart!
I agree with jjk,
a sitemap should not be necessary if the VM 2 code would produce good results!!!
Quote from: franzpeter on January 19, 2013, 15:44:44 PM
I agree with jjk,
a sitemap should not be necessary if the VM 2 code would produce good results!!!
If VM2 code would produce bad results, then why do most of my products show up in Google in first page positions, many of them at No1? (Usually ahead of my competitor's shops) ;D
I'm pretty shure, part of that ranking is because the of the canonical url in the generated source code of your product pages.
@jjk ... also when you're not logged in to your google account? If i'm logged in to google, my page also shows up in top 5 results, but logged out ... I drop back a few pages.
Quote from: P2 Peter on January 19, 2013, 16:31:25 PM
@jjk ... also when you're not logged in to your google account?
Yes :)
Doesn't seem to make a difference for me whether I'm logged in at Google or not. Maybe because my Google account is assigned to a different domain.
@jjk,
I agree with P2 Peter. If your google account is linked to your page, google shows different, in my case, better results. So without logged in to google account while doing a search, the results are not so good.
I think you misunderstand the purpose of canonical url. A canonical url is good if products are linked for example from a module or the VM start page to show crawlers the right way to reach a product. Canonical urls are not for showing crawlers the right way for every product, because you can reach it from everywhere by VM default functions. A large amount of canonical url may tell a crawler that you use a lot of misleading links (link farming). It is easily visible because those misleading links use the same meta information, the same name, the same sku a.s.o.. I would say that VM 2 does actually produces 80 % of the links in a wrong way and tries to hide something what is called link farming by adding canonical url. We do not need those 80% of those canonical url. Telling crawlers that a misleading link is wrong with an error page or, like prestashop, showing the page with the correct link is the better way.
Google definition of canonical url:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=139394
One problem with many website crawlers probably is, that they follow all links but fail to detect when they enter a loop. If you have a link from product A to product B and also a link from product B to product A, you have an infinite loop. If I understand that correctly, crawlers may generate a lot of wrong links (like described in the first post), but the canonical url for them remains the same. If you change i.e. the category name in your url to 'xxx', VM2 will still display it (I agree that it shouldn't), but the canonical url in the page source code doesn't change.
BTW - since I use generic child variants, which share the description with the parent product, Google recommends me to use differnt meta tag descriptions for the child products, but it certainly doesn't complain about duplicate content, because that NOT duplicate content according to the definition of duplicate content.
okay jjk ... but read carefully!
QuoteTo specify a canonical link to the page http://www.example.com/product.php?item=swedish-fish, create a <link> element as follows:
<link rel="canonical" href="http://www.example.com/product.php?item=swedish-fish"/>
Copy this link into the <head> section of all non-canonical versions of the page, such as http://www.example.com/product.php?item=swedish-fish&sort=price.
the fact is that VM does not use canonical tag correctly. It shows a canonical tag on every product page ... pointing it to the page itself. This is not proper use of the canonical tag as described by Google, so it gets ignored and even worse... in Bing your page could get rejected.
Quote(1) Google doesn't care if you use the canonical tag on hundreds or thousands of pages, where Bing says they will trust the tag less if you do.
(2) Google doesn't care if you link a canonical tag to the same page (i.e. a redirect back to the redirect), but Bing does and says don't do it.
how is this canonical coded? I'm willing to have a look at it.
@jjk,
first of all, Google is not the only web crawler.
As I can read in the provided link on the Google page they write:
Add a rel="canonical" link to the <head> section of the non-canonical version of each HTML page.
Go to a VM 2 product details page and take a look into the source code. I think the product details page for a specific product is the so called canonical url. So why does VM 2 write a rel=canonical into the head of that page too. As far as I understand what they write on the google page the rel="canonical" should be in the head of every page, which provides a link to the 'canonical page' except of the 'canonical page' itself. VM 2 does write that on every of those pages, even the page, which is the 'canonical page'.
So I only can say that VM 2 from the Google point of view produces link farming by exploiting the canonical element for every page it produces. So we have three misconceptions:
1. allowing to access products and categories from everywhere by just adding the final part of the link to any other link from startpage on
2. abusing the canonical tag for link farming by giving every page a canonical link - even the page with the final url, where the canonical link should direct.
3. the product sorting does not work. It does not work in the category page and it does not work in the product detail pages. Clicking on next or previous does not go to the next or previous product. And if you configure from the BE, that it should sort by product_sku as default sorting criteria, the previous/next links do disappear. It does not matter to activate or deactivate product navigation on the detail page, that only works if the default sorting method is product_name. Else, it disappears. There is no case method implemented for sorting in the default sort order definition, not for category and not for product.
And for all those who think it is a template or theme issue. I can reproduce all those bugs with the default Joomla templates and the default VM 2 theme with VM 2.0.18a on localhost or live server.
Quote from: franzpeter on January 19, 2013, 18:50:45 PM
@jjk,
first of all, Google is not the only web crawler.
Yes, but here in Germany it has a market share of approximately 90 percent. My own statistics (for my old shop) show that 85 of the search engine traffic are via Google. The only other relevant search engine here is Bing. Yahoo is being fed by Bing for quite a while already.
QuoteAs I can read in the provided link on the Google page they write:
Add a rel="canonical" link to the <head> section of the non-canonical version of each HTML page.
Yes, but you missed the second option:
"Indicate the canonical version of a URL by responding with the Link rel="canonical" HTTP header."
I think this is what VM2 uses. At least with my VM2 shop this works perfectly. Google indexed ONLY those VM2 products (pages) with exactly this type of specifying the canonical url:
<link href="http://mydomain/de/categoryname/productname-detail" rel="canonical" />All other possible urls (pages) are NOT indexed. So I just suppose it's ok.
Anyway, the devs are aware of the problem and if they are able to locate the problem(s) and find a solution they will take care of this. However, since they are basically just a two person team, you can not always expect an immediate fix. There might be more important tasks at present. As far as I know, nobody got unlisted or suffered from a deep fall in ranking in the search engine results - looks more like at least Google likes VM2 content.
@jjk,
no, I did not miss
Indicate the canonical version of a URL by responding with the Link rel="canonical" HTTP header
Google is clear in what they say:
You can specify a canonical URL in two ways:
Add a rel="canonical" link to the <head> section of the non-canonical version of each HTML page.
or
Indicate the canonical version of a URL by responding with the Link rel="canonical" HTTP header.
So one way is to add the canonical to not canonical url. The second way is to indicate, which page is the canonical version.
It does not say: add canonical to all pages. They use by saying there are two ways an or and not an and!
So the correct way would be to add a canonical link to all non canonical url or add only a canonical link to the canonical page.
VM 2 adds canonical everywhere, even to the canonical url.
The problem is in components>com_virtuemart>views>productdetails/view.html.php
if ($format == 'html')
{
// Set Canonic link
$document->addHeadLink(JRoute::_($product->canonical, true, -1), 'canonical', 'rel', '');
}
$uri = JURI::getInstance();
//$pathway->addItem(JText::_('COM_VIRTUEMART_PRODUCT_DETAILS'), $uri->toString(array('path', 'query', 'fragment')));
$pathway->addItem(strip_tags($product->product_name));
Instead of comparing, whether the url is already a canonical url or not it produces by default for everything a canonical url. If VM 2 would compare the url and the canonical url about equality and saying: if both are equal do not set the canonical tag, if both are unequal do set the canonical tag it sets it always.
Well, it seems like the difference between your and my opinion is that I do want a canonical url for each of my links to a product page. If I have one product assigned to two categories, I do this on purpose, because both different category topics match a product property and give the customer a better chance to find it. The same applies to search engine listings. If a customer searches i.e. for 'category name 2', the search engines will not display it if you have only your canonical url for 'category name 1' being indexed (If you use canonical urls, Google usually only indexes your canonical urls and ommits all others). VM2 does not generate any 'bogus' canonical urls i.e for category names which don't exist, but which you can manually type into the browser to access the product.
However, if you place a product in 50 different categories, that is ihmo more a webdesign bug similar to keyword stuffing.
That help for small pages with about 100 artikles an 3 categories.
Komponente
AceSEF - Joomla SEO Suite for 39 $
http://www.joomace.net/joomla-extensions/acesef-joomla-seo-sef-urls
and
Plugin
VirtueMart AceSEF for 15 $
http://www.joomace.net/downloads/acesef/extensions/virtuemart-acesef
But it has the same problem, and i don´t know if google is statisfied with this solution
For our homepage is it not possibly to mange so a lot of pages.
If´e test it an the generator for create SEF pages is overloaded and fails.
Also it show in all categories
But it ist possible to manage all url´s manually. NO WAY FOR LARGE PAGES
I Think it ist mor as important for the virtuemart team to bring a true
connancial solution or to revise the rework the vm core of this problem
Quote from: franzpeter on January 20, 2013, 09:52:41 AM
It does not say: add canonical to all pages. They use by saying there are two ways an or and not an and!
Yes, it is OR written by coders, not XOR, what you understand. When I ask you, do you want nutella or ham, and you say yes, it is for a coder, correct! Cause OR is not XOR and in the language we use the "or" often as an "either, or", which is a xor.
First I cannot reproduce the sorting bug, which belongs btw completly not to this thread. Be organized, do not mix topics.
Second, a crawler is only checking the links which are THERE, not any wildcard! So I do not understand why it is a problem if you have endless urls to access one product. That is not a problem. There is only a problem, if we display more than one valid link to a product. and even if we do, we set the canonical.
So the general problem for me here is that this whole thread is much too unprecisely. It is like talking about to optimise a tempomat keeping 70mph+-2, but we can only measure the speed with +-5.
Hello Milbo,
thanks for answering. I did find out what could lead to those incorrect paths, which VM accepts and shows as valid paths. It has something to do with VM categories in Joomla menus. Say I have a category structure in VM, which I do want to leave untouched.
An example: There are categories like:
Coffee Machines, Dishes, Kitchenware, Fridges, vaccum cleaners, washing machines, tumblers.
Say I put those categories in a Joomla top menu. So maybe I do not want such a big top menu. So I produce a top menu with coffee machines and kitchenware and laundry machine. I can create an additional menu but do not publish it with those missing categories like vacuum cleaner, washing machines, tumbler and dishes. I would not like to have sef urls like for example kitchenware/dishes and laundry machine/tumbler, laundry machine/washing machines. For short urls I would prefer to keep the root categories as root categories, no matter what I put into the top menu. A way to do so is to create alias links into the top menu which link to the not published other menu, where the true links reside. Using that way would produce a link like /tumber or washing machines instead of laundry machine/washing machines or laundry machine/tumbler. So those categories still remain (for SEF) root categories, even if they reside in top menu as sub categories of laundry machine.
Another example would be if selling printers: color laser, mono laser, inkjet, solid ink, multifunctional devices. So you can create for example a top menu item like printer and put those color laser, mono laser, inkjet, solid ink, multifunctional devices as sub categories into that menu item. But printers need consumables. So there is paper, ink, toner a.s.o. In the VM 2 category structure those items reside in the root category consumables. If you would like to keep those sef urls together with your Printer top menu item, you cannot simply add consumables as subcategory. You would create link structures like /printer/consumables/ink for example. To prevent that, you can create a second menu in Joomla, which is not published and serves only for sef purposes. You can built there the original category structure for the root category consumables and add just an alias to the Joomla top menu (Consumables for example). Doing so allows to call that top menu subcategory with the sef url /consumables/ink or consumables/toner. Without the second menu and without an alias it would display as: printer/consumables/ink for example.
But doing so produces problems with VM 2 SEF. It seems that VM 2 cannot identify a menu alias entry in the correct way. It works somehow but produces reachable links, which should not be reachable. You can call those categories with consumables or with printer/consumables, both work. VM2 does refer to the alias and does refer too to the hidden menu link.
So for now the best thing to prevent those problems is to not use alias links with VM 2 in a Joomla menu.