Author Topic: Googlebot and Adsbot crash our site  (Read 4740 times)

djreason

  • Jr. Member
  • **
  • Posts: 125
    • Game Room and Man Cave Supplies
Googlebot and Adsbot crash our site
« on: October 26, 2012, 20:56:59 pm »
I am running a very heavily hacked version of Joomla 1.5.15 and virtuemart 1.1.4.

I have about 80K SKUs and growing.  I have had to turn off the Googlebot and Google Adsbot from crawling our site through robots.txt as everytime they came to the site, it crashed our server.

Would anyone be willing to look at how our pages are rendering to see if there is a simple fix we can implement so we can go back to being indexed?

thanks

wcpdiesel.com

stinga

  • Contributing Developer
  • Full Member
  • *
  • Posts: 872
    • Squangle ltd
Re: Googlebot and Adsbot crash our site
« Reply #1 on: October 27, 2012, 12:01:55 pm »
How is your site hosted? We don't have an issue...
Stinga.
614869 products in 747 categories with 15749 products in 1 category.
                                             Document Complete   Fully Loaded
                Load Time First Byte Start Render   Time      Requests      Time      Requests
First View     2.470s     0.635s     1.276s          2.470s       31            2.470s      31
Repeat View  1.064s     0.561s     1.100s          1.064s       4             1.221s       4

jenkinhill

  • UK Web Developer & Consultant
  • Global Moderator
  • Super Hero
  • *
  • Posts: 27667
  • Always on vacation
    • Jenkin Hill Internet
Re: Googlebot and Adsbot crash our site
« Reply #2 on: October 27, 2012, 13:28:13 pm »
But they are very old versions...
Kelvyn

Jenkin Hill Internet,
Lowestoft, Suffolk, UK

Unsolicited PMs/emails will be ignored.

Please mention your VirtueMart, Joomla and PHP versions when asking a question in this forum

Currently using VM 3.6.11.10291 on Joomla 3.9.16 PHP 7.0.33

djreason

  • Jr. Member
  • **
  • Posts: 125
    • Game Room and Man Cave Supplies
Re: Googlebot and Adsbot crash our site
« Reply #3 on: October 27, 2012, 18:17:10 pm »
Here are the specs of our server with LiquidWeb.  This is on a cloud server that is unshared.

Size:   'Intel(R) Xeon(R) CPU E5606 @ 2.13GHz', 2133MHz, 4 Core, Sockets: 2   Image:   
CentOS 6.3 64-bit with cPanel Fully-managed
CPU:           Intel Dual Xeon E5606   
Speed:   2133MHz   
RAM:           3950MB
CPUs:   2 Physical CPUs   
Cores:   8 Total Cores   
RAID:   (none)
Disks:   1   
Size:   453GB   
Type:   SATA


stinga

  • Contributing Developer
  • Full Member
  • *
  • Posts: 872
    • Squangle ltd
Re: Googlebot and Adsbot crash our site
« Reply #4 on: October 29, 2012, 20:43:28 pm »
Indeed they are old, but that is not a reason for slowness! :-)
Machine spec looks OK.

So you need to tune the code. Have a look at the mysql slow log see what is slow and start adding indexes (maybe)
Also you need to make sure you have removed things you don't need. Do you have things enabled that offer no benefit to your customers.
Lets have a look at your site and might be able to offer some pointers.
Stinga.
614869 products in 747 categories with 15749 products in 1 category.
                                             Document Complete   Fully Loaded
                Load Time First Byte Start Render   Time      Requests      Time      Requests
First View     2.470s     0.635s     1.276s          2.470s       31            2.470s      31
Repeat View  1.064s     0.561s     1.100s          1.064s       4             1.221s       4

djreason

  • Jr. Member
  • **
  • Posts: 125
    • Game Room and Man Cave Supplies
Re: Googlebot and Adsbot crash our site
« Reply #5 on: October 29, 2012, 21:19:40 pm »
my site is wcpdiesel.com

stinga

  • Contributing Developer
  • Full Member
  • *
  • Posts: 872
    • Squangle ltd
Re: Googlebot and Adsbot crash our site
« Reply #6 on: October 30, 2012, 11:37:36 am »
Time to first byte is slow 1.69 secs not that bad though.

I would let google crawl and have a look at your web logs, tail them in real time and try and see what is going on.
google is pretty adaptive and will slow down if you site is slow.
What do you mean by crash?

Also what is loader.white.gif? it never finish loading on my broswer so the page never completed. It might be a broken gif?
ANd you have some broken images, google won't like that either.
Stinga.
614869 products in 747 categories with 15749 products in 1 category.
                                             Document Complete   Fully Loaded
                Load Time First Byte Start Render   Time      Requests      Time      Requests
First View     2.470s     0.635s     1.276s          2.470s       31            2.470s      31
Repeat View  1.064s     0.561s     1.100s          1.064s       4             1.221s       4

djreason

  • Jr. Member
  • **
  • Posts: 125
    • Game Room and Man Cave Supplies
Re: Googlebot and Adsbot crash our site
« Reply #7 on: October 30, 2012, 14:16:24 pm »
I have a plugin installed so that I can have tabs on my product detail page.  loader.white.gif is part of that extension.  If you can think of a better extension for tabs other than MagicTabs with corresponding Core Design Scriptgenerator Plugin, I am willing to try it to improve page performance.  I also removed the facebook "be the first to like this page" as it constantly slowed load speeds and nobody uses it anyway.

stinga

  • Contributing Developer
  • Full Member
  • *
  • Posts: 872
    • Squangle ltd
Re: Googlebot and Adsbot crash our site
« Reply #8 on: October 30, 2012, 20:38:49 pm »
I wonder if your livechatinc has anything to do with it. my browser keeps getting control.cgi, every 15 seconds.
Can't answer about the tabs, don't use them at the moment.

Standard reply for speed follows...
Remove and uninstall anything you don't need.
Look at every mod/plug left. Is it needed? if not remove.
Again...Look at every mod/plug left. Is it needed? if not remove.
Look at every mod/plug left. Does it enhance the customer experience? If not... remove.

Now you should have something very lean.
Try the default theme, is it any better? If so, contact theme developer to get it fixed. All themes are not created equal!

One think that will slow your site down is the count of products in a category, turn it off, classic example of something that adds nothing for the user (In most cases). Every time a category is shown it has to count the number of products in the category, daft really, I have a count held in the table itself for other reasons.
 
Stinga.
614869 products in 747 categories with 15749 products in 1 category.
                                             Document Complete   Fully Loaded
                Load Time First Byte Start Render   Time      Requests      Time      Requests
First View     2.470s     0.635s     1.276s          2.470s       31            2.470s      31
Repeat View  1.064s     0.561s     1.100s          1.064s       4             1.221s       4