Experiences with Hosting Downtime, Dreamhost, Googlebot, Shell

By Posted 2008 Updated   BloggingDomain HostingSite NewsTutorials

The site was down for 36 hours over the weekend as our web hosting provider Dreamhost disabled all sites on the account for overuse of CPU minutes. After multiple unclosed support tickets went with no response… I knew I had to fix it myself and here is how I proceeded …

Site Down

Why was the Site Down?

They had sent me an email for overuse of CPU minutes, which basically meant this site was using much more server resources and stretching the MySQL database more than it optimally should, which compromises functioning of other sites hosted on the same share hosting server. So hosting technical support followed the right course of action – better disable one site, rather than crash the server and make many other sites go down. So all pages were showing 403 Forbidden errors. Here is the email excerpt –

A normal user utilizes under 75 CP, a heavy user utilizes 75-100 CP, and normally problematic users utilize 100-150 CP. You, are utilizing well over double CP minutes as a problematic user would.

It looks like your site(s) have outgrown the shared hosting environment You should look into upgrading to a Private Server (http://www.dreamhostps.com ) ASAP. Or if possibly find out what is making your code over utilize so much resources (http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage ) and put this to a stop before re-enabling your sites.

Please respond to me in regards to this issue, so I can verify that you’re taking a pro-active approach to solving this problem. If you re-enable your sites without writing back, you may be in danger of forcing me to disabling your hosting account.

I thought that it was a stumbleupon traffic spike for our domain management article or 9rules entry, but the traffic stats showed that it was not the case, and the cause was elsewhere.

Searching Alternative Web Hosting

The long downtime also got me thinking about alternative hosting solutions like upgrading to Dreamhost PS (these are the Dreamhost Private servers where I would have to pay around $30 per month extra for 300MB / 300mhz private server, with lots of additional advantages, but its on a few weeks waiting list). I was suggested by readers to look at Mediatemple Grid Servers, Liquidweb VPS, Slicehost VPS hosting too.

But after QOT survived the BBC effect easily, it had to be something else that was causing server load to increase. I love Dreamhost hosting which has provided a good experience over 2 years and I would not abandon it for a little downtime.

How do Tech Support Disable Websites

Over the years, I have learnt some strategies that web hosting services adopted to disable this site.

Rename the Domain folder – They simply rename the domain folder via FTP to something else like domainname.com.old and your site goes offline. Its simple to rename it back via FTP and get it online.

– Block via .htaccess – A simple tweak in .htaccess file can change permissions needed to access the server and can disable the site. I had the opportunity to explore this option when DH blocked Googlebot. Simply edit your .htaccess in any text editor (after you enable viewing hidden files in your FTP client – I use Filezilla) and you are done.

This time both these strategies were not applicable as domain folders were ok and .htaccess was not tampered. However, a friendly DH tech support rlparker helped me out in DH forums and over PM gave me the most useful tech advice ever (with such immense clarity!) that this site is back online again. Much of what I could attempt here is thanks to his guidance.

How to inform the readers?

We maintain a QOT Status blog on Blogpsot, which serves a communication medium when the site is down. I simply redirect this status blog feed via the main QOT feedburner feed and it reaches the 15000 feed readers instantly. I could also use Yahoo Pipes to mashup multiple feeds like that of our tumblelog (which survives because it is hosted on tumblr via a DNS modification) and get some alternate content in with site updates. I could continue doing this till the site was down.

How was the site disabled?

The directory permissions at root were edited such that the site was inaccessible to users, but accessible to me such that I could fix things and identify the causes. This was good and a very sensible move because now you can check you logs, remove corrupted scripts, remove plugins, identify other causes of CPU over overload and fix them yourself. After fixing all these issues, you can now activate the sites yourself too.

How did I Get the Blog Online?

It was clear I needed to alter the directory permissions to let in users to get the site online. But I first had to fix the cause of CPU overload, as activating the site without any corrective measures would again mean a sudden burst on server load, possibly crashing it and sending other sites offline, which would have invited more severe disabling measures from Dreamhost.

This forced me to learn more about Shell and SSH, which was what I needed to fix this issue and I wanted to do it right the first time. First I need to change my Dreamhost user account to enable Shell access. In dreamhost dashboard, Go to User > Manage users > Edit

Activate Shell User

Then I grabbed PuTTY, a free SSH, Telnet, rlogin, and raw TCP client to connect to the server and found some settings for PuTTY that work well with DreamHost. I read lots of stuff about using SSH and Unix commands, the Linux BASH command line, UNIX file permissions and this amazing chmod tutorial.

First I needed to identify the causes of heavy usage, and so I accessed ~/logs/yourdomain.com/http and typed
cat access.log| awk '{print $1}' | sort | uniq -c |sort -n

for last 10000 hits, use
tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n

this revealed the IPs hitting the domain the most and we identified thousands of hits were coming from 66.249.67.132. Upon typing host 66.249.67.132, we identified it as googlebot.com

So how do you block googlebot? Googlebot behaves badly sometimes and it is simply blocked by adding this line to .htaccess on your domain top folder
<Limit GET HEAD POST>
order deny,allow
deny from 66.249.67.132
</LIMIT>

You can block 66.249 only to block all Googlebot IP’s. This is a temporary measure as you finally want Googlebot to index your site. You can also slow Googlebot crawl and help your shared hosting. Now we could restore our websites back. Go to the top “.” folder using PuTTy (cd ..), and “chmod” appropriately to get the file directory permission as drwxr-xr-x and the sites went live instantly.

What Dreamhost Should have done?

Instead of disabling the site, they could easily identify the cause of increased CPU usage as Googlebot (its very simple via log analysis), inform the site owner and block Googlebot via .htaccess and that takes care of it all. And of course respond faster to support tickets.

I had to read a lot of stuff about Shell, UNIX and SSH to attempt what I did, and now I have become much wiser and well versed with what seemed cryptic a day ago. And this was truly a learning experience.

NOTE: I am not an expert in technically managing hosting servers and site crashes. This was my personal experience in managing this event to bring the site online. The measures suggested below need expert knowledge and if improperly used can harm your site data and functioning irreversibly. I take no liability for your misadventures. Its always best to seek professional help before attempting any such actions.


12 comments on “Experiences with Hosting Downtime, Dreamhost, Googlebot, Shell

  1. Sumesh from Blog Creativity says:

    One more reason to switch to another host ;) I wonder why you are clinging on to Dreamhost, when a site of QOT’s traffic and stature deserves much better.

    I was on Dreamhost until March, but then they had that one day downtime. I was frustrated because
    – they didn’t inform it (atleast I wasn’t informed)
    – one day of hosting isn’t acceptable. Even maintenance takes only a few hours.

    Besides, downtime means fewer hits on the day, and that translates to less new visitors and ad clicks. So, I made the switch to a host that is totally Digg-proof without any caching plugins (I sustained 2238 diggs for one post), and has also got an impressive 100% uptime for most of their servers for the last few months.

    I’m a happy Dreamhost switcher now.

  2. Tom Asaro says:

    When you are ready to make the jump, check out Linode Virtual Servers: http://www.linode.com/

    We have many Dreamhost “switchers” who can’t believe they didn’t move sooner.

  3. Dj Flush says:

    Good to see QOT back and its original stable state. I had issues regarding CPU overload with BlueHost so I moved to SliceHost VPS and since then it has been working like a charm.

    Happy new life to QOT :)

  4. Davinder says:

    Managing blogs isn’t easy once they get little bigger. Inspite of downtime, I still feel dreamhost is better among shared hosts. As such no host can afford to keep heavy usage site online and crash number of other sites on the same server.

    I was surprised to know, dreamhost offer SSH feature. Usually shared hosts dont offer it. I am also trying to learn SSH – it is a wonderful thing to help you manage site. Thanks for heads up on SSH and dreamhost… time to do some experiments.

    Also, if you are looking for dedicated hosting – try [theplanet.com] I have been using it for 3 months now, I works very well!

  5. Ross says:

    Dreamhost has that weird issue with Googlebot. If you need something that can go beyond shared hosting I would look into Slicehost like your readers suggested.

    Ross
    -http://www.hostdisciple.com

  6. QuickOnlineTips says:

    Every web hosting has problems. Some advertise it and say sorry, some cover it up. I find Dreamhost is always transparent about its misadventures.

    I have been with Dreamhost for 2 years, and in all that time there has been a max 4-5 days of downtime, most commonly beacuse of Googlebot hammering and disabling a site becomes essential to keep other sites on the server working. We get angry, but it seems fair enough. I still recommend Dreamhost. If you search enough on the web, you can dig up unhappy customers about any web hosting.

    I would just wish they had a better ticket response time, just to know someone is looking into the issue. Maybe better weekend support too.

  7. Techblissonline Dot Com says:

    Why did googlebot hammer only your site on the server? Strange…

    btw…sumesh which webhost do you suggest…

  8. Tim Cat says:

    Media Temple was always a slow ISP for me although I loved their control panel. I had very much success with myhosting.com. They have a nice community site located at http://portal.myhosting.com. I recommend it…

  9. Syahid A. says:

    At least their transparent, mine is not like that.

  10. Robert says:

    Hello! I found this an interesting read. I actually work in the Tech Support dept. of a webhosting company (Visox.com). We have had to disable a few accounts there is no doubt, but the methods that they used to disable your account were really just warnings.

    When we do it, the apache.conf record for your domain(s) is changed, so that the location of your files is moved from /home/username/ to /home/closed/

    Is this way, there is no way you can get around your site getting closed, and you will have to take it up with tech support to get it fixed.

    It is strange though, that they simply closed your account like that. We almost never have to do that, since when ever the something like this happens, we always identify what the problem is before hand, and recommend solutions to get it fixed. Although I will admit that sometimes we can not determine a specific cause, we still give you time to consider a plan upgrade even if you are over your quota.

    We are rather small, we only have about 200 servers, and about 15 of which are running as virtuals with shared hosting. Dreamhost is much larger, so for them its not as much of a priority to identify the cause.

    Anyways, I enjoyed reading your article and gaining some insight into the workings of another company.

  11. Nick says:

    Dreamhost PS service sucks for medium sites.

    I had a site hosted that averages 1000GB ram usage.

    It was down 1-2 hours every day all they can say to have patience and wait.

    After migrating me to other PS all stopped working for 2 days because the migration was not complete.

    After the tech guy that took 2 days to see what was the problem they put all my user accounts mixed in the same dir erasing and replacing similar filenames.

    After correcting that i found out my Private Server has been hosting some other user accounts that wore not mine and using my payed resources.

    Problems after problems…

    Run from dreamhost PS servers!

    For small sites shared hosting is ok though.

    If you need a service more stable the recommended links in this forum seem good.
    I personally like mediatemple.net.

  12. Mightee says:

    same thing happened to me when i was using dream host
    they blocked my account but refunded my money later :D

Leave a Reply

Your email address will not be published. Required fields are marked *




css.php