Experiences with Hosting Downtime, Dreamhost, Googlebot, Shell

The site was down for 36 hours over the weekend as our web hosting provider Dreamhost disabled all sites on the account for overuse of CPU minutes. After multiple unclosed support tickets went with no response… I knew I had to fix it myself and here is how I proceeded …

Site Down

Why was the Site Down?

They had sent me an email for overuse of CPU minutes, which basically meant this site was using much more server resources and stretching the MySQL database more than it optimally should, which compromises functioning of other sites hosted on the same share hosting server. So hosting technical support followed the right course of action – better disable one site, rather than crash the server and make many other sites go down. So all pages were showing 403 Forbidden errors. Here is the email excerpt –

A normal user utilizes under 75 CP, a heavy user utilizes 75-100 CP, and normally problematic users utilize 100-150 CP. You, are utilizing well over double CP minutes as a problematic user would.

It looks like your site(s) have outgrown the shared hosting environment You should look into upgrading to a Private Server (http://www.dreamhostps.com ) ASAP. Or if possibly find out what is making your code over utilize so much resources (http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage ) and put this to a stop before re-enabling your sites.

Please respond to me in regards to this issue, so I can verify that you’re taking a pro-active approach to solving this problem. If you re-enable your sites without writing back, you may be in danger of forcing me to disabling your hosting account.

I thought that it was a stumbleupon traffic spike for our domain management article or 9rules entry, but the traffic stats showed that it was not the case, and the cause was elsewhere.

Searching Alternative Web Hosting

The long downtime also got me thinking about alternative hosting solutions like upgrading to Dreamhost PS (these are the Dreamhost Private servers where I would have to pay around $30 per month extra for 300MB / 300mhz private server, with lots of additional advantages, but its on a few weeks waiting list). I was suggested by readers to look at Mediatemple Grid Servers, Liquidweb VPS, Slicehost VPS hosting too.

But after QOT survived the BBC effect easily, it had to be something else that was causing server load to increase. I love Dreamhost hosting which has provided a good experience over 2 years and I would not abandon it for a little downtime.

How do Tech Support Disable Websites

Over the years, I have learnt some strategies that web hosting services adopted to disable this site.

Rename the Domain folder – They simply rename the domain folder via FTP to something else like domainname.com.old and your site goes offline. Its simple to rename it back via FTP and get it online.

– Block via .htaccess – A simple tweak in .htaccess file can change permissions needed to access the server and can disable the site. I had the opportunity to explore this option when DH blocked Googlebot. Simply edit your .htaccess in any text editor (after you enable viewing hidden files in your FTP client – I use Filezilla) and you are done.

This time both these strategies were not applicable as domain folders were ok and .htaccess was not tampered. However, a friendly DH tech support rlparker helped me out in DH forums and over PM gave me the most useful tech advice ever (with such immense clarity!) that this site is back online again. Much of what I could attempt here is thanks to his guidance.

How to inform the readers?

We maintain a QOT Status blog on Blogpsot, which serves a communication medium when the site is down. I simply redirect this status blog feed via the main QOT feedburner feed and it reaches the 15000 feed readers instantly. I could also use Yahoo Pipes to mashup multiple feeds like that of our tumblelog (which survives because it is hosted on tumblr via a DNS modification) and get some alternate content in with site updates. I could continue doing this till the site was down.

How was the site disabled?

The directory permissions at root were edited such that the site was inaccessible to users, but accessible to me such that I could fix things and identify the causes. This was good and a very sensible move because now you can check you logs, remove corrupted scripts, remove plugins, identify other causes of CPU over overload and fix them yourself. After fixing all these issues, you can now activate the sites yourself too.

How did I Get the Blog Online?

It was clear I needed to alter the directory permissions to let in users to get the site online. But I first had to fix the cause of CPU overload, as activating the site without any corrective measures would again mean a sudden burst on server load, possibly crashing it and sending other sites offline, which would have invited more severe disabling measures from Dreamhost.

This forced me to learn more about Shell and SSH, which was what I needed to fix this issue and I wanted to do it right the first time. First I need to change my Dreamhost user account to enable Shell access. In dreamhost dashboard, Go to User > Manage users > Edit

Activate Shell User

Then I grabbed PuTTY, a free SSH, Telnet, rlogin, and raw TCP client to connect to the server and found some settings for PuTTY that work well with DreamHost. I read lots of stuff about using SSH and Unix commands, the Linux BASH command line, UNIX file permissions and this amazing chmod tutorial.

First I needed to identify the causes of heavy usage, and so I accessed ~/logs/yourdomain.com/http and typed
cat access.log| awk '{print $1}' | sort | uniq -c |sort -n

for last 10000 hits, use
tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n

this revealed the IPs hitting the domain the most and we identified thousands of hits were coming from 66.249.67.132. Upon typing host 66.249.67.132, we identified it as googlebot.com

So how do you block googlebot? Googlebot behaves badly sometimes and it is simply blocked by adding this line to .htaccess on your domain top folder
<Limit GET HEAD POST>
order deny,allow
deny from 66.249.67.132
</LIMIT>

You can block 66.249 only to block all Googlebot IP’s. This is a temporary measure as you finally want Googlebot to index your site. You can also slow Googlebot crawl and help your shared hosting. Now we could restore our websites back. Go to the top “.” folder using PuTTy (cd ..), and “chmod” appropriately to get the file directory permission as drwxr-xr-x and the sites went live instantly.

What Dreamhost Should have done?

Instead of disabling the site, they could easily identify the cause of increased CPU usage as Googlebot (its very simple via log analysis), inform the site owner and block Googlebot via .htaccess and that takes care of it all. And of course respond faster to support tickets.

I had to read a lot of stuff about Shell, UNIX and SSH to attempt what I did, and now I have become much wiser and well versed with what seemed cryptic a day ago. And this was truly a learning experience.

NOTE: I am not an expert in technically managing hosting servers and site crashes. This was my personal experience in managing this event to bring the site online. The measures suggested below need expert knowledge and if improperly used can harm your site data and functioning irreversibly. I take no liability for your misadventures. Its always best to seek professional help before attempting any such actions.

Share with friends

12
Leave a Reply

avatar
12 Comment threads
0 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
12 Comment authors
MighteeNickRobertSyahid A.Tim Cat Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Sumesh from Blog Creativity
Sumesh from Blog Creativity

One more reason to switch to another host ;) I wonder why you are clinging on to Dreamhost, when a site of QOT’s traffic and stature deserves much better. I was on Dreamhost until March, but then they had that one day downtime. I was frustrated because – they didn’t inform it (atleast I wasn’t informed) – one day of hosting isn’t acceptable. Even maintenance takes only a few hours. Besides, downtime means fewer hits on the day, and that translates to less new visitors and ad clicks. So, I made the switch to a host that is totally Digg-proof… Read more »

Tom Asaro
Tom Asaro

When you are ready to make the jump, check out Linode Virtual Servers: http://www.linode.com/

We have many Dreamhost “switchers” who can’t believe they didn’t move sooner.

Dj Flush
Dj Flush

Good to see QOT back and its original stable state. I had issues regarding CPU overload with BlueHost so I moved to SliceHost VPS and since then it has been working like a charm.

Happy new life to QOT :)

Davinder
Davinder

Managing blogs isn’t easy once they get little bigger. Inspite of downtime, I still feel dreamhost is better among shared hosts. As such no host can afford to keep heavy usage site online and crash number of other sites on the same server. I was surprised to know, dreamhost offer SSH feature. Usually shared hosts dont offer it. I am also trying to learn SSH – it is a wonderful thing to help you manage site. Thanks for heads up on SSH and dreamhost… time to do some experiments. Also, if you are looking for dedicated hosting – try [theplanet.com]… Read more »

Ross
Ross

Dreamhost has that weird issue with Googlebot. If you need something that can go beyond shared hosting I would look into Slicehost like your readers suggested.

Ross
-http://www.hostdisciple.com

QuickOnlineTips
QuickOnlineTips

Every web hosting has problems. Some advertise it and say sorry, some cover it up. I find Dreamhost is always transparent about its misadventures. I have been with Dreamhost for 2 years, and in all that time there has been a max 4-5 days of downtime, most commonly beacuse of Googlebot hammering and disabling a site becomes essential to keep other sites on the server working. We get angry, but it seems fair enough. I still recommend Dreamhost. If you search enough on the web, you can dig up unhappy customers about any web hosting. I would just wish they… Read more »

Techblissonline Dot Com
Techblissonline Dot Com

Why did googlebot hammer only your site on the server? Strange…

btw…sumesh which webhost do you suggest…

Tim Cat
Tim Cat

Media Temple was always a slow ISP for me although I loved their control panel. I had very much success with myhosting.com. They have a nice community site located at http://portal.myhosting.com. I recommend it…

Syahid A.
Syahid A.

At least their transparent, mine is not like that.

Robert
Robert

Hello! I found this an interesting read. I actually work in the Tech Support dept. of a webhosting company (Visox.com). We have had to disable a few accounts there is no doubt, but the methods that they used to disable your account were really just warnings. When we do it, the apache.conf record for your domain(s) is changed, so that the location of your files is moved from /home/username/ to /home/closed/ Is this way, there is no way you can get around your site getting closed, and you will have to take it up with tech support to get it… Read more »

Nick
Nick

Dreamhost PS service sucks for medium sites. I had a site hosted that averages 1000GB ram usage. It was down 1-2 hours every day all they can say to have patience and wait. After migrating me to other PS all stopped working for 2 days because the migration was not complete. After the tech guy that took 2 days to see what was the problem they put all my user accounts mixed in the same dir erasing and replacing similar filenames. After correcting that i found out my Private Server has been hosting some other user accounts that wore not… Read more »

Mightee
Mightee

same thing happened to me when i was using dream host
they blocked my account but refunded my money later :D