Sitemap Generator WordPress Plugin Hits Server Memory Limits
April 9th, 2009Google (XML) Sitemap Generator Plugin is a popular wordpress plugin which can generate a search engine compatible sitemap of your blog in seconds … but it was leading to Dreamhost to terminating our wordpress scripts and resultant internal server errors.
Internal Server Error Causes
Following investigation into cause of our persistent internal server errors, Dreamhost support kept informing us repeatedly to identify the wordpress scripts which were tripping the server.
I’ve checked into why you’ve been receiving these internal server errors, and it seems your scripts have been getting automatically killed by our Process Watcher due to your sites going over Memory limits on the shared server.
I would highly recommend that you follow the steps in the following wiki article in order to reduce your usage: http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage
Also, please be aware that just because this site is the one getting killed, it isn’t necessarily the problematic one. For instance, if Site A is using 90% of your allotted memory usage, and then Site B attempts to use an additional 15%, Site B will be 5% over and will get it’s script killed.
I’d also recommend looking into any 3rd party plugins you may be running, especially if you happen to be running Wordpress installs as they can be notoriously poor at memory management. (Secret Tip: Removing them from the plugins directory is more effective then simply disabling them) Lastly, you may want to look into DreamhostPS, as you will be able to raise your memory limits to whatever limit you’d like.
Sitemap Generator Wordpress Plugin and Memory Usage
So how much memory was Sitemap Generator consuming everytime it generated a sitemap of over 3000 urls? Around 25MB!
And to top that it was eating up that much memory every time we edited, deleted or created new articles. That was a setting which is usually enabled to have the most current sitemap.
And when I often tried to rebuild the sitemap manually, I got this error
Basically the script was taking too much memory (from the entire server which also hosts other sites) everytime it generated a sitemap with so many urls and the Dreamhost Process Watcher used to stop the scripts automatically due to your sites going over Memory limits, which basically stopped all WordPress processed till some tech support guy manually released the process watcher and activated the site. It was only thanks to caching that the site traffic sustained despite these errors.
Though this happened several times over the last few weeks, I was able to identify the plugin causing the memory usage spikes only recently. Thereafter I deactivated the plugin. It is obvious that if we need to run this plugin with all that memory usage, it is not possible on shared hosting and you need to upgrade to Dreamhost Private Server where the memory limit is as high as you set it.
Advanced Sitemap Settings
But the sitemaps are essential to let search engines know about your latest urls. The plugin author has created a wonderful FAQ which explains all these issues. Some which relates to us is
- Limit the number of posts If you have problems with the maximum execution time or memory limit you can limit the number of posts which will be included in the sitemap. Newer posts are included first so your sitemap will stay up-to-date.
- Increase the memory limit Building the sitemap needs a lot of memory. If the memory size is limited via configuration and the script can’t finish the sitemap, you can try to increase this limit by entering a higher value. The values are in megabytes so you can start with “2M” for smaller sites and raise the number until it works. However, it could be that you don’t have the permission to change this value so if it still doesn’t work and you already tried a very high value like 16M, you will need to contact your webhoster and ask him to raise it for you.
- Increase the maximum execution time Like the memory, the maximum execution time can also be limited. If the script doesn’t finish, try to set the time limit to “0″ which means unlimited or a high value like “30″ seconds.
Since it is already consuming too much memory, raising the memory or script execution time might not help. So we can continue to use the plugin after limiting the number of posts, which seems like the most practical solution as of now.
We also upgraded from PHP4 to PHP5 and both these fixes should take care of our internal server errors. Keeping our fingers crossed …
NOTE: The Google (XML) Sitemap Generator plugin is a very useful and highly recommended plugin. This article is for informational purpose to inform that large wordpress blogs will require larger memory to generate sitemaps and sometimes it may be beyond allowed memory limits on shared hosting servers. So either you upgrade to VPS hosting or tweak the sitemap plugins to work on a limited basis.
Subscribe RSS feed
That’s excellent information, thank you.
that’s fantastic to know.
it’s a pain to have shared host for wordpress bloggers. but it’s what webhosting provided for blogging, they should be able to raise a few bars themselves.. I know shared-hosting is not a best processing power, but web hosting should provide what’s necessary too. I am thinking, dedicated server is the way to go..
great read.
Great to know this. I have to doublecheck my blog to see how much memory this plugin used to generate sitemap to avoid this issue. Thanks
Th sitemap plugin is a very useful Wordpress plugin. If that is the case as you state here, it means that for old blogs that grew large over time, VPS hosting is really the way to go. That’s where everyone is going to.
Now there is some insight I was not aware of. Thanks a lot for sharing. I wonder if the problem is persistent to all hosting companies or is it just dreamhost and a handfull others?
Time to investgate the issue further I guess. Thanks for the heads up once again.
I got “Internal server error”. How can I reapir this? I can not lod to admin area of wordpress.
Or, you could have emailed the plugin creator privately, rather than slating his plugin publicly. And it isn’t an “all users would still need to be told” issue, as wp-admin would have alerted them to an update to the plugin. Not to mention the fact that it’s your own setup issue. OR the fact that if you aren’t allowed to use more than 25mb on a running process on a site this big, you need to stop being so stingy and go and get a REAL web host.
Classy work.
@Linn – that is a limitation of shared hosting.
@Tinh – it depends on hiw much memory your host allows and how memory intensive other sites on your server are.
@deuts – VPS is the way to go when the site becomes large. Most excellent plugins will consume too much memory on larger sites.
@zulfikar- this problem is related to site size, as larger sitemaps will require more memory to generate sitemaps.
@Retroone – Firstly this is an excellent plugin. There is no bug in the plugin, so author was not contacted. I agree it is our hosting issue and this article explains how most sites on shared hosting can edit their settings and keep using this essential plugin.
@Retroone: You need to calm down. No one is trying to deny the value of this plugin. And it’s not that he isn’t allowed to have over 25 MB. If you had read the post and saw the screen shots you would have noted that dreamhost allows 90 MB.
Again, if you’d read the post, you would have seen where the dreamhost tech explained the scripts are terminated due to total usage and not individual. So this site is just the one that happens to tip it over the edge.
My site has only 223 posts and this same plugin reports that it uses 33 MB to generate my sitemap.
The “problem” (if you want to call it that. Frankly, I see this post as merely informative) is that shared hosts overcrowd their servers.
It’s not the fault of the plugin author, not the fault of the blog owner for “his setup”.
there is something wrong also with the latest GD Star ratings.
I’ve done a couple XML coding projects in the past and they always seem to use a shocking amount of memory. Is that what is happening with this plugin?
I’ve never noticed that the plugins page in the admin section shows the time and memory used and a bunch of other statuses.
That’s nice for trouble shooting.
Hello,
I have a problem with this plugin. The sitemap generated by it for my site has 120 URLs.If i make a sitemap with other software it says 154 URLs. I compared the sitemaps and it looks that this sitemap doesn’t include pagination.Ex:www.abc.com- is included but it doesn’t include http://www.abc.com/q/page/2…www.abc.com/q/page/6.Any idea why? Thanks.
Great info but i think the problem only on shared server because i has this issues before and my host keep complaining about memory usage..
when i moved to another host they never complained even though the usage was the same but later on they did so had to change the plugin and use differnet one instead of moving to expensive vps hosting,
but overall small shared server hosting company will complain about anything not as the large ones.
thanks again for the info
Great post, I really appreciate this information.
Our site was getting pulled by our host on several occasions and I couldn’t figure out what was causing such large memory spikes. Going through the error log this time I kept seeing googlesitemapgenerator as an issue and found your post explaining this memory usage.
HUGE help. Now I know what my options are. Keep using this plugin and upgrade our server, or find an alternative to this plugin that might not take up so much memory. Given that I doubt I’ll find that and that my site keeps growing, it looks like upgrading might be the reasonable option.
Retroone, your comment is completely wrong.
thanks for this information.
had this issue today. i upped the memory limit and it worked. though, i also complained to Dreamhost that i too have that 90 mb limit, but i am on PS, with way more allotted than 90 mb, so i just emailed them asking that’s up with that.