404 errors turning into 500 errors through internal redirects
-
I am using WordPress MU 2.7.1 and BuddyPress 1.0 and all over my error log is:
Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
So i traced the error to my access log and I found that these errors are being caused by some site requesting a file that no longer exists. Which should create 404 (not found) error, but instead it is redirecting the site to look somewhere else and then that place is telling it to look in the first place, causing an internal redirect which only stops once it reaches the maximum of 10 internal redirects. At this point it puts out a 500 internal error and when there are enough of them tends to hang my PHP and give me whitescreens on any page that isn’t cached. Only rebooting php and my server solves the problem until google or someone else searches for a file on my site that isn’t there, and brings my site to a screeching halt, usually about once or twice a day.
GET /wp-content/mu-plugins/bp-core/images/mystery-man.jpg HTTP/1.1" 500 612 "-" "Googlebot-Image/1.0
(BuddyPress moved from mu-plugins into plugins)
I have been told to check my .htaccess file, but I don’t see anything wrong with it:
<FilesMatch ".(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
Header set Expires "Thu, 15 Apr 2015 20:00:00 GMT"
</FilesMatch>
RewriteEngine On
RewriteBase /
#uploaded files
RewriteRule ^(.*/)?files/$ index.php [L]
RewriteCond %{REQUEST_URI} !.*wp-content/plugins.*
RewriteRule ^(.*/)?files/(.*) wp-content/blogs.php?file=$2 [L]
# add a trailing slash to /wp-admin
RewriteCond %{REQUEST_URI} ^.*/wp-admin$
RewriteRule ^(.+)$ $1/ [R=301,L]
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule . - [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(wp-.*) $2 [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(.*.php)$ $2 [L]
RewriteRule . index.php [L]
<IfModule mod_security.c>
<Files async-upload.php>
SecFilterEngine Off
SecFilterScanPOST Off
</Files>
</IfModule>
# BEGIN WPSuperCache
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
AddDefaultCharset UTF-8
RewriteCond %{REQUEST_URI} !^.*[^/]$
RewriteCond %{REQUEST_URI} !^.*//.*$
RewriteCond %{REQUEST_METHOD} !=POST
RewriteCond %{QUERY_STRING} !.*=.*
RewriteCond %{HTTP:Cookie} !^.*(comment_author_|wordpress|wp-postpass_).*$
RewriteCond %{HTTP:Accept-Encoding} gzip
RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{HTTP_HOST}/$1/index.html.gz -f
RewriteRule ^(.*) /wp-content/cache/supercache/%{HTTP_HOST}/$1/index.html.gz [L]
RewriteCond %{REQUEST_URI} !^.*[^/]$
RewriteCond %{REQUEST_URI} !^.*//.*$
RewriteCond %{REQUEST_METHOD} !=POST
RewriteCond %{QUERY_STRING} !.*=.*
RewriteCond %{HTTP:Cookie} !^.*(comment_author_|wordpress|wp-postpass_).*$
RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{HTTP_HOST}/$1/index.html -f
RewriteRule ^(.*) /wp-content/cache/supercache/%{HTTP_HOST}/$1/index.html [L]
</IfModule>
# END WPSuperCache
Options -IndexesAny help is appreciated.
-
So i traced the error to my access log and I found that these errors are being caused by some site requesting a file that no longer exists
While this issue is being worked out, you should either block the offending site(s) from attempting to accessing the non-existent files, or you should do a 301 redirect of the moved files–or at least the moved directories.
Furthermore, you can block the Googlebot-Image bot in your robots.txt file. Although the image bot should not be causing the issue since it is just searching your site gathering information. It is not requesting files, it is discovering them.
While this issue is being worked out
Do you mean while I get this situation worked out? Or do you mean that people are working on this because numerous people have been getting this issue?
you should either block the offending site(s) from attempting to accessing the non-existent files, or you should do a 301 redirect of the moved files–or at least the moved directories.
How do I block the offenders (especially when one of them is googlebot)? How do I do a 301 redirect of the moved files (this might be the best, except for when they dont exist anymore like old plugins that i no longer use and deleted) ?
Thank you for your prompt reply, too.
301 Redirects:
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=93633
http://www.seomoz.org/blog/url-rewrites-and-301-redirects-how-does-it-all-work
Using your robots.txt file to block googlebot-image from discovering your image files:
http://www.google.com/support/webmasters/bin/answer.py?answer=35308
Make sure you block only the image directories that you do not want indexed.
Also, if you do not use Google’s Webmaster Tools, you may want to investigate. This service offers many tools, including a list of which files are throwing 404s. http://www.google.com/webmasters/tour/tour1.html
Do you mean while I get this situation worked out? Or do you mean that people are working on this because numerous people have been getting this issue?
I’m not sure that the word numerous applies as I’ve only seen a half dozen or fewer threads with this particular issue. Remember, one person’s 404 is not necessarily the same as the next persons.
This type of issue often comes down to one or more factors: bad install, using too old a version of WPMU or BP, plugin conflict, custom theme issue, apache issue, ISP limitations.
We can try helping you with most of these but do not have any control outside the core of the BuddyPress code. We investigate as best we can with the information provided and if it is caused by BuddyPress, offer a fix in trunk.
I see in another thread that you are on Slicehost. I’m with them as well. Since you have to set up your own slice, there may be an apache issue. So, to better understand you particular issue, please provide some more details:
- Have you tried deactivating all non-buddypress plugins, wp-super-cache in particular?
- Have you tried using the default .htaccess file that ships with WPMU?
- Are there any PHP errors in your log files?
- what are the other listed errors in your apache log?
Okay, thank you for helping me, I’ll try to go through your questions one-by-one.
First I’ll let you know what I have:
Slicehost
1GB slice
Ubuntu Hardy LTS 8.04
Running Apache, PHP 5, MySQL 5.0 (followed Pickeled Onion’s walk-throughs on how to set it up, I am by no means an expert, and my very first barebones VPS was this one.)
I help run http://sportsblognet.com where we have WPMU 2.7.1 and BP 1.0, along with bbPress 1.0-alpha6
Have you tried deactivating all non-buddypress plugins, wp-super-cache in particular?
I deactivated all but buddypress, feedwordpress (because our theme will break), wp-super-cache, and doncha’s domain mapping plugin. But maybe I will remove them all for an hour to see if any errors get through
I am removing wp-super-cache right now, but it was tough to get rid of the wp-content/cache folder because it was owned by the root user. But i got it deleted and removed the wp-super-cache stuff in the .htaccess and I’ll guess i’ll have to wait to see how that goes.
Have you tried using the default .htaccess file that ships with WPMU?
I just tried to, but it won’t let me.
My current working .htaccess (minus wp-super-cache stuff)
<FilesMatch ".(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
Header set Expires "Thu, 15 Apr 2015 20:00:00 GMT"
</FilesMatch>
RewriteEngine On
RewriteBase /
#uploaded files
RewriteRule ^(.*/)?files/$ index.php [L]
RewriteCond %{REQUEST_URI} !.*wp-content/plugins.*
RewriteRule ^(.*/)?files/(.*) wp-content/blogs.php?file=$2 [L]
# add a trailing slash to /wp-admin
RewriteCond %{REQUEST_URI} ^.*/wp-admin$
RewriteRule ^(.+)$ $1/ [R=301,L]
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule . - [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(wp-.*) $2 [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(.*.php)$ $2 [L]
RewriteRule . index.php [L]
<IfModule mod_security.c>
<Files async-upload.php>
SecFilterEngine Off
SecFilterScanPOST Off
</Files>
</IfModule>and the WPMU standard one
RewriteEngine On
RewriteBase BASE/
#uploaded files
RewriteRule ^(.*/)?files/$ index.php [L]
RewriteCond %{REQUEST_URI} !.*wp-content/plugins.*
RewriteRule ^(.*/)?files/(.*) wp-content/blogs.php?file=$2 [L]
# add a trailing slash to /wp-admin
RewriteCond %{REQUEST_URI} ^.*/wp-admin$
RewriteRule ^(.+)$ $1/ [R=301,L]
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule . - [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(wp-.*) $2 [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(.*.php)$ $2 [L]
RewriteRule . index.php [L]
<IfModule mod_security.c>
<Files async-upload.php>
SecFilterEngine Off
SecFilterScanPOST Off
</Files>
</IfModule>it wont let me use the standard one, and the only difference that I can see is the expires header setting that i added.
Are there any PHP errors in your log files?
this is my current concern right now
Request exceeded the limit of 10 internal redirects due to probable configuration error.
ALERT - canary mismatch on efree() - heap overflow detected
for the ALERT error, I contacted slicehost and they upgraded my kernel, they said that has fixed other people’s problems (fingers crossed – but no errors since they did that this morning) if that doesnt work i have to go through and install php without the suhosin patch.
most of the other errors are of spam scripts trying to signup and not finding wp-signup (i changed the signup name)
what are the other listed errors in your apache log?
That’s pretty much it.
EDIT: it’s about 30 minutes later and an Internal Redirect Error just showed up in my logs with wp-super-cache out, and all but buddypress plugins out. So it doesn’t look like its a plugin.
t’s about 30 minutes later and an Internal Redirect Error just showed up in my logs with wp-super-cache out, and all but buddypress plugins out. So it doesn’t look like its a plugin.
So, you had all plugins deactivated except BuddyPress?
doncha’s domain mapping plugin
What are you trying to accomplish with this plugin?
Are you running this as a subdomain or subdirectory install?
I’d suggest that you also create a phpinfo file and provide the output of that file to a pastebin that some of us can look at.
Actually i did have some mu-plugins still running, so now everything is out, just buddypress.
With the domain mapping plugin I am using it so that bloggers can use their own domain (ie http://baseballpastandpresent.com) instead of the subdomain way (baseballpastandpresent.sportsblognet.com)
Currently we only have 2 domains which require this plugin, but I’m slated to convert at least 10 more to their own domains within the next week, but I’ve halted that until I get the site running correctly.
This is a subdomain install.
Here’s the phpinfo
Okay, I did not see anything out of the ordinary in your PHPinfo file. Please delete that from pastebin for security reasons.
Actually I did have some mu-plugins still running, so now everything is out, just buddypress.
Okay, with just BuddyPress running now, are you still getting this error?
Nope, the error is gone.
and i checked my access logs and all of the GET requests that were looking for files that weren’t there are now 404 instead of turning into 500.
So, now my plan of attack is to add in my plugins, one-by-one, in order of most needed to frivolous. And try to call files that aren’t on my site. And see how that goes.
Thank you very much. I’ll let you know if anything changes, or what the problem seems to be. I am thinking that it must be a conflict between two plugins, because most of the plugins I use are used by lots of people without problems.
You’re welcome. I’m glad that it is working now, but feel your pain in figuring out (and more importantly resolving) what 3rd-party plugin is causing the issue.
So, now my plan of attack is to add in my plugins, one-by-one
This process is the only approach. Once you figure out what plugin is causing the issue (and hopefully it is a single plugin and not some plugin combination), then you should contact the plugin developer and ask them to help you.
Please let us know what you discover as this will help us help others in the future.
Ok, well all is not well…
I jumped the gun in concluding no internal redirect errors…
after I methodically put a plugin in and tested it, i got an error, and then realized that there were two more earlier that i didnt see.
They happened at 4am, and the site was without all plugins from about 10pm or so last night until about 9am this morning.
So it doesn’t appear to be a plugin…
Please help! I took out all plugins again, except buddypress (and the bbpress support plugin from buddypress)
I really dont know what to do next.
Rich-
Just a couple more thoughts while I think about your situation.
Have you determined the offending party that keeps triggering the redirects? Is it Googlebot-image? Have you disallowed it from searching your site with a robots.txt file?
If so, then it probably is not actually the Googlebot-image spider but a spam spider attempting to masquerade as the Google image bot. A robots.txt file cannot stop a spider from crawling your site. Reputable search engines honor the robots.txt file requests, unscrupulous ones ignore it.
For your information:
http://www.webmasterworld.com/search_engine_spiders/3204487.htm
http://perishablepress.com/press/2009/03/29/4g-ultimate-user-agent-blacklist/
Are you using Firefox? With Firefox and the Web Develoepr Add-on, you can get a lot of information about errors.
The offending party is usually Googlebot-images (but it has been others in the past).
I did what you said a disallowed googlebot-images with robots.txt, it seems to be working, but googlebot might only come through every 6-12 hours so can’t tell yet.
But even if that does work, I can easily recreate the error myself.
If I type in (which doesn’t exist):
http://sportsblognet.com/wp-content/mu-plugins/bp-core/images/mystery-man.jpg
I get a 500 Internal Server Error
If I type in (which doesn’t exist):
http://rotoassist.com/wp-content/mu-plugins/bp-core/images/mystery-man.jpg
It comes up as the proper 404 error
the rotoassist site is a regular WP 2.7.1 site on Site5 (one-click install type)
On another topic…is it good to create a ginormous robots.txt file? I’m wondering if that slows down your site at all.
And yes I use Firefox and have the Web Developer Add-on. The only info it gives me with the error though, is that it is a 500 internal error.
Another question…would you know anything about MySQL? Or do you know who I could talk to? I have some questions on our database and the speed of it.
Thanks for the constant help.
On another topic…is it good to create a ginormous robots.txt file? I’m wondering if that slows down your site at all.
I would place only what you need in your robots.txt file. The link I provide was just to show you that there are many, many bad bots out there. I would not implement the robots.txt file they recommend.
The fact hat yu are getting 500 instead of 404 error codes is a good clue. There must be some specific error showing up in Apache’s error log whenever that is fired. Please create the 500 error again, notice the exact time, and then check the log. What does it say?
Also, sometimes you can get a 500 error if directory or file permissions are not properly set. Where appropriate, they should be 755. Everything in /plugins/ that is BuddyPress specific should be set to 755.
A couple of additional questions while I’m at it:
- Have you searched the WPMU forums for 500 Internal Server Error thread? I think this may be more related to you WPMu install than BuddyPress?
- Are you using some special permalink structure? That should not be causing the lack of 404’s but every bit of info helps.
- Did you carefully follow the readme.txt file that comes with WPMU?
- Is this a new WPMU 2.7.1 install, or did you upgrade from an older version?
The specific error that shows in the error log is
[Fri May 22 15:44:40 2009] [error] [client xx.xxx.xxx.xxx] Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace., referer: https://buddypress.org/forums/topic.php?id=2840
I just clicked the link from an earlier post and the 500 error happened.
xx.xxx.xxx.xxx - - [22/May/2009:15:44:40 -0700] "GET /wp-content/mu-plugins/bp-core/images/mystery-man.jpg HTTP/1.1" 500 612 "https://buddypress.org/forums/topic.php?id=2840" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10 (.NET CLR 3.5.30729) FirePHP/0.3"
the latter is from my access log.
My permissions are set correctly, that’s usually the first thing i check.
now onto a couple more of your questions:
Have you searched the WPMU forums for 500 Internal Server Error thread? I think this may be more related to you WPMu install than BuddyPress?
Yes, I have searched WPMU forums, and even posted on there my problem, but no one reponded. I was pretty sure that it was related more towards WPMU. But these errors only started happening a few weeks ago, after I upgraded to WPMU 2.7.1 and BP 1.0
Are you using some special permalink structure? That should not be causing the lack of 404’s but every bit of info helps.
No, everything I was using was either in the htaccess.dist and the wp-super-cache (which has since been taken out until i find the cause). The permalink structure that I use on http://sportsblognet.com is the “year/month/day/name” option, which is one of the standard ones.
Did you carefully follow the readme.txt file that comes with WPMU?
I just re-read it to be sure, yes, yes I did follow it.
Is this a new WPMU 2.7.1 install, or did you upgrade from an older version?
This is an upgrade from WPMU 2.7, it was first created with WPMU 1.3.3 over a year ago. We’ve been following buddypress since October of 2008 and have been using it since 12/08, when the first beta came out. I’ve upgraded pretty much everytime without problem. And these Internal Redirects only started after the upgrade to WPMU 2.7.1 and BP 1.0 at the beginning of May.
You’ve just got to tear out *everything*. All this domain mapping and other things. Get down to a bare install of bp 1.0 using the default themes and wpmu 2.7.1.
Did you remove your previous bp install from /mu-plugins? Why is it trying to get to an avatar image from there? I suspect it’s all your *other* addons in some combination with the upgrade to 2.7.1 from 1.x and the addition of bp into the mix that is causing your problems.
Take things back to a known, working state. Start with bare 1.x. Then upgrade to wpmu 2.7.1 incrementally. From everything I hear on the wpmu forums this is the recommended procedure. Jumping from 1.x to the latest version in one step isn’t recommended.
When wpmu 2.7.1 is operating correctly without any fancy domain mapping installed. Then install bp 1.0 and get that working. Gradually add in your other plugins and test before moving to the next one. That procedure, by itself, may solve your problems.
The upgrade and installation of new features procedure, that you have been practicing currently, has driven your site insane. You are trying to track down individual symptoms of a completely unknown state of affairs. You’ll just wind up chasing symptoms instead of curing what’s wrong.
You’ve just got to tear out *everything*. All this domain mapping and other things. Get down to a bare install of bp 1.0 using the default themes and wpmu 2.7.1.
I did that, took out all plugins except for buddypress 1.0, and changed the themes to the default buddypress theme, and default buddypress member theme, and used a vanilla .htaccess for wpmu 2.7.1.
Still got an internal redirect when Googlebot-Images came through and looks for something that is not there. I even recreated the error and was able to see the 500 error.
Did you remove your previous bp install from /mu-plugins? Why is it trying to get to an avatar image from there? I suspect it’s all your *other* addons in some combination with the upgrade to 2.7.1 from 1.x and the addition of bp into the mix that is causing your problems
Yes, I removed buddypress RC-1 from mu-plugins before proceeding with installing BP 1.0. I dont know why it is grabbing files that dont exist. But that’s not really why i’m worried about it, it’s more that if any sites links to tries to get a file that is not currently on the site, it creates a 500 error.
I didn’t upgrade from WPMU 1.x to 2.7.1, I went from 2.7 to 2.7.1, and 2.6.5 before that and basically following each and every update that there has been since 1.3.3 last april. Maybe it was confusing in the way I wrote it.
Take things back to a known, working state. Start with bare 1.x. Then upgrade to wpmu 2.7.1 incrementally. From everything I hear on the wpmu forums this is the recommended procedure. Jumping from 1.x to the latest version in one step isn’t recommended.
I went from WPMU 2.7 to 2.7.1, and it was working fine in 2.7 with BP RC-1
But I did have everything stripped out for a matter of 10 hours and i still got the same errors. I do know that some plugins leave tables in the database, should i drop those tables associated with a plugin and see if that works? Because even if I have a clean install of WPMU 2.7.1, then BP 1.0, i still have a database with all of the remnants of the old stuff.
The tables in the db don’t matter if the plugin that they belong to isn’t running. I guess I’m not understanding what the problem is. The site functions fine but you see errors about 404s and 500s in the log when bots come through looking for things that aren’t there?
If so then you’ll just have to wait until the bots index your current site properly. What’s the big deal? This isn’t a bp issue.
I just thought, in case anyone was wondering…
I started a server from scratch (Ubuntu Hardy, Apache 2, PHP 5 MySQL 5) and installed a fresh WPMU 2.7.1 with a brand new bare database, only using the default theme w/out any plugins (w/out buddypress too)…and was STILL able to reproduce the internal redirect.
I then thought to see if other MU installs would react the same way…
so I tested these (plus about 30 others, not all Buddypress sites, some just MU installs)
http://bp-dev.org/wp-cotent/plugins/bp-core/images/mystery-man.jpg
and…
http://www.flokka.com/wp-cotent/plugins/bp-core/images/mystery-man.jpg
(I misspelled “wp-cotent” and the error came)
It happened on MOST WPMU installs (buddypress.org and WordPress.com returned 404 errors)
I realize this is a WordPress Mu issue, so I reported on their forums
https://mu.wordpress.org/forums/topic/12676
But if anyone has an idea as to make these errors a 404 like buddypress and wordpress.com do, Please Help.
Hi,
I think I found tour support request at the MU forums. I just wanted to update the forum over here, too.
The suggested solution to return 400 errors and not 500 for non-existing directories is to search for:
RewriteRule . – [L]
in your .htaccess file and to comment it out:
#RewriteRule . – [L]
This solved the problem for me. But the question is, if this is breaking something else? It seems to work for me so far….
I just found another solution which seems to be a better solution than just removing this rule. I’m sure it’s in there for a reason…
I change this block:
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule . – [L]
to:
RewriteCond %{ENV:REDIRECT_STATUS} !^$ [OR]
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule . – [L]
Adding the top line, did the trick and presented a 404 error page (from Apache).
Cheers!
Bookmark post
there’s entirely new rewrite rules for multisite now.
- The topic ‘404 errors turning into 500 errors through internal redirects’ is closed to new replies.