Skip to:
Content
Pages
Categories
Search
Top
Bottom

Re: 404 errors turning into 500 errors through internal redirects


Jeff Sayre
Participant

@jeffsayre

Rich-

Just a couple more thoughts while I think about your situation.

Have you determined the offending party that keeps triggering the redirects? Is it Googlebot-image? Have you disallowed it from searching your site with a robots.txt file?

If so, then it probably is not actually the Googlebot-image spider but a spam spider attempting to masquerade as the Google image bot. A robots.txt file cannot stop a spider from crawling your site. Reputable search engines honor the robots.txt file requests, unscrupulous ones ignore it.

For your information:

http://www.webmasterworld.com/search_engine_spiders/3204487.htm

http://perishablepress.com/press/2009/03/29/4g-ultimate-user-agent-blacklist/

Are you using Firefox? With Firefox and the Web Develoepr Add-on, you can get a lot of information about errors.

Skip to toolbar