BuddyPress.org

The tables in the db don’t matter if the plugin that they belong to isn’t running. I guess I’m not understanding what the problem is. The site functions fine but you see errors about 404s and 500s in the log when bots come through looking for things that aren’t there?

If so then you’ll just have to wait until the bots index your current site properly. What’s the big deal? This isn’t a bp issue.

On another topic…is it good to create a ginormous robots.txt file? I’m wondering if that slows down your site at all.

I would place only what you need in your robots.txt file. The link I provide was just to show you that there are many, many bad bots out there. I would not implement the robots.txt file they recommend.

The fact hat yu are getting 500 instead of 404 error codes is a good clue. There must be some specific error showing up in Apache’s error log whenever that is fired. Please create the 500 error again, notice the exact time, and then check the log. What does it say?

Also, sometimes you can get a 500 error if directory or file permissions are not properly set. Where appropriate, they should be 755. Everything in /plugins/ that is BuddyPress specific should be set to 755.

A couple of additional questions while I’m at it:

1. Have you searched the WPMU forums for 500 Internal Server Error thread? I think this may be more related to you WPMu install than BuddyPress?
2. Are you using some special permalink structure? That should not be causing the lack of 404’s but every bit of info helps.
3. Did you carefully follow the readme.txt file that comes with WPMU?
4. Is this a new WPMU 2.7.1 install, or did you upgrade from an older version?

Rich-

Just a couple more thoughts while I think about your situation.

Have you determined the offending party that keeps triggering the redirects? Is it Googlebot-image? Have you disallowed it from searching your site with a robots.txt file?

If so, then it probably is not actually the Googlebot-image spider but a spam spider attempting to masquerade as the Google image bot. A robots.txt file cannot stop a spider from crawling your site. Reputable search engines honor the robots.txt file requests, unscrupulous ones ignore it.

For your information:

http://www.webmasterworld.com/search_engine_spiders/3204487.htm

http://perishablepress.com/press/2009/03/29/4g-ultimate-user-agent-blacklist/

Are you using Firefox? With Firefox and the Web Develoepr Add-on, you can get a lot of information about errors.

301 Redirects:

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=93633

http://www.seomoz.org/blog/url-rewrites-and-301-redirects-how-does-it-all-work

Using your robots.txt file to block googlebot-image from discovering your image files:

http://www.google.com/support/webmasters/bin/answer.py?answer=35308

Make sure you block only the image directories that you do not want indexed.

Also, if you do not use Google’s Webmaster Tools, you may want to investigate. This service offers many tools, including a list of which files are throwing 404s. http://www.google.com/webmasters/tour/tour1.html

Do you mean while I get this situation worked out? Or do you mean that people are working on this because numerous people have been getting this issue?

I’m not sure that the word numerous applies as I’ve only seen a half dozen or fewer threads with this particular issue. Remember, one person’s 404 is not necessarily the same as the next persons.

This type of issue often comes down to one or more factors: bad install, using too old a version of WPMU or BP, plugin conflict, custom theme issue, apache issue, ISP limitations.

We can try helping you with most of these but do not have any control outside the core of the BuddyPress code. We investigate as best we can with the information provided and if it is caused by BuddyPress, offer a fix in trunk.

I see in another thread that you are on Slicehost. I’m with them as well. Since you have to set up your own slice, there may be an apache issue. So, to better understand you particular issue, please provide some more details:

1. Have you tried deactivating all non-buddypress plugins, wp-super-cache in particular?
2. Have you tried using the default .htaccess file that ships with WPMU?
3. Are there any PHP errors in your log files?
4. what are the other listed errors in your apache log?

So i traced the error to my access log and I found that these errors are being caused by some site requesting a file that no longer exists

While this issue is being worked out, you should either block the offending site(s) from attempting to accessing the non-existent files, or you should do a 301 redirect of the moved files–or at least the moved directories.

Furthermore, you can block the Googlebot-Image bot in your robots.txt file. Although the image bot should not be causing the issue since it is just searching your site gathering information. It is not requesting files, it is discovering them.

A number of years ago I offered email to my users and in the end there were thousands of people using my servers for email, including spam bots.

Here is the biggest problem that I faced.

As there was so much ‘spam’ coming from my domains, I was blacklisted on aol, and yahoo, and a few others. To this day, I still cannot use those domains to send email to aol etc…. Once blacklisted, it seems to be forever.

While I think the idea is cool, I cannot see a reason that I would ever offer email to my users other than my admins/moderators for internal communication.

just my 2cents

I had a problem with spam without reCaptcha

reCaptcha kills spam bots and I have no more spam.

Good luck

I had an idea of another way to combat the constant comment spam and registrations from bots, though I don\’t know if it would work.

As the bots don\’t read the page like humans do, they just search for and find the forums to fill in, brings up an interesting idea.

What about using \’hidden\’ fields that are not displayed on the screen itself.

The idea being that if data is input into the field it could not possibly be a human, so we then know to \’not activate\’ or error out the comment/registration etc…

The ‘auto spam bot programs’ that I have come across all look for specific fields etc. We could do something like name the hidden field ‘password’ etc, so the bot thinks it’s legit, but it’s not.

Does anyone know if the bots are \’smart enough\’ to figure this out and get around it?

Ahoy!

Do I need to make any changes to make Burt\’s mod work on RC2 ?

And…will this prevent the search engine bots from grabbing member profile information?

Thanks for this guide, very helpful!

I was trying to get this working myself before i found this.

2 things that were a must for me…

Alpha version on bbpress was the only version that worked

Don’t touch the cookie integration, skip that bit and just do the DB integration.

Oh and there is a problem with duplicate content if you care about google spidering it if users find the default forum and link to it.

There just add a robots.txt file with

User-agent: *

Disallow: /bbpress/

You could probably put a header redirect on it too to stop people using it and only using the bp side.