BuddyPress.org

Yes, my BP brothers, there’s a plugin called BP Simple Private and I think it will solve your problems, without having to work with robots.txt

Prior to using BP Simple Private, my site’s member pages were way too accessible.. You could type http://mysite.com/members <- and you’d see the whole members directory.

Prior to using BP Simple Private, you could type http://mysite.com/members/aaronthomas1979, and you’d jump straight to my profile.

With this plugin, people (or search engine bots,) will be redirected to a page of your choosing, and the member pages will not be displayed.

If you already have the issue of member pages being displayed in search results, there’s nothing you can do to immediately fix that, but make the change (get the plugin) and wait for those bots to crawl your site again, but next time, they won’t find those pages.

It works for me..

Hi @Tranny,

Trust me, I completely hear you. You’re right. Ideally anti-spam controls like that should be built-in.

Those are great suggestions. We’ll take a hard look at possibly including those features in future versions of WP-SpamShield as well. One thing you’ll find by using a plugin like WP-SpamShield is that it will block 100% of bots, and it will prevent most human spammers from even registering (which keeps them from even getting to a place where they can spam other users), and it will make it difficult for them to post their spammy messages.

Let us know if we can help in any way.

– Scott

@tranny,

dealing with spammers is a long run work, to not say a never ending work. There is no miraculous plugin or trick to stop them.
And even if you would be a genius coder creator of an extra super original spam shield, you could be sure to became target #1 of all spammers, because in this case, you would represent the absolute challenger of all code breakers !

Back to real life.

Most of updates spam comes directly into the DB. Bots are clever and don’t need to login to do that.
Some spammers are real people, and once they are logged, they do their stuff manually. These people can be isolated, but to do this, you have to find them in the user list. Which is absolutely not easy and time comsumting. And of course, this is not prevention but intervention, after you where spammed.

You could also track IP‘s, but again, this can be helpfull only AFTER you where spammed. But getting ip’s on admin user list is a great way to gain time. Once you have the IP, you can consult many anti spam sites who store any bot and user known to be attackers. And eventually ban them with this plugin.

For now, first thing to do is to clean your user list. Whatever suspect username, like a589xdf or special to BP, Bill UNERHOOD, can be eliminated. The first example use alpha-numeric digits, the second a very well formed first and last name. It’s extremely rare that normal users enter such credentials. In addition to this, you can check their email. Why would you, for example, have members with a polish email (@blogmedykamenty.pl) if you’re in New Zeeland and your site relates about pets ? In this case, you can raisonably doubt about an interest between medicaments and pets ! You can fire such user.

All this may be good and well, but you have also to hardening WP. This means using another table prefix as wp_ at very first. And second, to not use “admin” as user name. Never !
Read also @venutius tutorial

You have to clean out the existing spammers, unfortunately manually. And to avoid upcoming spam.

To calm down the bots – in case you receive dozen of spam daily, close all comments and deactivate notices and messages component in bp settings for 2 or 3 weeks.

Also, in case you’re on a dedicated server, you really need to enforce his security. But this is out of the scope of this forum.

Spam is an endless discussion on the web over years. Search this forum, you’ll find many topics.

Half of my users registrations are reported by buddypress like a spam.
The question is why ? Aside, note that it is WP who controls the registration, not BuddyPress.

Do you use Askimet or some other plugin who controls your registry ?

So I think it’s sufficient? no?

Simply ? NO !

Captchas are helping to determine a human activity, but generally don’t spam users.

Many spam bots go through captcha and many other goes directly to your db. And many many, if not all, can send emails !
Spammers are even cleverer as most door keepers, it’s a sad fact and a great part of that “sport” to prove it continuously.

Here some common tasks explained to avoid spam and other unwanted content on your site. Note that one of the first thing to do is to use another table prefix as the universal knowed wp_ !

@mingjie0409
there has been some discussion of this here: https://buddypress.org/support/topic/bussdypress-title-and-seo-yoast-problem/#post-253510

and here:

https://premium.wpmudev.org/forums/topic/bp-meta-tite-description-for-groups-and-members-pages

and several other places including the suggestions / feedback area amongst others.

I’ve had it fixed a time or two but updates to either bp or themes have broken my fixes.
This is something that google’s webmaster tools screams at you as being bad, and hurts your site the more members you get.

so I added a noindex no follow to the robots.txt file to remove most of the bp pages from being indexed.

To block those files you would use robots.txt, not .htaccess.

you can block google using robots.txt, you can read about that here https://www.elegantthemes.com/blog/tips-tricks/how-to-create-and-configure-your-robots-txt-file

To prevent indexing by google, you need one or more robots.txt files.

Re private – Have you tried this plugin ?
https://wordpress.org/plugins/bp-simple-private/

You could activate some kind of registration honeypot to keep the spam bots out. This technique has worked well for me in the past.

Example: https://en-gb.wordpress.org/plugins/registration-honeypot/

Regarding the confirmation email problem, this seems to be a common problem and one that isn’t easily solved. Many things can result in email going into spam, most are usually related to your mail server setup or the user’s own spam filter settings.

The list goes in robots.txt, not htaccess

@nithin270 – any changes you make for search spiders is going to take weeks (at minimum) before they are reflected in the search results.

I suggest doing some robots.txt additions (will list mine below) – however realize that long ago google made a decisions that even if your robots.txt says to disallow crawling something, if another page on the web links to your subpage that is blocked by robots.txt, it will still show the url in the search results – but have a description something like “this sites robots.txt prevents google from displaying description of this result”.

There has been debate about that decision, but it is what it is.

The only way to really prevent a page showing up in results is to hide it behind a password (like htpasswd) – however google does normally remove results if that page (or header info of images) includes “noindex” in the head of the page (there is a tricky way to add this to images – it was pointed out to me in the google webmaster forums)

given that bp pages like members are kind of pseudo pages, using something like yoast (currently as far as I know) – will not give you an option to add noindex, nofollow to your member pages..

you may be able to modify the code I got from wpmudev that checks “if is member page, then add meta description as…” –
( http://premium.wpmudev.org/forums/topic/bp-meta-tite-description-for-groups-and-members-pages#post-806736 )
to… also check “if is members page” – then add “meta name=”robots” content “noindex, nofollow”..
(something like that)

that should remove your members pages next time google crawls your site and the crawlers send the info back to the main algo/index..

I think there is a way to log into google webmaster tools if you have claimed /verified your site and click on urls to ask the big G to remove them as well. (I have not messed with that stuff in a while )

I also suggest adding a robots.txt file similar to this:

Disallow: */activity/p/*
Disallow: /docs/
Disallow: *send-invites*
Disallow: */groups/*members*
Disallow: */groups/*media*
Disallow: *widget-title*
Disallow: *members/*activity*
Disallow: *members/*notifications*
Disallow: *members/*friends*
Disallow: *members/*groups*
Disallow: *members/*docs*
Disallow: *members/*media*
Disallow: *acpage*
Disallow: *messages*
Disallow: *friends*
Disallow: *settings*
Disallow: /*/comment-page*
Disallow: *register*
Disallow: *login*
Disallow: *profile*
Disallow: *admin*
Disallow: *includes*
Disallow: *content*

to prevent some other quirky indexing issues with bp.

If your member profile stuff is sacred, then I would hunt the forums here for what others have been messing with that prevents profile info from being displayed if a user is not logged in… as there are plenty of indexing spiders that will not follow the robots.txt or robots index rules in <head> – in fact some specifically look for these things and purposely crawl and scrape stuff that is blocked –

Disclaimer: I am not an expert, not a real coder. Research these things with other sources, your situation may vary.

You can add this to your robots.txt file:

User-agent: *
Disallow: /members/

Feel free to add more fields for the different areas of BuddyPress.

Note that it doesn’t guarantee Googlebot et al won’t index these pages, it just requests that they don’t.

@atfpodcast – quick chat is working 24/7 on my busiest bp site with 11,000 members.. only thing that screws up that page is when rtmedia has an update the screws with permalinks (messes up all of my static “pages” – have to change and resave.. but that is an issue with rtmedia and permalinks – nothing to do with quickchat.. it works fine and I deploy it on different sites in different ways. I did some write up about it on one of the comment threads I originally linked to in the first response to the original post I think..

anyhow, I think quickchat is the perfect text-chat plugin for a small web site that does not hammer the chat system with hundreds of simultaneous users.

It does chat, avatars, bad words filter, admin can delete stuff – it’s quite the package – I wish the original dev would of answered my emails offering some paid enhancements.. but it’s fine as it is. I did a long write up on one site talking about the drawbacks of ANY chat system that hooks into sql – but I doubt most people will run into those issues until they get more than 20 or so same-time chatters – that’s when you start to look at things to limit sql requests to keep your server from choking – which is usually more of an issue with robots and scrapers and brute force login stuff – comment spam – all that.. guess that’s more of a slightly different discussion – lol.