BuddyPress.org

@nithin270 – any changes you make for search spiders is going to take weeks (at minimum) before they are reflected in the search results.

I suggest doing some robots.txt additions (will list mine below) – however realize that long ago google made a decisions that even if your robots.txt says to disallow crawling something, if another page on the web links to your subpage that is blocked by robots.txt, it will still show the url in the search results – but have a description something like “this sites robots.txt prevents google from displaying description of this result”.

There has been debate about that decision, but it is what it is.

The only way to really prevent a page showing up in results is to hide it behind a password (like htpasswd) – however google does normally remove results if that page (or header info of images) includes “noindex” in the head of the page (there is a tricky way to add this to images – it was pointed out to me in the google webmaster forums)

given that bp pages like members are kind of pseudo pages, using something like yoast (currently as far as I know) – will not give you an option to add noindex, nofollow to your member pages..

you may be able to modify the code I got from wpmudev that checks “if is member page, then add meta description as…” –
( http://premium.wpmudev.org/forums/topic/bp-meta-tite-description-for-groups-and-members-pages#post-806736 )
to… also check “if is members page” – then add “meta name=”robots” content “noindex, nofollow”..
(something like that)

that should remove your members pages next time google crawls your site and the crawlers send the info back to the main algo/index..

I think there is a way to log into google webmaster tools if you have claimed /verified your site and click on urls to ask the big G to remove them as well. (I have not messed with that stuff in a while )

I also suggest adding a robots.txt file similar to this:

Disallow: */activity/p/*
Disallow: /docs/
Disallow: *send-invites*
Disallow: */groups/*members*
Disallow: */groups/*media*
Disallow: *widget-title*
Disallow: *members/*activity*
Disallow: *members/*notifications*
Disallow: *members/*friends*
Disallow: *members/*groups*
Disallow: *members/*docs*
Disallow: *members/*media*
Disallow: *acpage*
Disallow: *messages*
Disallow: *friends*
Disallow: *settings*
Disallow: /*/comment-page*
Disallow: *register*
Disallow: *login*
Disallow: *profile*
Disallow: *admin*
Disallow: *includes*
Disallow: *content*

to prevent some other quirky indexing issues with bp.

If your member profile stuff is sacred, then I would hunt the forums here for what others have been messing with that prevents profile info from being displayed if a user is not logged in… as there are plenty of indexing spiders that will not follow the robots.txt or robots index rules in <head> – in fact some specifically look for these things and purposely crawl and scrape stuff that is blocked –

Disclaimer: I am not an expert, not a real coder. Research these things with other sources, your situation may vary.

You can add this to your robots.txt file:

User-agent: *
Disallow: /members/

Feel free to add more fields for the different areas of BuddyPress.

Note that it doesn’t guarantee Googlebot et al won’t index these pages, it just requests that they don’t.

@atfpodcast – quick chat is working 24/7 on my busiest bp site with 11,000 members.. only thing that screws up that page is when rtmedia has an update the screws with permalinks (messes up all of my static “pages” – have to change and resave.. but that is an issue with rtmedia and permalinks – nothing to do with quickchat.. it works fine and I deploy it on different sites in different ways. I did some write up about it on one of the comment threads I originally linked to in the first response to the original post I think..

anyhow, I think quickchat is the perfect text-chat plugin for a small web site that does not hammer the chat system with hundreds of simultaneous users.

It does chat, avatars, bad words filter, admin can delete stuff – it’s quite the package – I wish the original dev would of answered my emails offering some paid enhancements.. but it’s fine as it is. I did a long write up on one site talking about the drawbacks of ANY chat system that hooks into sql – but I doubt most people will run into those issues until they get more than 20 or so same-time chatters – that’s when you start to look at things to limit sql requests to keep your server from choking – which is usually more of an issue with robots and scrapers and brute force login stuff – comment spam – all that.. guess that’s more of a slightly different discussion – lol.

You could configure your robots.txt to exclude the /messages/ directory:

User-agent: *
Disallow: /messages/

With time, Googlebot should stop hitting that area of your website and the 404 errors will disappear?

@bergblume

If I may suggest that instead you add to your robots.txt file some wildcard rules such as:
Disallow: *members/*activity*
Disallow: */activity/p/*

as to where I think it may be possible to pull some info from the activity posts to create page titles and other meta stuff, most activity posts are going to be short in most cases, and having a bunch of new pages indexed (even properly with title and meta stuff) is likely to add little value to your site overall – and in fact might create a situation where a bulk of your site pages are considered “thin content” – which would hurt your seo more than any proper page titles will on activity pages.

Considering you are also asking about methods for adding meta keywords, I think you are looking for SEO help that is waaaaaaay outdated – unless you are going for different search engines aside from the big G, bing / yahoo for some reason – in which case my knowledge of the other spiders and their indexing methods is tiny.

Now if you are trying to get proper page titles on your “groups” and member profile pages, I think that would be a worthwhile effort.

Just a few random suggestions, I am not a BP dev or prof coder yet – others may have different advice.

I think that getting and using this info will be better / easier perhaps about WP 4.4 or whenever they include the REST API in the core – and then it should be easy peazy for themes to pull and use this info – not that it should be rocket science to do it now… however I have had no luck getting help with similar issues on other BP pages in the past.

It usually boils down to BP peeps saying that page title and meta stuff should be handled by your theme.. your theme devs will say they use the standard wp_title function (or something like that – which has been flipped around recently) – and that kind of stuff if you want it to be beyond what is “standard wordpress” should be handled by an SEO plugin… my experience with several SEO plugins has been that they will not get into all the BP “pseudo pages” for whatever reason, perhaps because a taxonomy is not registered for them, and that makes it more difficult.

Random thoughts for ya

@modx – if you install wP at all thousands of zombie computers that are being used as part of multiple botnets will try to constantly log into your system.

It never hurts to do a clean install though, go for it.

Certinaly there is a chance that they have already gotten your database credentials, so remaking your database usernmae / pass and changing your salt is a good thing to do if you are concerned they may have gotten into your stuff in the past.

I can tell you that in the future if you want a really clean install, first thing you do when you put up a new wp is to add a robots txt that tells all spiders (including google) to not index things like “*p-admin*, *login* *register* – etc – this will help keep it down a bit.. of course ther really evil ones are going to get your login page url.. so starting off a fresh install on a fresh domain with a plugin that moves / renames your login.php would help… but all the evil bots have already indexed your site and put your registration and login urls into their list of things to attack.. so it is what it is.

You can make the lock stronger in several ways, but I don’t think you will stop them from trying to get in,

I’ve blocked countries, large blocks of IP allocations, entire ASNs.. I’ve cut down the amount of attacks – but they still find ways to keep on trying with different methods.

OF course if you can take their password guessing down from 10,000 a day to less than 100 – well the math is in your favor.

@modx – for the moment, I would put your htaccess back to the way it was before – from what I am seeing; you are talking about login brute force password cracking being your big issue.

Although I still suggest blocking those other engine bots – you can probably do that just fine with a robots.txt file at the moment.

IF you are using a login security plugin already – you should be fine… many like to use “limit login attempts” – I use that on some sites – just change the default settings to be more strict than the 4 / attempts.. might also want to add the “whitelist limit login attempts” to keep yourself from getting locked out.

I think succuri is an excellent one too – but there are many others… these login attempt blockers will prevent a bunch of the bot attempts to break it.. recently I have found that adding the plugin “ip geo block” ( https://wordpress.org/plugins/ip-geo-block/ ) is very helpful. Again change the default settings so it also blocks access to your plugins folder and others..

Thing is, every single one of our wordpress (And therefor also buddypress) sites are getting these non-stop password attacks all day, every day. You can try things like add a “captcha” to your login form to make it harder.. but they will keep trying and tieing up your server resources..

Strange to see from your posts that all of those attacks are coming from USA based proxy servers – usually most the attacks come from Ukraine, makes me think they already got into your site once before and are willing to spend a little extra to try to re-attack.

Since those companies are us based you could write them with abuse complaints, but I don’t think you will ever stop the hacking attempts so long as you have a CMS that allows an admin login.

If you are dumping buddypress for the moment and don’t need others to get through a login prompt, I suggest adding this bit of htaccess pwd magic –
http://support.hostgator.com/articles/specialized-help/technical/wordpress/wordpress-login-brute-force-attack

Saves me servers a ton of sql requests 😉

@modx – I have had a similar thing, but not sure exactly what you are describing in your situation, and your screen shot did not show up.

I suggest downloading a copy of your “raw access logs” – first and foremost.

You may have a situation like I ran into where you need to add a robots.txt to block the baiidu spider (all several of them) – and the sogu spider.. also the majestic spider and the hrefs ones.. you may also want to add a few allow/deny directives in your htaccess to block some ip addy blocks..

Again I am not sure if your exact situation, but I ran into something like this recently, and it led me to an unanswered support post at the wp forums – and I found the best way to patch the problem was these things, although I still think more could be done with some advanced htaccess blocking rules using regex – which I am not overly proficient with yet.

@pinkishhue – interesting code – I hope it works!

I’d love to see an exta if then statement for the header where someone in this situation could have a:

meta name ‘robots’ “noindex, nofollow” added – if on a “members” page – this would help fix the OPs problem even more / better.

@dangthrimble – no matter what you do to hide the admin username, the really good hack teams are running scans to get the admin names by running url checks like “yourdotcom /?author=2
?author=3

and scraping the details wp is providing both on page, and in meta fields.
Then adding those names to their pass cracking bot nets.
(look in your raw access logs, you will see it)

I’ve tried changing names on wp sites many times, the rssn hackers get getting the new names, you can tell if you check your fail log with “limit login attempts” plugin.

I’ve played with some code to change in wp themes to hide details there, but my php is slightly below beginner and my understanding of what the theme code is doing is same.

I tried a plugin from the wp-repo that is supposed to hide all that- but it’s not working.

(you could htaccess geo block ukrain and chna from your site completely and probably prevent 90% of these issues anyway from what I have seen by checking logs)

I found some htaccess someone posted that is supposed to suppress all requests for “/?auth [nc] or something like that – but since I do not understand what each part is doing, I have not deployed it.

I think it needs to be htaccess add and include anything with “author” and a number to work well (reg ex for numbers?)

Until I find a htaccess regex method I understand and trust, I have found that the best combo for prevention is:

Geo IP Block
(https://wordpress.org/plugins/ip-geo-block/ )
(default settings are okay, I think it’s best to change the drop downs to block by country the plugins area, theme area, admin ajax, etc as well – options in settings
Also some blogs may want to uncheck the “comment post” block by country
)

If this geoip block plugin author had a donate link I’d already sent him some bucks, it’s the most useful plugin I’ve found since… “good question”

and succuri is an informative add on as well
(shows that some bots have figured out how to bypass the limit login attempts max tries setting)

The way WP is handling question marks in urls (string queries I think it’s called) and giving up 200 status codes and extra info (including author names) to bots is a big issue for me, this kind of relates to the unanswered support question I posted here:
https://wordpress.org/support/topic/question-mark-url-return-200-not-404-string-query-noindex-or?replies=1

@baldarab – if you were to create duplicate content in order for it to get indexed, I would add a robots.txt that tells google and others to not crawl / index your pages. It’s bad to have pages with duplicate titles and no meta description- as is the issue with how groups are currently handled in your case and many others… however, having a second page on the site with the exact same content in order to get it indexed may cause the dreaded duplicate content penalty, and perhaps the “thin content” – lower ranking factoring.

Certainly it would be best to get the title and meta descriptions fixed with bp groups (and member pages) as the best solution, but seeing how you are currently using the groups as basically a page with an activity, votes, comments, and related ones.. with a “sign up to add your own” option, – I wonder if it would be best to just delete those “groups”…

Instead I would consider making them just a new “page” of the site.. or if you really want to get into the whole “different users sharing different projects” – kind of instructables kind of thing – with a twist, it may be better to turn on WP multi-site – and make each activity a new “site (blog)”

– doing so would give you the same features – a video, instructions, related activities (posts and sites) – ability to sign up and create your own, comments, etc…

and that would give you the standard WP search indexing options – good page titles, descriptions, etc (not the errors that bP is currently producing for those pages)

just a thought – not exactly sure how you are using the groups thing and what the goal is with it, just glancing at the setup and thinking what I may do…

@eskymo – if you have the money to burn get a basic vps somewhere and get it started.

my past experiences with “cloud hosting” did not work out – other hosts may differ.

If you have not even started I would not worry about bandwidth or the network growth – sure if you are going to run TV commercials saying join my social network and get free gold bars – then you probably want to worry about that..

I have a solid buddypress site running on a basic shared hosting environment (reseller package they call it I think – gives a little more resources)

I have one running on a VPS, and another on a dedicated server. Even if you get 10,000 members sharing pics and uploading vids I don’t think you will hit bandwidth and growth issues.

You can always copy your database and zip your files to a new host or larger plan at same host if you run into limits.. and of course also look at things to do that would minimize things (block bad robots indexing, cache plugins, etc)

Not sure why you would stick with looking in london hoping to get support on the phone during daytime hours – if I have a host that can’t answer me with support at 2am I move on (and have) to the next one.

With a reasonable host I think you could start at $10 a month.. if can afford get a VPS ~50 a month – dedicated shared resources – better if you are going to setup your BP to also be WP-MS (multi-site / multi-blog (giving users their own “tumblr” type blogs on top of the social networking stuff)

If you grow out of it – great – you can likely afford to go dedicated and look into tuning thing..

/random opinion \ thoughts

Go to your main groups page, and an individual group page.

On both do a “view source”

Is there anything in your source that has anything to do with “robots”
(like robots meta noindex)

That would be an issue – but I bet it’s just the way buddypress handles the
“meta name description”
and html page “title” for groups, and individual member’s profile pages

G’s Wb Tools always says it’s an issue for my bp sites..
(missing completely, or duplicate your other pages, or too short)

This issue was brought up and worked on a bit a few months ago here:
https://buddypress.org/support/topic/bussdypress-title-and-seo-yoast-problem/#post-236324

(since then there was some interesting discussion in a trac where I was impressed some of the devs thought about a lot of issues with this – can’t remember the trac # atm)

I also made a suggestion about this issue here:
https://buddypress.org/support/topic/page-titles-meta-desc-option-like-wp-permalinks/

Although there was some great work on this issue recently, I think it’s still far from what the big G wants to see.

Given the issues with each group’s lack of good (or any) meta name description, and lame page titles, it may actually be more beneficial to block their indexing with a robots.txt file – as having a bunch of issues there affects the ranking for the rest of your site from what I gather.

Hi @baldarab

What’s the content of your robots.txt file?

I found this answer to this issue by: @donalyza. He said:

“if you use phpmyadmin, you can easily export the 3 tables related to groups from site 1 and import them in site 2. phpMyadmin has natively import/export tools and avoids you to use an extra plugin to generate CSV or sql formated files.

xxx_bp_groups
xxx_bp_groups_groupmeta
xxx_bp_groups_members

xxx is the prefix you entered during the wp install. By default it is wp, but it is recommended to use another one. Spambots are too much in love with wp_ prefix. You’re warned ! :d”

I just tried it and it worked, however I had to edit my prefix accordingly to the database prefix I was uploading it to. I used “Coda” to do this, but any similar program works. The site I exporting from had “wp_bp_groups_members” and I changed it to “qou_bp_groups_members” which was the new site prefix.

Wide subject and several answered on this forum.

Askimet covers comment spam and doesn’t avoid clever spam bots to hit directly the DB.

Basic recommandation is to use table prefix different of the classic wp_. This calms down most bots.

A closed door is always a challenge for any spammer. WP is not fort Knox and depending your host, what YOU did and many other security details, this has no end in fact. If your site is Facebook, you’ll probably receive more spam than if it would be mykittycat homepage. Glory has a price ! 😉

Some htaccess rules against reputated spam server and one or to plugins aside what exist natively in WP should be enough to protect you a little from massive spam.

For example: buddypress honeypot + ban hammer for BP

or more simple and rought
http://mattts.net/development-stuff/web-development-stuff/wordpress/buddypress/anti-spam-techniques/registration-honeypot/

… + searching this forum, maybe you can find more tips. 😉

I’ve run into the same problem and asked for ideas around here and seen others that have posted about it in the yoast forums a few times – don’t know why yoast does not get into the bp pages thing. I got some help from the wpmu-dev guys that cobbled together some code that creates page titles and meta descriptions for members and groups pages – which is awesome!

Does not fix the member/username/messages and similar sub-sub pages (but I block those with robots.txt wildcard matching anyhow)

Would like to see buddypress pages added to some kind of taxonmie thing that yaost would auto pickup.. and love to see the options for setting templates within buddypress itself, so as to not rely on yaost for some this, and custom mu=plugins hack to fix other parts of it… google’s webmaster tools alerts missing info on many pages of an active bp site, and has for a long time.

Hi @olaska
you need to add some more validations to your registration page read this post for more info
https://buddypress.org/support/topic/registration-validation/#post-235934
plus install bp-security question plugin which throws a math challenge on registration page this will help you keep out few more bots
Thanks

The robots.txt just says

User-agent: *
Disallow: /wp-admin/

which is pretty standard. The profile pages are under the ‘/members’ directory
(Here’s a link to the site)

Hi @independent

What does your robots.txt file look like?