BuddyPress.org

You could configure your robots.txt to exclude the /messages/ directory:

User-agent: *
Disallow: /messages/

With time, Googlebot should stop hitting that area of your website and the 404 errors will disappear?

@bergblume

If I may suggest that instead you add to your robots.txt file some wildcard rules such as:
Disallow: *members/*activity*
Disallow: */activity/p/*

as to where I think it may be possible to pull some info from the activity posts to create page titles and other meta stuff, most activity posts are going to be short in most cases, and having a bunch of new pages indexed (even properly with title and meta stuff) is likely to add little value to your site overall – and in fact might create a situation where a bulk of your site pages are considered “thin content” – which would hurt your seo more than any proper page titles will on activity pages.

Considering you are also asking about methods for adding meta keywords, I think you are looking for SEO help that is waaaaaaay outdated – unless you are going for different search engines aside from the big G, bing / yahoo for some reason – in which case my knowledge of the other spiders and their indexing methods is tiny.

Now if you are trying to get proper page titles on your “groups” and member profile pages, I think that would be a worthwhile effort.

Just a few random suggestions, I am not a BP dev or prof coder yet – others may have different advice.

I think that getting and using this info will be better / easier perhaps about WP 4.4 or whenever they include the REST API in the core – and then it should be easy peazy for themes to pull and use this info – not that it should be rocket science to do it now… however I have had no luck getting help with similar issues on other BP pages in the past.

It usually boils down to BP peeps saying that page title and meta stuff should be handled by your theme.. your theme devs will say they use the standard wp_title function (or something like that – which has been flipped around recently) – and that kind of stuff if you want it to be beyond what is “standard wordpress” should be handled by an SEO plugin… my experience with several SEO plugins has been that they will not get into all the BP “pseudo pages” for whatever reason, perhaps because a taxonomy is not registered for them, and that makes it more difficult.

Random thoughts for ya

@modx – if you install wP at all thousands of zombie computers that are being used as part of multiple botnets will try to constantly log into your system.

It never hurts to do a clean install though, go for it.

Certinaly there is a chance that they have already gotten your database credentials, so remaking your database usernmae / pass and changing your salt is a good thing to do if you are concerned they may have gotten into your stuff in the past.

I can tell you that in the future if you want a really clean install, first thing you do when you put up a new wp is to add a robots txt that tells all spiders (including google) to not index things like “*p-admin*, *login* *register* – etc – this will help keep it down a bit.. of course ther really evil ones are going to get your login page url.. so starting off a fresh install on a fresh domain with a plugin that moves / renames your login.php would help… but all the evil bots have already indexed your site and put your registration and login urls into their list of things to attack.. so it is what it is.

You can make the lock stronger in several ways, but I don’t think you will stop them from trying to get in,

I’ve blocked countries, large blocks of IP allocations, entire ASNs.. I’ve cut down the amount of attacks – but they still find ways to keep on trying with different methods.

OF course if you can take their password guessing down from 10,000 a day to less than 100 – well the math is in your favor.

@modx – for the moment, I would put your htaccess back to the way it was before – from what I am seeing; you are talking about login brute force password cracking being your big issue.

Although I still suggest blocking those other engine bots – you can probably do that just fine with a robots.txt file at the moment.

IF you are using a login security plugin already – you should be fine… many like to use “limit login attempts” – I use that on some sites – just change the default settings to be more strict than the 4 / attempts.. might also want to add the “whitelist limit login attempts” to keep yourself from getting locked out.

I think succuri is an excellent one too – but there are many others… these login attempt blockers will prevent a bunch of the bot attempts to break it.. recently I have found that adding the plugin “ip geo block” ( https://wordpress.org/plugins/ip-geo-block/ ) is very helpful. Again change the default settings so it also blocks access to your plugins folder and others..

Thing is, every single one of our wordpress (And therefor also buddypress) sites are getting these non-stop password attacks all day, every day. You can try things like add a “captcha” to your login form to make it harder.. but they will keep trying and tieing up your server resources..

Strange to see from your posts that all of those attacks are coming from USA based proxy servers – usually most the attacks come from Ukraine, makes me think they already got into your site once before and are willing to spend a little extra to try to re-attack.

Since those companies are us based you could write them with abuse complaints, but I don’t think you will ever stop the hacking attempts so long as you have a CMS that allows an admin login.

If you are dumping buddypress for the moment and don’t need others to get through a login prompt, I suggest adding this bit of htaccess pwd magic –
http://support.hostgator.com/articles/specialized-help/technical/wordpress/wordpress-login-brute-force-attack

Saves me servers a ton of sql requests 😉

@modx – I have had a similar thing, but not sure exactly what you are describing in your situation, and your screen shot did not show up.

I suggest downloading a copy of your “raw access logs” – first and foremost.

You may have a situation like I ran into where you need to add a robots.txt to block the baiidu spider (all several of them) – and the sogu spider.. also the majestic spider and the hrefs ones.. you may also want to add a few allow/deny directives in your htaccess to block some ip addy blocks..

Again I am not sure if your exact situation, but I ran into something like this recently, and it led me to an unanswered support post at the wp forums – and I found the best way to patch the problem was these things, although I still think more could be done with some advanced htaccess blocking rules using regex – which I am not overly proficient with yet.

@pinkishhue – interesting code – I hope it works!

I’d love to see an exta if then statement for the header where someone in this situation could have a:

meta name ‘robots’ “noindex, nofollow” added – if on a “members” page – this would help fix the OPs problem even more / better.

@dangthrimble – no matter what you do to hide the admin username, the really good hack teams are running scans to get the admin names by running url checks like “yourdotcom /?author=2
?author=3

and scraping the details wp is providing both on page, and in meta fields.
Then adding those names to their pass cracking bot nets.
(look in your raw access logs, you will see it)

I’ve tried changing names on wp sites many times, the rssn hackers get getting the new names, you can tell if you check your fail log with “limit login attempts” plugin.

I’ve played with some code to change in wp themes to hide details there, but my php is slightly below beginner and my understanding of what the theme code is doing is same.

I tried a plugin from the wp-repo that is supposed to hide all that- but it’s not working.

(you could htaccess geo block ukrain and chna from your site completely and probably prevent 90% of these issues anyway from what I have seen by checking logs)

I found some htaccess someone posted that is supposed to suppress all requests for “/?auth [nc] or something like that – but since I do not understand what each part is doing, I have not deployed it.

I think it needs to be htaccess add and include anything with “author” and a number to work well (reg ex for numbers?)

Until I find a htaccess regex method I understand and trust, I have found that the best combo for prevention is:

Geo IP Block
(https://wordpress.org/plugins/ip-geo-block/ )
(default settings are okay, I think it’s best to change the drop downs to block by country the plugins area, theme area, admin ajax, etc as well – options in settings
Also some blogs may want to uncheck the “comment post” block by country
)

If this geoip block plugin author had a donate link I’d already sent him some bucks, it’s the most useful plugin I’ve found since… “good question”

and succuri is an informative add on as well
(shows that some bots have figured out how to bypass the limit login attempts max tries setting)

The way WP is handling question marks in urls (string queries I think it’s called) and giving up 200 status codes and extra info (including author names) to bots is a big issue for me, this kind of relates to the unanswered support question I posted here:
https://wordpress.org/support/topic/question-mark-url-return-200-not-404-string-query-noindex-or?replies=1

@baldarab – if you were to create duplicate content in order for it to get indexed, I would add a robots.txt that tells google and others to not crawl / index your pages. It’s bad to have pages with duplicate titles and no meta description- as is the issue with how groups are currently handled in your case and many others… however, having a second page on the site with the exact same content in order to get it indexed may cause the dreaded duplicate content penalty, and perhaps the “thin content” – lower ranking factoring.

Certainly it would be best to get the title and meta descriptions fixed with bp groups (and member pages) as the best solution, but seeing how you are currently using the groups as basically a page with an activity, votes, comments, and related ones.. with a “sign up to add your own” option, – I wonder if it would be best to just delete those “groups”…

Instead I would consider making them just a new “page” of the site.. or if you really want to get into the whole “different users sharing different projects” – kind of instructables kind of thing – with a twist, it may be better to turn on WP multi-site – and make each activity a new “site (blog)”

– doing so would give you the same features – a video, instructions, related activities (posts and sites) – ability to sign up and create your own, comments, etc…

and that would give you the standard WP search indexing options – good page titles, descriptions, etc (not the errors that bP is currently producing for those pages)

just a thought – not exactly sure how you are using the groups thing and what the goal is with it, just glancing at the setup and thinking what I may do…

@eskymo – if you have the money to burn get a basic vps somewhere and get it started.

my past experiences with “cloud hosting” did not work out – other hosts may differ.

If you have not even started I would not worry about bandwidth or the network growth – sure if you are going to run TV commercials saying join my social network and get free gold bars – then you probably want to worry about that..

I have a solid buddypress site running on a basic shared hosting environment (reseller package they call it I think – gives a little more resources)

I have one running on a VPS, and another on a dedicated server. Even if you get 10,000 members sharing pics and uploading vids I don’t think you will hit bandwidth and growth issues.

You can always copy your database and zip your files to a new host or larger plan at same host if you run into limits.. and of course also look at things to do that would minimize things (block bad robots indexing, cache plugins, etc)

Not sure why you would stick with looking in london hoping to get support on the phone during daytime hours – if I have a host that can’t answer me with support at 2am I move on (and have) to the next one.

With a reasonable host I think you could start at $10 a month.. if can afford get a VPS ~50 a month – dedicated shared resources – better if you are going to setup your BP to also be WP-MS (multi-site / multi-blog (giving users their own “tumblr” type blogs on top of the social networking stuff)

If you grow out of it – great – you can likely afford to go dedicated and look into tuning thing..

/random opinion \ thoughts

Go to your main groups page, and an individual group page.

On both do a “view source”

Is there anything in your source that has anything to do with “robots”
(like robots meta noindex)

That would be an issue – but I bet it’s just the way buddypress handles the
“meta name description”
and html page “title” for groups, and individual member’s profile pages

G’s Wb Tools always says it’s an issue for my bp sites..
(missing completely, or duplicate your other pages, or too short)

This issue was brought up and worked on a bit a few months ago here:
https://buddypress.org/support/topic/bussdypress-title-and-seo-yoast-problem/#post-236324

(since then there was some interesting discussion in a trac where I was impressed some of the devs thought about a lot of issues with this – can’t remember the trac # atm)

I also made a suggestion about this issue here:
https://buddypress.org/support/topic/page-titles-meta-desc-option-like-wp-permalinks/

Although there was some great work on this issue recently, I think it’s still far from what the big G wants to see.

Given the issues with each group’s lack of good (or any) meta name description, and lame page titles, it may actually be more beneficial to block their indexing with a robots.txt file – as having a bunch of issues there affects the ranking for the rest of your site from what I gather.

Hi @baldarab

What’s the content of your robots.txt file?

I found this answer to this issue by: @donalyza. He said:

“if you use phpmyadmin, you can easily export the 3 tables related to groups from site 1 and import them in site 2. phpMyadmin has natively import/export tools and avoids you to use an extra plugin to generate CSV or sql formated files.

xxx_bp_groups
xxx_bp_groups_groupmeta
xxx_bp_groups_members

xxx is the prefix you entered during the wp install. By default it is wp, but it is recommended to use another one. Spambots are too much in love with wp_ prefix. You’re warned ! :d”

I just tried it and it worked, however I had to edit my prefix accordingly to the database prefix I was uploading it to. I used “Coda” to do this, but any similar program works. The site I exporting from had “wp_bp_groups_members” and I changed it to “qou_bp_groups_members” which was the new site prefix.

Wide subject and several answered on this forum.

Askimet covers comment spam and doesn’t avoid clever spam bots to hit directly the DB.

Basic recommandation is to use table prefix different of the classic wp_. This calms down most bots.

A closed door is always a challenge for any spammer. WP is not fort Knox and depending your host, what YOU did and many other security details, this has no end in fact. If your site is Facebook, you’ll probably receive more spam than if it would be mykittycat homepage. Glory has a price ! 😉

Some htaccess rules against reputated spam server and one or to plugins aside what exist natively in WP should be enough to protect you a little from massive spam.

For example: buddypress honeypot + ban hammer for BP

or more simple and rought
http://mattts.net/development-stuff/web-development-stuff/wordpress/buddypress/anti-spam-techniques/registration-honeypot/

… + searching this forum, maybe you can find more tips. 😉

I’ve run into the same problem and asked for ideas around here and seen others that have posted about it in the yoast forums a few times – don’t know why yoast does not get into the bp pages thing. I got some help from the wpmu-dev guys that cobbled together some code that creates page titles and meta descriptions for members and groups pages – which is awesome!

Does not fix the member/username/messages and similar sub-sub pages (but I block those with robots.txt wildcard matching anyhow)

Would like to see buddypress pages added to some kind of taxonmie thing that yaost would auto pickup.. and love to see the options for setting templates within buddypress itself, so as to not rely on yaost for some this, and custom mu=plugins hack to fix other parts of it… google’s webmaster tools alerts missing info on many pages of an active bp site, and has for a long time.

Hi @olaska
you need to add some more validations to your registration page read this post for more info
https://buddypress.org/support/topic/registration-validation/#post-235934
plus install bp-security question plugin which throws a math challenge on registration page this will help you keep out few more bots
Thanks

The robots.txt just says

User-agent: *
Disallow: /wp-admin/

which is pretty standard. The profile pages are under the ‘/members’ directory
(Here’s a link to the site)

Hi @independent

What does your robots.txt file look like?

Spam bots are clever, that’s it.

What does your robots.txt file look like?

@sbraiden – I had similar 24 second page load times, and tried to get advice for enqueing, dergistering, and compacting the multiple and overlapping java and css all these plugins attached to buddypress are mixing in on top of the themes stuff.

Y Slow gives my BP site with basic plugins an “F” – WP professionals shrug it off – meh.

(more one all that here: http://premium.wpmudev.org/forums/topic/deregister-enque-compact-css-and-java-jquery-buddypress-load-time )

I have a sneaky suspicion however that my load times were an issue for me when logged in, and perhaps being logged in as an admin, I THINK that the (stupid) wpmudev dashboard plugin was slowing down my page load speed dramatically more than anything else. It was strange that after I complained of my long page loads, that 2 days later wpmudev had an update for their dashboard thing, and then my page load time went to like 3 seconds. – Coincidence, maybe, nothing definitive. – Everyone else said the pages loaded fine – so maybe it was just an admin thing – or maybe my ghostery blocking gravatar loads or something.

ANYHOW – in regards to hosting, I have a small buddypress site running fine on a shared server with amerinoc. I have one that is fairly busy running fine on a dedicated server at certified hosting.

I personally think that most important thing for a WP based site to perform well is blocking all the bad bots.

I have found that blocking all the naver and badu spiders (And most others) with a robots/txt file has decreased the sql over load (at peak times) on my servers by more than 80%.

I found that hosting a few wp sites on a shared server or dedicated could cause problems not just with spiders crawling pages too much too fast for indexing, but also all the attempting account creations / account brute force logins – even if they are blocked with something like sucuri or limit login attempts – every time they tried to login – they were using up server resources to load the login page, then hit the database to check credentials.

I also suggest using a pwd auth like explained here: http://support.hostgator.com/articles/specialized-help/technical/wordpress/wordpress-login-brute-force-attack

locking down the login with a double thing like that is fine for most WP installs, and a private family / friends BP site should be fine – when the bots can’t login through the first thing there is no need for wordpress to load a bunch of php / css files and pull from SQL a bunch of times just to give a bot a failed login – It becomes a problem for general open to the public buddypress comms I guess.

Now I set all my non-BP sites to use the double auth, I block all search engine bots aside from the top three selectively with robots.txt – and now just about any server can run fine with wp / bp – especially if some attention is paid to plugin overhead, wp-cacheing tools.

I have my fingers crossed the new bp-mediapress (sp? and Beta!) plugin thing will decrease the plugin overhead of rtmedia and offer a better alternative for pics and stuff.

Same random thoughts – I’m not an expert so take my 2 cents with a grain of salt or two..

It’s not buddypress but WP who manage registering and concerning bots, they go directly on your server.

A good starting point is to read the WP Codex:
https://codex.wordpress.org/Hardening_WordPress

And at least,
Never use “admin” as username
Never user wp_ as table prefix

If you search for “spam” on this forum, you will find many topics about this subject.
And the latest one, even if not directly related to your issue, is one of the sticky post on the forum homepage. https://buddypress.org/support/topic/this-is-why-we-cant-have-nice-things/

@@henrywright yes we essentially had to shut off access to the WP backend as the only effective way to cut off the bots as it was such an overwhelming and sustained onslaught, unlike anything I’ve seen before and I’ve experienced a few over the years. When access restored you should be able to post, hopefully whoever has had access previously i.e published pages was/is an author should find they continue to be able to edit/publish as they used to.