Search Results for 'bots'
-
AuthorSearch Results
-
February 11, 2016 at 5:35 pm #249773
In reply to: private event for exclusive company site
calvin
Participantyou can block google using robots.txt, you can read about that here https://www.elegantthemes.com/blog/tips-tricks/how-to-create-and-configure-your-robots-txt-file
February 8, 2016 at 10:29 pm #249628In reply to: making users profiles, forums and activity private
Sofia Woods
Participanthi @shanebp – thank you I will tend to the robots.txt. I had not heard of this plugin – I’ve just installed it and it does exactly what I need.
ThanksFebruary 5, 2016 at 7:21 pm #249515In reply to: making users profiles, forums and activity private
shanebp
ModeratorTo prevent indexing by google, you need one or more robots.txt files.
Re private – Have you tried this plugin ?
https://wordpress.org/plugins/bp-simple-private/December 21, 2015 at 8:10 pm #247951In reply to: Spammers let in/Real people kept out.
Henry Wright
ModeratorYou could activate some kind of registration honeypot to keep the spam bots out. This technique has worked well for me in the past.
Example: https://en-gb.wordpress.org/plugins/registration-honeypot/
Regarding the confirmation email problem, this seems to be a common problem and one that isn’t easily solved. Many things can result in email going into spam, most are usually related to your mail server setup or the user’s own spam filter settings.
November 20, 2015 at 3:01 pm #247008In reply to: No index buddypress pages
gurusurfer
ParticipantOk for the
is_buddypress()the functionsfunction for_buddypress() { if ( is_buddyPress() ) { echo "<meta name=”robots” content=”noindex, nofollow>"; } else { } } add_action( 'wp_head', 'for_buddypress' );Working perfectly, but when creating another function for the
bp_is_groups()I am getting error “call to undefined functionsbp_is_groups()Any help is much appreciated!
Thanks
November 19, 2015 at 8:45 pm #246988In reply to: No index buddypress pages
ParticipantI have also tried in a loop like in this codex, but breaking the page, what am i missing?
<?php if ((is_buddypress() ) { echo '<meta name=”robots” content=”noindex, nofollow>'; } ?>You help is much appreciated!
Thanks
November 19, 2015 at 4:34 pm #246961In reply to: No index buddypress pages
ParticipantThank you very much for the quick reply!
Sorry I am no php pro here, tried like this but breaks page!
add_action( 'wp_head','noindex_groups' ); function noindex_groups() { if ( bp_is_groups() ) { echo '<meta name=”robots” content=”noindex, nofollow>' } }add_action( 'wp_head','noindex_buddypress' ); function noindex_buddypress() { if ( bp_is_groups() ) { echo '<meta name=”robots” content=”noindex, nofollow>' } }Thanks
November 6, 2015 at 6:37 pm #246447In reply to: Hide Buddypress pages from google serch.
ModeratorThe list goes in robots.txt, not htaccess
September 11, 2015 at 5:57 am #244260In reply to: Hide Buddypress pages from google serch.
Nithin K Varrier
ParticipantThanks a lot djsteveb, danbp and Henry. I updated my robots.txt. Let’s wait for some days.
September 10, 2015 at 8:28 pm #244254In reply to: Hide Buddypress pages from google serch.
djsteveb
Participant@nithin270 – any changes you make for search spiders is going to take weeks (at minimum) before they are reflected in the search results.
I suggest doing some robots.txt additions (will list mine below) – however realize that long ago google made a decisions that even if your robots.txt says to disallow crawling something, if another page on the web links to your subpage that is blocked by robots.txt, it will still show the url in the search results – but have a description something like “this sites robots.txt prevents google from displaying description of this result”.
There has been debate about that decision, but it is what it is.
The only way to really prevent a page showing up in results is to hide it behind a password (like htpasswd) – however google does normally remove results if that page (or header info of images) includes “noindex” in the head of the page (there is a tricky way to add this to images – it was pointed out to me in the google webmaster forums)
given that bp pages like members are kind of pseudo pages, using something like yoast (currently as far as I know) – will not give you an option to add noindex, nofollow to your member pages..
you may be able to modify the code I got from wpmudev that checks “if is member page, then add meta description as…” –
( http://premium.wpmudev.org/forums/topic/bp-meta-tite-description-for-groups-and-members-pages#post-806736 )
to… also check “if is members page” – then add “meta name=”robots” content “noindex, nofollow”..
(something like that)that should remove your members pages next time google crawls your site and the crawlers send the info back to the main algo/index..
I think there is a way to log into google webmaster tools if you have claimed /verified your site and click on urls to ask the big G to remove them as well. (I have not messed with that stuff in a while )
I also suggest adding a robots.txt file similar to this:
Disallow: */activity/p/*
Disallow: /docs/
Disallow: *send-invites*
Disallow: */groups/*members*
Disallow: */groups/*media*
Disallow: *widget-title*
Disallow: *members/*activity*
Disallow: *members/*notifications*
Disallow: *members/*friends*
Disallow: *members/*groups*
Disallow: *members/*docs*
Disallow: *members/*media*
Disallow: *acpage*
Disallow: *messages*
Disallow: *friends*
Disallow: *settings*
Disallow: /*/comment-page*
Disallow: *register*
Disallow: *login*
Disallow: *profile*
Disallow: *admin*
Disallow: *includes*
Disallow: *content*to prevent some other quirky indexing issues with bp.
If your member profile stuff is sacred, then I would hunt the forums here for what others have been messing with that prevents profile info from being displayed if a user is not logged in… as there are plenty of indexing spiders that will not follow the robots.txt or robots index rules in <head> – in fact some specifically look for these things and purposely crawl and scrape stuff that is blocked –
Disclaimer: I am not an expert, not a real coder. Research these things with other sources, your situation may vary.
September 10, 2015 at 9:36 am #244221In reply to: Hide Buddypress pages from google serch.
ModeratorYou can add this to your robots.txt file:
User-agent: * Disallow: /members/Feel free to add more fields for the different areas of BuddyPress.
Note that it doesn’t guarantee Googlebot et al won’t index these pages, it just requests that they don’t.
August 22, 2015 at 1:12 am #243477In reply to: Good group chat plugin for bp?
Participant@atfpodcast – quick chat is working 24/7 on my busiest bp site with 11,000 members.. only thing that screws up that page is when rtmedia has an update the screws with permalinks (messes up all of my static “pages” – have to change and resave.. but that is an issue with rtmedia and permalinks – nothing to do with quickchat.. it works fine and I deploy it on different sites in different ways. I did some write up about it on one of the comment threads I originally linked to in the first response to the original post I think..
anyhow, I think quickchat is the perfect text-chat plugin for a small web site that does not hammer the chat system with hundreds of simultaneous users.
It does chat, avatars, bad words filter, admin can delete stuff – it’s quite the package – I wish the original dev would of answered my emails offering some paid enhancements.. but it’s fine as it is. I did a long write up on one site talking about the drawbacks of ANY chat system that hooks into sql – but I doubt most people will run into those issues until they get more than 20 or so same-time chatters – that’s when you start to look at things to limit sql requests to keep your server from choking – which is usually more of an issue with robots and scrapers and brute force login stuff – comment spam – all that.. guess that’s more of a slightly different discussion – lol.
August 18, 2015 at 12:47 pm #243342giorgosnl
ParticipantThanks Henry, I have added it to the robots.txt, though this is not a directory. I dont have any /sitename/messages directory on the site. Something (a custom function maybe) is generating this url. Any idea what it takes to have such url generated and picked by google?
August 18, 2015 at 8:02 am #243315ModeratorYou could configure your robots.txt to exclude the /messages/ directory:
User-agent: * Disallow: /messages/With time, Googlebot should stop hitting that area of your website and the 404 errors will disappear?
July 30, 2015 at 9:13 am #242565In reply to: SEO title for activity posts
ParticipantIf I may suggest that instead you add to your robots.txt file some wildcard rules such as:
Disallow: *members/*activity*
Disallow: */activity/p/*as to where I think it may be possible to pull some info from the activity posts to create page titles and other meta stuff, most activity posts are going to be short in most cases, and having a bunch of new pages indexed (even properly with title and meta stuff) is likely to add little value to your site overall – and in fact might create a situation where a bulk of your site pages are considered “thin content” – which would hurt your seo more than any proper page titles will on activity pages.
Considering you are also asking about methods for adding meta keywords, I think you are looking for SEO help that is waaaaaaay outdated – unless you are going for different search engines aside from the big G, bing / yahoo for some reason – in which case my knowledge of the other spiders and their indexing methods is tiny.
Now if you are trying to get proper page titles on your “groups” and member profile pages, I think that would be a worthwhile effort.
Just a few random suggestions, I am not a BP dev or prof coder yet – others may have different advice.
I think that getting and using this info will be better / easier perhaps about WP 4.4 or whenever they include the REST API in the core – and then it should be easy peazy for themes to pull and use this info – not that it should be rocket science to do it now… however I have had no luck getting help with similar issues on other BP pages in the past.
It usually boils down to BP peeps saying that page title and meta stuff should be handled by your theme.. your theme devs will say they use the standard wp_title function (or something like that – which has been flipped around recently) – and that kind of stuff if you want it to be beyond what is “standard wordpress” should be handled by an SEO plugin… my experience with several SEO plugins has been that they will not get into all the BP “pseudo pages” for whatever reason, perhaps because a taxonomy is not registered for them, and that makes it more difficult.
Random thoughts for ya
July 17, 2015 at 7:31 pm #241949In reply to: Brute force and Four-oh-four Attacks
Participant@modx – if you install wP at all thousands of zombie computers that are being used as part of multiple botnets will try to constantly log into your system.
It never hurts to do a clean install though, go for it.
Certinaly there is a chance that they have already gotten your database credentials, so remaking your database usernmae / pass and changing your salt is a good thing to do if you are concerned they may have gotten into your stuff in the past.
I can tell you that in the future if you want a really clean install, first thing you do when you put up a new wp is to add a robots txt that tells all spiders (including google) to not index things like “*p-admin*, *login* *register* – etc – this will help keep it down a bit.. of course ther really evil ones are going to get your login page url.. so starting off a fresh install on a fresh domain with a plugin that moves / renames your login.php would help… but all the evil bots have already indexed your site and put your registration and login urls into their list of things to attack.. so it is what it is.
You can make the lock stronger in several ways, but I don’t think you will stop them from trying to get in,
I’ve blocked countries, large blocks of IP allocations, entire ASNs.. I’ve cut down the amount of attacks – but they still find ways to keep on trying with different methods.
OF course if you can take their password guessing down from 10,000 a day to less than 100 – well the math is in your favor.
July 17, 2015 at 2:34 pm #241946In reply to: Content Invisible to Google
baldarab
ParticipantThanks for the tip r.e. robots.txt.
As to whether we change them to pages, actually once you’re a member we do use them A LOT as groups, so making them pages would be challenging. But good thought, thank you.
July 17, 2015 at 8:46 am #241939In reply to: Brute force and Four-oh-four Attacks
Participant@modx – for the moment, I would put your htaccess back to the way it was before – from what I am seeing; you are talking about login brute force password cracking being your big issue.
Although I still suggest blocking those other engine bots – you can probably do that just fine with a robots.txt file at the moment.
IF you are using a login security plugin already – you should be fine… many like to use “limit login attempts” – I use that on some sites – just change the default settings to be more strict than the 4 / attempts.. might also want to add the “whitelist limit login attempts” to keep yourself from getting locked out.
I think succuri is an excellent one too – but there are many others… these login attempt blockers will prevent a bunch of the bot attempts to break it.. recently I have found that adding the plugin “ip geo block” ( https://wordpress.org/plugins/ip-geo-block/ ) is very helpful. Again change the default settings so it also blocks access to your plugins folder and others..
Thing is, every single one of our wordpress (And therefor also buddypress) sites are getting these non-stop password attacks all day, every day. You can try things like add a “captcha” to your login form to make it harder.. but they will keep trying and tieing up your server resources..
Strange to see from your posts that all of those attacks are coming from USA based proxy servers – usually most the attacks come from Ukraine, makes me think they already got into your site once before and are willing to spend a little extra to try to re-attack.
Since those companies are us based you could write them with abuse complaints, but I don’t think you will ever stop the hacking attempts so long as you have a CMS that allows an admin login.
If you are dumping buddypress for the moment and don’t need others to get through a login prompt, I suggest adding this bit of htaccess pwd magic –
http://support.hostgator.com/articles/specialized-help/technical/wordpress/wordpress-login-brute-force-attackSaves me servers a ton of sql requests 😉
July 17, 2015 at 5:40 am #241926In reply to: Brute force and Four-oh-four Attacks
Participant@modx – I have had a similar thing, but not sure exactly what you are describing in your situation, and your screen shot did not show up.
I suggest downloading a copy of your “raw access logs” – first and foremost.
You may have a situation like I ran into where you need to add a robots.txt to block the baiidu spider (all several of them) – and the sogu spider.. also the majestic spider and the hrefs ones.. you may also want to add a few allow/deny directives in your htaccess to block some ip addy blocks..
Again I am not sure if your exact situation, but I ran into something like this recently, and it led me to an unanswered support post at the wp forums – and I found the best way to patch the problem was these things, although I still think more could be done with some advanced htaccess blocking rules using regex – which I am not overly proficient with yet.
July 15, 2015 at 9:29 am #241863In reply to: BP member pages visibility restriction?
Participant@pinkishhue – interesting code – I hope it works!
I’d love to see an exta if then statement for the header where someone in this situation could have a:
meta name ‘robots’ “noindex, nofollow” added – if on a “members” page – this would help fix the OPs problem even more / better.
July 12, 2015 at 1:46 am #241758Participant@dangthrimble – no matter what you do to hide the admin username, the really good hack teams are running scans to get the admin names by running url checks like “yourdotcom /?author=2
?author=3and scraping the details wp is providing both on page, and in meta fields.
Then adding those names to their pass cracking bot nets.
(look in your raw access logs, you will see it)I’ve tried changing names on wp sites many times, the rssn hackers get getting the new names, you can tell if you check your fail log with “limit login attempts” plugin.
I’ve played with some code to change in wp themes to hide details there, but my php is slightly below beginner and my understanding of what the theme code is doing is same.
I tried a plugin from the wp-repo that is supposed to hide all that- but it’s not working.
(you could htaccess geo block ukrain and chna from your site completely and probably prevent 90% of these issues anyway from what I have seen by checking logs)
I found some htaccess someone posted that is supposed to suppress all requests for “/?auth [nc] or something like that – but since I do not understand what each part is doing, I have not deployed it.
I think it needs to be htaccess add and include anything with “author” and a number to work well (reg ex for numbers?)
Until I find a htaccess regex method I understand and trust, I have found that the best combo for prevention is:
Geo IP Block
(https://wordpress.org/plugins/ip-geo-block/ )
(default settings are okay, I think it’s best to change the drop downs to block by country the plugins area, theme area, admin ajax, etc as well – options in settings
Also some blogs may want to uncheck the “comment post” block by country
)If this geoip block plugin author had a donate link I’d already sent him some bucks, it’s the most useful plugin I’ve found since… “good question”
and succuri is an informative add on as well
(shows that some bots have figured out how to bypass the limit login attempts max tries setting)The way WP is handling question marks in urls (string queries I think it’s called) and giving up 200 status codes and extra info (including author names) to bots is a big issue for me, this kind of relates to the unanswered support question I posted here:
https://wordpress.org/support/topic/question-mark-url-return-200-not-404-string-query-noindex-or?replies=1July 11, 2015 at 2:16 am #241701In reply to: Content Invisible to Google
Participant@baldarab – if you were to create duplicate content in order for it to get indexed, I would add a robots.txt that tells google and others to not crawl / index your pages. It’s bad to have pages with duplicate titles and no meta description- as is the issue with how groups are currently handled in your case and many others… however, having a second page on the site with the exact same content in order to get it indexed may cause the dreaded duplicate content penalty, and perhaps the “thin content” – lower ranking factoring.
Certainly it would be best to get the title and meta descriptions fixed with bp groups (and member pages) as the best solution, but seeing how you are currently using the groups as basically a page with an activity, votes, comments, and related ones.. with a “sign up to add your own” option, – I wonder if it would be best to just delete those “groups”…
Instead I would consider making them just a new “page” of the site.. or if you really want to get into the whole “different users sharing different projects” – kind of instructables kind of thing – with a twist, it may be better to turn on WP multi-site – and make each activity a new “site (blog)”
– doing so would give you the same features – a video, instructions, related activities (posts and sites) – ability to sign up and create your own, comments, etc…
and that would give you the standard WP search indexing options – good page titles, descriptions, etc (not the errors that bP is currently producing for those pages)
just a thought – not exactly sure how you are using the groups thing and what the goal is with it, just glancing at the setup and thinking what I may do…
July 9, 2015 at 9:42 am #241625In reply to: Advice on Hosting
Participant@eskymo – if you have the money to burn get a basic vps somewhere and get it started.
my past experiences with “cloud hosting” did not work out – other hosts may differ.
If you have not even started I would not worry about bandwidth or the network growth – sure if you are going to run TV commercials saying join my social network and get free gold bars – then you probably want to worry about that..
I have a solid buddypress site running on a basic shared hosting environment (reseller package they call it I think – gives a little more resources)
I have one running on a VPS, and another on a dedicated server. Even if you get 10,000 members sharing pics and uploading vids I don’t think you will hit bandwidth and growth issues.
You can always copy your database and zip your files to a new host or larger plan at same host if you run into limits.. and of course also look at things to do that would minimize things (block bad robots indexing, cache plugins, etc)
Not sure why you would stick with looking in london hoping to get support on the phone during daytime hours – if I have a host that can’t answer me with support at 2am I move on (and have) to the next one.
With a reasonable host I think you could start at $10 a month.. if can afford get a VPS ~50 a month – dedicated shared resources – better if you are going to setup your BP to also be WP-MS (multi-site / multi-blog (giving users their own “tumblr” type blogs on top of the social networking stuff)
If you grow out of it – great – you can likely afford to go dedicated and look into tuning thing..
/random opinion \ thoughts
July 7, 2015 at 5:34 pm #241548In reply to: Content Invisible to Google
ParticipantHi and thank you, @djsteveb. In answer to your questions:
We don’t have a main groups page, only individual ones. Looking at the source of an individual one (e.g. http://www.freakyrivet.com/activities/defying-gravity-making-people-think-you-can/), I’ve searched and there’s no ‘robots’ text anywhere in the file.
Looking at the links you’ve sent, I note that Yoast’s plugin comes up a lot – I have it on my site as well.
Thank you.
July 6, 2015 at 11:40 am #241489In reply to: Content Invisible to Google
ParticipantGo to your main groups page, and an individual group page.
On both do a “view source”
Is there anything in your source that has anything to do with “robots”
(like robots meta noindex)That would be an issue – but I bet it’s just the way buddypress handles the
“meta name description”
and html page “title” for groups, and individual member’s profile pagesG’s Wb Tools always says it’s an issue for my bp sites..
(missing completely, or duplicate your other pages, or too short)This issue was brought up and worked on a bit a few months ago here:
https://buddypress.org/support/topic/bussdypress-title-and-seo-yoast-problem/#post-236324(since then there was some interesting discussion in a trac where I was impressed some of the devs thought about a lot of issues with this – can’t remember the trac # atm)
I also made a suggestion about this issue here:
https://buddypress.org/support/topic/page-titles-meta-desc-option-like-wp-permalinks/Although there was some great work on this issue recently, I think it’s still far from what the big G wants to see.
Given the issues with each group’s lack of good (or any) meta name description, and lame page titles, it may actually be more beneficial to block their indexing with a robots.txt file – as having a bunch of issues there affects the ranking for the rest of your site from what I gather.
-
AuthorSearch Results
