BuddyPress.org

I think, overall, the core devs have approached BP development in the most effective order possible.

There really isn’t much point in adding spam protection to a platform nobody uses, and nobody would be using BP if the core developers had spent the past two years hardening it with spam protection instead of adding member-centric features.

But we’ve hit “critical mass” now …and we have to deal with the spam problem before BP is ready to move from a science experiment people use on hobby sites to a “platform” that developers can use for serious commercial endeavors. And I’m willing to throw a sizable chunk of dev time at it to help make that happen.

Anyhow, at this point I’ve written the first draft of the proposed changes to the forum, and what I need now is everyone to read through it and post their feedback.

Seriously.

Because if something in the proposal doesn’t work for your application …or you can think of a better way to do it… you’d better get a post up NOW before we start writing code.

Thanks!

^F^

PART 3 – STRONG -vs- WEAK METHODS

When it comes to spam on BP sites, you’ll see all sorts of stuff posted on blogs saying “change [whatever] on your site and your spam problem will disappear”.

Truthfully, a lot of these tricks will actually work …for a while… but eventually, the spammer makes a minor change to their bot, and they’re back in business. In fact, many of the leading blog spamming packages include sophisticated logging features to catch the errors that “uniquely configured” blogs generate and help the spammer quickly fix the “problem”.

If we’re going to have a reliable anti-spam solution for BuddyPress, we should probably focus on “Mathematically Strong” methods, not on “Obfuscation” and “Moving Things Around”. That way, we won’t have to constantly change our spam protection methods.

Changing Page Slugs

Many people recommend changing the page slugs on BP installations to reduce spam. While this is certainly easy to do, you of course need to give your users *links* to those page slugs somewhere on your site so they can actually visit the pages. And if users can follow the links, so can a spam bot.

Changing page slugs is kind of like boarding-up the front door of your house, installing a new door in the side of your house, and then attaching a piece of string from the front door to the side door of so everyone can find the new door.

The “change your page slugs” approach seems to come from the “change your admin menu URL” technique. Changing your admin menu URL is actually a *strong* protection technique. Since there is no link to it anywhere on the site and you’re the only one that knows the URL, it’s like having two passwords on your admin login. An attacker would have to try billions of URL’s to find it.

Not so with all the other URL’s on your site. They have to be linked off other pages so your users can find them.

Adding Fake Form Fields

Many people recommend adding a few extra fields to forms throughout your site (sign-up, login, post to group, etc) and “hiding” these fields using CSS. If any of the “trap” fields are filled out, in theory, you’ve just detected a bot, because a normal user would never see the fields and fill them out.

This approach *might* defeat a very simple bot that searches every web page it can find for forms, and fills every field in every form with random spam; but it will not defeat a bot that understands CSS or is specifically targeted at BuddyPress, especially considering that BuddyPress is *open source*.

Don’t think bots can analyze CSS? Read this: http://www.google.com/support/webmasters/bin/answer.py?answer=66353

A bot designer can simply read through the BP source code and discover the names of the fields that should be filled in and the names of the fields that should be left empty.

To use our “house” analogy, adding extra form fields is like installing 3 front doors on your house and rigging two of them with grenades …then hanging a big red “out of order” sign on the the two rigged doors so your friends don’t use them.

Obviously if your friends can read the signs, so can your enemies.

JavaScript Proof of Work

Javascript proof of work (Wp Hashcash) defeats spammers by making visitor’s web browsers solve a math problem in JavaScript before they are allowed to post.

Because everyone knows spam bots can’t run JavaScript.

http://forums.digitalpoint.com/showthread.php?t=1124949
http://www.scrapebox.com/
http://blogcommentdemon.com/
http://www.senuke.com
http://www.botmasternet.com/more1/

Except when they can.

There’s also the issue of what to do with visitors that don’t have JavaScript enabled.

The WordPress and BuddyPress development teams have put an epic amount of work into ensuring both platforms will work reliably when JavaScript isn’t available. Requiring users to have JavaScript to post any kind of content to the site nullifies much of this work.

Proof-of-work was a great idea back in 1997 when spammers ran hundreds of attack threads from a single server and solving the JavaScript math problems slowed it to a crawl.

In 1997, we’d be dealing with a single spammer running 1000 attack threads against the site. Because the spammer was running 1000 threads, each of which would have to solve the JavaScript problem, they would effectively be penalized 1000 fold over a normal user. The end result is they would only be able to run a few threads before their computer slowed to a crawl and their spamming abilities would be sharply limited.

Epic win for site.

Unfortunately, things are different in 2010.

Spam bots have become the tool of choice for basement SEO marketers. Instead of a few members of the “spam elite”, we’re dealing with tens of thousands of “do it yourself” spammers each running 1 attack thread using the new “automatic backlink software” they just picked up for $29.00 off some random SEO website. Instead of fighting one spammer splitting their resources across a thousand threads, we’re fighting a thousand spammers running a single thread dedicated *just to our site*.

Skipping a ton of math, what this means, is that in order to cause a spammer a 1-second delay while their computer solves our JavaScript challenge, we have to cause each of our *legitimate users* a 1 second delay while *their* computer solves our JavaScript challenge. And, considering the 3 to 5 second database lag I see on 90% of the BP sites I visit, the challenge would need to take much longer than a second to have any merit at all …otherwise page refresh time would be the limiting factor, not the JS challenge.

So what happens when a user visits the site using a computer that is much slower than a typical desktop …say a mobile phone or an old laptop? The challenge would take proportionally longer to complete. A challenge that requires 5 seconds to solve on a desktop PC, could take 30 seconds on an iphone …and 30 second response times would not make for an enjoyable user experience.

Overall, proof-of-work challenges are probably not a good choice in the 2010 Internet landscape.

Mathematically Strong Methods

In the next post, I’ll cover the specific details of the methods I’ve proposed for the BP spam solution, and why they will defeat most spam attacks.

^F^

@Erich73

Add a “refresh” button beside the captcha that allows the user to flip through multiple captchas until they find one they like.

^F^

@thekmen

Last I heard, BuddyPress does not run activity stream posts, or anything else, through Aksmet …it’s wide-open and that’s what’s causing the problem!

If you install the WP Akismet plugin, it runs *blog comments* through Akismet, but that’s it.

See why I’m really concerned and am putting work into this?

^F^

PART 2 – DEFEATING SPAMMERS

In the last post I covered why and how spammers attack BP installations. This post will cover how I propose to counter them.

Fast Attacks -vs- Slow Attacks

There are two basic kinds of spam attacks that get run on social networks: “fast” or “flood” attacks, and “slow” attacks.

In a fast attack, the spammer signs up for an account on the site, then sends thousands of messages as quickly as possible.

Obviously, the site admin will be deluged with complaints about the spam user and quickly delete their account …but in the hours (or days) it takes the admin to respond, hundreds and hundreds of people will read the spam messages. Then the spammer signs up for another account, and repeats the process.

In a “slow” attack, the spammer signs up for *hundreds* of accounts on the system, often over a period of many months, and only sends out spam messages one at a time …often days, weeks, or months apart.

“Slow” attacks are very difficult to counter using automation …at least without annoying legitimate users.

The best way people have come up with so far is just a “report spam” button which, when clicked, reports the member to an admin so they can investigate it and if necessary delete the account. This will be implemented as part of @francescolaffi ‘s BP content moderation plugin in a couple of months.

Unfortunately, a “report spam” button doesn’t work well against “fast” attacks.

This is because:

a) There is a delay while the admin responds to manually submitted spam reports, or,
b) When a consensus scheme is used (if X users report a member their account gets suspended), there is a delay while enough votes are accumulated to flag the member as a spammer.

During that time, people are reading the spam messages and the spammer is winning.

Goals of Proposed BP Core Anti-Spam Mods

The goal of the proposed core modifications is to counter “fast” attacks by the following means:

1) To make it difficult for a spammer to create large numbers of member accounts using automated means.

2) To make it difficult for a spammer that already has a member account to use automated means to:
a) send large volumes of PM’s
b) send large numbers of friend requests
c) create large numbers of groups
d) create large numbers of group posts
e) post large numbers of comments
f) post large numbers of status updates

3) To accomplish 1) and 2) without being annoying to legitimate users.

4) To make the system configurable, so it can be adapted to the needs of the site …for example: visually impaired users, or display on mobile phones.

5) To make the system “on by default” and “secure by default”

How We Can Accomplish This

1) New User Sign-up

a) Add a captcha on the new account sign-up screen.
b) If the “user” gets the captcha wrong on the first try, require *TWO* captchas to be solved before they can proceed. (If the odds of a bot solving ONE captcha with OCR are 1 in 100, the odds of the bot solving TWO captchas with OCR are 1 in 10,000. This is a technique Gmail uses.)

…set X to be a random number on each installation between 3 and 7…

c) If the user gets X captchas wrong in a row, block their IP for a random amount of time (15 minutes to 2 hours). (This is what Craigslist does)
d) If the user fails X captchas *again* after being blocked, permanently ban their IP and post it to akismet.
e) If a locally banned IP tries to sign-up, don’t throw an “error page”. Completely ignore the request and don’t send anything.
f) If an akismet banned IP tries to sign up, require *TWO* captchas to be solved on the first try, and if they get X captchas wrong in a row, permanently ban their IP and repost it to akismet.
g) Add an option field to the admin menu that limits the number of accounts that can be created per IP address. By default, set it at 2.

2) Existing User Sign-In

a) Use a “normal” password box on first sign-in attempt.
b) If the member gets their password wrong on the first try, require them to solve a captcha on the second try. Offer password recovery option.
c) If the member gets their password wrong on the second try, require *TWO* captchas to be solved before they can proceed. Offer password recovery option.

…set X to be a random number on each installation between 3 and 7…

d) If the user gets X logins / captchas wrong in a row, block the visitor’s IP for a random amount of time (15 minutes to 2 hours).

3) Private Messages

a) Add a field to the user table that allows PM limiting to be bypassed or set to a unique value on a user-per-user basis.
b) Add three option fields on the admin menu: allow “X” messages to be sent every 24 hours, averaged over the past “Y” hours with “Z” hysteresis
…when BP is installed, randomly set X, Y, and Z to allow a daily maximum of between 18 and 24 messages, averaged over between 2 and 24 hours, +/- 3 messages.
c) If the maximum is exceeded, require the member to solve a captcha before they can send another PM.
d) If they get the first captcha wrong, require them to solve two captchas before they can send another PM.

…set R to be a random number on each installation between 3 and 7…

e) If the user gets R captchas wrong in a row, block their IP for a random amount of time (15 minutes to 2 hours). (This is what Craigslist does)
f) If the user fails R captchas *again* after being blocked, permanently ban their IP and post it to akismet.
g) If a locally banned IP tries to visit the site, don’t throw an “error page”. Completely ignore the request and don’t send anything.

Consider how difficult the algorithm above makes it to send automated messages. A spammer can’t just send “12 messages a day” or “1 message an hour” and avoid triggering the system. Every BP installation will have a unique combination that will cause it to trip. Yet for a “normal” user, the system will hardly ever trip, and if it does, it takes all of 5 seconds to enter a captcha and continue. And the system can be bypassed entirely for edge cases, like paid advertisers or site news.

3) Friend Requests

a) Create a config option in BuddyPress that allows the admin to remove the member’s directory with one click. Disable the member directory by “default” on new installs. In my experience, the only people that use the member’s directory (in its default state, on a socially oriented site) are Spammers, Marketers, and Competitors. There’s a reason Facebook, MySpace, LinkedIn, and Twitter do not have “global” member directories.
b) Implement same scheme as private messages.

4) Group Creation

a) Add a field to the user table that allows Group limiting to be bypassed or set to a unique value on a user-per-user basis.
b) Add an option field on the admin menu that sets a maximum number of groups that can be created by a user. By default, set it at 5.

5) Group Posts

a) Add a field to the user table that allows group post limiting to be bypassed or set to a unique value on a user-per-user basis.
b) Create a “whitelist” field on the admin page that allows “trusted” media sharing URL’s like YouTube, Revver, Flickr, etc to be bypassed in spam protection.
c) Create an option that automatically “scrubs” URL’s and email addresses from group posts if they are not on the whitelist. Not just “nofollow” …complete removal. This will stop 90% of abuse dead in its tracks, because most spammers are just trying to get traffic to a site or replies to an email.
d) If the system detects a URL or email address embedded in a message, and it’s not on the whitelist, require a captcha to be solved before allowing the post.
e) If they get the first captcha wrong, require them to solve two captchas before approving the post.

…set R to be a random number on each installation between 3 and 7…

f) If the user gets R captchas wrong in a row, block their IP for a random amount of time (15 minutes to 2 hours).
g) If the user fails R captchas *again* after being blocked, permanently ban their IP and post it to akismet.
h) If a locally banned IP tries to visit the site, don’t throw an “error page”. Completely ignore the request and don’t send anything.

i) For posts that do not contain a URL or email address, run the post through akismet. If it passes, approve the post. If it fails, require a captcha to be solved before allowing the post.
j) If they get the first captcha wrong, require them to solve two captchas before approving the post.
k) If the user gets R captchas wrong in a row, block their IP for a random amount of time (15 minutes to 2 hours).
l) If the user fails R captchas *again* after being blocked, permanently ban their IP and post it to akismet.
m) If a locally banned IP tries to visit the site, don’t throw an “error page”. Completely ignore the request and don’t send anything.

6) Comments

a) Create an admin option that only allows users to comment on their *friend’s* items. Activate it by default on new BP installations.

7) Status Updates

a) Add a field to the user table that allows status update limiting to be bypassed or set to a unique value on a user-per-user basis.
b) Create a “whitelist” field on the admin page that allows “trusted” media sharing URL’s like YouTube, Revver, Flickr, etc to be bypassed in spam protection.
c) Create an option that automatically “scrubs” URL’s and email addresses from status updates if they are not on the whitelist. Not just “nofollow” …complete removal. This will stop 90% of abuse dead in its tracks, because most spammers are just trying to get traffic to a site or replies to an email.
d) If the system detects a URL or email address embedded in a message, and it’s not on the whitelist, require a captcha to be solved before allowing the activity stream post.
e) If they get the first captcha wrong, require them to solve two captchas before approving the activity stream post.

…set R to be a random number on each installation between 3 and 7…

f) If the user gets R captchas wrong in a row, block their IP for a random amount of time (15 minutes to 2 hours).
g) If the user fails R captchas *again* after being blocked, permanently ban their IP and post it to akismet.
h) If a locally banned IP tries to visit the site, don’t throw an “error page”. Completely ignore the request and don’t send anything.

i) For activity stream posts that do not contain a URL or email address, run the post through akismet. If it passes, approve the post. If it fails, require a captcha to be solved before allowing the post.
j) If they get the first captcha wrong, require them to solve two captchas before approving the post.
k) If the user gets R captchas wrong in a row, block their IP for a random amount of time (15 minutes to 2 hours).
l) If the user fails R captchas *again* after being blocked, permanently ban their IP and post it to akismet.
m) If a locally banned IP tries to visit the site, don’t throw an “error page”. Completely ignore the request and don’t send anything.

8 ) In All Cases

a) When a member account is banned, or repeatedly triggers spam protection measures, send an alert to the site administrator.
b) Allow admin alerts to be disabled if necessary, example: DDOS attack against the site.

9) CONCLUSION

While the list of modifications above may look incredibly complicated, really, it’s not.

I’d say “worst case” it’s about a week of work to research and make these modifications. Then we can push it out into beta testing with all the other new code to give it a proper shakedown.

I’m sure there are plenty of ways the algorithms above could be improved, so please go ahead and post your feedback!

Thanks!

^F^