Sitemaps for wikis, penalties for spammers

page 1 of 3123next »

Sitemaps for wikis, penalties for spammers

michal frackowiak 12 Jun 2008 09:35

We have just introduced a new nice feature for all wikis — automatic sitemaps. According to sitemaps.org:

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site.

Every wiki now features a special URL which contains a list of pages within a wiki to help search engines crawl and find your content easier. E.g. a sitemap for www.wikidot.com can be found at http://www.wikidot.com/sitemap.xml (note: this is an XML file, your browser might not display this correctly). Of course there are no sitemaps for private wikis — they are not public by definition.

But honestly, last week we have been fighting spam on Wikidot most of the time. About 10% of all content created on Wikidot within the last 2 years is spam (this is not that much, but could become a serious problem if left unattended). Our job is to find spam, hide it from you and discourage people from spamming. Therefore we have built a very nice automatic filter for detecting spam content and now we need to plug this to Wikidot to provide 2 kinds of functionality:

• it should help us marking some wikis as spam (those are mostly about bank credits, health insurances etc.)
• it should help wiki admins and blog authors to protect from spam (spam comments, forum posts, spam pingbacks etc.) — we will certainly need this once we introduce a real blogging platform

As an important step in discouraging spammers from starting spam wikis, every wiki marked as spam:

• blocks all web crawlers (including Google, Yahoo MSN and all others) from indexing the content (by using the robots.txt file),
• changes all external links into redirects with special rel="nofollow" attribute — such links are not visible to any web crawler.

Those certainly make creating "link farms" on Wikidot pretty useless.

Although sitemaps do not bring any new visible improvements, they will certainly improve your wikis' visibility in the Internet. Good luck to all Wikidotians!

---

Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me

Last edited on 13 Jun 2008 04:50 by michal frackowiak + Show more

Wikidotians

scottplan 13 Jun 2008 00:08

Far be it from me to question the King on this matter of highest importance…

but shouldn't that be Wikidotians?

Lilliput : Lilliputians = Wikidot : Wikidotians

Humbly submitted.

S.

Re: Sitemaps for wikis, penalties for spammers

michal frackowiak 13 Jun 2008 04:49

Well, I guess you are right! Although there is no "Wikidotians" entry in any dictionary, this is a good point.

Thanks!

(updated in the news above)

---

Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me

Re: Sitemaps for wikis, penalties for spammers

RMIGHTY1 13 Jun 2008 20:24

Aww, durn, I thought I was a wikidite or a wikidonian!

Re: Sitemaps for wikis, penalties for spammers

David Marseilles 15 Jun 2008 10:03

Keep up the good work. I'm psyched to see the blogging platform evolve.

Re: Sitemaps for wikis, penalties for spammers

wikario1 16 Jun 2008 01:22

Good news!

But rel = "nofollow" looks like something paid and closed (in sense of truly open source philosophy). I think it would be better (but 100 % are more difficult) to realise struggle against a spam and spamers through a rating. As soon as the rating goes down below certain level the ability to create pages is closed… Further - possibility to edit current pages disappears, and then the account goes blocked and spams-pages removes.

Thanks!

Last edited on 16 Jun 2008 01:26 by wikario1 + Show more

Re: Sitemaps for wikis, penalties for spammers

cold_blood3d 16 Jun 2008 01:54

great progress. I am continually impressed by Wikidot.

Re: Sitemaps for wikis, penalties for spammers

michal frackowiak 16 Jun 2008 07:32

Thanks for all the comments:

@wikario1:

We would like to avoid complicated "spam rating systems" simply because we do not want to start playing games with spammers. As soon as you set rules, ratings etc. people start finding workarounds, tricks and hacks. Also, it sometimes could hurt normal users. Somehow internally we are introducing various ratings through the statistical filter, but those are not used for anything else than spam detection and not necessarily "per user". In fact we are only feeding the filter with data and it figures the patterns itself. The only kinds of visible user levels are the karma levels we have introduced.

What we actually want is to know which wikis are violating our Terms and which are created to promote spam. We have a set of heuristic measures to find these and we are going to plug in our Bayesian filter too.

I think the more important thing is, once we start the mentioned blogging platform, to stop spam from appearing in comments, pingbacks etc. According to Akismet stats there is much more spam in the blogsphere than legitimate content. And nobody wants to look for and review spam manually. Also we have seen several forums affected by spammers where community acts as a filter and report spam manually. We want to help in this matter.

It appears that as a side-effect of working on spam we have developed a service (now in our internal use only) very similar to Akismet or TypePad AntiSpam — where users can send various content for spam identification. Not sure now if it will develop into a public service but it looks nice so far and has at least a few nice advantages over the competition (e.g. it is not bound to blogs only, has more flexible API…).

Anyway, I am getting back to work ;-) Thanks for your comments and all the support for the project!

Michal

---

Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me

Last edited on 16 Jun 2008 18:50 by Gabrys + Show more

Re: Sitemaps for wikis, penalties for spammers

wikario1 17 Jun 2008 16:14

@Michal:

Ohh… Now I have completely understood your point of view. Thanks for the answers. You do all correctly. Otherwise Wikido.com can turn in to another trash full by a spam. But personally I, as the supporter of open source and open internet don't like & welcome rel=nofollow methods. I agree, that spammers will try to bypass any filters. That's the shameful evolution of spam. And that's very bad! Cleaning and protection against spam is a difficult and long process which should be as much as possible automated without a damage to ordinary users. (for example, some of my wiki sites have got under the nofollow filter after the someone has placed there spam content (now it's deleted by me, and closed for editing).

So what's the pass for open and safe spam fighting? You know this much better than me, guys… I just discuss this hard problem from my angle :)

On my supervision on the Internet there is a refusal from nofollow - revival of live follow links.

—-
Regards,
Mike

Re: Sitemaps for wikis, penalties for spammers

michal frackowiak 17 Jun 2008 18:45

I can agree that rel="nofollow" is not a "standard", but rather looks like a quick hack proposed by Google to tell search engines: "do not follow this link". Also, the rel attribute should define relation between items, not behavior.

But both Yahoo! and MSN accepted this hack and are using it. And effectively DOES stop spammers from building spam link farms. Many major open-source software (WordPress, MediaWiki) also have support for rel="nofollow" attribute. So de facto it became a web standard.

Also, we are not (yet?) putting such anti-spam solutions into our open-source release.

Let's wait and see how it works.

---

Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me

page 1 of 3123next »