There is a real need for a reliability metric. For instance, a lot people recently tweeted rumors about Jeff Goldblum dying. Penalize those people. Other people actually were debunking the rumors, reward them. It will help quality to use an NLP grammar parser I believe. Every time there are rumors/scams on twitter you have a chance to improve your metric.
Problem with an NLP parser is since twitter restricts to 140 characters a lot of people don't post proper grammatically correct sentences. This makes the parser much more complex since the parser will then have to guess the intended meaning. A tweet like "Jeff Goldblum???" would be almost impossible for the parser to understand.
There really aren't that many ways to say "I heard Jeff goldblum just died???". You could probably make a list of like 100 different ways and get good accuracy. In fact, question marks alone are a good indication the person didn't do any private verification.
If you were going to create 100 different rules for that one sentence, imagine the costs of scaling up. It simply is not viable. The only way to make this possible is to maybe train the system for a corpus consisting of about a million tweets. But the problem with that would be that those million tweets would first have to be individually tagged manually by humans.
But how many ways can a rumor be stated? Say the rumor is someone got married: "did so-and-so just get married???". If you make a 100 sentence structures to look for could you reuse them for multiple rumors?
The problem is not how many different ways a rumor can be written, it's how many rumors can be written. Esp. considering the fact that you have websites which automate the process of rumor-manufacturing...
Thanks for your comments, I agree that something automated is more efficient and robust. However, I was also thinking doing something manual might be a quick way to build and launch a site, even if it has poor accuracy. If you want to discuss it further, send me an email. Its in my profile.
