It might look a bit crazy, but I'd like to do some stats on all tweets from the beginning.
I heard a team of scientists got them by asking Twitter directly, but I wondered if it was possible to get this data in another way.
At the end I will filter them, so I could also search in an archive file, but I need data fron a large time scale. Actually I don't need them all, the last year would be good (the more, the better).
This question Is there an app with a historical archive of twitter messages for a specific hashtag? suggests searching on Google, but I don't think I can "crawl" Google, so I'm not sure it's possible to fetch the data, or is it ?
Plus I would like the tweets not to be added twice, and I'd like to get all retweets as well, which might not be included in Google search.
Another source could be the Library of Congress which will archive twitter history, but I've looked for this data, but didn't find anything, apparently they don't have it yet.
So do you guys know another source which keeps track of all tweets, or how I could use one of the above ? I'd appreciate any suggestion.
Best Answer
I'm not sure it's possible to get ALL tweets without providing a search term. That would be a hell of a lot of data - even just for this year.
There are a list of services here however all them require you to provide a search term.
Your best bet may be Archivist as they actually allow you to download the source code of the site. You may be able to customise this to do what you want.
If all that fails, your only other option would be to contact Twitter directly and see if they can provide you with a years worth of data. I wouldn't hold out much hope for that though!
Good luck!