[Noisebridge-discuss] Share your Twitter Firehose?

[John Adams]

On Fri, Nov 26, 2010 at 5:01 PM, Sai <sai at saizai.com> wrote:
> I'd like to do a public deanonymization demonstration to prove a point
> about privacy (and a flaw in several major browsers that really ought
> to have been fixed by now).
>
> In order to do so, I want access to a reasonably up-to-date (i.e. up
> to within last couple days at any point) database of
> a) Twitter follower relationships, expired within a week or so
> b) URLs posted, de-shortened, by posting username, w/ tweet ID

I work for Twitter. Docs on the streaming API (aka firehose) are available here:
http://dev.twitter.com/pages/streaming_api

The social graph isn't directly available. You'll have to query each
user via the REST API for that, and it changes constantly. It is a
multi hundred million node graph with a few billion edges. Again, not
something you're going to be able to easily process.

For the tweets themselves, you'll be interested in the Site streams
feed, which we offer in a low-bandwidth, free mode called the
"spritzer." There's very little chance you could consume the full
firehose. We don't offer it to anyone except paid partners, and even
then it's bandwidth is in the 5-8 megabits/second range.

Seeing that I also work in the security group here, I'm also
interested in what you think you may be able to do with the feed.

-j