[Noisebridge-discuss] Distributed computing and storage (with some major caveats)

[Sai Emrys]

Hi all.

I'm interested in building something with the following primary features:

* operate on very distributed set of computers ("nodes")
* do arbitrary MapReduce-style computation and other segmentable actions
* do arbitrary mysql-style database operations (or worst case,
memcache-style plus some function for indexing)

The caveats - it must also be:
* P2P with no centralized components whatsoever (e.g. C&C, node
listing, discovery service, etc)
* self-organizing (e.g. for addition of a new node, it might know a
couple neighbors, but should be able to join in fully in a way that's
resilient against those neighbors going down)
* tolerant to any arbitrary number of nodes going down or entering at
any time with minimal interruption of data and calculations
* securely operable (e.g. crypted requests, verifiably signed data entry, etc)
* tolerant to (and preferably detecting of) bogus or outright
malicious nodes with minimal loss of data or calculations (e.g. by
confirming nodes' performance against other nodes?)
* low profile and cross platform (with nodes of any arbitrary
combination of platforms and CPU/RAM/HD capacities)

The closest two things I'm aware of are FreeNet (for storage) and
BOINC (for computation). MapReduce itself (and all variants I'm aware
of) fail on the centralization and node trust. FreeNet fails at
indexing, and possibly at persistence strategy (though that's a
difficult problem). AFAIK BOINC also fails at centralization.

Do any of you know other / better sources? Any of you interested in
building something like this?

My preferred language is Ruby, though I'd be willing to learn
something else for this.

Thanks,
 Sai