[Noisebridge-discuss] Distributed computing and storage (with some major caveats)

[Danny O'Brien]

On Mon, Jun 22, 2009 at 7:00 PM, Sai Emrys <noisebridge at saizai.com> wrote:

> Hi all.
>
> I'm interested in building something with the following primary features:
>
> * operate on very distributed set of computers ("nodes")
> * do arbitrary MapReduce-style computation and other segmentable actions
> * do arbitrary mysql-style database operations (or worst case,
> memcache-style plus some function for indexing)
>
> The caveats - it must also be:
> * P2P with no centralized components whatsoever (e.g. C&C, node
> listing, discovery service, etc)
> * self-organizing (e.g. for addition of a new node, it might know a
> couple neighbors, but should be able to join in fully in a way that's
> resilient against those neighbors going down)
> * tolerant to any arbitrary number of nodes going down or entering at
> any time with minimal interruption of data and calculations
> * securely operable (e.g. crypted requests, verifiably signed data entry,
> etc)
> * tolerant to (and preferably detecting of) bogus or outright
> malicious nodes with minimal loss of data or calculations (e.g. by
> confirming nodes' performance against other nodes?)
> * low profile and cross platform (with nodes of any arbitrary
> combination of platforms and CPU/RAM/HD capacities)
>
> The closest two things I'm aware of are FreeNet (for storage) and
> BOINC (for computation). MapReduce itself (and all variants I'm aware
> of) fail on the centralization and node trust. FreeNet fails at
> indexing, and possibly at persistence strategy (though that's a
> difficult problem). AFAIK BOINC also fails at centralization.


On the storage side, you might be interested in Tahoe. A lot of it is
orthogonal to your original request, but still kind of interesting, and
Zooko is a good guy for having these kind of discussions.

http://allmydata.org/trac/tahoe

d.



>
>
> Do any of you know other / better sources? Any of you interested in
> building something like this?
>
> My preferred language is Ruby, though I'd be willing to learn
> something else for this.
>
> Thanks,
>  Sai
> _______________________________________________
> Noisebridge-discuss mailing list
> Noisebridge-discuss at lists.noisebridge.net
> https://www.noisebridge.net/mailman/listinfo/noisebridge-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.noisebridge.net/pipermail/noisebridge-discuss/attachments/20090629/1039364e/attachment-0003.html>