[Noisebridge-discuss] Distributed computing and storage (with some major caveats)

[Ian]

we have a cloud mailing list for anyone not on it.

On Mon, Jun 29, 2009 at 3:17 PM, Danny O'Brien<danny at spesh.com> wrote:
>
>
> On Mon, Jun 22, 2009 at 7:00 PM, Sai Emrys <noisebridge at saizai.com> wrote:
>>
>> Hi all.
>>
>> I'm interested in building something with the following primary features:
>>
>> * operate on very distributed set of computers ("nodes")
>> * do arbitrary MapReduce-style computation and other segmentable actions
>> * do arbitrary mysql-style database operations (or worst case,
>> memcache-style plus some function for indexing)
>>
>> The caveats - it must also be:
>> * P2P with no centralized components whatsoever (e.g. C&C, node
>> listing, discovery service, etc)
>> * self-organizing (e.g. for addition of a new node, it might know a
>> couple neighbors, but should be able to join in fully in a way that's
>> resilient against those neighbors going down)
>> * tolerant to any arbitrary number of nodes going down or entering at
>> any time with minimal interruption of data and calculations
>> * securely operable (e.g. crypted requests, verifiably signed data entry,
>> etc)
>> * tolerant to (and preferably detecting of) bogus or outright
>> malicious nodes with minimal loss of data or calculations (e.g. by
>> confirming nodes' performance against other nodes?)
>> * low profile and cross platform (with nodes of any arbitrary
>> combination of platforms and CPU/RAM/HD capacities)
>>
>> The closest two things I'm aware of are FreeNet (for storage) and
>> BOINC (for computation). MapReduce itself (and all variants I'm aware
>> of) fail on the centralization and node trust. FreeNet fails at
>> indexing, and possibly at persistence strategy (though that's a
>> difficult problem). AFAIK BOINC also fails at centralization.
>
> On the storage side, you might be interested in Tahoe. A lot of it is
> orthogonal to your original request, but still kind of interesting, and
> Zooko is a good guy for having these kind of discussions.
>
> http://allmydata.org/trac/tahoe
>
> d.
>
>
>>
>>
>> Do any of you know other / better sources? Any of you interested in
>> building something like this?
>>
>> My preferred language is Ruby, though I'd be willing to learn
>> something else for this.
>>
>> Thanks,
>>  Sai
>> _______________________________________________
>> Noisebridge-discuss mailing list
>> Noisebridge-discuss at lists.noisebridge.net
>> https://www.noisebridge.net/mailman/listinfo/noisebridge-discuss
>
>
> _______________________________________________
> Noisebridge-discuss mailing list
> Noisebridge-discuss at lists.noisebridge.net
> https://www.noisebridge.net/mailman/listinfo/noisebridge-discuss
>
>