There seemed to be a lot of interest in the competition last night. We sorta splintered off into group discussions about the competition, but never really reconvened before Erin's talk started. Maybe we should report back over the mailing list on what thoughts everyone had about the competition?<br>
<br>Theo and I discussed two main areas...<br><br>Process:<br>- We shouldn't have a single approach to solving the problem. If people have ideas they should run with them and report back their success/failure to the group. The collaboration between our diverse ideas/approaches/experiences will be our strength in working together.<br>
- Since this is throw away code for this competition only, we need not get hung up on efficiency or elegant implementations. That said, if we hit a point where our code is not able to perform fast enough then we can address it at that point, instead of overengineering from the get-go.<br>
- Theo suggested that we start by using things like python/ruby scripts to massage the starting data set into something more useful (with more features), then analyse and visualize that using things like R.<br>- I'm wondering if people think it's legit to use the mailing list for discussion or if we should create a discussion list for the competition to prevent from spamming the main list with competition collboration?<br>
- Also, as we transform the dataset into different views, we are going to end up with some large files that we will be passing around to each other. Any suggestions on how to best do that? ML git repo?<br>
<br>Strategy (this is since just brainstorming level ideas):<br>- The dataset forms a graph of directed edges between vertices. At the core of this problem will performing analysis on that graph. The first intuitive approach we had come to mind was that the shorter the distance between two vertices using existing edges, the more likely it would be that an edge could/should exist between those vertices.<br>
- After the talk, Erin, Theo, and I stumbled on the idea that some vertices might be uber-followers (meaning more outbound edges than the average vertex) and that some vertices might be uber-followees (meaning more inbound edges than average). This reminded me of PageRank for link graphs, so perhaps we can draw from techniques in that vein. The application of this in our problem, might be in weighting since people who follow lots of people might be more likely to follow someone further out in their "network" where, someone who doesn't follow many people might less likely to follow someone outside their "network".<br>
- Since the edges are directional, we know that it's possible for people to "follow" someone with out that person "following back". At first glance it might make sense that the reverse edges would be likely in cases like this. However consider a "hub" user with lots of followers who doesn't reciprocate with edges back to his followers, then the information of who follows him is less important in determining who he would follow. Conversely, for a user who commonly reciprocates with followbacks, then the information on who follows her might be useful in suggesting who she follow.<br>
<br>Update:<br>- Last night I started thinking about this as a graph theory problem and started researching techniques. This section seemed useful for getting started: <a href="http://en.wikipedia.org/wiki/Graph_theory#Graph-theoretic_data_structures">http://en.wikipedia.org/wiki/Graph_theory#Graph-theoretic_data_structures</a><br>
- The data provided by kaggle is basically a "indicence list". Theo and I discussed converting the provided data in a form that maps outbound vertices to their list of inbound/target vertices, which it turns out is called a "adjacency list".<br>
- I wrote some ruby code last night to generate an adjacency list from the original training data. I dumped it to CSV format where the first column in a row is the outbound vertex, and all following columns for a given row are the list of inbound vertexs pointed to by the oubtbound vertex's edges. I can upload that somewhere once we figure out the best spot to hand off things like this...<br>
<br>So what wonderful ideas were happening on the other side of the room prior to Erin's talk? <br><br>Jared-<br><br><div class="gmail_quote">On Wed, Nov 10, 2010 at 2:32 PM, Joe Hale <span dir="ltr"><<a href="mailto:joe@jjhale.com" target="_blank">joe@jjhale.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hey,<div><br></div><div>I'll be going along to Noisebridge at 7.30 and will start having a look at the social network data in the 45 min before Erin's talk.</div><div><br></div><div>Laters,</div><div><br></div><font color="#888888"><div>
Joe</div></font><div><div></div><div><div><br></div><div><br><div class="gmail_quote">On 10 November 2010 13:19, Mike Schachter <span dir="ltr"><<a href="mailto:mike@mindmech.com" target="_blank">mike@mindmech.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Awesome everyone! Just so you know, I won't be in tonight or<br>next week, please keep me informed via email list and wiki about<br>what's going on if you can,<br><font color="#888888"><br> mike</font><div><div>
</div>
<div><br><br><br><div class="gmail_quote">On Wed, Nov 10, 2010 at 11:44 AM, Shahin Saneinejad <span dir="ltr"><<a href="mailto:ssaneine@gmail.com" target="_blank">ssaneine@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Hey, I'd really like to help but there's no way I can make it to the meeting tonight. My schedule's otherwise flexible in case everyone's open to meeting at a different time this week for the competition. If not, maybe I can catch up via project wiki notes or something.<br>
<font color="#888888">
<br>Shahin<br><br><br></font><div class="gmail_quote"><div><div></div><div>On Wed, Nov 10, 2010 at 11:11 AM, mnsqerr <span dir="ltr"><<a href="mailto:mnsqerr@webmail.co.za" target="_blank">mnsqerr@webmail.co.za</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div><div></div><div>
<p>Mike,<br>This sounds really fun. Lets do it! </p>
<p><br>The link you posted is not working for me, here is a working link: <a href="http://kaggle.com/component/taskmaster/?view=competition&task_id=2464" target="_blank">http://kaggle.com/component/taskmaster/?view=competition&task_id=2464</a> </p>
<p> </p>
<p>-Erin</p><br><hr>
South Africa premier free email service - <a href="http://www.webmail.co.za/" target="_blank">webmail.co.za</a>
<br><a href="http://b.wm.co.za/click.pwm?cid=20039230&loc=N-MT&seq=4cdaee66" target="_blank"><img src="" border="0"></a>
<br></div></div><div>_______________________________________________<br>
ml mailing list<br>
<a href="mailto:ml@lists.noisebridge.net" target="_blank">ml@lists.noisebridge.net</a><br>
<a href="https://www.noisebridge.net/mailman/listinfo/ml" target="_blank">https://www.noisebridge.net/mailman/listinfo/ml</a><br>
<br></div></blockquote></div><br>
<br>_______________________________________________<br>
ml mailing list<br>
<a href="mailto:ml@lists.noisebridge.net" target="_blank">ml@lists.noisebridge.net</a><br>
<a href="https://www.noisebridge.net/mailman/listinfo/ml" target="_blank">https://www.noisebridge.net/mailman/listinfo/ml</a><br>
<br></blockquote></div><br>
</div></div><br>_______________________________________________<br>
ml mailing list<br>
<a href="mailto:ml@lists.noisebridge.net" target="_blank">ml@lists.noisebridge.net</a><br>
<a href="https://www.noisebridge.net/mailman/listinfo/ml" target="_blank">https://www.noisebridge.net/mailman/listinfo/ml</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
ml mailing list<br>
<a href="mailto:ml@lists.noisebridge.net" target="_blank">ml@lists.noisebridge.net</a><br>
<a href="https://www.noisebridge.net/mailman/listinfo/ml" target="_blank">https://www.noisebridge.net/mailman/listinfo/ml</a><br>
<br></blockquote></div><br>