I wanted to send out more pointers to some more things in the vein of my talk from last week. SMART is probably targeted more for academics. Hackers will probably be more interested in some of the more practical tools for building real world relevance systems.<br>
<br>Specifically I wanted to introduce you all to Nutch, which is an Apache project provider crawling and searching utilities built on Hadoop and
Lucene (also Apache Projects). With Nutch you can configure fetcher, parser, indexer, and searcher
plugins to use it for any thing from a pimped out custom search engine for your website or relevance engine for any domain.<br><b><br>What is Hadoop?</b><br>Hadoop is open source software to use in building scabable, distributed computing systems. It has a Map Reduce implementation, which Nutch uses to run it's crawling, parsing and indexing work.<br>
<b><br>What is Lucene?</b><br>Lucene is full text document search engine project. At the core of Lucene search algorithmns is tf-idf. Nutch uses Lucene by generating Lucene indices as the output of its crawl process. Nutch extends the Lucene searcher with it's plugins, but the core the relevance algorithm comes down to Lucene's: <a href="http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html" target="_blank">http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html</a><br>
<br><b>What can you do with Nutch?</b><br>I've used Nutch to implement full text search for <a href="http://findlaw.com" target="_blank">http://findlaw.com</a>. They used a Google Search Appliance for years, but we were able to use Nutch to replace the applicance with a custom in-house search engine implementation built on Nutch.<br>
<br>We also used Nutch to build an ads relevence system for serving up the ads on FindLaw.com. This project required custom fetcher, parser, indexer, and searcher Nutch plugins on top of the Nutch basics, resulting in a domain-specific revelence system that took advantage of Hadoop scale and Lucene's tf-idf implementation, while being totally outside the normal application domain of Nutch (full text search of web pages).<br>
<b><br>Links:</b><br>Nutch: <a href="http://nutch.apache.org/" target="_blank">http://nutch.apache.org/</a><br>Hadoop: <a href="http://hadoop.apache.org/" target="_blank">http://hadoop.apache.org/</a><br>Lucene: <a href="http://lucene.apache.org/" target="_blank">http://lucene.apache.org/</a><br>
A tutorial: <a href="http://wiki.apache.org/nutch/NutchTutorial" target="_blank">http://wiki.apache.org/nutch/NutchTutorial</a><br><br>Jared-<br><br>PS: I'll move this thread to the wiki for archival purposes eventually...<br>
<br><div class="gmail_quote">
On Wed, Sep 15, 2010 at 11:26 PM, Jared Dunne <span dir="ltr"><<a href="mailto:jareddunne@gmail.com" target="_blank">jareddunne@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
The Slides:<br><a href="https://docs.google.com/present/edit?id=0Ae0pay6z9C6GZGYzZG1uMm5fNDFmdGt2YnhoYw&hl=en" target="_blank">https://docs.google.com/present/edit?id=0Ae0pay6z9C6GZGYzZG1uMm5fNDFmdGt2YnhoYw&hl=en</a><br>
<br><br>SMART<br>
Someone asked a good question after the talk about if there was a "generic" vector space model framework out there. We discussed "search appliances" such as Google's offerings, but you were looking for something that you could hand off the term vectors or data for given domain and then have a toolkit of these vector space algorithms provided to look at it. I mentioned Salton on the theory front, but I should of also mentioned the result of his research, SMART, which is an implementation of his work with sample data sets. Its probably a good thing to play around with in the vein of your question.<br>
<br>Lots of good links from its wiki page:<br><a href="http://en.wikipedia.org/wiki/SMART_Information_Retrieval_System" target="_blank">http://en.wikipedia.org/wiki/SMART_Information_Retrieval_System</a><br><br>SMART unixy bits via FTP:<br>
<a href="ftp://ftp.cs.cornell.edu/pub/smart/" target="_blank">ftp://ftp.cs.cornell.edu/pub/smart/</a><br><br>This tutorial looks promising (loving the old school html):<br><a href="http://www.tcnj.edu/%7Emmmartin/CSC485IMME321/Papers/SMART/SmartCourse.html" target="_blank">http://www.tcnj.edu/~mmmartin/CSC485IMME321/Papers/SMART/SmartCourse.html</a><br>
<br><br>I'll probably send out some additional stuff later on about other areas that we started to touch on in the discussion after the talk, like query and term expansion and spell correction.<br><font color="#888888"><br>
Jared-<br><br><br>
</font></blockquote></div><br>