[Rack] robots.txt

[Andy Isaacson]

On Tue, Apr 30, 2013 at 02:31:37PM -0700, Ben Kochie wrote:
> I added a robots.txt to https://noisebridge.net
> 
> User-agent: *
> Disallow: /wiki/Help
> Disallow: /wiki/MediaWiki
> Disallow: /wiki/Special:
> Disallow: /wiki/Template
> Disallow: /wiki/skins/
> 
> I noticed bingbot is uselessly crawling the entire contents of
> Special:RecentChanges.

Is robots.txt a prefix, or a directory based exclusion scheme?  Will
"Disallow: /wiki/Special:" cause bingbot to skip
"/wiki/Special:RecentChanges"?

-andy