[Rack] robots.txt

[Leif Ryge]

Yeah, I'm planning to upgrade mediawiki soon (perhaps today) and also
will work with gnusosa to setup the Mozilla Persona extension.

~leif

On Tue, Apr 30, 2013 at 02:54:25PM -0700, Ben Kochie wrote:
> The  Disallow: /wiki/Special: came from the mediawiki examples.
> 
> I added the additional /wiki/Special*
> 
> Also, would someone who likes doing this kind of thing update our
> mediawiki:
> 
> http://lists.wikimedia.org/pipermail/mediawiki-announce/2013-April/000127.html
> http://lists.wikimedia.org/pipermail/mediawiki-announce/2013-April/000129.html
> 
> -ben
> 
> On Tue, 30 Apr 2013, Jeff Tchang wrote:
> 
> >
> >Googlebot (but not all search engines) respects some pattern matching.
> >
> > *  To match a sequence of characters, use an asterisk (*). For instance, to block access to all
> >    subdirectories that begin with private:
> >
> >User-agent: Googlebot
> >Disallow: /private*/
> >
> >
> >So in your example
> >
> >User-Agent: *
> >Disallow: /wiki/Special*
> >
> >Will work for google. I am not sure bingbot obeys it.
> >
> >On Tue, Apr 30, 2013 at 2:38 PM, Andy Isaacson <adi at hexapodia.org> wrote:
> >      On Tue, Apr 30, 2013 at 02:31:37PM -0700, Ben Kochie wrote:
> >      > I added a robots.txt to https://noisebridge.net
> >      >
> >      > User-agent: *
> >      > Disallow: /wiki/Help
> >      > Disallow: /wiki/MediaWiki
> >      > Disallow: /wiki/Special:
> >      > Disallow: /wiki/Template
> >      > Disallow: /wiki/skins/
> >      >
> >      > I noticed bingbot is uselessly crawling the entire contents of
> >      > Special:RecentChanges.
> >
> >      Is robots.txt a prefix, or a directory based exclusion scheme?  Will
> >      "Disallow: /wiki/Special:" cause bingbot to skip
> >      "/wiki/Special:RecentChanges"?
> >
> >      -andy
> >      _______________________________________________
> >      Rack mailing list
> >      Rack at lists.noisebridge.net
> >      https://www.noisebridge.net/mailman/listinfo/rack
> >
> >
> >
> >

> _______________________________________________
> Rack mailing list
> Rack at lists.noisebridge.net
> https://www.noisebridge.net/mailman/listinfo/rack

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://www.noisebridge.net/pipermail/rack/attachments/20130430/83c4f80e/attachment.sig>