The Whitehouse.gov Website’s Robots.txt File Has 1839 Lines In It

By Daniel Miessler on January 23rd, 2007: Tagged as Internet | Security

1839?

I’m sure they’re not trying to hide anything, as doing so with a robots.txt file would be asinine, but 1839 entries is just insane. At some point you have to start wondering whether the content should be online at all if you don’t want it indexed.

I wonder if they’re using any of the directories as honeydirs (I just made it up). The idea being that you would have no links to a given directory, tell people in the robots.txt file absolutely NOT to go there, and then sit and log people who show up. It’d be easy to do in Apache:

grep $bad_directory access_log | cut -f1

...and there’s your list of “curious” people.

Probably not worth it for this kind of site, but an interesting thought. In fact, I’m going to implement this tonight for my site.

Comments

Some Joker

Looking at most of those entries, it looks like they're excluding pages which look to be designed for text only browsers/screen readers.. nearly every directory ends in /text

Disallow: /asia/2005/photoessay/china/text
Disallow: /asia/2005/photoessay/japan/text
Disallow: /asia/2005/photoessay/korea/text
Disallow: /asia/2005/photoessay/mongolia/text
Disallow: /asia/2005/photoessay/mrsbush1/text
Disallow: /asia/2005/photoessay/mrsbush2/text

and if you browse up one directory, you get the same story with pictures..

I'd say it looks like they are doing it to work around for a poor file structure or possibly to keep search engines from finding duplicate text (although without pictures)

*shrugs* I'm all for pointing out when the administration does something crooked, but I can't see fault in this one.. (granted, I've only checked out 20 or so of the links.. the only one that didn't go anywhere for me was /video/text )