Kottke made an interesting post about the differences between the old and new Whitehouse.gov websites robots.txt files yesterday. Robots.txt files tell search engines what to include or not to include in their index.
The new one is :
User-agent: *
Disallow: /includes/
an example from the old one:
Disallow: /earmarks/search
Disallow: /earmarks/query.html
2400 lines of disallows, more over on Kottke’s post.
I wonder if it really alludes to transparency on their part or just a hurry from the dev’s to get the website up in time.