Get exclusive CAP network offers from top brands

View CAP Offers

WH robots.txt

[bsa_pro_ad_space id=2]
  • This topic is empty.
Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #591288
    Anonymous
    Guest

    Check this out. It’s kind of interesting.
    http://www.whitehouse.gov/robots.txt

    #677289
    Anonymous
    Inactive

    Hmm, lots of stuff they don’t want spidered.

    Lookis like all of their Iraq stuff isn’t going to be spidered.

    #677296
    Anonymous
    Inactive

    Interesting indeed. But why would they want to put all that stuff up on the web in the first place if they don’t want people reading it?

    Bart

Viewing 3 posts - 1 through 3 (of 3 total)