Get exclusive CAP network offers from top brands

View CAP Offers

Unvalidated Robots.Txt Risks Google Banishment

[bsa_pro_ad_space id=2]
  • This topic is empty.
Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #606157
    Anonymous
    Inactive

    i subscribe to webpronews and received this article…kinda interesting:



    Unvalidated Robots.Txt Risks Google Banishment
    David A. Utter | Staff Writer

    The web crawling Googlebot may find a forgotten line in robots.txt
    that causes it to de-index a site from the search engine.

    Webmasters welcome being dropped out of Google about as much as
    they enjoy flossing with barbed wire. Making it easier for Google
    to do that would be anathema to being a webmaster. Why willingly
    exclude one’s site from Google?

    That could happen with an unvalidated robots.txt file. Robots.txt
    allows webmasters to provide standing instructions to visiting
    spiders, which contributes to having a site indexed faster and
    more accurately.

    Google has been considering new syntax to recognize within
    robots.txt. The Sebastians-Pamphlets blog said Google confirmed
    recognizing experimental syntax like Noindex in the robots.txt
    file.

    This poses a danger to webmasters who have not validated their
    robots.txt. A line reading Noindex: / could lead to one’s site
    being completely de-indexed.

    The surname-less Sebastian recommended Google’s robots.txt
    analyzer, part of Google’s Webmaster Tools, and only using
    the Disallow, Allow, and Sitemaps crawler directives in the
    Googlebot section of robots.txt.



    not sure if this has any validity, but interesting none-the-less.

    ps.

    #754947
    Lucretia
    Member

    I got this email too but I do not use restrictions for the robots.txt (standard) file

    I do not think I have a problem or do I ?

    Regards
    Rick

    #754952
    Anonymous
    Inactive

    not really. the only reason i got it setup is because once my programmer forgot to put a password to the link exchange admin section, and then google bot went through all of them and in particular the delete link…

    effectively this deleted all the links :roflmao: so now i just use it as back up in case my programmer goes incompetent for a short bit…

Viewing 3 posts - 1 through 3 (of 3 total)