Get exclusive CAP network offers from top brands

View CAP Offers

Htaccess question?

[bsa_pro_ad_space id=2]
  • This topic is empty.
Viewing 1 post (of 1 total)
  • Author
    Posts
  • #596902
    Anonymous
    Inactive

    I hired someone to fix my htaccess file to block all forms of the java bot which it sucessfully does. However, when checking my logs today I noticed the following, you will see that the first hit was sucessfully blocked and the bot was given a 403, however under the same ip the bot continued to crawl every page of my site but it does not have a user agent. So it looks like when the java bot learned it was being blocked, it automatically switched to an unknown user agent in order to bypass my htaccess block. Can anyone shed some light on whats going on here. This is a new site and it hasnt even been completely indexed by all the search engines yet, so my concern is that these thieves are going to get my content indexed first, then I will get hit with the dupe. Thanks in advance for any insight.

    Host: 38.99.203.110

    /robots.txt
    Http Code: 403 Date: Sep 13 01:26:01 Http Version: HTTP/1.1 Size in Bytes: –
    Referer: –
    Agent: Java/1.6.0-beta2
    |
    |
    |

    /
    Http Code: 200 Date: Sep 13 01:26:01 Http Version: HTTP/1.1 Size in Bytes: 13233
    Referer: –
    Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.0)
    |
    |
    |

    /sitemap.html
    Http Code: 200 Date: Sep 13 01:26:02 Http Version: HTTP/1.1 Size in Bytes: 9842
    Referer: –
    Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.0)
    |
    |
    |

    /poker-reviews/doyles.html
    Http Code: 200 Date: Sep 13 01:26:03 Http Version: HTTP/1.1 Size in Bytes: 6255
    Referer: –
    Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.0)
    |
    |
    |

    /poker-reviews/hollywood.html
    Http Code: 200 Date: Sep 13 01:26:04 Http Version: HTTP/1.1 Size in Bytes: 5574
    Referer: –
    Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.0)
    |
    |
    |

    /poker-reviews/absolute.html
    Http Code: 200 Date: Sep 13 01:26:05 Http Version: HTTP/1.1 Size in Bytes: 5318
    Referer: –
    Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.0)
    |
    |
    |

Viewing 1 post (of 1 total)