- This topic is empty.
-
AuthorPosts
-
September 13, 2006 at 2:20 pm #596902AnonymousInactive
I hired someone to fix my htaccess file to block all forms of the java bot which it sucessfully does. However, when checking my logs today I noticed the following, you will see that the first hit was sucessfully blocked and the bot was given a 403, however under the same ip the bot continued to crawl every page of my site but it does not have a user agent. So it looks like when the java bot learned it was being blocked, it automatically switched to an unknown user agent in order to bypass my htaccess block. Can anyone shed some light on whats going on here. This is a new site and it hasnt even been completely indexed by all the search engines yet, so my concern is that these thieves are going to get my content indexed first, then I will get hit with the dupe. Thanks in advance for any insight.
Host: 38.99.203.110
/robots.txt
Http Code: 403 Date: Sep 13 01:26:01 Http Version: HTTP/1.1 Size in Bytes: –
Referer: –
Agent: Java/1.6.0-beta2
|
|
|/
Http Code: 200 Date: Sep 13 01:26:01 Http Version: HTTP/1.1 Size in Bytes: 13233
Referer: –
Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.0)
|
|
|/sitemap.html
Http Code: 200 Date: Sep 13 01:26:02 Http Version: HTTP/1.1 Size in Bytes: 9842
Referer: –
Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.0)
|
|
|/poker-reviews/doyles.html
Http Code: 200 Date: Sep 13 01:26:03 Http Version: HTTP/1.1 Size in Bytes: 6255
Referer: –
Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.0)
|
|
|/poker-reviews/hollywood.html
Http Code: 200 Date: Sep 13 01:26:04 Http Version: HTTP/1.1 Size in Bytes: 5574
Referer: –
Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.0)
|
|
|/poker-reviews/absolute.html
Http Code: 200 Date: Sep 13 01:26:05 Http Version: HTTP/1.1 Size in Bytes: 5318
Referer: –
Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.0)
|
|
| -
AuthorPosts