Get exclusive CAP network offers from top brands

View CAP Offers

Slurp crawling but not indexing

[bsa_pro_ad_space id=2]
  • This topic is empty.
Viewing 11 posts - 1 through 11 (of 11 total)
  • Author
    Posts
  • #588101
    Anonymous
    Inactive

    Slurp is hitting my site on a daily basis and obviously crawling all my pages (i see the requests in logs) but it has only indexed my index and a couple of irrelevant pages.

    Any idea as to why this would be happening?

    Thanks! :cheers:

    #662904
    vladcizsol
    Member

    Its just a theory at this point, but I believe most if not all relevant pages are being targeted and in short order banned for duplicate content.

    See:

    http://www.casinoaffiliateprograms.com/bb/showthread.php?t=5378

    This could explain why undesirable terms are still easily obtained.

    #662907
    Anonymous
    Inactive

    How can you tell,by looking at your stats, if your site has been visited by a slurp? Is there a distinctive ip address that I should be looking for?

    #662909
    Anonymous
    Inactive

    Awstats lists all the robots like this:

    Inktomi Slurp 933+543 7.55 MB 15 Mar 2005 – 08:04

    The name, how often has it crawled this month, how often did it hit robots.txt files, what bandwidth does it use, when was the last visit.

    Awstats does not count robots as visitors. Some stats do that and it messes things up.

    Awstats is one of the stats programs I use and I find it quite useful.

    #662915
    Anonymous
    Inactive

    Awstats does not count robots as visitors. Some stats do that and it messes things up.

    Very true. That’s why I have very different figures with awstats and webalizer. But it’s a matter of configuration. It’s possible to configure webalizer to not count these robots as visitors.

    #662927
    Anonymous
    Inactive

    Good Morning,

    I must be missing something with Webalizer.

    Some people actually seem to find it useful.

    I have always found it to be a completely useless piece of crap!

    It gives me a bunch of words and terms that were used to find my site, but no indication as to whether those terms were entered at MSN, Google, Yahoo or wherever.

    It’s frustrating, as most hosting plans offer cpanel, which comes with Webalizer pre-installed. .. :rasberry:

    AWStats are a bit better .. but still not terribly comprehensive.

    I recently spent an entire week searching for hosting plans offering decent stats.

    Apollo offers Urchin for a one time installation fee of $35.

    I guess you get what you pay for .. Webalizer is free, so most don’t complain.

    Can anyone enlighten me as to what Webalizer may be good for?

    #662929
    Anonymous
    Inactive

    Hey Professor, thanks for the reply! Isn’t that issue affecting google only? AFAIK it is.

    Gravity, as Professor suggested above, you can use awstats or you can simply look at the raw access logs and search for the string “slurp!” which slurp uses as it’s useragent it hits your site.

    Hope this helps! ;)

    #662932
    Anonymous
    Inactive

    i’m using awstats…came with the hosting plan…and some of the robots are taken out of the stats, but others are not…inktomi slurp is taken out (they last visited this morning at 7:14)…msnbot and become.com visit often and are recorded as unique users, which is anoying…there are several others that aren’t as easy to figure out, but they are reported showing an equal number of page views as hits, which would be next to impossible for a real person to achieve…for instance i had dotnet3.orcsweb.com visit and go to 2 pages but only register 2 hits, so probably some sort of robot…

    #662933
    Anonymous
    Inactive
    LasVegasLady wrote:
    Can anyone enlighten me as to what Webalizer may be good for?

    Webalizer lists referrers even if they didn’t come from any place where they had to click on anything. Like I am here, and I suddenly decide to visit you and type your URL in the browser. Webalizer will still show you where I came from.

    Also, I have a very large site. I am interested in, let’s say, which the most popular game is, or which page is most often the one people leave from, or which one do they have bookmarked and enter through, etc. Webalizer is nice for that because you can configure it.

    I use all of the following, and I find something useful about all of them:

    Analog
    Awstats
    Bandwidth usage
    Error log
    Latest Visitors
    Raw Logs
    Raw Log Manager
    Urchin
    Webalizer

    #662935
    Anonymous
    Inactive

    I use Webalizer and like it as well, although it took a few changes to the default configuration to make it behave. In particular, the defaults have it counting internal pages as referrers, which is pretty useless. A one-line change in the config fixes that. (I can look it up if anyone needs it.)

    Dominique, Webalizer relies on the same weblogs as all the other webstats software you mention. It can’t give you referrer information on type-in traffic, unless the visitor is using a browser that transmits the current webpage as a referrer even if the user typed in a new address rather than clicking a link. I don’t know which browsers do that, but I know some have in the past.

    If that information is present in your server logs, any of the noted stats software can access it.

    I would think most current browsers would definitely not do that, as it’s potentially a security issue. Imagine if you’re visiting some site with a password and username in the URL, then type in another website. The webmaster on the second site could grab your username and password for the first site from his web logs.

    #663013
    vladcizsol
    Member
    Quote:
    Isn’t that issue affecting google only? AFAIK it is.

    It positively IS effecting Yahoo and conversely MSN

Viewing 11 posts - 1 through 11 (of 11 total)