|
The Relationship Between the Search
Engines, and Ways to Get Listed When all Your Past Attempts
Have Failed.
Identifying Some of the Spiders and Bots - Knowing
if your Page was Visited?
That is a difficult thing to determine. Many search engines
allow you to query their search results with your URL. You
will need to look around at their information to learn how.
Most of them have a page for webmasters to answer such question.
Another way to know is by looking at your access logs. If
you are on a free host, then chances are that you will not
have access to logs. If you pay for your web hosting, like
Sanders Consultation Group Plus, then your hosting company
should provide you access to your logs. We download ours on
a daily basis to see where people are coming from and who
referred them. If you have a free host, then there are free
services out there that will help track this information for
you. Below are a few of them for you to choose from.
Add
Free Stats
Counted
FreeStats
MasterStats
Sitetracker
When you look at your logs, if you see something out of place,
then chances are that it was probably a spider or bot of some
kind that visited your page or site. Most of them do not show
a URL. They show names instead. For instance, if you find
a reference in your logs to Scrubby/2.1, then chances are
that your page or site has been spidered by Scrub
The Web. If you find reference to Googlebot,
I am sure you can figure that one out on your own. A simple
solution to find out is to visit your favorite search engine
and do a search on the abnormal listing. Remember that abnormal
listings are those that will usually refer to a name, and
usually do not have a full URL included. Others we have found
in our logs include Gigabot (Gigablasts indexing agent), Marvin
v0.3 (http://www.hon.ch/MedHunt/Marvin.html
and I have NO idea why he visited my site), and Claymont.com
(Claymonts indexing agent). To see our list of present user
agents (spiders and bots) you can check out our user
agent page. Special thanks to Gary
Keith for his continued dedication to this project, and
without him, we would not be able to make this page available.
If you prefer not to seach for the obscure note in your logs,
you can just drop by Gary's site instead. Should you find
something that isn't indexed in his database, please do us
a favor and submit
it to Gary. He has made it a life's work to track these
critters, and your help will allow him to identify them for
all of us. Thanks again Gary.
Search engine companies are competitive, but each of them
started out as small kids on the block. When they start out,
it's hard to get the information they need to supplement their
index. Adding sites is a time involved process, especially
if they do not have their own spiders and bots to automate
the process. Algorithms and filters need written to make the
results relevant. Some of them use other engines to supplement
their indexes. What that means to site owners is that by being
included in one search engines index you could show up in
many others indexes. This is a back door approach to search
engine positioning. It is a way to get inclusion to search
engines you have not been able to get results with. By paying
close attention to the submission requirements of an easier
search engines submission policy, you can ensure your inclusion
with them and possibly gain inclusion in a more popular search
engine. To take advantage of this, you need to know how the
search engines are related, and which search engine to concentrate
on to use as a stepping stone to the next.
Primary, Secondary, Directory, and Paid Results - More
Terminology to Understand the Relationships Between the Search
Engines
There are different types of results to take into consideration.
The list includes primary, secondary, directory, and paid
results. Each search engines has a primary and secondary results
list. The primary results lists include pages they feel are
most relevant for the searched keywords. The secondary results
are those which might still apply to the searched keywords,
but the relevancy does not rank as high as the primary results.
This can be compared to the first string and second string
on a football team. The first string is where a team begins
the game. They put out their best players because they want
to win. The second string players are usually then substituted
in the event that a first string player is not available.
This is the same with search engines primary and secondary
results.
Directory results are classified separately because there
are search engines, and there are directories. DMOZ isn't
really a search engine, they prefer to be called a directory.
It is a human project initiative, and you will find notes
to this throughout their site. DMOZ is the human factor in
the search engine game. Each of their indexed pages are visited
by a human being (editor), and then the editor assigns the
description of each page. There is no automation in this process.
That is why it is so hard to get high rankings in DMOZ. Unless
the editor considers your page great and the content excellent,
then you will have no problems with your page.
Lastly there are paid results. These results are for pages
which have been submitted through the guaranteed submission
process. Remember before I said there was only one sure way
of guaranteeing inclusion. The page owners have paid to have
these pages listed. The search engines also swap these pages
back and forth.
The Relationships Between the Search
Engines - How Some of Them Work Together to Provide Their
Visitors Search Results
The major suppliers behind the scenes include DMOZ, fast,
Google, Inktomi, and Overture . In the overall relationship,
DMOZ is the backbone to most of the prominent search engines.
Overture only provides paid inclusion results to the search
engines they support. DMOZ fast, and Inktomi are the most
insulated, they receive no kind of results from anyone. Strictly
stated, these three are suppliers. Bottom tier receivers include
AOL, Ask Jeeves, HotBot, Netscape, Lycos, iwon-search, Yahoo,
alltheweb, MSN, and Alta Vista. These search engines receive
results from other search engines. Some of them allow you
to submit your page to them, and others do not. If they do
not allow submission, then you need indexed by a search engine
that supplies them with results.
Search engines who may have decent traffic but supply no
results to anyone else include Alta Vista, MSN, alltheweb,
Lycos, iwon-Search, Netscape, HotBot, AOL Search, and Ask
Jeeves. These are what I will call dead end search engines.
Get indexed with these and enjoy the traffic from their visitors,
but they do nothing to help you get listed with any of the
others. When I look at places I want to get a site indexed
at, I do searches to see how relevant their search results
are. The ones with the highest search relevance results are
the places that I target first to get indexed with. The places
with the best search result relevance are going to be the
place people use the most to find what they are searching
for.
You should keep something else in mind before reading further
into this article. If you are sent to another search engine
results as the efforts of being indexed, that does not mean
that you will be ranked the same way in their results. For
instance, say that DMOZ indexes you as a primary result and
send you to Google. That does not mean that Google is going
to send you to Netscape as a primary result. Although presently
Google has no secondary results agreement with Netscape, it
could mean that you would be sent to one of their "partners"
as a secondary result. The same holds true for any other combination.
Back
to Table of Contents
Paid
Inclusion (Article Continues)
About the Author
James R. Sanders is the owner of Sanders
Consultation Group Plus. He has been a webmaster and website
designer since 1997. He has also been involved in self employment
ventures since 1992. He is presently a contributing author
of NewbieHangout. His writing is targeted to webmasters, would
be webmasters, website designers, would be website designers,
self employed, or those researching information looking for
solutions to questions associated with design, business operations,
and promotion today. His goal is to provide practical information
based upon his years of experience to help webmasters, website
designers, and self employed people achieve their goals in
today's competitive global market. You can subscribe to his
free newsletters at SCGP
- Newsletter.
|
|
