Search engines like google, yahoo, msn etc help people find relevant information on the Internet. Major search engines maintain huge databases of web sites that users can search by typing in some text.
To compile their databases, search engines rely on computer programs called spiders or robots. These programs crawl across the web by following links from site to site and indexing each site they visit. Each search engine uses its own set of criteria to decide what to include in its database.
Also search engines uses unique criteria to organize information for its users.There are many other ways to organize results, and most search engines use a combination of several of them.
There are some Factors that are very obvious and others that are less obvious. Furthermore, each search engine weights each of these factors differently, and places the special attention in different spots.

On page factors include :

content, placement of the title tag
Keyword frequency, weight, Keyword prominence and proximity
Meta description tag
ALT tags
Comment tags
Keywords in URL names
Alphabetical placement

Off page factors include :

Link popularity
Click popularity
Themes
Overall site design

Search Engines Look in a Web Page

Title tag - You need a relevant title, Use it for 5 key words. Title Tag gives the overview or theme of your site. This can increase the Ranking of the Site.
Meta description tag - Most engines look at this tag. Use distinct ones throughout your site, and distinct ones for each page. Make them particular to that page. Every web site in existence is built on what is called HTML code. Within a HTML based web page is the ability to use what are called Meta tags.
Keyword - Some engines use them directly, some check them as part of a validation process.
Heading - The search engines view < h> tags as they give weight to the words within them.
Bold - Lesser importance than < h> tags. This can be helpful to Search Engine for better search Results.
Alt text - Use descriptive short sentences in your alt tags.
Comment tags - Some engines use comment tags for content. So in Building the Web Site this point should be kept in mind.
Traffic - The search engines do keep track of how many people follow their links.
Link Popularity - How many other web pages around the Internet point to your web site, Are they considered valuable resources ?
Keyword frequency – Keyword across all pages. Does the content really talk to the subject. So this point is also important in building Web Site.
Robot.txt - Crawler first search for Robot.txt file in root folder. It is file in which instruction for robots are given.

How Does A Search Engine Works ?

>> The search Engines like Google, Yahoo, and AltaVista have programmed to build their own Search Engine Spiders or crawlers, which are released on the WWW [WORLD WIDE WEB]. Once they are on the web they visit all the Web Sites existing on the web. These spiders visit the websites submitted to them through the Submit URL form.
>> The search engine spider will visit the site immediately and schedule the site for inclusion in the search engine’s index.
>> Within a few weeks, the engine will place the site in their index.
>> The spider will revisit the site, to include any changes. Once the site is included, the spider will usually revisit every two weeks.
>> When someone uses a search engine, they type “keywords” into the search box. They are submitting a query to a search engine.
>> Then Search Engine Looks into the Index and provide back the result of Query to the user in the form of Lists of the Sites with that particular Keyword.

Search Engine Crawlers

GoogleCrawler

FRESH GOGGLE BOT: Crawl only index page of site and go back. It doesn’t check other pages of web site. if Page has PR>4 then fresh bot comes every 3rd or 4th day. Collect data in BYTES
DEEP GOGGLE BOT: It comes on the basis of fresh goggle bot. when the updation are made in web site then deep bot crawl each page of web site. It comes after every 20 days or monthly basis. Collect data in Kilobytes
IMAGE GOGGLE BOT: Read images on web sites and maintain database. It reads filename of image and ALT tag. It comes on monthly basis.

Other Crawlers Name

Google : Google Bot, Google Image Bot
Yahoo : Yahoo IM Crawler, Yahoo Seek
Netscape : Mozilla Compliable Agent
MSN : MSN Bot
Alexa : Ia_Archiver