Crawlers (or bots) are used to assemble information available on the web. By using web site navigation menus, and reading inside and external links, the bots start to grasp the context of a web page. Of course, the words, pictures, and different information on pages additionally help search engines like google and yahoo understand what they’re ab