Crawler

A search engine program that systematically browses or "crawls" through website content and follows links to other pages in an ongoing effort to accurately index pages.