A crawler, also known as a bot or a spider, is a program that is utilised by search engines to search the internet for information.

The content of a website is then crawled when the bot searches a website and its linked pages.

The more fresh content, ideally exclusively unique content, the crawler discovers, the higher the Google ranking the website achieves.

Glossary index