Web crawlers are computer programmes. They are also called bots. These index internet content and information. They crawl through websites and search engines. They download and catalogue information so obtained. When users make a search enquiry, the information is retrieved and reviewed by the crawlers. They also validate HTML code and hyper-links of websites they crawl.

Also called spiders or spiderbots, they are operated by search engines for the purpose of web indexing.

Working

First they download website’s robot.txt.file. They find new pages via hyperlinks. They add newly discovered URLs. They index every page at that URL.

Web scraping extracts data from one or more sites. In crawling, URLs are found, and links are found. If web is to be extracted, crawling and scraping can be combined. Crawling is for indexing purposes.

Tools for Web Scraping

Beautiful soap is a Python library for web scraping. It pulls data from HTML/XML files. Scrapy is web crawling through Python. Selenium is browser automation tool for testing web apps. Pandas is a Python library for data manipulation and analysis. Octopause is a modern visual web data extraction software. It can extract data from any website.

Challenges in Scraping

There are changes in structure or layout. Websites take antiscraping measures. The data quality of extracted data varies. Scraping can be illegal. It is technically challenging when one deals with huge data or complex websites.