Parsing with Python. Web Scraping in Action

Parsing with Python. Web Scraping in Action If programming is magic, then web scraping is true sorcery. By writing a simple automated program, you can send requests to web servers, receive responses with data, and then analyze this data to... extract the necessary information. This updated third edition of the book will not only introduce you to web scraping but also serve as a comprehensive guide to collecting virtually any type of data on the modern Internet. Part I focuses on the mechanics of web scraping: how to send requests to web servers with Python, process responses, and automate interactions with websites. Part II explores more specific tools and applications that will be useful in any web scraping scenario. The book will show you how to: • analyze complex HTML pages; • develop web crawlers using the Scrapy framework; • store data obtained through scraping; • read and extract data from documents; • clean and normalize poorly formatted data; • read and write information in natural languages; • perform searches on forms and login pages; • scrape JavaScript code and work with APIs; • write and use programs to convert images to text; • bypass anti-scraping traps and bot blockers; • test your websites with scraping.