Web Scraping with Python: Successfully scrape data from any website with the power of Python

by Richard Lawson

Successfully scrape data from any website with the power of Python This book is aimed at developers who want to use web scraping for legitimate purposes. Prior programming experience with Python would be useful but not essential. Anyone with general knowledge of programming languages should be able to pick up the book and understand the principals involved. The Internet contains the most useful set of data ever assembled, largely publicly accessible for free. However, this data is not easily reusable. It is embedded within the structure and style of websites and needs to be carefully extracted to be useful. Web scraping is becoming increasingly useful as a means to easily gather and make sense of the plethora of information available online. Using a simple language like Python, you can crawl the information out of complex websites using simple programming. This book is the ultimate guide to using Python to scrape data from websites. In the early chapters it covers how to extract data from static web pages and how to use caching to manage the load on servers. After the basics we'll get our hands dirty with building a more sophisticated crawler with threads and more advanced topics. Learn step-by-step how to use Ajax URLs, employ the Firebug extension for monitoring, and indirectly scrape data. Discover more scraping nitty-gritties such as using the browser renderer, managing cookies, how to submit forms to extract data from complex websites protected by CAPTCHA, and so on. The book wraps up with how to create high-level scrapers with Scrapy libraries and implement what has been learned to real websites. This book is a hands-on guide with real-life examples and solutions starting simple and then progressively becoming more complex. Each chapter in this book introduces a problem and then provides one or more possible solutions.

Genres: Programming

151 pages, Paperback

First published August 3, 2015

Reviews

[Emilia Greendevald · Rating 4 out of 5]

This book is a good starting point for beginners. However, your overall approach will always vary depending on your business's focus. Competitive intelligence has become a key asset in the telecom industry. Telecommunications scraping allows companies to track product launches, new tariffs, and promotional campaigns in near real time. This data can feed analytics dashboards and forecasting models. Strategy teams can then identify emerging trends before they become mainstream. For companies operating in saturated telecom markets, this insight can be extremely valuable.