Beautifulsoup Web Scraping

Web scraping deals with extracting or scraping the information from the website. Web scraping is also sometimes referred to as web harvesting or web data extraction. Copying text from a website and pasting it to your local system is also web scraping. However, it is a manual task. In this Web Scraping tutorial: Grasp Python Web Scraping fundamentals. Use BeautifulSoup & Requests to scrape & crawl Craigslist with Python. Learn how to save your scraped output to a CSV / Excel file. Understand the difference between Web Scraping Python libraries and frameworks. So let's start your Web Scraping real-life project.

APIs are not always available. Sometimes you have to scrape data from a webpage yourself. Luckily the modules Pandas and Beautifulsoup can help!

Related Course:Complete Python Programming Course & Exercises

Web scraping

Pandas has a neat concept known as a DataFrame. A DataFrame can hold data and be easily manipulated. We can combine Pandas with Beautifulsoup to quickly get data from a webpage.

If you find a table on the web like this:

We can convert it to JSON with:

And in a browser get the beautiful json output:

Converting to lists

Rows can be converted to Python lists.
We can convert it to a dataframe using just a few lines:

Pretty print pandas dataframe

You can convert it to an ascii table with the module tabulate.
This code will instantly convert the table on the web to an ascii table:
This will show in the terminal as: