Web Scraping with Python

Image for post
Image for post
Photo by Markus Spiske on Unsplash

What is web scraping?

Why scrape the web?

Step 1: Inspect the Website

Image for post
Image for post
How to Go to Developer Tools on Google Chrome
Image for post
Image for post
Web Inspector on Google Chrome

Step 2: Scrape the Website using Selenium

pip install selenium
from selenium import webdriver as wddriver = wd.Chrome(‘path to your webdriver’)link = “the website link”driver.get(link)content = driver.page_source

Step 3: Parse the HTML code using BeautifulSoup4

pip install beautifulsoup4
from bs4 import BeautifulSoupsoup = BeautifulSoup(content)example = soup.find(id = ‘someHTML-ID’)
x = soup.findall(‘div’, attrs={‘class’:’someHTML-Class’})info_data = []for i in x:info1 = x.find(‘span’, attrs={‘class’:’someClass1'}))info2 = x.find(‘div’, attrs={‘class’:’someClass2'}))info3 = x.find(‘span’, attrs={‘class’:’someClass3'}))info_data.apppend([info1, info2, info3])

〖A data geek 📊〗〖Life-long learner〗〖ESFP-T〗〖✨ŸØⱠØ✨〗

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store