Web Scraping Basics with Python

Web scraping means using a program to read a web page and pull out the data you want. In Python, the classic combo is requests (to download the page) and BeautifulSoup (to parse the HTML). Here's the whole idea in a few lines:

import requests
from bs4 import BeautifulSoup

# 1. Download the page
response = requests.get("https://example.com")

# 2. Parse the HTML
soup = BeautifulSoup(response.text, "html.parser")

# 3. Extract what you want — here, the page title
print(soup.title.text)     # "Example Domain"

Before anything else: scrape responsibly. Let's cover the basics and the etiquette.

Setting up

These libraries aren't part of the standard library, so install them with pip first:

pip install requests beautifulsoup4

requests fetches the raw HTML; beautifulsoup4 turns that HTML into something you can search.

Step 1: Fetch the page

requests.get() downloads a page. Always check the status code — 200 means success:

import requests

url = "https://example.com"
response = requests.get(url)

# 200 = OK, 404 = not found, etc.
if response.status_code == 200:
    print("Page downloaded successfully")
    html = response.text     # the raw HTML as a string
else:
    print(f"Failed with status {response.status_code}")

Step 2: Parse the HTML

Feed the HTML to BeautifulSoup so you can search it by tag, class, or id:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")

# Find the first <h1> tag
heading = soup.find("h1")
print(heading.text)

# Find ALL <a> (link) tags
links = soup.find_all("a")
for link in links:
    print(link.get("href"))   # the URL each link points to

Step 3: Target specific elements

Real pages have many elements, so you target them by class or id:

from bs4 import BeautifulSoup

Imagine this is a snippet from a page

html = """

<div class="product"> <h2 class="name">Python Book</h2> <span class="price">499</span> </div> """

soup = BeautifulSoup(html, "html.parser")

Find by class name

name = soup.find("h2", class_="name").text price = soup.find("span", class_="price").text

print(f"{name} costs {price}") # Python Book costs 499

You can also use CSS selectors with `soup.select(".product .price")`, which many find more intuitive.

## A complete example

```python
import requests
from bs4 import BeautifulSoup

# A practice site made for scraping
url = "https://quotes.toscrape.com"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

# Each quote sits in a <span class="text">
quotes = soup.find_all("span", class_="text")

for i, quote in enumerate(quotes, start=1):
    print(f"{i}. {quote.text}")

Sites like quotes.toscrape.com and books.toscrape.com exist specifically for practice — use them while learning instead of hammering real websites.

Scrape responsibly

Web scraping comes with responsibilities:

• Check robots.txt (e.g. example.com/robots.txt) — it states what the site allows bots to access.
• Read the site's terms of service. Some sites forbid scraping.
• Don't overload servers. Add a small delay (time.sleep(1)) between requests so you don't flood the site.
• Prefer official APIs when a site offers one — they're more reliable and explicitly permitted.
• Identify yourself with a clear User-Agent header when appropriate.

import time
import requests

# Polite scraping: pause between requests
for page in range(1, 4):
    response = requests.get(f"https://quotes.toscrape.com/page/{page}/")
    # ... process the page ...
    time.sleep(1)    # wait 1 second to be gentle on the server

Common mistakes

• Not checking the status code: If you skip response.status_code, you might parse an error page and get confusing results.
• Forgetting to install BeautifulSoup correctly: The pip package is beautifulsoup4, but you import it as from bs4 import BeautifulSoup. The names differ.
• Assuming the HTML structure never changes: Websites update their layouts. A scraper that worked yesterday can break tomorrow — expect to maintain it.
• Scraping JavaScript-rendered pages: requests only sees the initial HTML. If content loads via JavaScript, you may need tools like Selenium or Playwright instead.
• Ignoring legality and etiquette: Scraping without checking robots.txt and terms of service can violate a site's rules. Always scrape ethically.

FAQ

Is web scraping legal? It depends on the site and your jurisdiction. Always check the site's terms of service and robots.txt, and prefer official APIs. Scrape public data respectfully.

What if the data loads with JavaScript? requests won't see it. Use a browser-automation tool like Selenium or Playwright that runs the page's JavaScript first.

How do I store the scraped data? Write it to a file (CSV or JSON) using Python's file handling, or save it to a database as your projects grow.

---

Web scraping combines Modules, Packages & pip for the libraries and Regular Expressions in Python for cleaning extracted text. More projects await on the Python learning hub.

Want to learn this properly? Join the waitlist for our Python course — taught in Jalgaon, beginner-friendly.