Cybersecurity News Scraper

The problem

Instead of visiting several sites every morning for cybersecurity news, I wanted one place that collected their main headlines, descriptions, and links.

I tried Scrapy and Beautiful Soup before choosing requests and lxml for this version. Pandas handles cleanup, combines the results, and exports the finished report to CSV.

The approach

The script requests each page, parses its HTML, and uses site-specific XPath expressions to select the title, description, and article link. It strips extra characters, removes duplicate links, and stores each site's results in a pandas DataFrame.

A label row identifies the source before the individual DataFrames are combined into the final Summary.csv file.

Modules and source list

Pandas handles tabular cleanup and export, requests retrieves each page, and lxml parses the HTML. After testing Scrapy and Beautiful Soup, I chose lxml because it fit this small, XPath-driven project well.

The URLs are kept in one list so the main loop can process each source in turn.

import pandas as pd
import requests
import lxml.html

url = [
    "https://cyware.com/cyber-security-news-articles",
    "https://threatpost.com/",
    "https://thehackernews.com/",
    "https://www.securitymagazine.com/topics/2236-cyber-security-news",
    "https://www.bobbythings.com",
]

Request, parse, and select

For each URL, the script downloads the page and converts the response into an lxml document. Because every publisher structures its HTML differently, each source needs its own XPath selectors for titles, descriptions, and links.

for p in url:
    page = requests.get(p)
    doc = lxml.html.fromstring(page.content)

    if p == "https://cyware.com/cyber-security-news-articles":
        title = doc.xpath(
            '//h1[@class="cy-card__title m-0 cursor-pointer pb-3"]/text()'
        )
        descrip = doc.xpath('//div[@class="cy-card__description"]/text()')
        links = doc.xpath('//div[@class="cy-panel__body"]//a/@href')

Clean and structure the results

The selected strings are trimmed and split, duplicate links are removed, and alert URLs are filtered out. The cleaned values are stored in a DataFrame, converted back from nested lists into readable strings, and prefixed with a row naming the source.

titlesplit = [item.lstrip().rstrip().split(",") for item in title]
descripsplit = [item.lstrip().rstrip().split(",") for item in descrip]

nodupelink = list(set(links))
linkclean = [item for item in nodupelink if "alert" not in item]
linkssplit = [item.split(",") for item in linkclean]

df = pd.DataFrame({
    "Title": titlesplit,
    "Description": descripsplit,
    "Link": linkssplit,
})

df["Title"] = df["Title"].str.join(", ")
df["Description"] = df["Description"].str.join(", ")
df["Link"] = df["Link"].str.join(", ")

site_row = pd.DataFrame({
    "Title": "Cyware",
    "Description": " ",
    "Link": " ",
}, index=[0])

df = pd.concat([site_row, df]).reset_index(drop=True)

Handle other sources and export

The remaining publishers follow the same pattern with selectors tailored to their markup. If a URL has no matching branch, the script reports it. Finally, the source DataFrames are combined and written to Summary.csv.

elif p == "https://threatpost.com/":
    title2 = doc.xpath(
        '//div[@class="c-border-layout"]//h2[@class="c-card__title"]//a/text()'
    )
    descrip2 = doc.xpath('//div[@class="c-border-layout"]//p/text()')
    links2 = doc.xpath(
        '//div[@class="c-border-layout"]//h2[@class="c-card__title"]//a/@href'
    )

    df2 = pd.DataFrame({
        "Title": title2,
        "Description": descrip2,
        "Link": links2,
    })
else:
    print(f"Something went wrong with ${p}")

finaldf = pd.concat([df, df2, df3, df4])
finaldf.to_csv("Summary.csv", index=False, header=True)

What I’d improve next

• Add better error handling for unavailable pages and unexpected responses.
• Highlight stories containing specific keywords.
• Schedule the script and deliver the report automatically each morning.
• Move to Scrapy if the list of sources grows significantly.
• Make selectors more resilient because a site's HTML changes can break an XPath.