Building a Web Technology Detector with Scrapy

For prospecting, I was opening sites one by one to guess their marketing stack. I built an engine with Scrapy that detects these technologies across thousands of domains, by accumulating signals rather than searching for a word in the HTML.

The problem

Inspect the source code, look at loaded scripts, hunt for marketing tags: I was doing this by hand for every company. Yet a website says a lot:

Detected signal	What it often tells you
Klaviyo	Modern e-commerce stack
HubSpot	Inbound / CRM approach
Segment	Data culture
Heavy GTM usage	Tracking that’s become hard to maintain
No CMP	Low GDPR maturity

Existing tools didn’t fit:

	Tool	Main problem
	BuiltWith	Closed, no clean API, expensive at scale
	Wappalyzer	Browser extension-oriented, poorly suited to automation

Above all, I wanted to understand how detection worked: add my own signatures, analyze false positives, plug everything into other systems. It all started on my Windows PC, with Python, Scrapy and a few JSON files.

From an if to a bundle of signals

The first version fit in two lines:

if "hubspot" in html:
    print("HubSpot detected")

Then came dynamically injected scripts, shared CDNs, technologies visible only through cookies, JS frameworks returning nearly empty HTML. The question became: do several independent elements strongly suggest HubSpot?

flowchart TD
    A[Domain list] --> B[Scrapy spider]
    B --> C[HTML] & D[Scripts] & E[Headers] & F[Cookies] & G[Meta tags]
    C & D & E & F & G --> H[Signature engine]
    H --> I[Scoring]
    I --> J[Detected technologies]
    J --> K[JSON export] & L[Supabase] & M[n8n workflows]

Scrapy detects nothing: it collects signals, the logic lives elsewhere. I chose it because it handles several hundred pages/minute locally without JS rendering, with retries, throttling, concurrency and pipelines built in. Playwright alone was too heavy, Selenium too slow, requests + BeautifulSoup only good for one-off jobs.

Signatures and scoring

A first-version signature was "html": "hubspot": mass false positives, no notion of confidence. Today, each technology combines several types of evidence:

{
  "name": "Klaviyo",
  "html": ["klaviyo"],
  "script": ["static.klaviyo.com"],
  "cookies": ["__kla_id"],
  "confidence": { "html": 0.10, "script": 0.30, "cookie": 0.45 }
}

Scoring adds up the weights, capped at 1.0. A cookie is very reliable (someone had to install it), a CDN script reliable but sometimes shared, a word in the HTML may come from editorial content. I tested more statistical approaches: mostly, they made the system harder to debug. I stuck with readable scoring.

Result on one domain: Shopify at 0.97 (cdn.shopify.com, _shopify_y, shopify-checkout), Klaviyo at 0.91 (static.klaviyo.com, __kla_id). Enough to score a company’s tech maturity, build ICPs or track stack changes.

What broke

Modern JavaScript

On some Next.js sites, Scrapy literally got <div id="__next"></div>: everything else arrived after hydration, and some technologies became invisible. The compromise: a conditional Playwright fallback, triggered only when detection is insufficient.

Mode	Pages/min (approx.)	RAM
Scrapy only	~300-400	Low
Scrapy + Playwright	~20-50	High

The hard part wasn’t the code but deciding when to trigger the fallback, on which domains and with which timeouts. It’s the most important trade-off in the project, and it remains hard to scale.

Dirty data and false positives

Absurd redirects, invalid certificates, timeouts, anti-bot protection, broken HTML, strange encodings: handling edge cases takes more time than the business logic.

After several thousand domains: lots of half-broken sites, far more WordPress than expected, very repetitive stacks by industry, and dozens of dead legacy scripts.

What I would do differently

My signatures ended up as signatures.json, signatures_v2.json, signatures_final_final.json. Today I would organize them by category (/crm/hubspot.json, /marketing/klaviyo.json…) with validation from the start. And I tested by hand on random domains for too long: you need a list of reference domains with a known stack, HTML snapshots and regression tests, because a technology can change CDN or script overnight.

Key takeaways

I thought I was building a technology detector; mostly, I built a web signal collection system, which only sees what leaks on the frontend (backend and infrastructure stay invisible). An understandable, stable pipeline beats a “smart” system that’s impossible to maintain.