Sales databases tell you a company exists, rarely why it deserves a call right now. So I built a system that collects public traces (WTTJ, LinkedIn, Trustpilot, websites), qualifies them with an LLM and turns them into context SDRs can use. This page links the four technical notes it is made of.
Related Technical Notes
- Detecting Web Technologies with Scrapy
A stack detection engine built from HTML, scripts, cookies, headers and signatures.
→ Read the note - The SDR Lead Machine
Company database, Hunter.io, pain points, customer references, n8n, Microsoft Lists and Power BI to prepare prospecting.
→ Read the note - Multi-Source Competitive Monitoring
Trustpilot, LinkedIn, websites, Supabase and an LLM to extract competitive signals.
→ Read the note - Expansion Signals with WTTJ, LinkedIn and n8n
WTTJ via Algolia, LinkedIn, news pages and Redis to spot growth indicators.
→ Read the note
The problem
Salespeople rarely lack data: name, industry, size, contacts, supposed technologies, funding rounds. What they lack is what has changed. A database describes a company; a business signal describes a movement: a wave of hiring on WTTJ, LinkedIn posts about a new market, recurring negative Trustpilot reviews, a marketing stack change.
Before this system, I spent my time opening websites, reading source code, browsing job ads and cross-checking tabs to come up with a credible message. Each task was simple; the whole was slow and impossible to industrialize.
Existing tools (B2B databases, intent data, sales intelligence platforms) did not fit: static data, opaque scoring, high cost. I wanted to change sources, rules, prompts and thresholds freely, and above all to see the reasoning.
A signal is not the truth
A job ad does not prove hypergrowth, a negative review does not prove a competitor is struggling, a script in the HTML does not prove a technology is actually used. The system formulates hypotheses, not certainties.
What I built
The principle: collect, structure, and only then call the AI.
flowchart LR
A[Public sources] --> B[Collection]
B --> C[Cleaning and extraction]
C --> D[AI qualification]
D --> E[Explainable scoring]
E --> F[(Supabase)]
F --> G[SDR interface]
The stack: n8n for orchestration, Scrapy (with Playwright as a fallback) for crawling, Supabase to store companies and signals over time, OpenRouter with gpt-4o-mini for qualification, Apify for LinkedIn, Hunter.io for contacts, Redis to cap concurrency on APIs, Microsoft Lists and Power BI on the team side.
The building blocks, in order:
- Technical signals. Scrapy collects HTML, scripts, cookies and headers; a signature engine produces a confidence score per technology and keeps its evidence. Details: web technology detector with Scrapy.
- Enrichment and personalization. Hunter.io provides contacts. The LLM never starts from a blank page: it gets the detected technologies, pain points prepared per competing technology and customer references from the same industry. I do not ask it to “write an email” but to “suggest a careful, verifiable angle for an SDR”. Details: the SDR lead machine.
- Expansion signals. For WTTJ, querying the Algolia index the site itself calls (public search-only key) is far more stable than parsing HTML (under an hour of work, versus several days of parsing). The trap: without the right company slug, the API returns zero results with no error, and the signal silently disappears. Details: expansion signals with WTTJ, LinkedIn and n8n.
- Competitive monitoring. A single review is anecdotal; when Trustpilot, LinkedIn and the website tell the same story, the signal becomes solid. Details: multi-source competitive monitoring.
- SDR output. Microsoft Lists with simple statuses, Teams notifications, tracking in Power BI. Not glamorous, but understood by a team already on Microsoft.
Scoring is deliberately readable: points by source, freshness (under 30 days), multiple sources and ICP fit, capped at 100. It is there to sort, not to predict.
The numbers
After several weeks, signals came 35% from LinkedIn, 25% from WTTJ, 20% from websites and news, 12% from Trustpilot and 8% from technical detection. Estimated relevance is almost the reverse:
| Source | Estimated relevance | Comment |
|---|---|---|
| Technical detection | ~70% | Precise if signatures are good |
| WTTJ | ~65% | Reliable, depends on slug and deduplication |
| Trustpilot | ~55% | Biased toward extreme experiences |
| Websites / news | ~45% | Heterogeneous, blogs often abandoned |
| ~25% | Lots of generic employer branding |
What got in the way
- Web data is dirty. Redirects, expired certificates, JavaScript rendering, undated blogs, slugs that look nothing like the company name. The cost is not the first crawler, it is handling the exceptions.
- Calling the AI too early. Sending raw content to the LLM produced long answers, hallucinations and JSON that was hard to parse. Cleaning, extracting the useful fields, then enforcing strict JSON stabilized the output.
- Duplicates. The same post fetched twice, a reposted WTTJ job, a page crawled on several dates. A hash of
company_id:source:title:published_atsolved it. - Volume. 200 weak signals a week is unusable. The right metric is the number of actionable signals, not the number detected.
- n8n getting bloated. Very fast for prototyping, but once business logic grows you need sub-workflows, naming, logs and normalized outputs.
If I started again, I would version prompts from day one (a prompt_version column on every signal), build a set of business test cases (“generic employer branding post → not relevant”) and move business logic out of n8n sooner.
Takeaways
A useful RevOps system is not the one that automates the most, it is the one that makes the right context available at the right moment. AI is there to classify and summarize already-structured data; if a simple rule is enough, I keep the rule. And the decision stays human: the SDR sees the source, can invalidate the signal and can choose not to reach out.