Market research agents for retail banking
An LLM crawler reads product and rate pages across 20+ bank websites, extracts structured data, and checks its own work against screenshots before anything reaches an analyst.
- Result
- 99.3%
extraction accuracy across 20+ bank websites - Role
- Designed and built the crawler and validation pipeline
- Domain
- Enterprise bank
- Period
- 2025
Problem
Analysts compared competitors' products and rates by visiting every bank's website by hand. A full pass took about two weeks, and the sites changed layout often enough that scrapers kept breaking.
Approach
Let a language model read pages the way an analyst does, instead of hard-coding selectors, and never trust a single extraction: validate every value against what the page actually shows.
What I built
- A crawler (Crawl4AI and Playwright) that navigates each bank's product pages and hands rendered content to an LLM extraction step with a fixed output schema.
- A validation pass that screenshots the source page and asks a vision model to confirm each extracted value, flagging disagreements for review.
- Normalized output for analysts, deployed on Azure.
Outcome
- 99.3% extraction accuracy across 20+ bank websites.
- A market analysis that took two weeks now runs in under an hour.