ScraperAI — LLM Web Extraction Platform on .NET 10

What it does

Users connect their own AI provider, write a prompt that says what to read and what to extract, and test it in the app. A prompt's URL can take {{parameters}}, so one prompt covers a whole family of pages, such as every vehicle on a dealer's site.

Each run fetches the page by direct request or headless browser, sends the content to the chosen model, and returns structured JSON, XML, CSV, or Markdown. Every job records the parameters, timings, tokens, and cost.

Once a prompt works, any program can call it by ID through the REST API, wait for the result or poll for it, and receive a signed webhook when the job finishes. Claude Code can call it too, through MCP.

Live updates with SignalR

Jobs start from many places: the web app, the REST API, or Claude Code through MCP. A SignalR hub keeps every open page in step with all of them. Each signed-in user joins their own group and hears only about their own jobs: a new job appears in the list the moment it's created, each step of a run shows as it's recorded, and ratings, deletes, and usage counts refresh without a reload.

Messages carry only what changed; the page then reloads that job through the normal API, so access rules live in one place. Polling is kept only as a fallback while the live connection is down.

The hub is locked down like the rest of the API: it requires a signed-in, active user, closes connections when the session expires, refuses requests whose Origin names another site (WebSockets can't carry the CSRF header), and has its own rate-limit bucket so reconnects don't use up the app's request budget.

Screenshots

Stage 1

Dashboard & AI Providers

Each account sees its recent jobs and usage at a glance. Users connect their own AI provider (OpenRouter, Anthropic, OpenAI, Gemini and others). Keys are stored encrypted and never shown again.

⤢ ⤢

Stage 2

Prompts with Parameters

A prompt holds the target URL, the extraction instructions, the model, the return type (JSON, XML, CSV, Markdown, or text), and the fetch method (static or headless browser). {{name}} placeholders let one prompt cover a whole family of pages. The test run shows the exact URL it will fetch.

⤢ ⤢ ⤢

Stage 3

Jobs & Results

Every run becomes a job you can filter, follow live over SignalR, and run again. Job pages show each step with its timing, the exact prompt sent and the model's reply, the extracted result, the page content the model read, and the run details: parameters, tokens in and out, and cost.

⤢ ⤢ ⤢ ⤢

Stage 4

API, Webhooks & Integrations

Prompts run from any code through a REST API authenticated with per-app API keys. Callers can wait for the result or queue and poll, and signed webhooks report finished jobs. The docs generate ready-to-run cURL, Python, Node.js, .NET, and PowerShell calls, and an MCP server lets Claude Code run prompts directly.

⤢ ⤢ ⤢ ⤢

Stack

.NET 10 and ASP.NET Core back end, React front end, Playwright for headless crawling, and SignalR for live job updates. Bring-your-own-key AI providers (OpenRouter, Anthropic, OpenAI, Gemini, and others) with encrypted key storage. Public REST API with API keys, signed webhooks, and an MCP server for Claude Code. Accounts use two-factor authentication.

The same platform later became the server for colorFluff, with the crawl pipeline swapped for an AI image pipeline.

Live demo: scraperai.fly.dev