Web Crawler API
Turn any page, PDF, DOCX, or authenticated member portal into clean markdown - on a queue, with webhooks.
What this solves
Pulling content out of the web for downstream use (search, RAG, digests, archiving) usually means hand-rolling a fetcher, a rate limiter, an HTML-to-text pass, and a job runner so it doesn't block your request thread. This API is that pipeline as a service: submit a URL, a whole site, a sitemap, or an authenticated member portal, and get back AI-converted markdown once it's ready - polled or delivered via webhook.
Crawl modes
- Single URLOne page → markdown
- WebsiteRecursive crawl, depth + page limits
- SitemapFans out one job per URL
- Member LoungeAuthenticated events, resources, discussions
- CSAESame, for CSAE-hosted portals
Live status
Checking…
–Waiting
–Active
–Completed
–Failed
–Delayed