Authenticated crawling for operator-led application reconnaissance.
A single-job FastAPI service that runs pre-crawl authentication when needed, runs Katana directly, and turns its JSONL artifact into a sitemap your team can inspect or feed into the next testing step.
Katana handles repeatable crawling. AI is used only for authentication and ambiguous browser workflows.
Install dependencies, run the API, and create a crawl job. The service drains one job at a time and persists each completed sitemap atomically.
# Install runtime dependencies uv sync --extra test # Run the API uv run tenzai-crawler-server # Create a crawl job uv run tenzai-crawler create https://example.com # Check status and read the completed sitemap uv run tenzai-crawler status <job_id>
The orchestrator owns job state. Authentication is optional, crawling stays auth-agnostic, and standard Katana, shared-Chrome pure-headless Katana, and browser-guided discovery contribute evidence to one persisted result.
Tenzai Crawler is focused on repeatable crawl capture for trusted operators, not broad public exposure.
Use manual headers when the operator already has a session, or run the auth agent when credentials or a login URL are supplied.
Scope rules bound target domains while built-in, operator, and auth-derived Katana exclusions protect destructive or session-ending paths. Fixture ledgers expose traffic outside that Katana boundary.
Job state lands in SQLite, request logs stay on disk, and completed or checkpointed cancelled jobs expose a deduplicated sitemap through the API.