Tenzai Crawler docs

Overview

The service accepts crawl jobs through an API, stores job state in SQLite, runs work serially, runs standard and pure-headless Katana against a shared browser profile, performs browser-guided gap discovery, and returns a normalized sitemap when the job completes.

Quickstart

Install dependencies and run the API from the repository root.

# Install dependencies
uv sync --extra test

# Start the API server
uv run tenzai-crawler-server

Create a job from the CLI.

uv run tenzai-crawler create https://example.com
uv run tenzai-crawler list
uv run tenzai-crawler status <job_id>
uv run tenzai-crawler cancel <job_id>

API

The API exposes a small job lifecycle: create, list, inspect, and cancel. The generated FastAPI reference is available at /docs when the service is running.

# Create a crawl job
curl -X POST http://localhost:8000/jobs \
  -H 'Content-Type: application/json' \
  -d '{"target_url":"https://example.com"}'

# Read job status or completed sitemap
curl http://localhost:8000/jobs/<job_id>

Authentication

Authentication is optional. Header-only configuration is manual-header mode and goes directly to the crawl. Credentials or a login URL run the auth agent before crawling.

Manual-header mode

{
  "target_url": "https://example.com",
  "auth_config": {
    "headers": [
      "Authorization: Bearer $TOKEN",
      "Cookie: session=abc"
    ]
  }
}

AI-auth mode

{
  "target_url": "https://example.com",
  "auth_config": {
    "login_url": "https://example.com/login",
    "credentials": {
      "email": "{{env:APP_EMAIL}}",
      "password": "{{env:APP_PASSWORD}}"
    },
    "instructions": "Login and stop once the dashboard is visible.",
    "success_indicator": "Dashboard"
  }
}

Secret templates are resolved in memory before authentication. The submitted auth config is persisted and returned by the job API, so protect the jobs database and API as sensitive when submitting plaintext values.

Architecture

The orchestrator owns authentication, the job browser, known-file discovery, both Katana lanes, browser-guided discovery, checkpoints, and job state. Crawler, process, and parser adapters keep external tooling isolated.

flowchart TD
      API[FastAPI routes
app.main] --> Store[(Job store
app.db)]
      Store --> Orch[Orchestrator
app.orchestrator]

      Orch --> Auth{Auth required?}
      Auth -->|credentials or login_url| AuthAgent[Auth Agent
app.auth_agent]
      Auth -->|manual headers or no auth| Session[Job browser and auth context]
      AuthAgent --> Session

      Session --> Known[Bounded robots and sitemap discovery]
      Known --> Standard[Standard Katana
JavaScript, forms, classification]
      Standard --> Pure[Pure-headless Katana
shared Chrome plus passive CDP]
      Pure --> Guided[Hash-route queue plus Playwright and LLM gap discovery]
      Guided -->|stable seeds| Standard
      Standard --> Logs[Per-lane JSONL and terminal summaries]
      Pure --> Logs
      Guided --> Parser[Exact evidence aggregation
app.parser]
      Logs --> Parser
      Parser --> Sitemap[Sitemap result]
      Sitemap --> Store
      Store --> API
          

• API: validates input and records jobs.
• Orchestrator: owns status transitions, checkpoints, budgets, and the single-job queue.
• Auth agent: uses Playwright browser controls when login is required.
• Crawler: runs standard and pure-headless Katana with target, scope, headers, and seeds.
• Browser discovery: validates same-document hash routes, explores remaining workflow controls, and returns stable seeds to Katana.
• Parser: aggregates controlled-fetch, Katana, and passive-CDP evidence.

Threat model

Tenzai Crawler assumes a trusted operator, a trusted deployment boundary, and untrusted target websites. If that operator boundary changes, API authentication, authorization, and network egress policy become deployment requirements rather than optional hardening.

Output

Completed jobs expose a sitemap object with a flat list of observed request entries, a path tree for UI rendering, and a terminal discovery result. A discovery that stops before fixpoint is returned with partial completeness and a warning naming its stop reason. Log artifacts remain on disk for later review.

{
  "job_id": "6b1a...",
  "status": "completed",
  "target_url": "https://example.com",
  "sitemap": {
    "entries": [],
    "tree": {
      "children": {},
      "pages": []
    },
    "discovery": {
      "outcome": "fixpoint",
      "stop_reason": "complete_round_added_nothing"
    }
  }
}