Concept
Tenzai Crawler keeps repeatable work deterministic and uses AI only when a page requires interpretation.
Thesis
A crawler already knows how to fetch pages, follow links, apply scope, deduplicate URLs, and record evidence. Replacing that work with a model would add cost and uncertainty without adding useful judgment.
Some application states do require judgment. Login forms suck and come in dozens of versions. Controls can be ambiguous. Wizards and multi-step workflows may reveal routes only after a sequence of browser actions.
Where ordinary crawlers stop
Login
Session state hides routes that a public crawl cannot reach.
Safety
Built-in, operator, and auth-derived exclusions keep Katana off destructive paths.
Wizards
Branching steps expose states that do not have stable links.
Multi-step flows
Useful requests may appear only after interactions across pages.
Division of work
Deterministic core
The orchestrator and Katana own scope, crawl execution, budgets, job state, checkpoints, and sitemap evidence.
Bounded AI
AI selects login actions when authentication is requested and handles ambiguous controls after deterministic discovery.
Runtime verification
Browser and network evidence decide whether authentication worked, an action changed state, or a URL is a useful crawl seed.
Manual-header jobs do not invoke AI authentication.
Discovery loop
The crawler establishes a baseline first. Playwright then handles distinct client-side states, using deterministic actions when they are unambiguous and a model only for the remaining workflow gaps. Verified GET URLs return to Katana for another crawl stage.
flowchart LR
Baseline[Deterministic baseline
Katana] --> Gaps[Unresolved
browser state]
Gaps --> Action[Bounded action
code first, AI when needed]
Action --> Verify[Runtime
verification]
Verify --> Seeds[Stable GET seeds]
Seeds --> Baseline
Baseline --> Result[Checkpointed sitemap]
The loop stops at a discovery fixpoint or a configured budget. Partial results retain a stop reason instead of claiming complete coverage.
Evaluation
The design is evaluated across four dimensions.
Coverage
How much of the application did the run reach?
Safety
Did the run stay within scope and avoid harmful outcomes?
Speed
How much time and model work did discovery require?
Reliability
Does repeated execution produce usable, explained results?
Read the DEFCON 34 Red Team Village presentation, Improving crawling with an LLM (PDF).