Crawler policy/bot

What LektaBot does on your server

LektaBot fetches a page only after somebody asks Lekta to audit it. It identifies itself as LektaBot in every request, parses robots.txt under RFC 9309, and never impersonates a browser or another crawler. This page is the operator-facing record of that behaviour.

policy v1.1in force since 2026-08-16Obeys robots.txt

Identity

Two lanes, one token

Every audit fetches the same URL twice — a plain HTTP GET and a full Chromium render — so the two can be compared through one extractor. Both requests carry the same product token, so a single robots.txt rule governs both.

01Plain GET

No JavaScript, no cookies, no stored session. This is the lane that mirrors what major AI crawlers receive.

Mozilla/5.0 (compatible; LektaBot/0.1; +https://lekta.dev/bot)

02Chromium render

Headless Chromium, scripts executed. The LektaBot declaration is appended to the real Chromium UA — no impersonation — and the token is the same, so one rule covers both lanes.

<Chromium UA> LektaBot/0.1 (+https://lekta.dev/bot)

Request behaviour

Measured limits

TriggerOnly a URL somebody submitted through the form or the API, or a re-audit a member scheduled for a saved site. Nothing is discovered by crawling.user-initiated

VolumeTwo requests for the target URL — one plain GET, one Chromium render — plus one for /robots.txt.2–3 req

RateOne audit at a time per host, at most 1 per minute and 5 per hour per hostname — across all users, not per user.1 concurrent

TimeoutsA typical audit finishes in 5–30 s; the whole run is abandoned at the hard limit and reported as a timeout.75 s max

RetriesNo retry loops. A refused, failed or rate-limited response is recorded as a finding, not requested again.0 retries

RedirectsUp to five hops, each new location revalidated by the same SSRF guard. The final URL is the one that gets graded.5 hops

RobotsRFC 9309 parsing, longest-match precedence. A disallow is written into the report and scheduled re-audits skip the site entirely; a single audit a human explicitly requested is delivered as a user-triggered fetch, block noted.RFC 9309

RetentionRaw HTML lives in memory for the duration of the audit and is then discarded; the report keeps only derived findings.no raw HTML

BasisRFC 9309 §2.2 for robots parsing, RFC 9110 §15.3.1 for status handling.Values verified 2026-08-17

What LektaBot never does

Hard rules

Impersonate a browser, Googlebot, or any other crawler user-agent.

Crawl beyond the submitted URL — no link following, no sitemap walks, no discovery of new pages.

Submit forms, click buttons, or POST anything to your server.

Attempt authentication, reuse cookies, or send credentials of any kind.

Probe for admin paths, files, or endpoints that were not submitted.

Republish your content: reports are private and served with X-Robots-Tag: noindex.

Blocking, allowing, slowing down

robots.txt

LektaBot reads /robots.txt at the start of every audit; a repeat of the same URL within 15 minutes is answered from the result cache without touching your server. Any of these rules take effect on the next fresh audit of your site.

Block every audit

User-agent: LektaBot
Disallow: /

One rule stops both lanes. The block is written into every report of this host, and scheduled re-audits skip it entirely — no badge exception.

Allow public pages only

User-agent: LektaBot
Disallow: /admin/
Disallow: /checkout/
Allow: /

Most specific rule wins, per RFC 9309 §2.2.2 — the same precedence the audit reports.

Slow it down

User-agent: LektaBot
Crawl-delay: 5

Crawl-delay is not part of RFC 9309 and LektaBot does not need one: per-target limits — 1 per minute, 5 per hour, one at a time — are already stricter than any delay you could write here.

Verifying a request was really us

No spoofing

The user-agent string can be forged by anyone, so do not trust it alone. There is no reverse-DNS zone to resolve against; what you can check is the declaration, the request pattern, and — for anything doubtful — our own records.

01Read the user-agent. Every genuine request declares LektaBot by name and links this page.

Mozilla/5.0 (compatible; LektaBot/0.1; +https://lekta.dev/bot)

02Check the shape. One audit is two page requests plus a robots.txt read, one at a time — a stream of requests is not LektaBot.

2–3 requests · concurrency 1 · 1/min per host

03Still unsure — send the timestamp, source IP and log lines. Request records are short-lived, so write soon after the event.

hello@lekta.dev

No published IP listEgress addresses are not published as a static range, and there is no reverse-DNS zone; the checks above and our own request records are the authoritative trail.

Policy changes

Dated log

2026-08-17Policy v1.1Policy restated in the design system. Measured limits now read live from the engine configuration; behaviour unchanged.

2026-08-16Compliance contractWritten commitment added: scheduled re-audits obey robots.txt in full and skip a disallowing site — no badge exception.

2026-08-16Policy v1.0First published crawler policy — the address the user-agent declares became a real page.