The method, as one document.

The conflict over AI and content is usually reported as a series of lawsuits. That framing hides its structure. What actually happened between 2023 and 2026 is that a new extraction economy formed faster than the infrastructure for governing it, and three separate layers of that infrastructure then matured at three very different speeds. This article maps the war as a system: who takes, who loses, what has been built in response, and where the structural gap remains.

The extraction economy

Large language models are trained on, and increasingly answer from, content taken from the open web. Three properties make this different from every previous generation of web consumption.

First, the value transfer is invisible at the moment it happens. A human reader arrives, consumes, and leaves a trace that the publisher’s own analytics can see. A crawler arrives under a self-declared name, takes the corpus, and the value surfaces later, somewhere else, as a model answer or a training improvement. The party that took the content holds the only detailed record of the taking.

Second, the loss is real and measurable in aggregate. Setting company self-reporting aside, independent measurement through 2025 showed the ratio of bot traffic to human traffic on publisher sites collapsing from hundreds-to-one toward tens-to-one, while click-through from AI answer surfaces fell to a fraction of traditional search referrals. The pattern is consistent across every serious study: machine consumption up, human referrals down.

Third, the taking is deniable at the level of any individual event. A user agent string is written by the client. Network addresses are rented. This is covered in depth in The attribution problem, but the consequence belongs here: the economy runs on events that are individually unprovable with the tooling most publishers have.

The three-layer response stack

Everything built in response falls into one of three layers, and the layers matured in order.

Layer one: blocking

Blocking matured first because it maps onto infrastructure that already existed. Content delivery networks sit in front of a large share of the web, they classify traffic for a living, and switching classification from “serve” to “refuse” is an afternoon of product work. By mid-2025 the largest CDN had flipped to blocking identified AI crawlers by default across millions of sites, and default-blocking has since become table stakes across the industry.

Blocking is necessary and insufficient. It answers “how do I stop this” and says nothing about “what already happened” or “what should this cost”. It also created an arms race: blocked crawlers respond by becoming less identifiable, which degrades the very classification blocking depends on, and pushes the attribution problem downstream to everyone else.

Layer two: metering

Metering rails arrived through 2025 and 2026: pay-per-crawl schemes operated at the CDN edge, independent licensing intermediaries between publishers and AI companies, and machine-readable licensing standards that let a site declare terms in a form a crawler can parse. The direction of travel is clear: from a binary allow-or-block toward priced access with declared terms.

Metering has the same structural limitation as blocking, from the evidence point of view: its records are operational telemetry inside somebody’s platform. They price the compliant traffic. They do not produce a portable, independently verifiable account of the non-compliant traffic, which is exactly the traffic disputes are about.

Layer three: evidence

The third layer is the one this project occupies, and as of 2026 it is substantively empty. There is no widely deployed instrument that records machine traffic at the moment it happens, in a form a third party can verify without trusting the recorder. The full argument for why the incumbents of layers one and two are structurally unlikely to build it is made in Instrument, not witness; the short version is that every large player in this traffic is a party to it, and evidence signed by an interested party inherits the interest.

The money

The licensing market is real and highly concentrated. Public reporting between 2023 and 2026 described deals ranging from single-digit millions per year for individual publishers to a reported quarter-billion-dollar multi-year arrangement at the top of the market, with aggregate industry commitments in the low billions. Three observations matter more than any individual number.

• Concentration. A few dozen large rightsholders captured essentially all licensing revenue. The long tail of the web supplied the bulk of the corpus and received approximately nothing.
• Leverage asymmetry. Deals were struck almost exclusively by parties large enough to credibly threaten litigation. The correlation is not incidental: licensing follows leverage, and leverage follows the ability to prove the taking.
• Settlement gravity. The largest single payment of the period, a settlement of one and a half billion dollars in a book-piracy class action, was not a licence for training. It was compensation for how the material was obtained. The market prices provenance failures, not training itself.

Timeline

Where this leaves the instrument

Read as a system, the war has a shape. The taking is industrial and individually deniable. The response stack solved prevention and is solving pricing. The unresolved layer is proof, and proof is upstream of both of the others: priced access needs enforceable terms, and enforceable terms need provable violations. The rest of this wiki is about building that layer properly: the format that carries the proof, the practice that captures it, and the law that decides what it is worth.

Every dispute about machine traffic eventually reduces to one question: who was that? This article treats the question with the seriousness it deserves, because most of the industry does not. The claim “GPTBot crawled my site” contains, hidden inside it, four separate factual assertions with four different evidentiary strengths, and disputes are routinely lost in the gaps between them.

Identity on the web is self-declared

The User-Agent header is a string the client composes about itself. Nothing in HTTP verifies it, nothing in TLS binds it, and nothing prevents a scraper from presenting a browser’s string, a competitor’s string, or a different string on every request. This is not an edge case or an abuse of the protocol. It is the protocol.

The consequences run in both directions:

• False positives. Traffic claiming to be a well-known crawler may be anyone. Every popular bot name is widely impersonated, which is precisely why serious operators publish verification mechanisms for their own crawlers.
• False negatives. Traffic claiming to be a browser may be a fleet. An operator that does not want its fetches attributed simply does not announce them.

Any record that stores only the claimed agent has recorded an assertion, not a fact. This is why the ingress profile stores the claim and the network evidence side by side and never merges them.

The verification ladder

Attribution evidence comes in grades. From weakest to strongest:

1. The claimed string alone. Worth almost nothing in contested proceedings, for the reasons above.

2. IP range matching. Some crawler operators publish the address ranges their fleets use. Matching the peer address against a published range is meaningful evidence, with two caveats: publication is voluntary and incomplete across the industry, and ranges change, so the match must be made against the ranges as published at the time of the visit, which itself requires a pinned snapshot. Several major crawl operations publish no ranges at all, and at least one widely deployed agent-name is defined by its operator as purely declarative, with no network identity to check against. For such agents, network verification is impossible in principle, and a record can honestly say only that.

3. Forward-confirmed reverse DNS. The strongest widely available method, used by the major search crawlers’ own verification guidance. The procedure: take the peer address, resolve it backwards to a hostname, check the hostname belongs to the operator’s stated domain, then resolve that hostname forwards and confirm it returns the original address. The forward step is what defeats spoofed reverse records, because the party controlling the reverse zone for an address cannot make someone else’s forward zone agree with it.

4. Cryptographic request signing. The emerging end state: crawlers signing their requests with published keys, so identity becomes verifiable per-request rather than inferred per-address. The IETF standardisation of HTTP message signatures for bot authentication points in this direction. Adoption is early; a capture instrument should record such signatures when present and expect their absence for years.

The laundering economy

The hard cases are not sloppy crawlers. They are supply chains built to make attribution fail.

Residential and mobile proxies. Commercial proxy networks rent access to tens of thousands of consumer addresses, so that industrial fetching arrives at the target dressed as scattered household traffic. Public complaints in the 2025 and 2026 litigation wave describe exactly this: scraping intermediaries advertising browser-like user agents and rotating residential exits as product features. The addresses are real households; the reverse DNS is the ISP’s; every per-request signal says “person”, and every one of those signals is truthful about the address while being useless about the principal.

Third-party scraping intermediaries. A model company that buys data from a scraping vendor acquires content plus deniability: the fetches were made by someone else, from someone else’s infrastructure, under someone else’s name. Litigation has begun treating this as a laundering structure, and pleading it requires exactly the kind of cross-correlated records that almost nobody holds.

Cache and dataset intermediation. Content can reach a training corpus without its taker ever visiting the source: through public crawl archives, resold datasets, or another model’s outputs. A site-side capture instrument records what visited the site; it cannot, and must not claim to, trace what happened downstream. This boundary is stated honestly in What a record proves.

What the courts have accepted so far

The litigation record to date shows attribution being established, when it is established at all, through four routes: the defendant’s own declared crawler hitting the plaintiff’s servers in volume; canary content that could only have been obtained through one path, surfacing in a model’s output; subpoenas to intermediaries whose internal volumetrics no plaintiff could observe from outside; and the defendant’s own documents produced in discovery. Three of these four routes depend on someone else’s records. The entire premise of the ingress instrument is to move the strongest available evidence into the hands of the party that owns the content, recorded at the moment of the visit and verifiable by the party’s opponent.

Design consequences

The attribution ladder dictates several non-negotiable properties of the record format, each covered in depth elsewhere in this wiki:

• Claim and evidence are separate fields, never merged into a conclusion (the six groups).
• Verification steps performed at capture time are recorded with their outcomes, including failures.
• Anomalies, such as a claimed agent on an unconfirmable network, are first-class data rather than discarded noise.
• The record never states “this was operator X”. It states what was claimed, what was checked, and what the checks returned. The inference belongs to the expert and the court, and a record that pre-draws the conclusion is weaker for it, not stronger.

The EU AI Act gave the data war its first hard regulatory deadline, and it did so in a very particular shape: a disclosure regime with real fines and, so far, no verification layer. Understanding that shape precisely matters for both sides of the instrument, because the gap between what must be declared and what can be proven is where signed records earn their keep.

The obligations, precisely

Article 53 places four duties on providers of general-purpose AI models placed on the EU market. Two of them concern this wiki directly.

The copyright policy, 53(1)(c). Providers must put in place a policy to comply with EU copyright law, and in particular to identify and respect rights reservations expressed under the text-and-data-mining provisions of the 2019 Copyright Directive. In plain terms: when a rightsholder machine-readably says “not for mining”, the provider’s crawlers are expected to honour it.

The training-data summary, 53(1)(d). Providers must publish a “sufficiently detailed summary” of the content used for training, using a mandatory template published by the AI Office in July 2025. The template is more concrete than the statute suggests: it asks for data modalities and size bands, the principal datasets, and, for web-scraped data, the names and behaviour of the crawlers used, the collection periods, and the top domains scraped by volume, with reduced thresholds for smaller providers. Measures taken to respect rights reservations are described in narrative form.

The timeline arrived in three steps: obligations attached to new models from August 2025, enforcement powers with fines up to three percent of worldwide turnover or fifteen million euros activated in August 2026, and models placed on the market before August 2025 have until August 2027.

The first year: declaration without verification

The record of the first compliance year is instructive. Independent academic review of the published summaries, conducted with a civil-society partner and presented at a major accountability conference in 2026, found the early landscape sparse: months into the obligation, only a handful of summaries existed at all, frontier providers were largely absent, and the summaries that did exist were graded mediocre on transparency and usefulness. Several major labs disclosed through unstructured prose rather than the mandatory template. Filing activity accelerated visibly only as the enforcement date approached.

Two structural observations follow.

First, the regime runs on self-description. The template requires a provider to say which crawlers it used and describe how reservations were honoured. Nothing in the mechanism tests those statements against events. The code of practice that accompanies the Act obliges signatories to publish crawler information and honour machine-readable reservations, but it prescribes no request-level record-keeping and no evidentiary standard for demonstrating compliance.

Second, both sides of an Article 53 dispute currently argue from weak evidence. A rightsholder who believes its reservation was ignored holds unsigned server logs and industry-level measurement showing that identified AI fetchers reach disallowed paths at material rates. A provider that honestly honoured reservations holds internal logs that no outside party has a reason to believe, authored by the party they would exonerate. Neither side can produce a record the other is obliged to take seriously.

Where records enter

The instrument maps onto this regime symmetrically.

On the rightsholder side, signed ingress records convert “we believe our reservation was ignored” into a chained series of specific, verifiable events: this agent, this network, this path, this robots state pinned by digest at the moment of the fetch. That is the difference between a complaint the AI Office must investigate from scratch and a complaint that arrives pre-evidenced. It also feeds the private-law routes that run in parallel: the German collecting-society judgments of 2025 and 2026 showed courts entirely willing to find infringement around AI training and outputs, and notably unwilling to accept regulatory compliance postures as a defence.

On the provider side, a signed crawl ledger is the artifact the regime implicitly asks for and does not yet name: a record of reservations honoured, requests refused, and terms consulted, kept in a form an inquiring authority can verify without trusting its author. A provider that adopts this discipline converts its copyright policy from a document into a demonstrable practice. The strategic read is developed further in Instrument, not witness: defence adopts what offence made credible.

Honest boundaries

Three cautions, stated plainly because overclaiming is how this wiki would lose its value.

• No enforcement case law exists yet. As of this article’s date, no fine or formal proceeding under these provisions has been publicly reported. The AI Office’s stated posture begins with compliance dialogues. How much evidentiary weight it will assign to signed records is a question the first cases will answer.
• The template’s escape hatches are real. Size bands, top-percentile domain lists and best-effort clauses cap how much disclosure pressure the summary itself exerts. The verification vacuum is a market opening precisely because the disclosure floor is low.
• A reservation is not a wall. The legal weight of an ignored robots directive differs by claim and by jurisdiction, and a prominent US ruling treated it as a sign rather than a technical barrier. Records of ignored reservations feed contract, copyright and regulatory theories with different force in different rooms. The mapping belongs to counsel.

This article digests the public litigation record that bears on one narrow question: what evidence about machine traffic and model behaviour has actually held up, and what has failed? Each digest states the public facts neutrally and then draws the design lesson. Nothing here is legal advice, and nothing here takes a side in any live matter; the material is cited because instrument design must answer to it.

Training, acquisition, and the provenance split

A 2025 federal ruling in a US authors’ class action drew a line that reorganised the entire dispute landscape: training a model on lawfully obtained books was held to be fair use, while acquiring those books from pirate libraries was not. The subsequent settlement, finally approved in 2026 at one and a half billion dollars, priced the acquisition failure rather than the training itself.

Lesson. The legally load-bearing question moved upstream from “did you train on it” to “how did you obtain it”. That is a provenance question, and provenance questions are decided on records of acquisition events. An industry whose largest payment to date turned on how content was taken is an industry that will keep litigating exactly the events an ingress instrument records.

The robots.txt ruling

In late 2025, a US federal court consolidated several publisher claims and rejected an anti-circumvention theory built on ignored robots.txt directives, reasoning that the file functions as a sign rather than a technological barrier. Access-control statutes protect locks; robots.txt is a notice.

Lesson. A record of an ignored reservation is not evidence of circumvention, and an instrument that markets it as such would discredit itself. The same record remains fully potent for what it actually shows: notice given, notice received machine-readably, and conduct after notice. Those are the raw materials of contract, copyright and regulatory theories. Precision about which door the evidence opens is part of the product.

The engineered-exhibit problem

Two public events define this category. In a UK image-library case decided in 2025, outputs generated by the plaintiff’s own team were discounted with the observation that there was no evidence the depicted behaviour had occurred in the wild. And in the flagship US news case, the plaintiff withdrew its own headline exhibit of verbatim model outputs rather than defend, in discovery, the process by which those examples had been produced.

Lesson. This is the single most consequential pattern for evidence design. Both failures share one property: the exhibits were reconstructions, produced by an interested party after the dispute began, with no contemporaneous record of the process. Courts did not reject the underlying claims; they rejected the evidentiary mode. The design answer is capture-or-nothing: a record exists from the moment of the event or it does not exist, and the session profile exists precisely to make model-output evidence contemporaneous, parameterised and reproducible instead of curated.

The attribution pleadings

The 2025 and 2026 complaints from a major forum operator against scraping intermediaries and an answer engine read as a case study in attribution failure and its workarounds. Public filings describe scrapers advertising rotating residential exits and disguised agents; a canary post visible through exactly one channel that surfaced in an answer engine within hours, functioning as a marked bill; and pre-suit subpoenas to a search company for volumetrics that no outside party could have measured. Separately, a CDN’s 2025 honeypot investigation publicly documented an answer engine’s undeclared fetching through rotated agents and networks, and that report was quoted verbatim in subsequent litigation.

Lesson. Every successful attribution route in the record was either borrowed (someone else’s logs, obtained by subpoena), improvised (canary content), or built by an infrastructure incumbent as a one-off exposé. None of it was a repeatable instrument in the hands of the injured party. The gap between “attribution is possible for the well-resourced” and “attribution is routine for anyone with a capture node” is the product.

The compliance-defence rulings

German collecting-society litigation produced two 2025 and 2026 judgments in which courts found infringement in AI training and outputs, and in the later of which the defendant’s posture of regulatory compliance was rejected as a defence to copyright claims.

Lesson. Regulatory declarations and private-law liability run on separate tracks. A provider cannot argue its way out of an infringement claim by pointing at its filed summaries; conversely, a provider with a signed crawl ledger holds something no declaration provides: event-level proof of the discipline it claims. Records serve where statements fail, on both tracks.

The discovery asymmetry

Across the flagship US cases, the most probative material has repeatedly surfaced only through discovery: internal acquisition records, download logs, and compelled production of model interaction logs at a scale of tens of millions of conversations. Discovery works, but it is slow, adversarial, expensive, and available only after a suit survives its early motions.

Lesson. The entire evidentiary economy of this war currently runs on records held by defendants and extracted under compulsion. A capture instrument inverts the default: the party with the standing injury holds contemporaneous records from day one, and discovery becomes corroboration rather than the sole source of proof.

Synthesis: the design constraints courts have already written

Read together, the record dictates the instrument’s constitution. Evidence must be contemporaneous, because reconstructions get discounted. It must be process-transparent, because exhibits die in discovery about how they were made. It must record claims separately from verification, because identity is self-declared and courts know it. It must be verifiable by the adversary, because records that require trust in their author inherit their author’s interest. And it must be precise about what it proves, because the one theory the courts have already closed, circumvention-by-robots.txt, is exactly the kind of overclaim that takes an exhibit down with it. Every one of these constraints is load-bearing in the format’s design rationale.

Evidence has a price, and so does its absence. This chapter works through the economics that make a capture instrument rational, without a single invented number: the structure of the costs is public, and the structure is what decides.

The cost curve of a dispute

Every machine-traffic dispute runs the same cost curve, and the shape matters more than the absolute figures. It begins cheap: a letter, a takedown, a licensing approach. It becomes expensive at exactly one point: the moment factual assertions must survive challenge. From there, costs compound through pleading standards, expert work, and above all discovery, which in the flagship AI cases has meant multi-year fights over logs, corpora and internal records, with litigation budgets publicly reported in the tens of millions for a single matter.

The economic function of contemporaneous records is to move work from the expensive segment of the curve to the cheap one. A complaint that opens with verifiable exhibits spends less time surviving motions built on “upon information and belief”. An expert who certifies a published process bills fewer hours than one reconstructing events from fragments. A discovery fight narrows when one side’s records are already on the table, chained and checkable. None of this eliminates cost; it re-times it, and re-timing is most of the game, because the expensive segment is precisely where under-resourced parties abandon meritorious claims.

The asymmetry, priced

The case-law chapter documents the pattern: probative material in these disputes has lived overwhelmingly on the defence side and surfaced only under compulsion. Priced out, the asymmetry looks like this. The party without records pays for investigation (what happened at all), for reconstruction (assembling a narrative from unsigned fragments), for the credibility discount every reconstruction carries, and for the discovery lottery. The party with records pays for storage and a verification walkthrough. The first bill scales with the dispute; the second scales with traffic, and traffic is cheap: a capture node’s marginal cost per recorded visit is a signature and a few kilobytes.

This asymmetry also explains who currently wins. The war chapter noted that licensing revenue concentrated among parties large enough to threaten litigation credibly. Credibility here is not courage; it is expected-value arithmetic. Leverage in a licensing negotiation is the counterparty’s estimate of what you could prove, discounted by what it would cost you to prove it. Records held in advance move both terms of that product at once, which is why the same instrument that serves litigation serves parties who never intend to litigate: the quiet majority of evidence value is realised in rooms where no judge ever appears.

Three budget lines, one instrument

The purchase decision lands differently in three budgets, and the honest analysis respects the differences.

Litigation support. For a party already in or near a dispute, the comparison is direct: instrument cost against the reconstruction hours it displaces and the exhibit risk it retires. The engineered-exhibit rulings put a concrete price on that risk: exhibits have been discounted and withdrawn, and with them, months of position.

Compliance. For a provider under the Article 53 regime, the comparison is against the cost of answering an inquiry with prose plus internal logs, in a fine environment denominated in percentages of turnover. A signed ledger is insurance whose premium is an engineering afternoon.

Insurance against the unpriced. For everyone else, the instrument is an option: cheap while nothing happens, decisive if something does. Options on rare, expensive events are systematically underbought when the buyer must imagine the event; the use-cases page exists because every one of its six mornings is such an event, imagined too late.

What this does not argue

Economic honesty cuts both ways. A capture node does not make a weak claim strong; it makes a true claim provable, and the difference only pays when truth and provability were the binding constraint. A party whose losses are speculative buys little. A party whose reservation was never machine-readable buys evidence of ambiguity. And the instrument’s own cost is not zero: keys must be kept, retention governed, exports supported. The economics clear easily where machine traffic already touches money someone answers for, which is exactly the boundary drawn around the segments this product serves.

A format is a set of decisions frozen into structure. This article records the decisions behind the passport envelope, the constraints that forced them, and, at least as importantly, the roads not taken. A reader who disagrees with a decision should be able to find the constraint it serves and attack that instead; this is what it means for a format to be examinable.

The constraints

Five requirements were treated as non-negotiable, in this priority order.

C1. Adversarial verifiability. The party with the strongest motive to discredit a record, the opposing side of a dispute, must be able to verify it completely, locally, offline, without any relationship with the issuer or with us. Every design that required a live service, an account, or trust in an operator was disqualified by this constraint alone.

C2. Capture-time existence. A record must be creatable only at the moment of the event, and its integrity must be checkable ever after. Retroactive fabrication cannot be made impossible, but it must be made expensive, visible and narrow (see the threat model).

C3. Court shape. The record must decompose into the elements existing evidence practice already examines: content digests, timestamps, a describable generation process, and a certifiable chain of custody. Novel cryptography is acceptable; a novel evidentiary theory is not, because the first court to see the format should recognise the pattern (see Admissibility).

C4. Semantic honesty. The format must be structurally incapable of stating more than was observed. Claims and verification outcomes are separate fields. Anomalies are data. There is no field for conclusions.

C5. Boring implementability. Any competent engineer must be able to write an independent verifier in an afternoon from the specification alone. Interoperability is the format’s growth strategy, so exotic dependencies are a tax on adoption.

The decisions

One envelope, profiled vocabularies. A single container (version, id, issuer, six claim groups, chain, signature) serves ingress, session and observation capture, with profiles fixing the vocabulary inside each group.

The alternative, separate formats per capture direction, was rejected because verifiers, tooling and case law accumulate per format, and fragmenting that accumulation across three formats would triple the adoption cost for a marginal gain in per-field precision. Profiles are specified in The three profiles.

Six groups as the claim taxonomy. Source, observation, method, context, quality, rights. The taxonomy came from working backwards through dispute transcripts: who was it, when was it, how was it captured, against which terms, what was actually served, and what follows legally. Every field in every profile answers one of those six questions, and any proposed field that answers none of them is rejected by construction.

Ed25519 over a canonical JSON body. Chosen for C5 as much as for security: the algorithm is ubiquitous, misuse-resistant, fast, and available in every language’s standard tooling. The canonical form, sorted keys and no insignificant whitespace, is deliberately unsophisticated; the full reasoning, including why the more elaborate canonicalisation standards were not adopted wholesale, is in Canonical form.

A linear hash chain, not a tree, not a blockchain. Each record commits to the digest of its predecessor. Linear chaining delivers the property that actually matters for evidence, tamper-evidence across a series, at near-zero complexity cost. Merkle-tree checkpointing is an anticipated upgrade path for high-volume nodes and third-party witnessing (Chain mechanics). A public blockchain as the primary trust anchor was considered and rejected: it imports volatility of perception into a product whose entire brand is sobriety, it solves global ordering, which is not the problem, and it does not solve capture-time truthfulness, which is. A qualified timestamp from an accredited authority achieves the legal effect with an authority courts already recognise (Time).

Signing by the operator, not by us. The instrument’s maker holds no keys and signs nothing. This decision looks commercially self-defeating, an evidence company refusing to be the root of trust, and it is the single most important decision in the design. The moment the toolmaker signs records, the toolmaker becomes a witness with an interest, and every record inherits the question “why believe them?”. The full doctrine is in Instrument, not witness.

No conclusions in the record. The format has no field for “this was operator X” or “this violated the terms”. It stores the claimed agent, the checks performed, their outcomes, and the terms as served. This was contested internally, because customers want conclusions. But a record that draws its own conclusions is an argument, and arguments are attackable in a way that observations are not. The conclusion is the expert’s job; the record’s job is to make the expert unnecessary to trust.

Roads not taken

• Central capture service. A hosted service capturing on customers’ behalf fails C1 and C2 at once: it centralises trust in an interested operator and turns the company into the single point of subpoena, compromise and doubt.
• Watermarking and content fingerprinting. These identify content reuse downstream; they say nothing about the acquisition event and cannot carry terms-in-force. Complementary, not competitive: a fingerprint match plus an ingress record is a stronger pair than either alone.
• zkTLS-style transcript proofs. Cryptographically stronger for outbound observation (proving what a server said without trusting the observer), and genuinely admirable work. Rejected as the foundation for the ingress instrument on three grounds: the deployed protocols address the observer-of-a-server topology rather than the server-recording-its-visitors topology; the toolchain has no presence in evidence practice, failing C3 today; and the trust reduction it buys is smallest exactly where our records live, on the recorder’s own edge, where the recorder’s interest is disclosed rather than hidden. The technology remains on the watchlist, and the format’s profile mechanism can absorb transcript proofs as an additional evidence grade when practice matures.
• Per-record trusted hardware attestation. Attestation of the capture environment adds real strength against the “modified node” attack and equally real dependency on vendor attestation chains. Deferred, not rejected: the envelope reserves room for environment attestations as an optional hardening layer.

What this buys

The result is a format with a small spec, a boring cryptographic core, an adversary-first verification story, and a shape courts have seen before. None of these properties is impressive in isolation. Together they define an instrument that can be handed to the party most motivated to break it, which is the only kind of evidence instrument worth building.

A digital signature does not sign a document. It signs bytes. If two honest parties can derive different bytes from the same record, the signature proves nothing to one of them, and an evidentiary format has failed at its only job. Canonicalisation is the machinery that guarantees byte-identity, and it deserves more respect than it usually gets, because most of the historical failures of signed-document systems trace back to exactly this layer.

The problem, stated precisely

A passport travels as JSON: through APIs that re-serialise it, editors that pretty-print it, databases that reorder keys, and languages that have opinions about numbers. JSON the data model survives all of this; JSON the byte sequence does not. Signing requires a function that maps the data model to one and only one byte sequence, such that every independent implementation agrees.

The verification contract is: strip the signature member, canonicalise the remainder, verify Ed25519 over those exact bytes. Every pitfall below is a way for two implementations to disagree about “those exact bytes”.

The classic pitfalls

Key order. JSON objects are unordered by specification, and every serialiser picks its own order. Canonical form must fix one. Ours: lexicographic sort by key at every nesting level.

Insignificant whitespace. Pretty-printers insert it freely. Canonical form emits none.

Number re-encoding. The deepest pitfall. The same numeric value has many spellings (1, 1.0, 1e0, 0.1 vs 1e-1), floating-point round-trips can alter spellings, and some ecosystems silently lose integer precision above 2^53. A measured value that gets “helpfully” re-encoded en route breaks the signature while remaining semantically identical, which is precisely the kind of failure that surfaces two years later in a deposition. Our rule is prevention rather than cure: numbers are never re-encoded after issuance, profiles keep measured quantities in integer units (bytes, milliseconds, sequence numbers), and verifiers must treat the received spelling as authoritative.

Unicode. Two byte sequences can render as the same string (composed and decomposed forms), and escape choices (é versus the raw character) vary by serialiser. Canonical form emits UTF-8 without normalisation and with a fixed minimal-escaping rule: what was issued is what is signed, and no layer in between is permitted an opinion.

Duplicate keys. Legal in lax parsers, poison in signed documents, because parsers disagree on which duplicate wins. Canonical form rejects them outright.

Relationship to RFC 8785

An IETF standard for JSON canonicalisation exists (JCS, RFC 8785), and the obvious question is why not simply adopt it. The answer is nuance rather than disagreement. JCS is excellent and battle-tested for the object-and-string layer, and our rules coincide with it there. Its number canonicalisation, however, is defined in terms of IEEE 754 double-precision serialisation, which makes the canonical spelling of a number a function of float semantics. For a format whose numbers are measurements destined for cross-examination, we prefer the stricter discipline: integer units in the profiles, no re-encoding anywhere, received spelling authoritative. A future revision may declare full JCS compatibility for the non-numeric layer formally; the current specification keeps its four rules small enough to reimplement from the prose alone, which is its own kind of safety.

Try it

The widget below runs the actual canonicalisation used by the site’s verifier, in your browser, and digests the result with SHA-256. Reorder keys, add whitespace, pretty-print: the canonical bytes and the digest do not move. Change any value, even trivially, and the digest changes completely.

Why this layer decides cases

Consider the deposition question every signed-record system must eventually survive: “Your witness says this record verifies. My expert serialised the same record and got different bytes. Which of you is wrong?” Under a loose canonicalisation, that question has no clean answer, and the record’s credibility bleeds out in the ambiguity. Under a four-rule canonical form that both experts can implement from one page of prose, the question is settled by running the rules in front of the court. Determinism is not an implementation nicety. It is the difference between a signature that proves and a signature that argues.

Every record carries the digest of the record before it. That single field turns a pile of individually signed objects into a series with history, and it is worth being exact about what that buys, because hash chains are surrounded by folklore. This article states the guarantees, the non-guarantees, and the hardening layers in order of cost.

The core property

Let each record commit to chain.prev = sha256(canonical(previous record)). Then modifying any historical record changes its digest, which contradicts the commitment stored in its successor, which would need rewriting, which changes its digest, and so on to the head of the chain. The formal property: any edit to a non-head record is detectable by anyone holding any later record, at the cost of one digest comparison per link.

Three practical consequences:

• Quiet correction is impossible. An operator can destroy records, but cannot surgically improve one without visibly orphaning everything issued after it.
• Every shared record is a checkpoint. The moment record N leaves the operator’s hands (sent to counsel, filed with a complaint, emailed to a counterparty), the entire history before N is pinned against rewriting, because the recipient’s copy commits to it transitively.
• Detection is cheap and mechanical. Verifying a chain segment is a loop any junior expert can run; no judgement calls are involved until an inconsistency appears.

What the chain does not prove

Honesty about limits is a design feature of this wiki, so here is the complete list.

It does not prove completeness. A chain proves that what exists was not reordered or edited. It cannot prove that a record was never created, because an operator running a capture node can decline to record an event, and the chain closes seamlessly over the gap. Selective capture is the instrument’s most fundamental residual risk and is treated at length in the threat model; the mitigations are statistical (continuous sequence numbers, volume accounting against independent telemetry) rather than cryptographic.

It does not prove wall-clock time. Digests order records relative to each other; they say nothing about when the series ran. An operator could, in principle, fabricate an entire chain last night with backdated timestamps. This is why time is its own hardening layer: qualified timestamps countersign records with an authority whose clock carries legal presumption, converting “the operator says Tuesday” into “an accredited third party attested no later than Tuesday”.

It does not prevent truncation. An operator can present a chain that simply stops, or starts late. Sequence numbers make gaps and late starts visible; witnessing (below) makes silent truncation contradict published commitments.

Hardening layers, in cost order

1. Sequence numbers (already in the envelope). Free; makes gaps countable.
2. Distribution as checkpointing. Every export to an outside party pins history. A practice as simple as monthly delivery of the head digest to opposing-friendly custody (counsel, an escrow, a counterparty) bounds the rewrite window at no infrastructure cost.
3. Qualified timestamp countersignatures on the head at intervals. Bounds backdating with a legally presumed clock.
4. Public witnessing. Publishing periodic head digests where they cannot be quietly replaced (a company page with archive coverage, a transparency log). This is the cheap end of the transparency-log idea, and the format’s roadmap anticipates it.
5. Merkle-tree accumulation. For high-volume nodes, batching records into trees whose roots are chained and witnessed gives logarithmic inclusion proofs: a single visit can be proven to belong to a witnessed batch without shipping the whole day’s chain. This is the certificate-transparency architecture applied to capture, and it is the designed upgrade path rather than a v0.1 feature, because the linear chain’s properties are sufficient for the record volumes of early deployments and its verification story is radically simpler to examine in court.

The doctrine

The chain’s purpose is frequently misstated as “making records immutable”. Nothing makes bytes immutable. The chain’s actual purpose is narrower and more useful: it makes dishonesty expensive, compounding and detectable, and it converts every act of sharing into an act of notarisation. An instrument whose operator’s hands are visibly tied is worth more than one that promises its operator is honest; the chain is how the tying is done in public.

Disputes turn on time. Terms changed on the 14th; was the fetch on the 13th or the 15th? The licence lapsed at midnight; the crawl ran at 23:58 or 00:02? A capture instrument that cannot defend its clock cannot defend anything else, because every other field in the record inherits the timestamp’s credibility. This article covers the full stack: how the moment is measured, how it is made expensive to lie about, and how it acquires legal weight.

The threat, stated as an adversary would

The operator of a capture node controls its hardware clock. A sceptical expert must therefore assume, until shown otherwise, that observed_at is an assertion like any other: the operator says it was Tuesday. Two distinct lies are possible. Backdating: fabricating a record now and claiming it was captured earlier, to manufacture history. Forward-dating is rarer but real: claiming a later capture to dodge a terms window. A serious format bounds both directions with different tools, because no single mechanism bounds both.

Layer one: measuring honestly

The observed_at_source field records how the moment was established, and the capture node’s discipline is to derive time from sources outside its own host: consensus across multiple NTP servers, with modern deployments adding a cryptographically authenticated time protocol where available, so that a compromised or misconfigured local clock is detectable at capture rather than discoverable at deposition. Honest measurement does not defeat a deliberately lying operator; it defeats the far more common failure, drifted and broken clocks, and it leaves an audit trail of which authorities the node consulted. Precision requirements are deliberately modest: disputes are decided at the granularity of seconds and terms windows, not microseconds, and a format that promises microsecond truth invites attacks on a promise it never needed to make.

Layer two: bounding the lie with countersignatures

The strong tool against backdating is older than the web and courts already trust it: the timestamping authority. The mechanism, standardised as RFC 3161, is elegantly simple. The node sends a digest of the signed record to an independent authority; the authority signs “I saw this digest at time T” with its own key. The record demonstrably existed no later than T, on the word of a party with no stake in the dispute, who never even saw the record’s contents, only its digest.

Chained records amplify this cheaply: a countersignature on record N transitively bounds every earlier record in the chain, because they are committed by N’s digest lineage. A node does not need to countersign every visit; countersigning the chain head at sensible intervals bounds the entire series at a fraction of the cost. The interval is a policy knob: tighter for high-stakes capture, looser for archival deployments.

Note the asymmetry: countersignatures prove no later than. They do not prove no earlier than, and the format does not pretend otherwise. What bounds forward-dating in practice is the combination of chain order, sequence continuity, and the countersigned bounds of neighbouring records: a record claiming Thursday, sandwiched in a chain whose Wednesday head was countersigned, has an arithmetic problem.

Layer three: legal presumption

Here the engineering meets the law, and the EU built something specific. Under the eIDAS regulation, a qualified electronic timestamp, one issued by an accredited trust service provider operating under audited requirements, enjoys a statutory presumption of the accuracy of its date and time and the integrity of the bound data. Presumption is a procedural superpower: the burden of proof flips, and it is the challenger who must show the timestamp wrong, rather than the proponent who must prove it right. Non-qualified timestamps are not excluded from evidence, eIDAS guarantees them admissibility, but they are argued rather than presumed.

This is why the format’s v0.2 envelope reserves a countersignatures member specifically for qualified timestamps: for records destined for EU proceedings, a QTSP countersignature converts the time question from testimony into presumption. In the United States no direct statutory analogue exists, but the same countersignature slots cleanly into the self-authentication framework for records of a reliable process, and practitioner literature on electronic evidence has treated trusted timestamping as best practice for years. Other jurisdictions vary, and several have extended timestamp recognition to distributed-ledger anchoring; the format is agnostic about the authority type and precise about recording which authority was used.

Why not a blockchain anchor

Anchoring digests into a public chain does bound backdating, and one European court has accepted such an anchor as proof of anteriority. The format nevertheless treats ledger anchoring as an optional extra rather than the spine, for reasons argued in the design rationale: the legal recognition is patchier than the QTSP route, the operational story imports third-party volatility, and the audience this instrument serves, courts, regulators, opposing experts, already has a decade of practice with timestamping authorities. Where a ledger anchor helps a particular matter, nothing prevents adding one alongside; the envelope’s countersignature list is plural on purpose.

The compound guarantee

Assembled, the layers give the record’s time three independent legs: measured from consensus sources and labelled with its method; ordered and bounded by the chain; and countersigned, where the stakes warrant, by an authority whose word carries presumption. An adversary must now defeat all three at once, and must do so against the threat model’s observation that every shared record pins the series. Time remains the most attacked field in the format. That is exactly why it is the most defended.

One envelope, three directions of capture. A profile fixes the vocabulary inside the six claim groups for one direction, and this article is the reference: every field, its meaning, and, because this is an evidentiary format, the dispute question each field answers. A field that answers no dispute question does not survive review; several proposed fields have already died by that rule.

The six groups ask, in order: who, when, how, against what, with what result, and with what legal consequence. They keep those meanings in every profile.

Ingress: a machine visit to your content

The profile of the first product: a passive node beside a publisher’s edge records an arriving machine request. The specimen on the front page and in the format page is an ingress record.

source. Who came asking?

observation. When did it happen? observed_at (the instant), observed_at_source (how time was established: consensus NTP, authenticated time, see Time), capture_point (where in the serving path the copy was taken), duration_ms.

method. How was the capture made? transport (how the request arrived), capture (passive_mirror), interference (none: the recorded visit was served exactly as it would have been unrecorded), site (the property being recorded). This group exists because the first thing an opposing expert examines is whether observation altered the observed.

context. Against which terms? requested_url, robots_txt_sha256 (the policy file as served at that second, pinned by digest), robots_state (allowed or disallowed for this claimed agent, as evaluated by the published parsing rules), license_signals (machine-readable reservations in force: licensing-standard declarations, TDM reservations). The design principle: terms are captured as data, contemporaneously, never reconstructed from memory (Terms as data).

quality. What was actually served? http_status, content_sha256, content_bytes, response (served or refused), anomalies (an open list: agent-network mismatches, malformed requests, protocol oddities worth preserving). The response is part of the fact: a refusal recorded is evidence of restraint, which matters on the provider side.

rights. What follows? basis (publisher terms, licence, agreement), terms_url and terms_sha256 (the exact prose version in force), reserved (uses reserved by the holder), jurisdiction.

Session: what a model was asked and what it answered

The second profile, drafted, records interactions at a model’s API boundary. Its reason to exist is the engineered-exhibit problem: courts have discounted curated model outputs precisely because nothing documented how they were produced. A session record makes the production process itself the evidence.

• source identifies the caller and the endpoint: which API, which account context, which client.
• observation timestamps each turn of the exchange, not merely the session.
• method is the group that answers “you engineered it”: the complete prompt chain including system context as visible to the caller, model identifier and version, and the sampling parameters (temperature and friends) that determine reproducibility. A cherry-picked output dies in discovery; a fully parameterised one invites re-running.
• context pins the terms under which the model was accessed.
• quality digests each output, records refusals and truncations, and preserves response metadata.
• rights carries the same envelope as ingress.

An honest limitation, stated as always: a session record proves what this caller sent and received. It does not prove what the model would say to others, and non-zero sampling temperatures mean exact reproduction is probabilistic. The profile’s value is that every one of those caveats is visible in the record rather than buried in a methodology fight.

Observation: an outbound fetch

The original v0.1 profile: a machine observing the open web, recording what it consumed, from where, under which rights. Where ingress records the taking of your content, observation records the sourcing for your decisions, and the two are mirror images: final_url and redirect chains instead of requested_url alone; egress network and region instead of peer analysis; freshness and cache posture in observation; corroboration by independent observations in quality. A provider running the crawl ledger is running this profile over its own fleet, with refusals recorded as first-class events.

Cross-profile invariants

Whatever the direction, five rules hold, and verifiers may rely on them:

1. Claims and checks never merge. An identity claim and its verification outcome are always distinct fields.
2. Digests pin every external artifact. Content, terms, robots files: anything the record references, it references by hash.
3. Absence is explicit. A check not performed is recorded as not performed, never omitted, so silence cannot be read as success.
4. No conclusions. No field states a violation, an identity, or a liability. Records observe; experts conclude.
5. Unknown fields must not break verification. Profiles may extend; the canonical form signs whatever is present; verifiers reject only unknown envelope versions, not unknown claim fields. This is how the format grows without orphaning old verifiers.

Every guarantee in this format reduces, eventually, to a statement about a private key: records signed under it were made by whoever controlled it. That reduction is the point, and also the exposure. This article specifies the key lifecycle an operator is expected to run, and the reasoning, because in an evidentiary system key hygiene is not an operational detail. It is the chain of custody’s first link.

Whose keys these are

The instrument doctrine decides this before cryptography enters: the operator of a capture node generates, holds and uses its own signing keys. The toolmaker never holds them, never escrows them, and cannot sign on anyone’s behalf. The demonstration key signing this site’s specimen records is labelled as such in the public registry, and the label is load-bearing: a verifier reading status: demo knows exactly how much weight the signature carries, which is the format working as designed.

Generation and custody

Ed25519 keys are 32 bytes of entropy; generating them well is a solved problem and the specification simply requires a cryptographically secure source. Custody is where discipline lives, and the requirements scale with what the records are worth:

• Baseline: the private key lives on the capture node, readable only by the capture process, never in configuration management, never in backups that leave the operator’s control.
• Hardened: the key lives in a hardware module or platform keystore, and the capture process requests signatures without ever holding key bytes. This converts “attacker read a file” into “attacker had to own the hardware”, a materially better deposition answer.
• Witness-grade: hardware custody plus dual control over key ceremonies, with the ceremony itself logged. Warranted when a node’s records are expected to anchor high-stakes proceedings.

The specification mandates none of these tiers; it mandates that the operator be able to describe its tier, because the description is what a qualified person certifies and what an opposing expert probes. An honest “baseline” survives cross-examination better than an aspirational “hardened”.

Rotation

Keys rotate on schedule, not on suspicion. Scheduled rotation bounds the blast radius of an undetected compromise, keeps the rotation machinery exercised, and produces a registry whose history looks like discipline rather than emergency. The registry entry for each key carries its validity window, and each record names its key by key_id, so verification is always “was this record’s key valid at that record’s position in the chain”, never “is this key valid today”.

Rotation interacts with the chain elegantly: the first record signed under a new key sits in a chain whose predecessor was signed under the old one, so the succession itself is tamper-evident. An operator cannot quietly retcon a key change.

Publication, and the permanence rule

Public keys live at a stable, well-known address per issuer, in a registry listing key id, algorithm, validity window and status. Two rules give the registry its evidentiary force.

Retired keys are published forever. A record signed in 2026 must verify in 2033 against the key that signed it, regardless of how many rotations intervened. Deleting a retired public key would orphan every record under it; the registry therefore only ever grows. status moves through active, retired, and where necessary revoked, but entries never vanish.

The registry itself is history-pinned. A registry an operator could silently rewrite would be the soft underbelly of the whole scheme: swap the published key, swap the history. Registry snapshots are therefore digestible and pinnable like everything else, and third parties who verify records are encouraged to retain the registry state they verified against. Public web archives do this incidentally; counsel does it deliberately.

Compromise

The honest scenario planning, because “it will never happen” is not a custody policy:

1. Suspected leak. The key is revoked in the registry with a timestamp. Records signed before the revocation instant retain their weight to the extent the chain and countersignatures bound them there; records surfacing later under the revoked key are self-discrediting. This is where qualified timestamps earn their keep: a countersigned chain head from before the incident cleanly separates the trustworthy past from the contaminated window.
2. Compromise with backdating attempt. The attacker holding the key tries to forge history. They collide with the chain (existing shared records commit the sequence), with countersigned heads (an accredited authority’s clock bounds the past), and with distribution (every copy in outside hands is a witness). The forgery has to beat all three; the threat model walks the arithmetic.
3. Loss without compromise. A destroyed key stops issuance but destroys nothing: verification needs only the public half, which is published. The operator rotates forward and the chain records the seam.

The one-sentence summary

Keys are where an evidentiary format either becomes real or stays a whitepaper: generated by the party who answers for the records, held to a describable standard, rotated on schedule, published forever, and revocable in a way that separates past from contamination. Everything else in this wiki assumes this article is being taken seriously.

An evidentiary format has a constraint ordinary formats do not: its oldest artifacts matter most. A record captured today may face its examination in five years, against a verifier written in between, by a party hostile to surprises. Format evolution must therefore be governed like an API with a decade of consumers, and this chapter states the governing rules.

The versioning contract

The envelope carries passport_version, and the contract around it has three clauses.

Verifiers reject unknown envelope versions. A verifier that encounters a version it does not implement must fail closed, loudly. Guessing across envelope versions is how signature scope bugs are born, and a wrong “valid” is catastrophically worse than a refused check.

Envelope versions change rarely and mean it. The envelope bumps only when the signing perimeter changes: a new top-level member inside the signed body, a canonicalisation amendment, an algorithm transition. Each bump ships with a delta document and test vectors, because the canonical-form chapter applies in full to every version separately.

Claims evolve without envelope bumps. Profiles may add fields freely, and here the rule inverts: verifiers must tolerate unknown claim fields, verifying the signature over whatever is present. The signature covers unknown fields automatically, since canonicalisation serialises what exists. An old verifier checking a new record verifies correctly and simply does not interpret the new fields; interpretation is the expert’s layer, not the mathematics’.

This split, strict on the envelope and tolerant on the claims, is what lets the format grow vocabulary for years while every verifier ever shipped keeps returning correct answers.

Additions, deprecations, and the one forbidden move

Adding a field requires: a name that answers one of the six group questions, semantics documented in the profile, and an “absent means unrecorded” reading, never “absent means false”. The profiles chapter states the invariant: absence is explicit, silence proves nothing.

Deprecating a field means new records stop emitting it; documentation keeps its semantics forever, because records that carry it are still alive in someone’s evidence locker.

The forbidden move is redefinition. A field’s meaning, once published, is frozen. If reality changes, a new field is added and the old one deprecated. The reasoning is evidentiary, not aesthetic: an expert certifying a 2026 record against 2030 documentation must never discover that robots_state meant something subtly different when the record was made. Semantic drift is how honest records become impeachable.

Algorithm transitions

Signature algorithms age on decade scales, and the format’s answer is already structural: every record names its algorithm and key id, the registry binds keys to algorithms and validity windows, and verification is always performed against the record’s own declared suite. A future transition therefore looks like key rotation, not migration: new records sign under the new suite, old records keep verifying under the old one, and the chain records the seam. What is deliberately excluded is re-signing history: a record’s evidentiary weight is anchored to its capture-time signature, and “we re-signed everything with a stronger key” is indistinguishable, to a sceptic, from “we rewrote everything”. If a legacy algorithm is ever catastrophically broken, the honest posture is the archival one: the chain, countersigned time, and distributed copies continue to bound the records’ integrity even where the primary signature’s strength has decayed.

The compatibility promise, in one paragraph

Any record ever validly issued remains verifiable by the tools of its era; any conformant verifier gives correct answers about any record whose envelope version it implements; unknown claims never break verification; meanings never change under a name; and every change to any of this ships as a numbered, diffable document. This paragraph is the whole promise, and everything above it is the mechanism. A format that keeps this promise for ten years becomes infrastructure; a format that breaks it once becomes a cautionary tale in someone else’s wiki.

Every objection to single-party capture eventually lands on the same square: the operator recorded it, the operator benefits, discount accordingly. The threat model answers with visibility machinery; this chapter covers the stronger answer, available whenever the world cooperates: a second, independent observer. Corroboration is where the format’s per-record honesty starts compounding into something no single witness could provide.

What independence means here

Two records corroborate only to the extent their failure modes are independent. Real independence requires separate operators (different interests), separate keys (different compromise surfaces), separate infrastructure (different clocks, different classification code paths), and ideally separate vantage points. A publisher’s two mirrored nodes on one edge are redundancy, valuable operationally, nearly worthless evidentially. Two publishers independently recording the same crawler campaign, or a publisher’s ingress records meeting a provider’s egress ledger, are the genuine article: for both series to be wrong identically, their operators would have to be colluding, and collusion is a claim an opponent must then actually make, with its own burdens.

Agreement as evidence

When independent series agree, the agreement upgrades several fields from asserted to established:

• Time. Two operators with different clocks and different countersignature authorities bracketing the same events collapses the clock-manipulation surface to nearly nothing.
• Attribution. The same claimed agent, from the same network neighbourhoods, exhibiting the same verification outcomes across unrelated targets converts an anecdote into a campaign profile. This is precisely the shape of the industry measurement that found identified fetchers on disallowed paths at material rates: aggregation is corroboration at scale.
• Completeness. The chain’s inability to prove absence (its stated limit) softens when a second series covers the same window: gaps in one observer’s record that the other observer fills are now explained gaps, and selective capture must fool both books at once.

The strongest special case is cross-side corroboration: a publisher’s ingress record and a crawl operator’s egress ledger describing the same fetch. When adversarial parties’ records agree, the agreed facts effectively leave the dispute, and the argument narrows to what those facts mean. Narrowing arguments is what evidence is for.

The disagreement protocol

Corroboration’s value depends on taking disagreement seriously rather than averaging it away. When two series conflict, the protocol is diagnostic, in order: check verification of both records (a failed signature ends the inquiry); compare capture points and classifier versions, because most honest divergence is positional (a CDN-relayed address versus a direct one, pattern three); compare clock disciplines and countersignature bounds; and only then reach for the adversarial explanations. The format supports the protocol by making every one of those comparison points an explicit field. A disagreement between records that each carry their method, vantage, clock source and toolchain versions is a solvable engineering puzzle; a disagreement between two bare log files is a credibility contest, which is exactly the genre this instrument exists to retire.

Format hooks, present and planned

The observation profile already carries a corroboration structure (independent observation counts and divergence) for multi-fetch outbound capture. For cross-operator corroboration the working pattern is reference by digest: a bundle simply includes the corroborating segments, and any verifier confirms both chains and compares the events. The planned refinement, deliberately deferred until practice justifies it, is a corroboration assertion: a signed statement by one operator that its record N describes the same event as digest D from another operator’s chain, giving the pairing itself tamper-evidence. The deferral is the versioning discipline at work: fields are added when deployments demonstrate the need, not when a chapter can imagine it.

The strategic point

Corroboration is the format’s network effect, stated without the marketing register: each independent deployment makes every overlapping deployment’s records stronger, because the probability that campaigns visible to both are artifacts of either collapses. A market where one publisher runs a node produces exhibits. A market where a hundred publishers, three intermediaries and two providers run nodes produces something closer to a shared factual record of machine traffic, which no single participant controls and every participant can verify. That is infrastructure in the exact sense this wiki uses the word, and it is reached one boring deployment at a time.

The capture node is a small piece of software with one unusual requirement: it must be able to explain itself, in detail, to a hostile expert. This article is that explanation, written before the argument starts. It covers the mirror mechanism, the node’s observability boundaries, its failure behaviour, and the two deployment directions.

The mirror principle

The node never sits in the serving path. It sits beside it, receiving a copy of traffic that the edge was going to serve anyway. In the reference deployment this is a single reverse-proxy directive: the edge mirrors each request to an internal endpoint where the node listens, and the mirror subrequest’s fate is decoupled from the original by design. The visitor’s request is answered exactly as it would be with no node present.

This buys the property the method group records as interference: none, and it is worth being precise about why it matters twice over. Operationally, it means the node cannot take the site down: if the node dies, mirroring fails silently and serving continues. Evidentially, it means the record describes a visit that was not shaped by being recorded: no added latency profile, no altered responses, no honeypot dynamics unless deliberately deployed. An expert probing “did observation alter the observed” gets a clean architectural answer, not a statistical argument.

The trade, stated honestly: a mirror is a copy, and a dropped copy is an unrecorded visit. Passive capture chooses serving integrity over capture completeness, and the format’s answer to the completeness question is accounting, not architecture.

What a mirror can and cannot see

A capture node’s knowledge boundary is set by where it stands, and the record must never imply otherwise.

• It sees decrypted requests, because it stands behind the edge’s TLS termination. It records the request line, headers, peer address as presented to the edge, and response disposition. Where the edge is a third-party CDN, the “peer” may be the CDN’s own address with the client’s address in a forwarded header; the record stores both and labels which is which, because the difference is exactly the kind of thing cross-examination finds.
• It does not see intent, principals, or downstream use. The attribution ladder ends at network evidence; who operates the network is inference, and inference lives outside the record.
• It does not see other people’s edges. One node instruments one property. Statements about industrial-scale behaviour across the web belong to aggregated analyses, not to any single record.

Completeness accounting

Since cryptography cannot prove a negative (“no visit went unrecorded”, see chain mechanics), the node makes completeness measurable instead:

• Continuous sequence numbers make the record count per interval a checkable figure.
• Heartbeat records: the node periodically issues a signed record of its own liveness and counters (requests mirrored, records issued, records dropped, by reason). A silent gap in heartbeats is itself evidence of an outage window, honestly bounded.
• Reconciliation hooks: mirrored-request counters can be compared against the edge’s own access-log line counts. The two systems fail differently, so agreement between them is meaningful, and a divergence is a flag raised by the operator’s own records rather than discovered by an adversary.

The doctrine: the instrument does not claim perfect capture. It claims known capture, with the gaps on the record.

Classification: deciding what to keep

Recording everything is neither necessary nor wise; a publisher’s evidence interest is machine traffic, and storage of ordinary visitor traffic is pure liability. The pipeline runs: cheap syntactic filters first (known bot tokens, protocol fingerprints), then the verification ladder for anything claiming or resembling a crawler (forward-confirmed rDNS, published-range checks), then operator policy: record verified crawlers, record claim-failures (agents whose verification failed, which are often the interesting ones), drop confirmed human traffic unrecorded.

Two rules keep classification honest. What is dropped is dropped before recording, so the chain contains no ghost of it and retention policy stays clean. And the classifier’s version and ruleset identifier travel in the method group, because “why was this visit recorded” must have a reproducible answer years later.

Storage, retention, export

Records are append-only by construction; the store’s job is to preserve chain order, serve segment exports, and enforce retention. Retention follows the rights group’s retention_max_days where terms impose one. Export produces a self-contained bundle: the record segment, the issuer registry snapshot it verifies against, and any countersignatures, so that the bundle verifies on an air-gapped laptop. The export path is a first-class feature because the product’s entire purpose is to be handed to someone hostile.

The ledger deployment

Point the same node at a crawler fleet’s egress instead of a publisher’s ingress and the instrument becomes the provider-side ledger: every fetch the fleet makes is recorded under the observation profile, and, critically, every fetch the fleet declines is recorded as a refusal with the reservation that triggered it. Architecture is identical, evidentiary direction is inverted, and the refusal records are the exhibit a copyright-policy inquiry actually wants. One codebase serving both sides is not an accident; it is the symmetry argument made executable.

Failure modes, enumerated

For the record, the honest list: mirror drops under load (bounded by accounting); node crash (serving unaffected, gap bounded by heartbeats); clock trouble (bounded by time discipline); key exposure (bounded by custody and revocation); malicious operator (the subject of the threat model, and the reason the format assumes the operator is interested rather than saintly). None of these is hidden behind the word “reliable”. An instrument that lists its failure modes in public is harder to ambush with them.

Half of every machine-traffic dispute is about conduct. The other half is about what the rules were at the moment of the conduct, and that half is routinely lost for a mundane reason: nobody can prove what the rules said at 09:14 on a Tuesday eight months ago. Policy files change, licensing signals get added, prose terms get revised, and by dispute time the parties are arguing from memory and archive scraps. The instrument’s answer is a discipline: terms are captured as data, contemporaneously, pinned by digest, alongside every recorded event. This article covers what exactly gets captured and the semantics that make it meaningful.

robots.txt, with the folklore removed

The robots exclusion protocol was standardised in 2022 after three decades as a de facto convention, and the standard settles several points that dispute narratives regularly get wrong.

Group matching is by agent token, not by string equality. A crawler is expected to match the group whose User-agent line best matches its token, falling back to the wildcard group. The practical consequence: whether a fetch was “disallowed” depends on which agent it claimed to be. The ingress record therefore stores robots_state as an evaluation for the claimed agent, computed by the published matching rules, together with the file digest, so the evaluation is reproducible by anyone from the pinned file plus the claim.

Longest-match wins between allow and disallow. The standard resolves conflicting rules by path-match length. An evaluation that ignores this produces wrong answers on real files, which is why the node embeds a conforming evaluator and records its version.

The file is advisory, and the law knows it. A prominent US ruling treated robots.txt as a sign rather than a technical barrier, foreclosing circumvention theories built on it (the case-law digest). This does not reduce its evidentiary value; it specifies it. A pinned robots file plus a recorded fetch against it is proof of machine-readable notice and of conduct after notice, which is the factual predicate for contract, copyright and regulatory theories. The record proves notice; counsel picks the door.

Fetching the file is part of the conduct. Well-behaved crawlers fetch robots.txt before crawling; several documented stealth-crawling episodes involved fetchers that never requested it at all. An ingress node records requests for robots.txt like any other machine visit, so “they never even asked” becomes a provable statement rather than an inference from absence.

Beyond robots: the reservation stack

The 2025-2026 period layered several machine-readable signals on top of the exclusion protocol, and the license_signals field exists to capture all of them as found:

• Licensing-standard declarations. The really-simple-licensing family lets a site publish terms, including payment terms, in machine-readable form referenced from robots.txt. A pinned copy of the declaration is a pinned offer, which changes the character of a subsequent uninvited fetch.
• TDM reservations. The EU text-and-data-mining regime makes machine-readable reservation the trigger for the Article 53 obligations around copyright policy. The reservation’s presence and form at fetch time is exactly what a provider must honour and a rightsholder must prove; both needs are served by the same pinned capture.
• Response-level signals. Header and meta conventions expressing no-training preferences, plus content credentials embedded in assets. Individually weak, but the record’s job is inventory, not advocacy: everything reservation-shaped that was in force gets captured, and weight is argued later.
• Prose terms. The rights group pins terms_url and terms_sha256: the human-readable terms as they stood. Prose is where enforceability usually lives; the digest is what kills the “the page said something different back then” argument.

The pinning discipline

Mechanics matter here, because a sloppy pin is worse than none.

1. Capture from the same vantage as the visit. The terms recorded are the terms as served by the property at that moment, fetched by the node itself, not retrieved later from an archive. Archives are corroboration, not source.
2. Digest everything, store once. Policy files change rarely relative to traffic; the node stores each distinct version once, addressed by digest, and each visit record references the digest in force. Storage stays negligible while every record stays complete.
3. Change events are records too. When the node observes a new robots or licensing version, the transition itself is issued as a signed record: old digest, new digest, observation time. The terms history becomes its own tamper-evident series, which is precisely what “the licence lapsed at midnight” disputes need.
4. Evaluation is stored next to inputs. robots_state is derivative; the file digest and claimed agent are primary. Storing both means a challenged evaluation can be re-run in front of anyone, and an evaluator bug, if one is ever found, damages the derived field without contaminating the primary evidence. Layered honesty of exactly the kind the format’s rationale requires.

What this makes possible

With terms-as-data running, statements that are currently unprovable become mechanical: this agent fetched this path 4,112 times during a window in which the pinned policy disallowed it for that agent token; the reservation signal was present in every one of those windows; the policy transitioned twice that quarter and the conduct did not change. No memory, no reconstruction, no archive archaeology: a chained series of signed observations, each carrying the rules of its own moment. That sentence is the product, and every clause of it depends on pinning done properly.

The architecture chapter describes one node beside one edge. Real deployments come in five shapes, and each shape forces a different set of decisions about placement, keys, and what the records will be able to say. This chapter is the field guide.

Pattern one: the single property

The reference case: one site, one edge, one node, one key. Placement is a mirror directive on the origin server; the chain is one series; the key belongs to the property’s operator. The single decision that matters here is classification policy: which traffic to record. The default posture, record verified crawlers and verification failures, drop confirmed human traffic, keeps the evidence sharp and the privacy exposure minimal.

Pattern two: the publisher fleet

Twenty properties under one publisher. The choice is one chain per property versus one central chain, and the answer is firmly per property: chains should follow the boundaries of legal interest, because exhibits are requested per property, retention rules may differ per property, and a single merged chain forces every future export to drag unrelated properties’ existence into scope. Keys may be per property or shared at the operator level; per property is cleaner for the same reason, at the cost of registry size, which is trivial. What is centralised is monitoring, the heartbeat records, and the export tooling. Fleet deployments also unlock the first corroboration opportunity: sibling properties observing the same crawler campaign produce independently signed series whose agreement is itself evidence.

Pattern three: behind a CDN

Most publishers of size sit behind a content delivery network, which changes what the origin-side mirror sees: the peer address is the CDN’s, with the client’s address relayed in forwarded headers. The record stores both, labelled as what they are, and the source group’s verification ladder runs on the relayed address with an explicit note that it is relayed. This is honest and slightly weaker than direct observation, and the chapter’s advice is to say so rather than hide it: an expert who finds the distinction in the record trusts the rest of it more, and an expert who finds it concealed trusts none of it. Operators wanting direct observation can additionally capture at the CDN’s edge where the CDN offers log push with client transport details, treating that stream as a corroborating unsigned source rather than a record: the signed record remains the origin node’s own observation.

Pattern four: the provider egress ledger

The mirror turned around: a crawl operator records its own fleet’s fetches and refusals under the observation profile. Two decisions dominate. First, completeness posture: unlike a publisher, a provider wants to demonstrate coverage, so the ledger runs in record-everything mode and the accounting machinery reconciles against the fleet’s request counters, because a compliance ledger with unexplained gaps is worse than none. Second, refusal semantics: refusals are first-class records carrying the reservation that triggered them, since the refusal series is the exhibit an Article 53 inquiry actually wants. Egress ledgers also want tighter countersignature intervals: the party most suspected of self-serving records benefits most from an external clock.

Pattern five: the agent platform

Autonomous agents fetch on behalf of many principals, which adds a dimension no other pattern has: records must be attributable per principal without leaking one principal’s activity into another’s exports. The deployment answer is chain partitioning by principal, session-profile capture at the platform’s boundary, and export tooling that produces per-principal bundles. The platform holds the keys, and the principals inherit the evidence: an inversion of the publisher case worth being explicit about in the platform’s terms.

Decisions that recur everywhere

Across all five shapes, four knobs always need a deliberate setting, and the honest defaults are:

• Retention: follow the rights group’s constraints where terms impose them, and set an explicit policy where they do not, because “we keep everything forever” is a liability posture, not a strength (privacy chapter).
• Countersignature cadence: proportional to stakes; daily heads for active-dispute properties, weekly for archival ones.
• Witnessing: at minimum, periodic head digests delivered outside the operator’s own custody; the chain chapter lists the ladder.
• Classifier updates: versioned, recorded in method, and never applied retroactively; a reclassification is a new observation policy, not a rewrite of old records.

The pattern behind the patterns: place chains along the lines where legal interest divides, place keys with whoever answers for the records, and let everything operational centralise. Deployments that follow those three sentences export cleanly years later; deployments that do not, spend those years explaining themselves.

A capture node records network traffic, and in Europe the first professional to review that sentence is a data protection officer. This chapter is written for that review. It takes the strict view throughout, because an evidence instrument that gets breezy about data protection hands its opponents a discrediting theme, and because the strict view turns out to be comfortably survivable. The usual disclaimer applies with force here: this is design documentation, not legal advice, and a deployment’s actual assessment belongs to its controller.

What the records actually contain

Start with the inventory, because the analysis is impossible without it. An ingress record contains: an IP address and its network metadata, a user-agent string, a requested path, timing, content and policy digests, and verification outcomes. Under the GDPR’s expansive reading, an IP address can be personal data, and a user-agent string contributes to device fingerprinting. So the strict assumption is: ingress records can contain personal data, and the deployment is processing it. The comfortable observation follows immediately: the processing is narrow, purpose-bound, and overwhelmingly concerns traffic that is not natural persons at all, which is where design does its work.

Minimisation by classification

The classification pipeline is the privacy architecture, not just the relevance filter. The node’s default posture records two categories: traffic that verifies as an operator’s declared crawler, and traffic that claims or resembles machine access and fails verification. Confirmed human browsing is dropped before a record exists, unrecorded and unrecoverable. Three properties of this posture matter to a DPO:

• The target population is corporate infrastructure. A forward-confirmed crawler address is a datacenter machine operated by a company; records about it are about an enterprise’s conduct, which is the GDPR’s easiest terrain.
• The hard cases are the point of the product. Laundered scraping through residential proxies means some recorded addresses belong to consumer connections whose holders are unaware participants (the attribution chapter). These records are precisely the evidence of the laundering, and the analysis below has to carry them honestly, because they cannot be filtered out without deleting the product.
• Minimisation is provable. Because dropping happens pre-record, the chain itself documents the policy: a body of records containing only machine-classified traffic is its own compliance exhibit.

The legal basis

For a publisher-side deployment, the natural basis is legitimate interest: establishing, exercising and defending legal claims, protecting property against unauthorised industrial access, and complying with the evidentiary demands of licensing and regulation. The balancing test runs favourably for exactly the reasons above: the interest is weighty and recognised, the data is minimal and purpose-locked, the subjects are overwhelmingly corporate actors, and the residual natural-person exposure (proxy exits) exists because a third party routed industrial activity through consumer connections, a fact that itself weighs in the balance. Deployments should document this assessment; the format helps by making the processing description concrete, versioned and small.

Two disciplines keep the basis clean. Purpose lock: records exist for evidence and compliance, and the deployment must not quietly repurpose them into analytics or marketing, because purpose creep is what collapses legitimate-interest arguments. Access control: records are evidence, held like evidence; the export bundle, not the raw chain, is the unit of disclosure.

Retention, deletion, and the chain

Retention is where evidence instincts and privacy instincts genuinely collide: evidence wants forever, minimisation wants brevity. The resolution is honest tiering. Records tied to live interest (an active dispute, a notified claim, a regulatory window) are retained under the legal-claims basis for as long as that interest runs. Records with no attached interest age out under a published schedule, and here the chain’s append-only nature meets the right to erasure without contradiction: deletion removes record content while preserving the digest skeleton, so the series stays verifiable, the gap is visible and explained, and no personal data survives in the deleted span. A tombstoned record honestly says “a record existed here, was disclosed to retention policy, and its digest still binds the chain”. That sentence satisfies both the mathematician and the DPO, which is why the format was built to be able to say it.

Subject requests and the proxy problem

A person whose consumer connection was rented into a scraping pool could, in principle, surface as a data subject. The deployment’s honest answers: access requests are answerable because records are indexed and small; the data held is the observation itself, whose accuracy is cryptographically self-demonstrating; and erasure requests meet the legal-claims exemption where live interest exists, and the tombstone path where it does not. The scenario is rare, the answers are prepared, and preparing them in public, in this chapter, is itself part of the instrument’s posture: an evidence company that cannot answer a subject request cleanly has no business lecturing anyone about records.

The one-line summary for the DPO

Machine-traffic evidence, minimised at capture by verified classification, purpose-locked to legal claims and compliance, held under documented retention with erasure-compatible tombstoning, concerning corporate infrastructure in the overwhelming case and documenting third-party laundering in the residual one. Every clause of that sentence is a design decision described elsewhere in this wiki, which is what it means for privacy to be architecture rather than paperwork.

An evidentiary instrument that cannot articulate its own path into evidence is a hobby. This article maps that path for the two legal systems the instrument primarily targets, states the open questions plainly, and ends with the practical checklist the format was shaped around. One disclaimer, load-bearing rather than ritual: this is engineering literature about legal terrain, not legal advice, and the mapping for any actual matter belongs to counsel.

The United States: authentication by process

The threshold question for any exhibit is authentication: is this what its proponent claims? For machine-generated records the modern route is the pair of self-authentication rules added to the Federal Rules of Evidence in 2017.

Rule 902(13) covers records generated by an electronic process shown to produce accurate results; 902(14) covers data copied from a device or system where authenticity is shown by digital identification, and its committee notes name hash verification as the paradigm. Together they were designed for exactly this shape of evidence: a describable process, a digest that pins the data, and a written certification instead of a live foundation witness.

The certification comes from a qualified person, and the bar is deliberately practical: someone who understands the system and can describe how it produces its output, not necessarily its author or operator. Three properties of the format exist specifically to make that certification writable:

• The process is public and small: mirror, classify, canonicalise, sign, chain (architecture notes). A certifier describes a published mechanism, not a proprietary black box.
• Verification is reproducible by the opponent: the certification invites checking rather than trust (the doctrine).
• The digests and signatures are the identification 902(14) contemplates, with the chain extending the identification across a series.

Authentication is only the door. Behind it wait the hearsay rules, and here machine records enjoy a structural advantage worth understanding: statements generated entirely by a machine process are generally treated not as hearsay at all, because there is no human declarant; where human-composed content rides inside a record, the business-records exception and the layered-hearsay analysis do the remaining work. The format’s separation of machine observation from human assertion in distinct fields is what keeps that analysis clean.

Also behind the door: the reliability challenge. A 902 certification can be met with a demand for a live witness, and courts facing unfamiliar technology sometimes insist on one. Federal rulemaking through 2025 and 2026 has been actively debating machine-generated evidence, with a proposed reliability rule sent back for redrafting; the climate is one of heightened attention to process reliability, which cuts for instruments with published mechanisms and against black boxes. The honest statement of the state of play: the doctrinal fit is strong, and the direct precedent is absent. No published opinion yet analyses a signed, chained capture record under these rules, in either direction. The first proponent will be arguing fit, with the pattern (digest, timestamp, described process, certification) firmly on their side.

The European Union: presumption and free evaluation

EU evidence law runs through two layers. The eIDAS regulation provides the floor and the ceiling for the cryptographic artifacts: electronic documents and non-qualified signatures and timestamps cannot be denied effect merely for being electronic, and qualified timestamps from accredited providers carry a statutory presumption of time accuracy and data integrity (the time article covers the mechanics). This is why the format’s countersignature slot is aimed at qualified authorities: for EU proceedings it converts the record’s most attackable field into the challenger’s burden.

The second layer is national procedure, where free evaluation of evidence is the norm and traditions differ: notarial and bailiff capture protocols in some member states, expert-mediated assessment in others, and a growing body of national recognition for cryptographic timestamping, including at least one court accepting a public-ledger anchor as proof of anteriority. The practical consequence for instrument design: an EU-bound export bundle leans on the qualified timestamp for presumption, keeps the full verification story available for the expert the court will actually ask, and never assumes one member state’s ritual travels to another.

The pattern beneath both systems

Strip the citations and both systems ask the same four questions, which is why one format can serve both:

1. Is the data what it was? Answered by digests and signatures, verifiable by the challenger.
2. When did it exist? Answered by chain position and countersigned time, with presumption where qualified.
3. How was it made? Answered by a published, small, describable process.
4. Who vouches, and with what interest? Answered by the operator’s key, the operator’s certifier, and a toolmaker deliberately absent from the trust chain.

The staged-exhibit rulings show what happens when any of the four goes unanswered.

The honest unknowns

• No direct case law on this record class exists anywhere yet. Analogy is strong; analogy is not precedent.
• A first-impression court may demand live testimony regardless of certification quality. The format makes that testimony easy to give; it cannot make it unnecessary.
• Self-serving capture, records made by an interested party on its own infrastructure, is a weight argument that survives admissibility. The design answer is the threat model’s visibility machinery, but weight is won in each courtroom, not in a specification.
• Jurisdictions outside these two systems vary widely, from statutory ledger-evidence regimes to strict notarial monopolies. The format records enough to serve them; the strategy targets the two systems above first.

The practitioner’s checklist

What the export bundle hands to counsel, mapped to the questions above: the record segment with chain continuity; the issuer registry snapshot it verifies against; countersignatures, qualified where obtained; the published process description at the version in force; the classifier and evaluator identifiers for reproducibility; and the operator’s custody-tier statement for the certification. Every item exists because a specific rule, ruling or deposition pattern demanded it. That is what it means for a format to be court-shaped rather than merely cryptographic.

An evidence instrument earns trust by publishing the ways it can fail. This article is the standing adversarial review: each attack on a capture record, classified by who mounts it, what it costs, what stops it, and what honestly remains. The attacks are grouped by the attacker’s position, because position determines power.

Attacker one: the third party disputing a genuine record

The commonest adversary: a party confronted with authentic records and motivated to discredit them.

“The record was altered after the fact.” Defeated mechanically. Any post-issuance edit breaks the Ed25519 signature over the canonical body; a coordinated rewrite breaks the chain against every previously shared copy. This attack fails against the mathematics and is the reason the mathematics exists.

“The process is unreliable.” Met with publication: the mechanism is small, public and reimplementable (architecture, canonical form), and the record carries its component versions. The attack degenerates into specific bug claims, which is exactly where an honest instrument wants the fight: specific claims are testable.

“The claimed identity is wrong; that was not our crawler.” Not an attack on the record at all, and the format’s discipline makes this visible: the record never asserted an identity, only a claim, a network, and verification outcomes (attribution). A challenger disputing operatorship is disputing an inference the record deliberately never drew.

Attacker two: the interested operator

The serious case, because the operator controls the node, the key and the environment. The design assumes the operator is a party with a stake, and builds accordingly.

Fabrication: inventing visits that never happened. The operator can sign anything; nothing cryptographic prevents a lie at capture time. What bounds it is the corroboration surface a fabricated visit must survive: the peer network claimed must make sense against external routing reality; volumes must reconcile with edge logs and any CDN’s independent telemetry; the fabricated series must sit consistently inside a chain whose head may be countersigned and whose segments may already sit in other hands. A single invented visit is cheap; a litigation-grade body of invented traffic that survives discovery against the counterparty’s own logs is expensive and fragile. Residual risk: nonzero, stated plainly, and shared with every recording technology from CCTV to server logs; the format’s contribution is making fabrication brittle rather than impossible.

Selective capture: recording the convenient, dropping the rest. The chain closes seamlessly over an unrecorded event, so this is the fundamental limit (chain mechanics states it formally). Bounds: continuous sequence numbers make record counts checkable; heartbeat records bound outage windows; reconciliation against edge counters converts “trust me it was everything” into an arithmetic claim. And an adversary in discovery holds the other half of the traffic picture: selective capture that contradicts the counterparty’s own logs impeaches the whole series, which is a catastrophic cost for a marginal gain. Residual risk: real for operators who control all corroborating systems; materially smaller for the typical deployment behind third-party infrastructure.

Backdating: manufacturing history. Bounded from three directions: chain order, countersigned heads with qualified time, and every previously distributed copy. The rewrite must beat all three simultaneously; the arithmetic is walked in the chain article’s interactive.

Clock manipulation at capture. Subtler than backdating: run the node with a skewed clock so genuine events carry wrong times. Bounded by consensus time sourcing recorded in observed_at_source, by countersignature intervals that sandwich the drift, and by the mundane fact that skewed capture times diverge from every correlated external system. Residual: small skews inside a countersignature interval are hard to exclude; the format records enough to expose them when correlation becomes available.

Staged traffic: real records of theatrical visits. The operator hires a crawl against itself and records it faithfully. The records are true and the narrative is false. No format detects intent; what the format does is preserve the forensics that unravel staging in discovery: full network evidence per visit, terms state, timing patterns. Staging leaves statistical fingerprints, and a record system this granular preserves them against its own operator. Residual risk: real, bounded by discovery, and identical to the risk borne by every category of party-generated evidence courts already weigh.

Attacker three: the compromiser

Key theft. Covered in custody: hardware tiers raise the cost, revocation separates past from contamination, countersigned history survives the incident. The exposure window between compromise and detection is the honest residual, shared with all signature systems.

Node compromise. An attacker owning the capture host can do everything the malicious operator can, with the same bounds, plus one more defender: divergence between the node’s output and the operator’s other telemetry now works for the operator. Environment attestation is the designed hardening for high-assurance deployments (design rationale).

Upstream spoofing. Feeding the mirror falsified traffic requires either edge compromise (equivalent to node compromise) or network-path injection against TLS-terminated connections (equivalent to breaking the transport the whole web relies on). Not the weakest link; listed for completeness.

What the instrument never claims

The boundary statements, collected: a record proves an observation at a point, not global truth; it cannot prove absence of events; it cannot name principals behind networks; it cannot trace content downstream of the visit; and it cannot manufacture credibility for an operator whose other conduct destroys it. Every one of these limits appears in the product’s public materials, because an instrument that overstates its guarantees hands its opponents the weapon of its own marketing. The bet underneath this entire page is that in evidence, published limits are a strength: the expert who reads this article and fails to find an undisclosed weakness has just done our credibility’s work for us.

Records earn nothing while they sit in the chain. Their entire value is realised at the moment of export: the day a segment leaves the operator’s custody and lands in front of counsel, a counterparty, a regulator, or an opposing expert. This chapter specifies that artifact, because an export that forces its recipient to trust anything has failed the doctrine, and an export that cannot survive a hostile walkthrough has failed the product.

Contents of a bundle

A conformant bundle is a self-contained directory that verifies on an air-gapped laptop. It contains, and its manifest inventories by digest:

1. The record segment. The records at issue, plus enough chain context to demonstrate continuity: the segment’s boundary records and the sequence numbers that frame it. Records outside the disclosed scope appear as digest-only stubs, which preserves chain verifiability without over-disclosing, and dovetails with the tombstone mechanics when retention has already pruned content.
2. The registry snapshot. The issuer’s key registry as of export, covering every key_id the segment references, including retired keys (custody chapter).
3. Countersignatures. Every timestamp countersignature bearing on the segment, qualified ones flagged as such with their authority identified (time chapter).
4. Referenced artifacts. The pinned terms the records cite: robots.txt versions, licensing declarations, prose terms, each stored once by digest (terms chapter).
5. The process documents. The format specification, the capture-node description and the classifier ruleset, at the exact versions the records name. These are what the qualified person certifies against.
6. The manifest itself, digesting all of the above, so the bundle as a whole is one more integrity-checkable object.

Nothing in the bundle is a conclusion. It is observations, keys, time, terms, and process, which is all a court needs and all the format permits.

The walkthrough the other side runs

The bundle is designed around the assumption that its most important reader is hostile. Their expert’s session, in order: verify the manifest digests; run the open verifier across the segment (signatures, canonical forms, chain links); confirm key validity windows against the registry; check countersignature bounds; re-run the robots_state evaluations from the pinned files and claimed agents; and diff the process documents against the published versions. Every step is mechanical, none requires contacting the operator or the toolmaker, and the entire session is scriptable. The design goal is precise: the opposing expert should finish the walkthrough with nothing left to ask for except things the format honestly does not claim. What remains after a clean walkthrough are the weight arguments, self-serving capture, selective capture, staging, and those are answered where they should be, in the threat model and by corroboration, not by the bundle pretending they do not exist.

The cross-examination it is built for

The recurring questions, and where the bundle already holds the answer. “Who made this record?”: the operator, under this key, with this custody statement; the toolmaker is nowhere in the chain of proof. “Could you have edited it?”: not without breaking these signatures or orphaning these shared checkpoints; here is the arithmetic. “Could you have omitted visits?”: omission is the stated limit; here are the sequence numbers, heartbeats and reconciliation counters that bound it. “Why should we trust your clock?”: no one asks you to; here are the countersigned bounds. “What does this record prove?”: exactly its fields, no more; the inference layer is the expert’s, and the record deliberately contains no conclusions to attack. An exhibit whose answers are pre-committed in public documentation deprives cross-examination of its favourite weapon, surprise, and that is the quiet reason this wiki exists at all.

Operational notes

Exports are events worth recording: a bundle’s creation is itself issued as a record (what was disclosed, to whom, when, manifest digest), which both serves later disputes about disclosure and, per the chain chapter, turns every export into a checkpoint pinning history. Bundles are immutable once issued; corrections are new bundles, referencing the old by digest. And counsel’s copy should be treated as the canonical checkpoint of record: boring custody habits, applied to the one artifact whose custody is the whole point.

The admissibility chapter maps the two systems the instrument targets first. Records, however, follow disputes, and disputes follow the internet, which respects no docket. This survey covers how the format’s evidence travels elsewhere: not exhaustively, and with the standing disclaimer at full strength, but thoroughly enough to justify the design’s jurisdictional agnosticism.

Common-law cousins

England and Wales admit electronic records under a pragmatic framework: authenticity is a matter of weight for the court, business records travel well, and civil procedure expects parties to agree electronic-evidence protocols. The practical carrier for a capture record is the expert report: a verification walkthrough exhibited by a qualified expert, which the export bundle was designed to make short. Notably, the UK hosted one of the defining engineered-exhibit rulings, so the contemporaneity argument lands with particular force there.

Canada and Australia run evidence statutes with explicit electronic-records provisions built on system-integrity presumptions: evidence that the recording system was operating properly grounds admissibility. A published process, versioned toolchain identifiers and heartbeat records are close to a purpose-built answer to that statutory question.

Civil-law traditions: the notarial route

Much of the civil-law world channels high-stakes evidence through officers of proof: notaries, bailiffs, court-appointed experts. France’s bailiff constat tradition, with its standards for digital observations, is the archetype (touched in the admissibility chapter); Italy, Spain and much of Latin America have cognates. The design consequence is composition rather than competition: a capture record does not replace the officer, it gives the officer something categorically better to observe. A bailiff who attends a verification session, runs the open verifier, and records the outcome has converted a party’s records into an official observation, and the format’s offline verifiability is what makes that session possible at all. Deployments expecting these venues should bias toward qualified countersignatures and keep bundles small enough for an officer’s session.

Germany deserves its own line: free evaluation of evidence, a strong court-expert tradition, and, as the case-law chapter records, courts already comfortable finding for rightsholders in AI matters. German proceedings will test process claims rigorously, which is the environment this wiki’s publication discipline was built for.

Statutory ledger regimes

A scattering of jurisdictions have legislated directly on cryptographic evidence: distributed-ledger timestamp recognition in Italian law, ledger-records statutes in several US states, and, most consequentially for practice, China’s internet courts, which since 2018 have accepted hash-verified and ledger-anchored evidence under published rules and handle it at volume. The design consequence is modest but real: the format’s countersignature list is plural and authority-agnostic, so a deployment facing such a venue can add a ledger anchor alongside the qualified timestamp without touching the envelope. The design rationale’s scepticism about ledgers as the spine coexists comfortably with using them as one more anchor where a venue rewards it.

Cross-border reality

Machine-traffic disputes are structurally transnational: the crawler, the proxy exit, the model operator and the rightsholder routinely sit in four countries. Three design properties do the heavy lifting when evidence crosses borders. Self-containment: a bundle that verifies offline survives translation into any procedure, because it asks nothing of local infrastructure. Authority pluralism: time attestation can be stacked (qualified EU timestamp plus ledger anchor plus plain TSA) to cover multiple venues’ preferences at once. Conclusion-freedom: because records state observations rather than legal characterisations, they do not import one jurisdiction’s legal concepts into another’s proceedings; the local expert supplies the local frame. The pattern echoes the versioning promise: the format stays small and neutral precisely so that bigger, locally varying machinery can be built on top of it without renegotiating the core.

What this survey changes at home

Almost nothing, which is the point. The instrument was designed against the strictest requirements of its two primary systems, and the survey’s finding is that those requirements are a superset of what most venues ask: contemporaneity, integrity, describable process, independent verifiability, credible time. A deployment that satisfies the threat model and ships clean bundles is, with local counsel and occasionally a bailiff’s afternoon, ready for more of the world than its operators are ever likely to need.

Every company in the evidence business faces the same question sooner or later, usually in a deposition: why should anyone believe you? This article explains why our answer is “you should not have to”, why that answer had to be structural rather than rhetorical, and what it implies commercially. It is the closest thing this wiki has to a founding document.

The witness trap

Consider the natural-seeming design: a trusted service captures traffic on customers’ behalf, signs the records with its own keys, and testifies to their authenticity when disputes arise. This is how most evidence vendors are built, and it works, up to a ceiling. The ceiling is that the vendor has become a witness, and witnesses are examined: their interest (they are paid by one side), their infrastructure (subpoenaable, compromisable, a single point of doubt), their jurisdiction, their personnel, their history. Every record the vendor ever signed shares the fate of the vendor’s credibility on its worst day.

The trap deepens in this particular market. The parties who could build capture at scale tomorrow, the CDNs, the bot-management vendors, the platforms, are all combatants: they block the traffic, meter it, sell access to it, or crawl it themselves. A record signed by a combatant inherits the combat. This is the structural reason the evidence layer stayed empty while blocking and metering matured (anatomy of the war), and it is not a gap a large player can close with a feature release, because the disqualification is positional, not technical.

The inversion

The doctrine inverts the natural design at every joint:

• The operator captures. The node runs on the interested party’s own infrastructure, and this interest is disclosed in the record itself rather than laundered through a “neutral” intermediary whose neutrality would then need defending.
• The operator signs. Keys belong to whoever answers for the records (custody). The toolmaker holds nothing and can vouch for nothing, which is precisely what makes the toolmaker unattackable.
• Anyone verifies. The format and verifier are open; the check runs offline against published keys (the design constraints). The adversary’s expert is the intended user.
• A qualified person certifies. Evidence practice supplies the human layer: a certifier who understands the published process and says so under the applicable rules (admissibility). The certifier is the party’s expert, chosen by them, examinable like any expert. The witness role is filled, but filled by the side that carries the burden anyway, using a mechanism courts already govern.

The result dissolves the deposition question. “Why believe LENTRA?” has no purchase, because no claim of ours is in the chain of proof. The questions that remain, is the process sound, was the key held properly, does the mathematics check, are all answerable by examination rather than trust. A witness asks to be believed. An instrument asks to be examined. The entire product is downstream of preferring the second sentence.

Precedent for the position

The instrument-maker position is not an invention; it is how forensic tooling already works. The makers of DNA sequencers, breathalysers, and disk-imaging suites do not testify that any particular sample is what it is; they publish validated processes, and operators and experts carry the evidentiary weight using those processes. The web-evidence market’s older segment works the same way: capture software operated by the litigant’s side, certified by qualified persons, accepted routinely. What was missing was this architecture applied to machine traffic, with cryptographic verification replacing vendor reputation as the integrity mechanism. We did not have to convince courts to accept a new trust topology. We had to notice that the accepted one had an empty seat.

The economics of neutrality

The commercial objection writes itself: the witness position looks monetisable (a service, recurring, sticky) and the instrument position looks leaky (open format, open verifier, keys we never hold). The response has three parts.

Neutrality is the moat, not the sacrifice. Every incumbent that could commoditise the capture layer is disqualified from the credibility layer by its own position. Neutrality cannot be added in a sprint; it must be chosen instead of a bigger business, which is exactly why it will not be chosen by anyone with a bigger business to protect.

Openness is distribution. Every independent verifier implementation, every opposing expert who learns the format, every regulator who checks a complaint’s records locally is adoption we did not pay for. Standards win the habits of the people who examine evidence; an examinable format recruits its examiners.

What is sold is the instrument, and instruments have owners. Nodes, deployment, hardening tiers, export tooling, certification support, the boring operational excellence around a public format. The revenue shape is the shape of every successful instrument company, and it composes with, rather than competes against, the trust architecture.

The discipline the doctrine imposes

Positions decay unless defended, so the doctrine has standing rules. The toolmaker never signs production records, never holds customer keys, never operates capture on a customer’s behalf in a dispute, and never publishes conclusions about any live matter. The format never gains a field for conclusions. The public materials never claim more than the threat model supports. Each rule has a price, and each price purchases the only asset this company actually has: the ability to hand its output to the most hostile reader in the room and say, in complete honesty, check.

Everything in this wiki stands on published work. This list collects it, with the two annotations that matter: what the thing is, and what role it plays in this method. It is organised by layer, and it doubles as a map of the method itself: a reader who works through this list can reconstruct most of our decisions, which is the intended effect.

Protocol layer

RFC 9309, the Robots Exclusion Protocol. The 2022 standardisation of robots.txt: agent-token group matching, longest-match rule resolution, fetch behaviour. Role here: the evaluation semantics behind robots_state; the reason a pinned file plus a claimed agent yields a reproducible allowed-or-disallowed answer (Terms as data).

RFC 9110-9112, HTTP semantics. The definitions of requests, headers and status codes that every capture field ultimately references. Role here: the reason claimed_agent is honestly described as a client-composed header and nothing more (Attribution).

HTTP message signatures and the web-bot-auth work. The signature mechanism for HTTP messages and the IETF effort applying it to crawler identification, backed by the major infrastructure operators. Role here: the anticipated top rung of the attribution ladder; the format records such signatures when present and is designed for the long transition during which they are absent.

Cryptographic layer

Ed25519 (RFC 8032). The signature scheme: fast, compact, misuse-resistant, universally implemented. Role here: the signature over every record (Design rationale).

RFC 8785, JSON Canonicalization Scheme. The IETF’s answer to deterministic JSON. Role here: the benchmark our four-rule canonical form is defined with and against, including the deliberate divergence on number handling (Canonical form).

RFC 3161, the Time-Stamp Protocol. Digest-based timestamping by an authority. Role here: the countersignature mechanism that bounds backdating and, in qualified form, carries presumption (Time).

Certificate Transparency (RFC 9162). The architecture of public append-only logs with Merkle inclusion proofs. Role here: the designed upgrade path for high-volume chains and public witnessing (Chain mechanics).

Legal layer

Federal Rules of Evidence 902(13) and 902(14), with committee notes. Self-authentication for records of reliable electronic processes and for data identified by digest, certified by a qualified person. Role here: the US door the export bundle is shaped for (Admissibility).

Regulation (EU) 910/2014, eIDAS, articles on electronic timestamps and documents. Non-discrimination for electronic evidence; statutory presumption for qualified timestamps. Role here: the EU door, and the reason the v0.2 envelope reserves qualified countersignatures.

Regulation (EU) 2024/1689, the AI Act, Article 53, with the AI Office template and the GPAI code of practice. The disclosure regime for general-purpose AI: training-data summaries, copyright policies, crawler disclosure. Role here: the regulatory demand curve for both sides of the instrument (the Article 53 article).

Directive (EU) 2019/790, Article 4. The text-and-data-mining exception and its machine-readable reservation trigger. Role here: what a license_signals capture is legally for.

The public litigation record, 2023-2026. The rulings and filings digested in the case-law article: the fair-use-with-provenance split, the robots-as-sign holding, the staged-exhibit discounts, the attribution pleadings. Role here: the design constraints written by judges.

Provenance and content layer

C2PA specifications. Content credentials: manifests signed at creation, an assertion vocabulary including training preferences. Role here: the adjacent standard for content provenance; our records export observation fields into its manifest rails, with the honest caveat that we are an observer, not an author (Standards page).

W3C PROV. The vocabulary for describing provenance relations. Role here: the export mapping that lets enterprise provenance systems ingest records without a bespoke adapter.

The really-simple-licensing family. Machine-readable licensing declarations referenced from robots.txt. Role here: a primary license_signals source, and the standard whose enforcement gap ingress records close.

Adversarial prior art

The zkTLS and web-proof literature. Protocols proving statements about TLS sessions without trusting the observer. Role here: the strongest alternative architecture for outbound observation, tracked and deliberately not yet adopted, for the positional and practice-maturity reasons argued in the design rationale.

The stealth-crawling forensics of 2025. The public honeypot investigations documenting undeclared fetching through rotated agents and networks. Role here: the empirical proof that the attribution ladder’s failure modes are exploited in production, not hypothetical (Attribution).

The measurement literature on AI crawling and referral collapse. Industry telemetry on bot-to-human ratios and answer-engine click-through. Role here: the aggregate loss picture behind the war’s anatomy; always cited as reported, never re-measured by us.

How to use this list

Reading order for a newcomer who wants the method from first principles: the war articles for the demand, then design rationale, canonical form and chain mechanics for the object, then time, keys and admissibility for the weight, then the threat model to try to break it all. The wiki’s standing invitation applies especially to this list: if something load-bearing is missing from it, that absence is a bug, and we would like the report.