Contextum: the document pipeline to Markdown for AI agents

The agent opens a file — the model reads text

An agent receives a file from the user. That file now has to reach the model’s context, and here the pipeline breaks: a model does not read PDF. It reads text. Something has to happen between “the file arrived” and “the text is in the context” — and usually nothing does.

The classic path looks like this: the agent or the application picks a library that “can read PDF”, gets a string back and hands it to the model. The quiet corruption starts there. Tables fall out of the document, merged cells, colour, underline, super- and subscript, vector images, charts. Nobody reports any of it, because there is nothing to report: the library returned a string, the string looks sensible, and the agent answered.

A month later the user gets an answer with half a contract missing, and blames the model. The model is not at fault: it did exactly what it was given.

The second edge of the problem is volume. Even perfectly extracted text from a long document does not fit into a context window, and cutting “the unnecessary parts” out of a document is not safe — you do not know which unnecessary part will matter two questions later.

So an agent and a model need a layer that does three things: brings the document into a form the model can read, and shrinks it so that what matters fits into the window. And says what was lost in the process.

Why this is a separate service, not an agent function

Conversion is CPU-bound. Rendering pages, parsing tables and building Markdown take the processor core away from everything else. When the agent process does this, heavy work competes with orchestration for the same resources: model calls, dialogue and memory all end up waiting for a render to finish.

AnetLab Contextum moves exactly that layer — file handling — into a service of its own. Dialogue orchestration, memory, tools and model selection stay with the agent layer. The split is not architectural decoration: each side scales on its own terms and fails on its own terms.

The unit of scaling here is a role, not a process. The CONVERT role is responsible for conversion and nothing else, so if it falls behind, another CONVERT process is added and COMPRESS, EMBED, INDEX and VAULT stay untouched. Inside one process the concurrency limit is 4 jobs at a time; beyond that the CPU is the constraint. The gateway scales horizontally: job state lives in Postgres, so replicas are safe.

The gateway is the only component that touches the storage directory. Workers expose diagnostics and nothing else. Between them runs gRPC over a protobuf contract — a single source of truth for Go and .NET.

Five stages and a flag for honesty

The pipeline has five stages, and they are not equivalent:

Beyond the five stages, ANON ships separately as an optional worker — PII anonymisation: it finds and masks or removes names, phone numbers, addresses and identity document numbers before the text reaches any analytical processing. It is not part of the base contract.

While a stage from the table runs on the built-in simulator it has to return simulated = true, and the console shows a SIMULATION badge. This is not reporting, it is part of the contract: a stage that does nothing has to say so. Presenting invented bytes and tokens as measured ones is not an option.

The consequence for an integrator is stated plainly: Markdown (CONVERT) and compressed Markdown with chunks (COMPRESS) can be taken from Contextum as a source of truth. Embeddings, vector search and knowledge-base writes cannot, until those stages are replaced with real engines. Three of the five stages are stubs today, and that is visible in the response rather than in the documentation.

One more detail: PPTX is converted to the MARP dialect, which expresses more than GFM does — slide background colour, background image, text colour, deck size. A presentation therefore loses less, and that is a property of the format, not an oversight.

The loss registry: what did not survive Markdown

Markdown does not express everything. Whatever was in the source and did not make it into the artifact goes into the loss registry: colour, font, underline, super- and subscript, alignment, merged cells, charts, vector images. The registry is returned with the conversion result, written to the worker log and visible in the converter’s response.

The second half of the rule matters more than the first: something disabled by an option is not a loss. Formulas with includeCellFormulas = false are the user’s choice, not a conversion defect. The difference between “we could not express it” and “we chose not to express it” is fundamental, and the registry keeps that difference.

The point of all this is that the agent layer can make an informed decision. An answer that dropped half a contract is a defect. An answer that dropped a part the customer deliberately switched off is expected behaviour. Without the registry the two cases are indistinguishable, and both look the same: “the LLM did not understand the document”.

Delivery that cannot run a stage twice

The broker gives at-least-once delivery: a message may arrive more than once. That is not a fault-tolerance feature, that is the normal mode of NATS JetStream. Which means the receiver has to be idempotent, and this is solved not by an “already processed” flag but by three different mechanisms:

The claim table is keyed by dispatch_id, and it is deterministic:

dispatch_id = hex(sha256(job_id + "|" + role)[:16])

This is not an optimisation, it is a condition of correctness. A repeated publication after a restart has to produce the same identifier — otherwise the worker runs the stage again as a new one, and the second result simply overwrites the first. 32 hex characters are the canonical UUID without dashes, which is why the column is declared as uuid.

The default lease is 600 seconds, and that value has to be clearly longer than the maximum stage duration. The owner in a row changes in exactly two cases: the row is in DONE state (replay) or the lease expired (the owner died). In every other case the arriving party gets HeldElsewhere, and the delivery is skipped with a delay rather than executed on top of someone else’s work. Expired leases are released by a background cycle — otherwise lease expiry would only be visible at the moment of the next claim attempt.

Redis deserves a separate note: it is pure optimisation. When it is unavailable the error is handled as a cache miss, not as “already processed”. Guessing “seen” would silently discard real work, and that is exactly the class of failure you cannot notice from the logs.

Metrics: measurement, estimate and claim

Three different things live in this product, and they are easy to confuse. The separation is made in the contract, because in reporting these three numbers sit next to each other.

Measurement — bytes in and bytes out, stage duration, chunk count. These are facts.

Estimate — tokens. They are counted with the heuristic “1 token ≈ 4 characters”, and this is not the GPT, Claude or Gemini tokenizer. Any number signed “tokens” is an estimate based on text length. It is useful for a rough idea of how much context an agent will consume and useless for exact billing, and calling it “model tokens” would be a lie.

Savings claim — a property of the COMPRESS stage and of that stage alone. CONVERT honestly measures bytes and tokens but claims no savings: compression has not happened yet. The summary metrics come from the result of the last measured stage, not from the maximum across stages, and the summary carries two pointers to its source — outputStage and savingsStage. If an integrator builds an interface for “context savings”, it should read totals of the last stage: the maximum across stages produces a number compression did not earn.

We do not publish savings percentages until we have a measurement on a real corpus. The documentation contains an example JSON, and an example is not a measurement. A number that cannot be reproduced looks like confidence we do not have on a product page.

Who this is for

Contextum is a B2B product: it is bought by an organisation rather than by a person, and not per document. The buyer is a company that already has an LLM contour or is about to build one.

• Companies with a working LLM contour. The model is wired in, and documents still reach it as a plain string that has lost its tables and structure.
• Companies building an LLM contour. The contour is in the plan, and there is no document layer in it yet — which is exactly the moment to put one in.
• AI providers boosting their platforms. The file layer becomes part of what the provider ships to its users, and the quality of that layer becomes the provider’s own responsibility.
• AI portal applications and internal AI projects that work with documents. Contracts, specifications, invoices, regulations, reporting — they are brought to what the model can read before the model call, not after.

The honest limit: for a one-off case — one file, one answer — a library is enough, and it is cheaper. Contextum is needed where documents flow in bulk, where it matters what exactly was lost, and where conversion must not block the agent’s orchestration.

Boundaries we do not hide

Contextum is in POC status, and the list of limitations is part of the product, not a footnote. It answers the question “can this be embedded today”, and for some items the answer is no.

• No authentication. Not over HTTP, not over gRPC, not over NATS. Anyone who knows the gateway address can read and upload documents. Acceptable only inside a trusted network.
• No TLS or mTLS — the network is assumed trusted.
• The upload limit is 64 MiB, above which the request comes back as 413 TOO_LARGE.
• No recognition. The scope covers extracting the text layer. OCR and scanned pages are not claimed: the contract has neither a recognition engine nor an ocr loss kind, and the documentation does not describe what happens to a scan.
• PDF page anchors are a loss, not a feature. pdf-page-anchor sits in the loss registry next to color, font, chart and vector-image: anchors exist in the source PDF but may not survive into Markdown. How many pages, sheets and slides a document has is reported by unitCount and unitKind (page / sheet / slide).
• Progress is polling only. No webhooks and no SSE: state is fetched through GET /api/v1/jobs/{id}.
• No automatic artifact cleanup — the integrator needs a TTL collector.
• One process, one role. No vertical scaling and no dynamic role registration.
• The vector dimension and the embedding model are undecided — the INDEX stage and the index choice depend on them.
• Job state survives a gateway restart only when ODOC_POSTGRES_DSN is configured. Without it the state lives in memory and is lost, which is unacceptable for an integration with an agent layer.

One subtlety in the contract deserves its own paragraph. Cancelling a job is a request: CANCEL_REQUESTED means the request was accepted, not that the job has been cancelled. The final status becomes CANCELLED, but you can only see it on the next poll. Anyone who does not know this detail writes code that cancels a job and immediately reads status — and gets something other than what they expected.

Where we changed our mind

Three decisions changed along the way.

The summary comes from the last stage, not the maximum. The rule is fixed in the contract, and it was the subject of a separate correction: while the summary could still be assembled as the maximum across stages, an integrator risked showing the user “savings” that compression, not conversion, had produced. The maximum is a convenient metric and a wrong one.

Tokens are called an estimate. This used to be called tokens, and the wording drifted away from reality: what was counted was text length, not model tokenisation. The documentation now says literally that any number signed “tokens” is an estimate based on text length and that it does not work for exact billing. Renaming a metric in the description is not a detail — it removes the temptation to compare our numbers against a model’s limits.

Honesty is in a flag, not in prose. While a stage runs on the built-in simulator it returns simulated = true. We could have written in the README which stages are real, but prose diverges from code at the first change and nobody notices. A flag in the response is checked by machine.

On the default stage list: it includes all five, so that the stage order does not depend on the configuration of each deployment. The price is that the user sees more stages in the list than they can use as a source of truth. We kept the full default and separated honesty with a flag, but if the product ever goes to GA, that is the first thing to revisit.

Related Articles

• Building a RAG pipeline on .NET: a practical walkthrough
• RAG on .NET: an architecture that survives production
• Production RAG: a 50-point checklist for production
• RAG evaluation on .NET: Recall@5, MRR and the release gate

CTA

The product card with the version, status, license and the list of stages — Contextum.

Planning how to embed a document layer in an agent system — write to us. We can look at which stages you actually need, which limitations are critical for your setup and how this sits next to the orchestration layer. While the product is in POC, we will talk about what it does and what it does not do.