# Day 04: Stop parsing prose. Make the model return a struct

*The working version of cartographer answered in paragraphs, so to get anything countable out of it I wrote a parser. Twelve lines of regex against the bullet list the model kept producing: a backticked name, a dash, a sentence. It worked on the first six files I pointed it at.*

*Then I reworded one line of the prompt and it returned nothing. Not an exception — an empty list. The model had drifted from a bulleted* `- name — role` *to a bolded* `**name**: role`*, and my regex had no opinion about bold.*

*The regex was not the mistake. The mistake was building on a format nobody had agreed to. Bullet points were a habit the model happened to have that week, and I had shipped a parser on top of a habit and called it an interface.*

*What replaces it is three things that only work together, and only the middle one is what people mean when they say structured output.*

---

## The Short Version

**Code for this day:** [cartographer at day-04](https://github.com/Harshavardhan17/cartographer/tree/day-04)

- Prose is **unfalsifiable by construction**. You cannot diff two runs of it, count what it found, or write a test that fails for the right reason.
- A parser over prose is **a contract nobody agreed to**. It breaks on a rewording, and it breaks by returning zero rather than raising.
- A JSON Schema on the request, a machine-readable description of the fields and types the reply must have, is **a request, not a guarantee**. Not every model behind an endpoint honours it, and the ones that do not fail differently from each other.
- So **the validator is the guarantee**, and it lives in your code beside the schema and the sentence explaining the schema. Split those three and they drift.
- Write the validator by hand, because **its error messages are prompt text**. They go straight back to the model on the repair turn.
- Repair works only if **the rejected answer stays in the transcript** next to the complaint about it. Retry without that and you get three identical failures.
- `True` is an `int` in Python, so **a** `line: int` **check accepts** `"line": true` unless you say otherwise.

---

## The parser you will write, and the day it returns nothing

Here is the shape of it. Everyone writes this, and it looks reasonable.

```python
import re

SYMBOL_LINE = re.compile(r"^[-*]\s+`(\w+)`\s*[-:]\s*(.+)$")

def symbols_from_prose(reply: str) -> list[tuple[str, str]]:  # Broken. Do not ship this.
    found: list[tuple[str, str]] = []
    for line in reply.splitlines():
        match = SYMBOL_LINE.match(line.strip())
        if match:
            found.append((match.group(1), match.group(2)))
    return found
```

A parser that raises is annoying. A parser that returns `[]` is a liar, and this one does so for every input it does not recognise — a file with genuinely no interesting symbols and a model that switched to bold are the same event as far as your program is concerned.

You cannot tell absence from failure to parse absence.

## The schema goes on the request, not in the prompt

Asking for JSON in the prompt is better than nothing, but it is not the mechanism. The endpoint takes a schema as a first-class field, and providers that support it constrain decoding, the token-by-token generation of the reply, to that schema rather than merely reading about it.

```python
RESPONSE_FORMAT = {
    "type": "json_schema",
    "json_schema": {"name": "file_report", "strict": True, "schema": REPORT_SCHEMA},
}
```

All of this lives in one new module, `src/cartographer/records.py`, beside the frozen dataclasses `Symbol` (name, kind, file, line, role) and `Finding` (claim, evidence, confidence) that the reply becomes. Two rules make a strict schema behave, and they have to hold at every level of nesting, which is exactly the sort of thing you forget on the fourth object. So build them once:

```python
from typing import Any

KINDS = ("module", "class", "function", "constant", "entrypoint")

def json_object(properties: dict[str, Any]) -> dict[str, Any]:
    return {
        "type": "object",
        "properties": properties,
        "required": list(properties),
        "additionalProperties": False,
    }

SYMBOL_SCHEMA = json_object(
    {
        "name": {"type": "string"},
        "kind": {"type": "string", "enum": list(KINDS)},
        "file": {"type": "string"},
        "line": {"type": "integer"},
        "role": {"type": "string"},
    }
)
```

Every property required, no additional properties. Optional fields are where a schema stops constraining anything: a model that finds nothing to say will simply omit them, and you are back to distinguishing absence from failure.

The schema constrains shape. It cannot carry meaning, and meaning is most of what you need — which of the five kinds a symbol is, that `evidence` should quote a line rather than paraphrase it, that two empty lists are a legitimate answer. That has to be prose, in the instruction, and it has to live in the same module as the schema it describes. Check the field names against [the provider's structured-output docs](https://openrouter.ai/docs/features/structured-outputs) before you change them; that is the part of the file that rots without anyone editing it.

## The validator is the guarantee

The schema is advisory in a way that is easy to miss when it works on the first model you try. Behind a router, the same code reaches models that decode strictly, models that treat the schema as a suggestion, and models that fence correct JSON in a code block. All three must end at the same typed record.

```python
from typing import Any, TypeVar

T = TypeVar("T")

class InvalidReport(ValueError):
    pass

def require(payload: dict[str, Any], key: str, expected: type[T]) -> T:
    if key not in payload:
        raise InvalidReport(f"missing required field {key!r}")
    value: object = payload[key]
    if isinstance(value, bool) or not isinstance(value, expected):
        raise InvalidReport(
            f"field {key!r} must be {expected.__name__}, got {type(value).__name__}"
        )
    return value
```

`require` fetches one field and checks its type: a missing key and a wrong type each raise an `InvalidReport` naming the field, and typing `value` as `object` makes the type checker demand the `isinstance` proof. `parse_report` strips any Markdown fence, parses the JSON, and builds every record through helpers like this, so the first problem found becomes one precise message. The `isinstance(value, bool)` line looks like paranoia and is not. `bool` subclasses `int`, so `"line": true` satisfies a plain `int` check and you end up with a `Symbol` whose line number is `True`. It will not raise anywhere. It will render as `line True` in your map three weeks later.

## Error messages are prompt text

There is no `jsonschema` dependency, and not to save a dependency: when a reply fails, the message you raise is not for you. It goes into the array as the next user turn.

`missing required field 'line'` is an instruction the model can act on. `KeyError: 'line'` is not, and neither is a library's generic path-based complaint about `$.symbols[0]`. A hand-written validator lets you write its error messages as prompts, which is worth more than the twenty lines it costs.

```python
def describe_file(path: pathlib.Path, attempts: int = 3) -> FileReport:
    messages: list[dict[str, str]] = [{"role": "system", "content": SYSTEM}]
    content = f"{INSTRUCTION}\n--- {path} ---\n{path.read_text()}"
    for attempt in range(1, attempts + 1):
        reply = turn(messages, content, RESPONSE_FORMAT)
        try:
            return parse_report(reply)
        except InvalidReport as error:
            print(f"[repair {attempt}/{attempts}] {error}", file=sys.stderr)
            if attempt == attempts:
                raise
            content = REPAIR.format(error=error)
    raise InvalidReport(f"attempts must be at least 1, got {attempts}")
```

`describe_file`, added to `src/cartographer/agent.py`, starts a fresh transcript per file and sends the instruction, a label and the file's text with the schema attached. A valid reply returns records. An invalid one is logged, and the next turn carries the repair prompt with the validator's message filled in; on the last attempt the error is re-raised, and the final line only runs if `attempts` was zero. `main` now prints aligned rows and writes `maps/records.json`. The loop reuses the same `turn` that appends both sides of the exchange, so the rejected answer is still sitting in the transcript when the complaint arrives. That ordering is the whole mechanism. The model must see its own bad output immediately above the complaint; without it the correction has nothing to attach to, and you pay for three near-identical failures.

The test worth writing, in the replaced `tests/test_agent.py`, is not "does it retry". It is: the second request contains the bad answer at position `-2` and the complaint at `-1`; `tests/test_records.py` adds seven offline tests, one per malformed reply. [Today's dev notes](https://github.com/Harshavardhan17/cartographer/blob/main/docs/devnotes/day-04.md) explain every line.

## Three ways to get a shape out of a model

|  | Prose plus a parser | "Reply with JSON" in the prompt | A schema on the request |
| --- | --- | --- | --- |
| **The contract lives in** | your regex, agreed to by nobody | one paragraph the model may skim | the request body, enforced while decoding |
| **How it fails** | silently, returning zero | fenced blocks, trailing apologies, a missing key | a provider that ignores the field entirely |
| **Right when** | never, for anything a program reads | always, as the floor under everything else | the shape matters and the provider supports it |

The columns are cumulative. The third does not replace the second, because the schema cannot explain itself, and neither replaces the validator.

## What I'd do

1. **Delete the parser before you write it.** If the output is going into a program, prose is not an intermediate format worth passing through.
2. **Keep the dataclass, the schema and the instruction in one module.** They are one contract wearing three costumes, and in three files they will disagree within a month.
3. **Make every field required and forbid extras**, at every level. Optional fields quietly reintroduce the ambiguity you paid a schema to remove.
4. **Write the error messages as prompts.** Name the field, say what was expected and what arrived. That string is the whole repair turn.
5. **Keep the rejected answer in the transcript.** The correction needs something to correct.
6. **Add a switch that turns the schema off** and check the validator still gets you a clean record. That run tells you what the schema is actually buying, measured instead of assumed.

---

*If this was useful, a clap helps other people find it. I would like to hear the failure side of this: what is the strangest thing a model has handed you inside a response that was supposedly schema-constrained?*

---

*All views in this post are my own. The scenarios are illustrative composites, and code examples are not drawn from any specific codebase.*