Classify a customer ticket in a format your application understands

You are developing a support tool. To organize requests, your application expects a category, priority, and product. With a conventional form, you would provide three dropdowns and validate their values server-side. But here, the user writes a free-form message: the model must identify all three pieces of information.

Can you simply ask it to “respond in JSON” and pass its answer to the rest of the application? Not quite. The model can add a sentence, omit a key, or propose a value your program does not recognize. And even when it follows the format perfectly, it can choose the wrong category.

We will therefore separate two problems: obtaining a response in a permitted form, then checking that it matches the ticket. For the first, we intervene during generation by restricting which tokens the model may choose next. For the second, we retain an independent check.

Download the complete project

The project runs locally with Python, Docker, and an NVIDIA GPU. It builds a classification experiment, not a web interface or ready-to-deploy support service. It makes no business decisions: classifying a ticket does not authorize refunding an order or modifying an account.

The problem and what we will build

Three fields, a limited set of answers

The corpus contains 240 fictional English tickets: 48 for prompt development and 192 for testing. The prompt is the set of instructions and text submitted to the model. Here, development fixes those instructions; it does not train the model's weights.

Every response must combine these three fields:

Field	What it represents	Permitted values
category	Request family	billing, delivery, technical, unknown
priority	Priority level	low, normal, high
product	Product concerned	P01, P02, P03, unknown

We retain these English names and values: they belong to the program's contract. For example, technical and technique are not interchangeable spellings for the validator.

To follow the mechanism, take an educational example added to the explanation, separate from corpus tickets and measured outputs:

My P02 device will not start. This is urgent.

For this example, we define the reference answer as category technical, priority high, and product P02:

{"category":"technical","priority":"high","product":"P02"}

This reference is an annotation: an expected answer defined independently of what the model will produce. In a real application, deciding what deserves high priority would require an explicit business rule.

There are four category choices, three priorities, and four products. The program can therefore enumerate 4 × 3 × 4 = 48 combinations. It writes each in a stable order without variable spaces: this produces 48 canonical strings, meaning 48 textual representations chosen as generation references.

The 48 possible answers and 48 development tickets have distinct roles. The first number comes from the contract; the second from the corpus split. Every ticket can be classified into one of the same 48 answers.

Requesting, constraining, checking: three different operations

An instruction asks the model to follow the format. A decoding constraint prevents choosing a next token that would leave permitted paths. A validator then examines the resulting response.

We therefore do not repair incomplete JSON afterward. At every step, the program retains only tokens extending one of the 48 canonical strings. The end signal becomes permitted only once a string is complete.

This small grammar is deliberately narrower than JSON. It allows neither extra free text, new keys, nor arbitrary amounts. General syntax is described in RFC 8259; our goal is a finite language for this specific contract, not a parser for all JSON.

Useful concepts before running the GPU

The model proposes the next piece of text

A language model produces a sequence of tokens. These are not necessarily characters or whole words: a token may represent part of a word or punctuation. The tokenizer converts text into numerical identifiers and allows conversion back to text. The Transformers tokenization documentation explains these splits.

At a given moment, the model receives the text already available and computes a score for each vocabulary token. That score is called a logit. In our workflow, choosing the highest score chooses the next token; argmax returns its index.

A logit is not a confidence percentage for the ticket's correct category. It helps choose the next text fragment, not validate a business decision.

The trie describes permitted continuations

A trie is a prefix tree. A prefix is simply the beginning of a sequence. Answers that begin the same way share the start of a path, then branch where their tokens differ.

Imagine autocomplete that permits only continuations from a closed catalog. After each choice, you advance through the remaining paths. The trie does not read the ticket or know the right category: it knows only which tokens can still lead to a permitted answer.

If A(s) denotes the tokens permitted after prefix s, we retain their logits and replace every other one with negative infinity, written -inf. Thus argmax chooses among admissible continuations, provided at least one has a finite score. We will follow a numerical example in the code.

Ending generation is also a decision

EOS, or end of sequence, is a special token indicating completion. It is not JSON's closing brace }. The program must distinguish “the object is complete” from “generation ended according to the intended rule.”

Allowing EOS at the beginning would let the model stop without writing anything. Here, EOS is offered only at a terminal trie node, after a complete sequence. A length limit can still interrupt execution before that point; we will measure this case too.

Three questions to ask of a response

Is it readable? Can the parser convert the text into a JSON value? Does it comply? Are all three keys and their permitted values present? Is it accurate? Do the values match the ticket annotation?

In our example, replacing P02 with P03 breaks neither syntax nor contract: P03 is a permitted product. Yet the response becomes incorrect. This is precisely the distinction the experiment makes visible.

Docker and GPU prerequisites

You should be able to read Python functions, lists, and dictionaries, use JSON and a terminal, and know Docker basics. Decoding concepts are explained along the way; no prior training experience is needed.

The validated environment uses Linux x86_64, Docker with Compose, NVIDIA Container Toolkit, and a 24 GiB RTX 3090. Allow 16 GiB of RAM and 16 GiB of disk space, excluding shared Docker layers. Video memory, or VRAM, belongs to the graphics card; it does not replace workstation RAM. No CPU execution is presented as equivalent to this GPU workflow.

The model is Qwen2.5-0.5B-Instruct, at fixed public revision 7ae557604adf67be50417f59c2c2f167def9a775. “Instruct” refers to its adaptation for following instructions; it does not guarantee error-free compliance. Transformers loads it locally without remote code. The model card at this revision provides further description.

The model already contains weights, numerical parameters acquired during training. Here we use them to compute outputs: this is inference. Building the trie or changing permitted tokens does not train those weights.

Extract the archive and open a terminal at its root. Every command in the article starts from this same directory. setup.sh prepares directories and writes only UID/GID to .env: these identifiers concern user and group file permissions.

acquire downloads public files, checks them, then places them under their final names only after validation, using atomic renaming. models/ retains this local version, called a snapshot; outputs/ receives measurements, and artifacts/ receives project checkpoints. Do not confuse these control files with new training.

sh scripts/setup.sh
docker compose config --quiet
docker compose build acquire gpu
docker compose run --rm acquire python scripts/acquire.py --config configs/minimum.yaml
docker compose run --rm gpu python scripts/check_environment.py --config configs/minimum.yaml

The five commands have different responsibilities. config --quiet checks Compose configuration. build builds the required images. acquire.py prepares model files. Finally, check_environment.py must pass before inference.

CUDA runs computations on the NVIDIA GPU here. A card being visible on Linux does not establish that Docker can use it. If CUDA does not initialize inside the container, start with the NVIDIA Container Toolkit documentation rather than modifying the decoder.

Existing Docker layers may shorten image building. This preparation time is not a model-decoding measurement.

Checking the program before the engine

Before asking the model to classify tickets, let us check the surrounding rules. The public suite covers the schema, trie, and failure scenarios. Run it after the GPU check:

docker compose run --rm gpu python -m pytest -q

A pass means the program's tested behavior matches expectations. It does not yet mean the model understands tickets. This is the same distinction as testing a form validator versus checking the accuracy of entered information.

Do not skip this stage to obtain your first JSON faster: plausible output would not reveal a forbidden branch accepted by mistake.

Data and output contract

Building possible answers without knowing the right one

To constrain the model, we need every permitted answer, not the expected answer for a particular ticket. The following function therefore receives neither tickets nor annotations:

def enumerate_canonical_outputs():
    return [
        json.dumps(dict(zip(SCHEMA, values)), separators=(",", ":"))
        for values in itertools.product(*SCHEMA.values())
    ]

In this snippet, SCHEMA maps field names to their value lists. The name does not mean it is a standard JSON Schema document: the code iterates over an enumerated contract here.

SCHEMA.values() supplies the choice lists. The asterisk in itertools.product(*SCHEMA.values()) passes them as separate arguments. The Cartesian product creates every combination, like three nested loops: a category, then a priority, then a product. The itertools.product documentation details this operation.

For a combination such as ("technical", "high", "P02"), zip(SCHEMA, values) pairs each name with its value. dict(...) builds the object; json.dumps(...) converts it to text. Finally, separators=(",", ":") enforces compact formatting without spaces between elements.

We obtain the promised 48 strings. We have not chosen the one appropriate for the ticket. Building a trie containing only its annotation would instead give the decoder the answer. That would be target leakage: the measurement would no longer evaluate what the model can identify.

Why fix the representation?

These two representations express the same object:

{"category":"technical","priority":"high","product":"P02"}

{
  "product": "P02",
  "priority": "high",
  "category": "technical"
}

Yet they are not the same text string. Our trie does not try to permit every equivalent formatting: it retains one canonical representation per combination. This simplifies generation paths without changing the information represented by the object.

The corpus also includes ambiguous wording, literal braces, and untrusted text attempting to change the instruction. The prompt asks that tickets be treated as data, never commands. That instruction alone is not complete protection: malicious text can still try to steer selection toward an incorrect but permitted value.

Now prepare the tickets, canonical objects, and prompt lock:

docker compose run --rm gpu python -m json_decode.cli prepare --config configs/minimum.yaml

At this stage, we fix the experiment's components. We are neither correcting a prediction nor teaching annotations to the model.

The core mechanism: baseline, trie, and generation

Comparing two paths on the same tickets

We begin with the unconstrained baseline. This is the comparison behavior: the same model receives the same prompt, but without trie-based vocabulary restrictions.

docker compose run --rm gpu python -m json_decode.cli baseline --config configs/minimum.yaml
docker compose run --rm gpu python -m json_decode.cli inspect-trie --config configs/minimum.yaml
docker compose run --rm gpu python -m json_decode.cli generate --config configs/minimum.yaml

baseline first runs the 48 development tickets and freezes the prompt before testing. Errors remain recorded: we do not adjust instructions after reading test results to make the comparison look better.

inspect-trie makes tokenization checks visible. generate runs both modes and preserves raw responses and stop reasons. The 192 test tickets are identical on both paths.

From strings to a token tree

Before generation, the project checks the text–token round trip to preserve the intended canonical strings. That check belongs to preparation and inspection: the class below contains no tokenizer. It receives identifier sequences already constructed.

class TokenTrie:
    def __init__(self, sequences):
        self.children = [{}]
        self.terminal = set()
        for sequence in sequences:
            node = 0
            for token in sequence:
                if token not in self.children[node]:
                    self.children[node][token] = len(self.children)
                    self.children.append({})
                node = self.children[node][token]
            self.terminal.add(node)

    def allowed(self, node, eos_ids):
        return sorted(
            set(self.children[node])
            | (set(eos_ids) if node in self.terminal else set())
        )

    def advance(self, node, token):
        if token not in self.children[node]:
            raise ValueError("Token forbidden by trie")
        return self.children[node][token]

self.children is a list of dictionaries. Each position represents a tree node. The first dictionary, at index 0, is the root. Within a dictionary, the key is a token identifier and the value is the next node's index.

For each answer, the loop starts again at the root. If a branch already exists, it follows it; otherwise, it creates a node. Shared beginnings are therefore not duplicated. At the end of each sequence, the current index is added to self.terminal.

The allowed() method supplies possible tokens from a node. It adds EOS identifiers only if that node is terminal. Numerical sorting stabilizes the returned list: it does not rank tokens by the model's preferences.

Finally, advance() follows an ordinary token through the tree and rejects a missing branch. EOS needs separate handling in the generation loop: it is a stop signal, not automatically a branch to traverse with advance().

Following a small tree by hand

Take two fictional sequences, [10, 20, 30] and [10, 40, 30], with 99 as EOS. These numbers only illustrate the mechanism: they are neither observed Qwen identifiers nor actual tokenization of our JSON.

Prefix already selected	Permitted next tokens	Why?
None	10	Both sequences begin this way.
10	20 or 40	The two paths branch here.
10, 20	30	Only one compatible path remains.
10, 20, 30	99, meaning EOS	The sequence is complete.

After 10, 20, choosing 40 would make completing either stored sequence impossible, so the program rejects it. Allowing 99 there would instead permit stopping halfway through the path.

The real trie applies exactly this principle to longer sequences from the 48 JSON strings. It does not infer their structure character by character from this illustration; it uses the model's verified tokenization.

Keeping useful scores, blocking the rest

The trie knows admissible tokens. The model supplies logits. The following function brings the two together:

def select_token(logits, allowed):
    if not allowed:
        raise ValueError("Impossible grammar: no allowed token")
    index = torch.tensor(sorted(allowed), device=logits.device, dtype=torch.long)
    scores = logits[index]
    if (
        torch.isnan(scores).any()
        or torch.isposinf(scores).any()
        or not torch.isfinite(scores).any()
    ):
        raise ValueError("No usable finite allowed logits")
    masked = torch.full_like(logits, -torch.inf)
    masked[index] = scores
    return int(masked.argmax()), masked

index places the permitted identifier list on the same device as the logits. scores = logits[index] extracts their scores. torch.full_like creates a same-shaped array filled with -inf, then masked[index] = scores restores only admissible values.

Here is an invented numerical example with a vocabulary reduced to four tokens:

Token identifier	Proposed logit	Permitted?	Logit after masking
0	9	No	-inf
1	2	Yes	2
2	5	No	-inf
3	-1	Yes	-1

Without the constraint, the model would choose token 0. With it, argmax returns 1, the identifier of the best remaining permitted token. It does not return its score 2. The PyTorch argmax documentation describes this behavior.

Why not replace forbidden scores with zero? Because allowed scores can be negative. A forbidden zero could then win. -inf excludes a possibility rather than merely assigning it a low score.

What do numerical checks protect?

An empty allowed list indicates a grammar that cannot continue. The function explicitly stops rather than selecting an arbitrary token.

It then examines allowed scores. NaN denotes an undefined numerical value; +inf is positive infinity. Their presence triggers an error. Having no finite score also triggers an error.

The nuance matters: an allowed score of -inf remains acceptable if at least one other allowed score is finite. The snippet therefore does not indiscriminately reject every non-finite value. It aims to retain at least one usable choice without letting a numerical anomaly become a decision.

Both modes use argmax. Unconstrained mode applies it to the entire vocabulary; constrained mode applies it after masking. This is a local, token-by-token choice, not a global comparison of all 48 complete answers. The constraint closes paths; it does not say which one correctly describes the ticket.

Measuring without confusing form and content

Transformers generation APIs expose decoding parameters and mechanisms. Their official documentation places this work in a broader setting. For this experiment, however, keep the archive's commands and code: the explanations do not require replacing the generation engine.

Reading the scoring function

row contains, among other things, generated text and the stop reason. label contains the expected object for this ticket. We compare the response with this reference only during evaluation:

def parse_result(row, label):
    try:
        value = json.loads(row["text"])
        valid_json = True
    except (ValueError, TypeError):
        value = None
        valid_json = False
    valid_schema = validate_schema(value)
    complete = row["finish_reason"] == "eos"
    return {
        "json_valid": valid_json,
        "schema_valid": valid_schema,
        "completed": complete,
        "length": row["finish_reason"] == "length",
        "exact_tuple": complete and valid_schema and value == label,
        "fields": {
            k: bool(complete and valid_schema and value[k] == label[k]) for k in SCHEMA
        },
    }

The first block tries to convert row["text"] using json.loads. If it fails, json_valid is False. We do not remove an introductory sentence, strip a Markdown block, or add a closing brace to rescue the result.

The validate_schema(value) check asks a second question: does the object meet the contract? Its implementation is in the project; this snippet shows its use, not every internal check.

The complete variable depends only on finish_reason == "eos". A length reason means the generation limit was reached. It prevents full success even if the text is already decodable.

Finally, exact_tuple requires all three conditions: EOS completion, a compliant object, and equality with the annotation. The “tuple” refers to the three business values; the response handled here remains a Python dictionary.

Equality value == label compares decoded objects, not byte-for-byte identity of raw text. The two JSON examples above, with different key ordering and spacing, therefore yield the same object for this comparison. Canonical generation is narrower than this equality criterion.

The detail that changes how field scores are read

Look at the start of the expression in fields: complete and valid_schema. A field earns a point only if the entire response complies and ends with EOS.

If the response contains two correct values and an allowed but incorrect product, those two fields can receive credit. But if the product is P99, outside the contract, all three fields score zero: the validator rejects the entire object.

Likewise, three exact values in generation stopped by length receive no field points under this protocol. These scores depend on compliance and generation completion; they are not a tally of every correct fragment visible in the text.

Comparing six responses to the same example

Return to our educational annotation technical / high / P02. The following table describes cases constructed to explain the function, not captured model outputs.

On mobile, scroll the table horizontally.

Cases A and D show why we retain the stop reason. B and C show the difference between an allowed but incorrect value and a forbidden value. Case F reminds us that an exact object does not necessarily have canonical formatting in unconstrained mode.

A detail about the parser used

In this snippet, json_valid means “accepted by json.loads with default settings.” Python accepts certain non-finite constants and retains the last value for a repeated key. The json module documentation explains these behaviors. This call alone therefore does not strictly check every RFC detail.

The 48 canonical strings introduce neither these constants nor duplicate keys. For more general input, an explicit policy would need to be defined and tested. That would be a project change, not a reason to silently modify reference responses or scores.

Measurements before results

Keeping every ticket in the calculation

For each mode, evaluate uses all 192 test tickets as its denominator. It calculates the valid-JSON rate, valid-schema rate, proportion of EOS endings, tuple accuracy, and accuracy for each field. No failed ticket is removed.

For example, the "Correct product" rate is the number of tickets for which fields["product"] is True, divided by 192. It is neither an average over only compliant responses nor a percentage obtained by asking the model to grade itself.

To check that the rule is clear, let us take an educational mini-batch of four responses: cases A, B, D and E from the previous table. Three texts can be decoded and comply with the schema: A, B and D. Three endings are EOS: A, B and E. Yet only A has an exact tuple.

We therefore obtain 3/4 = 75 % decodable JSON, 75 % compliance, 75 % EOS endings, but 1/4 = 25 % exact tuples. Category and priority receive credit for A and B, or 2/4 = 50 % each; product only for A, or 25 %.

This small batch is not a measurement of Qwen. It shows why each indicator answers a different question. Removing E, which cannot be decoded, would change the denominator and artificially improve several rates.

Running the evaluation and reading the observed measurements

docker compose run --rm gpu python -m json_decode.cli evaluate --config configs/minimum.yaml

The following values are locally observed measurements with this revision, this model, and this synthetic corpus. They do not predict the success rate on your company’s tickets.

On mobile, scroll the table horizontally.

What the result establishes — and what it does not establish

The first observation is clear: in this experiment, the constrained path produces 100 % valid JSON and 100 % compliant objects. All its generations end with EOS; none reaches the maximum length limit.

The second observation is just as important: only 48 of the 192 tuples are exact, or 25 %. The other 144 constrained responses respect the format but get at least one value wrong. The constraint solved the formatting problem for this batch, not the whole classification problem.

The product is correct in 75 % of cases, priority in 64.06 %, and category in 47.40 %. These rates cannot be added together to obtain a successful-ticket rate: the fields must be correct together, on the same ticket. The tuple score measures exactly that requirement.

The zeros in unconstrained mode call for greater caution. The field score already requires a compliant object and an EOS ending. A category score of zero therefore does not mean that the expected word never appears in raw responses. It means no response meets all the conditions for that field to receive credit. Reading the raw outputs is still necessary to distinguish a presentation issue from a content error.

Why can the trie guarantee form here without guaranteeing a correct answer? Because its paths are built from only the 48 authorized strings. This property assumes generation follows the trie through to an authorized ending. An interruption or numerical failure must remain a failure, not become a successful response by default.

Finally, the table does not let us attribute errors to a single cause or conclude that one mode is faster than the other. It reports observations from this protocol without isolating the effect of model size, the prompt, or each family of tickets.

Checking rejected branches

A demonstration where everything appears to go well is not enough. You also need to check operations that must fail: that is the role of negative cases.

The project checks a language that cannot be continued, forbidden prefixes, and the EOS authorization rule. Run these checks after the evaluation:

docker compose run --rm gpu python -m json_decode.cli negative-cases --config configs/minimum.yaml

For a web developer, this is equivalent to a test that deliberately submits a forbidden value to a form. The test passes when the expected rejection occurs, not when the invalid input is accepted.

If an absent branch becomes traversable or EOS becomes possible too early, the mechanism is broken, even if a few tickets still produce attractive JSON. An expected error that is no longer detected is a regression.

What we learned

We transformed a three-field contract into a set of responses, then into token paths. The model proposes scores; the trie identifies permitted choices; the selector chooses among them. None of these stages consults the ticket annotation to supply the correct answer.

At the end, responsibilities remain separate. The parser checks whether the text can be read. The schema checks the permitted keys and values. The annotation lets us measure accuracy. The stop reason indicates whether generation ended as expected.

The result of 100 % compliance and 25 % exact tuples summarizes this distinction: closing off response possibilities does not give the model knowledge of the ticket. We have a verifiable mechanism for controlling form, not proof of business correctness.

Exercise: state a hypothesis before running the command

Now imagine a change: EOS would be authorized at every trie node, including the root.

Before running the command, write down your prediction. If the logits favor EOS, could the model stop immediately? Could it end after an incomplete JSON prefix? Would the empty text be accepted by the parser?

The module src/json_decode/exercise.py already compares the two rules with controlled logits that favor EOS. You do not need to modify the reference decoder. We are deliberately testing a stopping rule, without running a new campaign intended to improve scores.

docker compose run --rm gpu python -m json_decode.cli exercise --config configs/exercise.yaml

Compare unsafe_immediate_eos, safe_first_token, and empty_json_valid with your predictions. The first lets you observe immediate termination under the unsafe rule; the second shows the initial choice under the protected rule; the third checks whether an empty text can be read.

The identifier for safe_first_token belongs to the project’s tokenization: do not replace it with the 10 from our fictional tree. What matters is that it is an authorized beginning rather than a premature stop.

To explain the result, distinguish authorizing a continuation from accepting a completed object. An empty text is not a JSON document. The text "", which contains two quotation marks, represents an empty JSON string, but still not the three-field object required here. Syntax and contract remain two separate checks.

Adapting the mechanism and preparing for production

When this finite trie remains relevant

This mechanism is easy to inspect because the contract is small and closed. To modify it, update the value list, serialization, tests, and independent validator together. Keep development and testing separate before changing the prompt.

The number of combinations can nevertheless grow quickly. For illustration, four categories, three priorities, and 10,000 possible product values would give 4 × 3 × 10 000 = 120 000 responses to enumerate. Common prefixes can be shared, but every combination must still be considered by this construction.

A free-form comment field can no longer be described by the tutorial’s 48 outputs. It then requires a suitable grammar, complexity and length limits, and tokenization tests. This extension changes the problem; it does not consist of removing a guard until the program accepts the new field.

An allowed value is not authorization to act

unknown is provided for category and product, not for priority. Its presence offers a possible value; it does not guarantee the model will choose it when uncertain. Nor is it a calibrated measurement of the model’s uncertainty.

For real tickets, you need authorized, versioned data, reviewed annotations, and explicit rules for handling ambiguity. A rejection policy and, where action is sensitive, human oversight or validation remain necessary.

The application must also handle authentication, access control, observability — useful logs, errors, and measurements — load testing, backups, and dependency auditing. None of these properties follows from the score table. A structured output remains untrusted input until the business domain has validated it.

Troubleshooting and cleanup

The environment check fails. Stop at check_environment.py. Check the driver, NVIDIA Container Toolkit, and the GPU exposed to Docker. A GPU access issue is not fixed by changing the allowed classes.

Trie inspection fails. Do not proceed to generate. First check canonical serialization and the tokenization/decoding round trip. The tree must describe the intended strings, not a similar-looking approximation.

Negative cases no longer catch rejections. Examine forbidden prefixes and the EOS rule before reviewing model scores. An isolated demonstration that produces JSON does not fix this regression.

To finish the walkthrough, generate the figures, check the checkpoints, preview cleanup, and stop the services. Run these four commands in this order, always from the project root:

docker compose run --rm gpu python lesson/figures/plot.py
docker compose run --rm gpu python scripts/check_checkpoints.py
docker compose run --rm gpu python scripts/clean.py --scope generated --dry-run
docker compose down --remove-orphans

The --dry-run option requests a preview: it lets you inspect the planned cleanup, not start a deletion. Do not remove it until you have inspected what would be deleted. The final command stops the project’s Compose resources; no global purge of other Docker projects is needed.

Primary sources

The format and Python code. RFC 8259 — the JSON format describes the general language. The Python json module documentation details serialization, decoding, and their particularities; the itertools.product documentation explains combination enumeration.

The model and decoding. The Qwen2.5-0.5B-Instruct model card at the revision used, the Transformers tokenizer overview, the Transformers generation API, and the torch.argmax function provide further detail on the components studied.

The environment. The NVIDIA Container Toolkit installation guide covers GPU access from containers.

These references supplement the explanation. To reproduce the measurements, retain the code, model revision, and fixed parameters in the r1 archive: current documentation pages do not constitute a newly validated tutorial configuration.