You are developing a management screen that needs to retrieve an amount from a PDF. The assistant answers 120.00. Fine… but where does that number come from? The estimated cost, the approved cost or another line in the document? Before filling in the form, you would like to click the answer and see the area supporting it.
That is this tutorial’s starting point. We will follow the fictional Orion Service Record form, which contains a date, two serial numbers and an approved cost. Our program must answer a question about the page, indicate where the value appears and abstain when the requested field is absent.
We will adapt a model capable of processing images and text using LoRA, then validate its outputs in Python. You do not need prior knowledge of this technique: we will see what it changes, how it learns and, above all, how to verify the result.
The workflow is page image → question → adapted model → answer and rectangle → validation → evaluation. An answer and an attractive box will not suffice: we will measure text, localization and abstention separately.
The concrete problem
Reading a page and answering a question are different tasks. A system can recognize several amounts without knowing which corresponds to ‘approved service cost’. It can also provide the correct amount while pointing to the wrong occurrence.
Here we perform VQA, or Visual Question Answering: answering a question from an image. The PDF is rendered as a page image, which accompanies the question. The expected result is not a free-form document summary, but structured data the program can accept or reject.
For a web developer, the parallel is simple: as with an API, we define an output contract. But compliant JSON does not guarantee truthful data. We must also compare predictions with reference annotations, then examine the indicated area on the page.
The corpus is synthetic: its forms are generated for a controlled experiment. This lets us know the expected answers and provoke certain errors. It does not automatically validate processing your customers’ PDFs.
Concepts before the code
An answer and its evidence form a single contract
Our running question asks for the approved service cost. The model must produce not only 120.00, but also a box: the rectangle around the area containing the requested value.
Here is an illustration of the expected format. The coordinates in this example are chosen for explanation; this is not a measured corpus prediction.
{
"answer": "120.00",
"box": [600, 700, 800, 750]
}
answer is a string, even for an amount. box contains four coordinates in the order left, top, right, bottom. They are expressed on a scale from 0 to 1000, not directly in pixels.
For an absent field, the second permitted form is:
{
"answer": null,
"box": null
}
This is abstention: the system suggests neither a value nor a location. It must be complete. A value without a box or a box without a value is rejected. Likewise, the string "null" is not the JSON value null: it would be text, not absence.
Why keep both elements together? Because we want to avoid two shortcuts. An exact amount paired with the wrong rectangle does not let us verify the correct occurrence. Conversely, a well-placed rectangle does not make an incorrect amount acceptable.
In this article, ‘localized evidence’ means the area proposed as support. The term does not mean the program has proven the answer true. The model predicts coordinates too: they can be wrong. The contract makes this proposal inspectable; evaluation will tell us whether it is correct.
What LoRA actually adapts
The base model is Qwen2-VL-2B-Instruct. It is a vision-language model, or VLM: it receives visual and textual information to generate text. Its official model card presents the variant used; we keep the project’s variant rather than replacing it with a newer model.
Its visual encoder turns the image into numerical representations. Its text decoder uses those representations and the question to produce answer elements sequentially. A token is a text unit handled by the model: a word, word fragment or symbol may occupy different tokens.
The tokenizer converts text into numerical identifiers. The processor also prepares images for the model. The Transformers Qwen2-VL documentation illustrates this multimodal preparation. You do not need to write a visual encoder in this tutorial: we use the pretrained model’s capabilities.
Weights are the numerical parameters learned by the network. Training adjusts them using examples and expected answers. Adapting an already trained model to a particular task is called fine-tuning.
LoRA, short for Low-Rank Adaptation, performs this adaptation without changing every weight. In our project, base weights remain frozen, meaning non-trainable. We add small matrices that learn a correction in selected layers. This collection of extra parameters forms the adapter.
Think of an existing component whose main processing is retained while a specialized correction is added. The analogy ends there: this correction is not a list of manually written conditions, but learned numbers. The PEFT conceptual LoRA guide describes this separation between frozen weights and adaptation matrices.
Why this choice here? We already have a vision-language model and want to study targeted adaptation to the Orion contract. LoRA reduces the number of trainable parameters and makes the saved adaptation identifiable. The workflow compares the frozen model with its adapted version; it does not compare LoRA with full fine-tuning. We will therefore not conclude that LoRA is always the best method.
The Orion corpus contains 400 pages. The test set contains 240 separate questions: 180 answerable and 60 concerning absent fields. Pages and questions are different units: these numbers do not describe 240 additional PDFs.
Preparation renders pages, builds question/answer pairs and sets expected rectangles. These reference answers, sometimes called gold, serve for learning and evaluation. Inference, meaning prediction generation, does not mount the annotations. The program must use the intended inputs, not reread the solution.
The inspect-batch command lets us examine a batch: image, question, target JSON and box. This is the right moment to understand the examples we ask the model to learn, before starting training.
An example of localized evidence
Return to our illustrative box [600, 700, 800, 750]. On a 1000 × 1000 pixel image, it corresponds to a rectangle extending from 600 to 800 pixels horizontally and 700 to 750 pixels vertically.
On the same page rendered at 2000 × 2000 pixels, this rectangle becomes [1200, 1400, 1600, 1500] in pixels. Yet its normalized representation remains identical. To recover a horizontal coordinate, calculate x_pixels = x_normalized × width / 1000; for a vertical coordinate, use height.
This scale resembles percentages, with an upper bound of 1000 rather than 100. It allows the same relative area to be specified at different resolutions, provided framing and orientation remain the same. Cropping changes the coordinate system; increasing resolution also does not automatically make the model more precise.
A page may contain ‘service cost’ and ‘approved service cost’. A box on the first field does not support an answer about the second, even if the amounts coincide. Hence the value of measuring overlap with the expected box.
The measure used is IoU, or Intersection over Union: the area shared by the rectangles divided by the total area covered by either. A value of 1 means they coincide; 0 means they share no area. We will calculate this by hand after the validation code.
Code LoRA and choose which tokens to learn
Two small matrices instead of one large unrestricted correction
A linear layer transforms one list of numbers into another. Its coefficients are stored in a matrix W. A matrix is a two-dimensional array; a tensor is the more general term PyTorch uses for numerical arrays.
LoRA retains W and adds a correction of the form (alpha/r) × B × A. Matrix A transforms the input into a small intermediate dimension r; B then transforms it into the output dimension. The correction is called low-rank: its construction limits the number of independent directions it can represent.
In the project, r = 8 and alpha = 16, so the multiplicative factor is 16/8 = 2. This factor controls the correction’s amplitude; it is not the optimizer’s learning rate.
The targeted layers are q_proj, k_proj, v_proj and o_proj in the text decoder’s attention. To locate them without detailing the entire Transformer: Q and K projections contribute to matching between positions, V supplies information to combine, and O transforms the combination’s result. The attention computation documented by PyTorch illustrates the combination of Q, K and V. These names designate network layers, not four PDF-reading tools. The visual encoder remains frozen.
Take a numerical example independent of Qwen2-VL’s actual dimensions. For a projection from 1024 inputs to 1024 outputs, W contains 1024 × 1024 = 1 048 576 coefficients. With rank 8, A contains 8 × 1024 = 8192 coefficients and B contains as many: 16,384 adaptation parameters, 64 times fewer than the coefficients in W, excluding any bias.
This calculation concerns one illustrative projection, not the model’s total parameter count. The large matrix remains present and in use. Reducing trainable parameters therefore means neither dividing all memory by 64 nor eliminating the base model’s cost.
Read the LoRALinear layer
Here is the class used in src/pdfvqa/lora.py. Imports and the code inserting it into the model are provided by the complete project.
class LoRALinear(nn.Module):
def __init__(self, base, rank=8, alpha=16, dropout=0.05):
super().__init__()
if not isinstance(base, nn.Linear) or rank <= 0 or not 0 <= dropout < 1:
raise ValueError("Invalid LoRA layer")
self.base = base.requires_grad_(False)
self.A = nn.Parameter(
torch.empty(
rank, base.in_features, device=base.weight.device, dtype=torch.float32
)
)
self.B = nn.Parameter(
torch.zeros(
base.out_features, rank, device=base.weight.device, dtype=torch.float32
)
)
nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
self.scale = alpha / rank
self.dropout = nn.Dropout(dropout)
def forward(self, x):
original = self.base(x)
update = torch.nn.functional.linear(
torch.nn.functional.linear(self.dropout(x.float()), self.A), self.B
)
return (original.float() + self.scale * update).to(original.dtype)
The first part retains the original layer. requires_grad_(False) tells PyTorch not to compute gradients for its parameters. A gradient helps determine how to adjust a parameter to reduce training error. Frozen weights still participate in the calculation; they are simply not updated by adaptation.
The second part creates the parameters to learn. nn.Parameter registers A and B as module parameters. Their dimensions follow the path ‘input dimension → rank → output dimension’. They are placed on the same device as the original layer.
A receives random initialization, while B starts at zero. Initially, passing through B therefore gives a zero correction: the adapter does not yet change the layer’s output. Training then learns this correction instead of starting by arbitrarily disturbing the model.
The forward() method computes both branches. self.base(x) produces the original output. The two linear() calls apply A and B in succession. The PyTorch expression follows the convention described by torch.nn.functional.linear, where weights are stored in output, input order. Finally, the correction is multiplied by self.scale and added to the original output.
Dropout operates only in the adaptation branch. During training, its value of 0.05 gives each element a 5% probability of being set to zero; retained elements are rescaled by the module. In evaluation mode, it no longer changes its input. This is the documented behavior of nn.Dropout, not permanent parameter deletion.
Finally, x.float(), matrices in torch.float32 and original.float() make the correction and sum calculation explicitly FP32, a 32-bit format. .to(original.dtype) returns the result to the original output’s type. Do not confuse numerical format handling with freezing weights: one concerns number representation, the other which parameters are trained.
Learn the answer, not copy the whole context
We have defined what can change in the model. Now we must define which text positions contribute to the error calculation.
During training, a sequence contains context—including the question and image-associated elements—followed by the assistant’s expected answer. The loss function measures the gap between predictions and tokens to learn. We want to train the JSON answer, not ask the model to copy the question or padding positions.
Padding adds positions to organize sequences of different lengths in arrays. Those positions are not content to predict. The code therefore uses the special value -100 in training labels for positions ignored by the loss.
def assistant_labels(input_ids, attention_mask, prompt_length, forbidden_ids):
if (
input_ids.ndim != 1
or attention_mask.shape != input_ids.shape
or not 0 < prompt_length < len(input_ids)
):
raise ValueError("Invalid completion boundary")
labels = input_ids.clone()
labels[:prompt_length] = -100
labels[~attention_mask.bool()] = -100
for identifier in forbidden_ids:
labels[input_ids == identifier] = -100
if (labels != -100).sum() == 0:
raise ValueError("No assistant target token")
if (labels[:prompt_length] != -100).any():
raise ValueError("Prompt included in loss")
return labels
The initial check expects a one-dimensional sequence, an attention_mask of the same shape and a prompt_length boundary after the start but before the end. This boundary separates context from answer.
clone() copies identifiers to build labels without modifying the input. The next three operations mark the prompt prefix, positions where the attention mask is zero and identifiers in forbidden_ids as -100. If no target remains, the function rejects the example instead of starting training with no answer to learn. The final check verifies that the prompt is excluded.
Take a teaching example of identifiers, unrelated to the tokenizer’s actual vocabulary. The input is [101, 501, 502, 601, 602, 603, 0], the prefix occupies the first three positions, the last is padding and 603 is a forbidden identifier. Labels become [-100, -100, -100, 601, 602, -100, -100].
Only targets 601 and 602 contribute directly to the loss. The context has nevertheless not been removed from the input. The model can still use it to produce the answer. Masking a training target does not mean erasing the question or image.
The value -100 is therefore neither a token the model should generate nor a synonym for JSON null. It means only ‘ignore this position for the loss’, under the ignore_index mechanism documented by PyTorch.
The loss then shifts targets: the prediction at one position learns the next token. This shift is not performed in assistant_labels(). The function shown builds the mask; the loss calculation uses it afterward. In the target JSON, values and coordinates are thus learned through their token representations.
Remember: LoRA chooses which parameters change; the mask chooses which targets contribute to learning; the validator checks the form of the generated output. These mechanisms have different roles.
Validate the generated answer
After inference, we receive raw text. It may be incomplete, malformed or contain impossible coordinates. We check it before considering it an answer usable by the program.
The following excerpt comes from src/pdfvqa/validate_answer.py. It does not consult the expected answer to fix the prediction.
"""Validate raw model output, without repairing it from the expected answer."""
import json
def validate_answer(raw, terminated=True):
if not terminated:
return {"valid": False, "error": "length", "value": None}
try:
value = json.loads(raw)
except (ValueError, TypeError):
return {"valid": False, "error": "json", "value": None}
if not isinstance(value, dict) or set(value) != {"answer", "box"}:
return {"valid": False, "error": "schema", "value": None}
answer, box = value["answer"], value["box"]
if answer is None and box is None:
return {"valid": True, "error": None, "value": value}
if answer is None or box is None:
return {"valid": False, "error": "asymmetric_null", "value": None}
if not isinstance(answer, str) or not answer.strip():
return {"valid": False, "error": "answer", "value": None}
if (
not isinstance(box, list)
or len(box) != 4
or any(type(x) is not int for x in box)
):
return {"valid": False, "error": "box_type", "value": None}
if not 0 <= box[0] < box[2] <= 1000 or not 0 <= box[1] < box[3] <= 1000:
return {"valid": False, "error": "box_bounds", "value": None}
return {"valid": True, "error": None, "value": value}
def iou(a, b):
intersection = max(0, min(a[2], b[2]) - max(a[0], b[0])) * max(
0, min(a[3], b[3]) - max(a[1], b[1])
)
union = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - intersection
return intersection / union if union > 0 else 0.0
From raw text to an accepted or rejected object
terminated indicates whether generation ended correctly. When it is False, the function returns the length error, even if the start of the output looks usable. The caller must pass this state: the validator cannot infer from the string alone how the model stopped.
json.loads() then decodes the text. Failure yields json. After decoding, the program requires a dictionary containing exactly answer and box: a missing or additional key yields schema.
The pair is checked next. Two None values, originating from the two JSON null values, form valid abstention. Only one null value produces asymmetric_null: abstention is incomplete. Otherwise, answer must be a nonempty string after checking whitespace.
The box must be a list of four integers, then satisfy 0 ≤ left < right ≤ 1000 and 0 ≤ top < bottom ≤ 1000. Strict inequalities prohibit zero width or height. The check type(x) is not int notably rejects a boolean in place of a coordinate. Errors distinguish an incorrect type (box_type) from geometry outside the contract (box_bounds).
The function uses answer.strip() to detect an empty string but returns the original object: it does not systematically clean the answer. Nor does it convert an amount, check a currency or clamp out-of-page coordinates to a boundary.
valid=True therefore means ‘compliant with the contract checked here’, not ‘correct in the document’. An incorrect amount paired with a well-formed rectangle can pass. Likewise, abstention can be structurally valid while the page contains the answer. Correctness evaluation must distinguish these cases.
Calculate IoU for a small shift
The iou() function first calculates intersection width and height. The max(0, ...) calls prevent negative areas when rectangles do not touch. The union is the sum of both areas minus their intersection, so the common area is not counted twice.
Return to an illustrative numerical example, not an Orion measurement. The reference box is [600, 700, 800, 750]. The prediction shifts 100 units right: [700, 700, 900, 750].
Each rectangle has area 200 × 50 = 10 000. Their intersection is only 100 × 50 = 5000. The union is 10 000 + 10 000 − 5000 = 15 000. IoU is therefore 5000 / 15 000 = 1/3, approximately 0.3333.
The text could remain exactly 120.00 in both answers. The IoU decrease does not measure a text change: it measures degraded localization. The final exercise will deliberately create this separation in project outputs.
The iou() function assumes usable boxes for this calculation; it does not replace validate_answer(). Predicted coordinates are never corrected after generation using gold references. Otherwise, we would measure an answer assisted by the solution, not the model’s prediction.
Prerequisites and commands
You need to be able to read Python, use a terminal and run Docker. The preceding excerpts explain the mechanisms; the ZIP supplies the complete project needed to execute them.
The reference profile uses Linux, Docker Engine, Compose, an NVIDIA driver and NVIDIA Container Toolkit, on a BF16-compatible 24 GiB RTX 3090. The workflow recommends 48 GiB of RAM and 55 GiB of disk space. These are preparation recommendations; the CPU profile is not validated.
VRAM is graphics-card memory, distinct from the machine’s RAM. BF16 is a 16-bit numerical format among PyTorch’s numerical types. Its presence in the profile does not mean every computation uses it: we just saw the LoRA correction explicitly prepared in FP32.
The driver enables GPU computation on the host. NVIDIA Container Toolkit provides Docker integration. Seeing the card with nvidia-smi confirms its visibility on Linux, not yet that the model works inside the container. check_environment.py performs that second check. The official NVIDIA documentation details system-appropriate installation; project checks do not silently modify the system.
Extract the ZIP and open a terminal in its root folder. All commands below run from this root, in order. The acquisition service retrieves model assets, tutorial prepares data, trains and evaluates, while inference produces predictions without consulting labels.
Step 1 — prepare
We check the environment, prepare the project, build services and acquire the expected files before running tests.
sh scripts/preflight.sh
sh scripts/setup.sh
docker compose config --quiet
docker compose build acquisition tutorial inference
docker compose run --rm acquisition python scripts/acquire.py --config configs/minimum.yaml
docker compose run --rm tutorial python scripts/check_environment.py
docker compose run --rm tutorial python -m pytest -q
Each command must finish with a zero exit code, meaning no reported error. These checks serve different purposes: Compose checks configuration, the build creates images, locked acquisition retrieves the intended assets, then the environment check runs the model inside Docker. Finally, tests verify program behaviors.
A successful build replaces neither the GPU check nor tests. And passing tests do not mean every model answer will be correct: that measurement comes later.
Step 2 — build the prediction
Let us prepare pages and questions, inspect a batch, then produce an initial reference without adaptation.
docker compose run --rm tutorial python -m pdfvqa.cli prepare --config configs/minimum.yaml
docker compose run --rm tutorial python -m pdfvqa.cli inspect-batch --config configs/minimum.yaml
docker compose run --rm inference python -m pdfvqa.cli baseline --config configs/minimum.yaml
prepare builds inputs and targets. inspect-batch lets us inspect what the model receives and what it is asked to produce. For the running question, locate the approved cost, then check that the target points to that field rather than the other cost.
baseline means comparison reference. Here, it uses the frozen model without Orion adaptation, but with the same output contract. This stage answers an essential question: what can the model already do before our training?
Open the intermediate outputs. Inspection helps understand data and errors, not rewrite a reference answer to match a prediction.
Step 3 — measure and check
We can now train the adapter, check resumption after interruption, then produce and evaluate adapted answers.
docker compose run --rm tutorial python -m pdfvqa.cli train --config configs/minimum.yaml
docker compose run --rm tutorial python -m pdfvqa.cli train-continuous --config configs/minimum.yaml
docker compose run --rm tutorial python -m pdfvqa.cli train-resume --config configs/minimum.yaml
docker compose run --rm tutorial python -m pdfvqa.cli verify-resume --config configs/minimum.yaml
docker compose run --rm inference python -m pdfvqa.cli predict --config configs/minimum.yaml
docker compose run --rm tutorial python -m pdfvqa.cli evaluate --config configs/minimum.yaml
train learns LoRA parameters and saves a checkpoint, a state preserving the work performed. The train-continuous and train-resume commands compare continuous execution with resumed execution. verify-resume checks that interruption did not change the experiment.
Correct resumption requires more than reloading a few matrices. The optimizer, which controls updates, also has state; random generators must be restored. The r001 reference result reports EXACT resumption. This describes that workflow’s check, not a promise of numerical identity on every device or after dependency changes.
predict then runs the adapted model in the inference service. evaluate joins predictions and annotations to calculate measurements. Labels therefore grade after prediction, rather than help the model while it answers.
Measurements before numbers
Read the metrics in the right order
Start with JSON and contract validity. The name might suggest simple JSON decoding, but the check shown goes further: exact keys, types, the answer/box pair and coordinates. A rejection must remain visible; do not salvage only fragments that give an impression of success.
Answer accuracy compares text with the reference after the normalization rule specified by the project. It is not a free-form assessment of meaning. Do not replace that rule with amount conversion or rewording chosen after seeing results.
IoU measures box overlap for answerable questions. It does not apply to absent fields: there is no expected rectangle to compare with abstention.
Joint correctness requires answer, schema and localization checks to pass together. A correct value therefore does not compensate for a wrong box. The numerical localization acceptance threshold is not detailed in this presentation: retain the ZIP evaluator’s rule, without deriving it from mean IoU or substituting an arbitrary threshold.
Finally, abstention precision asks ‘among emitted abstentions, how many actually concern absent fields?’ Abstention recall asks ‘among absent fields, how many correctly led to abstention?’
A teaching example: for ten absent fields, a system correctly abstains eight times, but also abstains once when the answer is present. Its recall is 8/10 = 0.8; its precision is 8/9, approximately 0.8889. The measures describe different errors. This example is not the Orion model’s result.
A system that always answers may overlook absences; a system that always abstains may miss every present value. That is why we retain both populations and multiple measurements.
Interpret outputs step by step
After predict, find the raw JSON answer, its validation status, the box and the overlay. This is the rectangle drawn on the image to help inspection, not additional model validation.
For an accepted output, compare the text, requested field and boxed area. For a rejected output, keep the error: json, schema, box_bounds or another reason. Turning an invalid result into a plausible sentence would lose the diagnosis.
Here are the r001 reference results on the synthetic Orion corpus’s 240 test questions:
On mobile, scroll the table horizontally.
| LoRA metric | Population concerned | Observed result |
|---|---|---|
| Answer accuracy | 240 questions | 1.0000 |
| JSON and contract validity | 240 questions | 1.0000 |
| Joint correctness | 240 questions | 1.0000 |
| Mean IoU | 180 answerable questions | 0.9857 |
| Abstention precision | Abstentions on the test | 1.0000 |
| Abstention recall | 60 absent fields | 1.0000 |
The frozen model achieves joint correctness of 0.3000, versus 1.0000 after LoRA adaptation. Across 240 questions, that corresponds to 72 joint successes versus 240 under this workflow’s criterion.
How should we interpret perfect joint correctness alongside IoU of 0.9857? Every answer passes the evaluator’s localization criterion, but not every rectangle exactly coincides with its reference. Passing the criterion does not mean perfectly equal coordinates.
These results show an adaptation gain over the frozen model here. They do not predict performance on handwriting, degraded scans or different customer field labels. Nor do they demonstrate superiority over methods this workflow does not compare.
Troubleshooting and moving toward a product
The model cannot run inside Docker? Return to the environment check, GPU access and NVIDIA integration. An execution problem cannot be diagnosed from an answer metric.
JSON is rejected? Inspect the prompt, generation limits and model version. An answer stopped too early can be incomplete. An out-of-page box remains invalid: do not truncate it to pass validation or retrieve the value from the reference.
The text is exact but the box is misplaced? Present this output as insufficiently supported. The interface can help expose the error; it must not hide it under an overall score.
In a product, every suggestion should remain linked to an immutable PDF, a page, an area, a model version and a decision log. These are elements to plan around the tutorial, not new fields to silently add to the tested JSON contract.
What we have learned
LoRA learns a low-rank correction in targeted projections without changing base weights. The training mask preserves context in the inputs but restricts the targets over which loss is calculated.
Once the answer is produced, Python checks its contract. Evaluation then checks something else: the right text, the right area and the right decision to answer or abstain. None of these dimensions replaces the others.
On Orion, adaptation substantially improves joint correctness over the frozen model. To understand why a textually exact answer remains insufficient, we will now shift only its visual support.
Exercise — move the evidence without changing the text
Before running the exercise, write your hypothesis. What happens to text accuracy, IoU and joint correctness if every box shifts horizontally by 100 normalized units without changing answers? What should happen to abstentions, which have no box?
Think of the earlier small calculation: two rectangles can remain within the page while overlapping much less. ‘Permitted coordinates’ and ‘correct localization’ are not synonyms.
Then run the supplied experiment without modifying the validated code:
docker compose run --rm tutorial python -m pdfvqa.cli exercise --config configs/exercise.yaml
docker compose run --rm tutorial python lesson/figures/plot.py
docker compose run --rm tutorial python scripts/check_checkpoints.py
The first command performs the exercise. The next produces figures, and the last checks expected checkpoints. Compare outputs with your prediction before settling for the final score.
Observed result: all 240 texts remain identical and text accuracy is 1. Mean IoU falls to 0.1692 and joint correctness to 0.25. The 60 abstentions remain correct, and no box is out of bounds.
The 0.25 is particularly revealing: the 60 abstentions represent 60/240 = 0.25 of the test. They account for every remaining joint success. The text did not change; the localization of present answers no longer passes the joint check.
This does not contradict the validator. A shifted rectangle can retain four integers, positive width and coordinates between 0 and 1000. It remains well-formed while pointing to the wrong area.
This output perturbation is distinct from the exercise’s two additional pages, where a field actually changed position. In one case, we move a box after prediction; in the other, we change the document the model must read. Do not present the first experiment’s figures as measurements of the second.
Choose an action after evaluation
An insufficiently supported answer should be presented to an operator with the PDF and overlay, not automatically sent to an accounting system. Abstention must not be replaced with whichever approximate value seems closest.
However, distinguish an evaluation metric from a check available in production. Joint correctness and IoU against gold require a known annotation. On a new PDF without a reference, the program does not suddenly have these scores to decide by itself. Rejection rules, business checks and human review must therefore be designed and evaluated separately.
Likewise, the tested contract expresses abstention through two null values; it contains no field explaining the reason. Retaining a verified reason in a log or interface is additional integration work, not information to invent from null.
To adapt the experiment to your forms, separate data by document family and template. Set the test aside before iterations, then measure answer, schema, localization and abstention separately. The aim is not merely a better number, but knowing which errors it may hide.
Expected outputs and useful errors
After prepare, locate the separate pages, questions and targets. After baseline, retain frozen-model outputs to compare the same cases. After predict, examine valid, error, then answer and box in the structured response when available. In the validation excerpt, this response is stored under value, which is None on rejection.
evaluate specifies populations and distinguishes test from exercise. This prevents new questions from being mixed with artificial perturbation of existing predictions.
An asymmetric_null result indicates an answer without a box or a box without an answer. box_type indicates a representation problem; box_bounds, a coordinate-bound or ordering problem. These errors are useful program outputs. Making them disappear without understanding their cause would make the system less observable, not more accurate.
Production, limitations and cleanup
The tutorial provides an experiment on fictional forms. To move to real documents, assemble authorized, representative examples with verified answers and areas. Variations in labels, image quality and layout must be evaluated rather than assumed covered.
Retain the model version, document, page, areas, units and rejection rules. Add access rights, monitoring and human review for costs or contractual decisions. Absence, an ambiguous unit or an invalid box must never be turned into certainty merely by the interface’s presentation.
Before cleanup, save useful exports. The script below targets only outputs/:
docker compose down
docker compose run --rm tutorial python scripts/clean.py --outputs-only
docker compose down
The first docker compose down stops project services. The next container performs targeted cleanup, then the final shutdown ends the workflow. These commands do not request a global Docker purge.
For deeper study, the official PEFT and LoRA documentation presents adapters on pretrained models. The original LoRA paper details the low-rank correction idea. These references supplement the explanation; the implementation shown here remains the ZIP’s LoRALinear class, not a rewrite using another library.
The documentation linked in this article helps explain the mechanisms. To reproduce r001, retain the supplied project’s commands, configuration and dependencies: a documentation example is not an invitation to modify the reference environment.
What we have learned: a value becomes inspectable when it stays connected to its page, an area and explicit validation rules. The final step is not to trust the displayed box, but to verify what it shows.