Imagine a back-office screen showing several records for what appears to be the same document. Titles look similar, references are not always entered with the same reliability, and some records correspond to different versions. You would like to suggest grouping them to make navigation easier. But certainly not delete a useful version or combine two distinct documents because they share a few words.
This is the problem we will explore with four fictional records. A-17 and B-92 describe logical document VC-204; D-40 is its version 2. C-11 describes VC-205, with an almost identical title. The expected grouping keeps A-17, B-92, and D-40 together while leaving C-11 separate.
The word logical matters: here it denotes the document identity to which several records and versions refer. It does not mean their files are byte-for-byte identical. Linking records is not merging their files. In this tutorial, every file and version is preserved; the program produces a proposal without updating the DMS, the document management system.
We will build this pipeline: records → candidate pairs → CamemBERT scores → threshold decisions → proposed and checked groups. An initial filter reduces comparisons. A model then learns to examine two records together. Finally, code checks that grouping does not bypass a reliable administrative conflict.
The workflow's value lies in understanding where each component acts, then measuring its contribution. An excellent classifier does not automatically recover candidates the filter missed. And a few incorrect pairs can suffice to form an excessively large group.
Measure each stage separately
Blocking: choose comparisons, not yet answers
Comparing all records pairwise seems simple. But for n records, we must examine n × (n - 1) / 2 distinct pairs: a record is not compared with itself, and A/B and B/A are not counted twice.
With only 648 records, this already gives 209,628 pairs. Blocking is a preselection step: it keeps a smaller set of pairs worth examining in depth. For a web developer, think of initial candidate selection before expensive processing rather than a definitive identity decision.
Teaching example, separate from the ZIP's rules: if only records with exactly the same reference were compared, a missing or mistyped reference could prevent two records of the same document from ever meeting. The model after that filter would have no chance to recognize them. The Python Record Linkage Toolkit indexing documentation presents this pair-reduction principle; it does not describe our project's specific rules.
In the r001 workflow, constrained grouping can request additional scores to check a proposed union. We will therefore distinguish candidates from the initial filter from verification comparisons added during grouping. The latter are not a new exhaustive search across all records.
A model that reads both records together
CamemBERT is a language model pretrained on French. A tokenizer splits text into units called tokens, sometimes smaller than a word. The model turns these units into embeddings: vectors, or lists of numbers, representing text in context. The official CamemBERT documentation describes its use for classifying text and sequence pairs among other tasks.
Here, we use a cross-encoder: both records are presented together to the model, which produces a score for their pair. The first record's text can therefore interact with the second's during computation. This is not the same operation as computing and storing an independent vector for each record, then comparing those vectors.
The intended advantage is joint examination of fields. The trade-off is cost: a new pair requires a new model computation. The Sentence Transformers cross-encoder documentation explains the distinction. Here it helps explain the architecture, not add a library or another model to the workflow.
Learning is supervised: a target is 1 when both records describe the same logical document, and 0 otherwise. A positive pair therefore does not necessarily mean “two identical copies of the same file.” Version 2 of a document can belong to the same group as its previous version.
The synthetic corpus contains 4,320 records for 1,440 logical documents, including 648 test records. Reference identities, often called gold, identify the expected pairs and groups. They serve training or evaluation depending on the split, never to provide the answer directly during inference.
Precision and recall: which pairs are we talking about?
A true positive is a pair proposed as identical that really is identical in the reference. A false positive wrongly joins two distinct documents. A false negative is an expected positive pair the system fails to find.
Precision asks “How many proposed links are correct?” Recall asks “How many expected links did we recover?” F1 combines them through a harmonic mean: 2 × precision × recall / (precision + recall).
We will use three complementary metrics:
| Metric | Calculation | Question |
|---|---|---|
| Blocking recall | Positive candidates / all expected positives | Does the first filter retain the right comparisons? |
| Classifier candidate recall | True positives / positives among candidates | Does the model recognize the correct pairs it receives? |
| End-to-end pair recall | True positives / all expected positives | What does the preselection and classification pipeline recover? |
Teaching example: ten positive pairs are expected, the filter retains seven, and the model recognizes all seven without error. Its candidate recall is 7/7 = 1, but end-to-end recall is 7/10 = 0.7. There is no contradiction: the denominators differ.
More generally, for this pair pipeline, end-to-end recall is the product of blocking recall and candidate recall. Improving the classifier alone therefore does not remove the limit imposed by preselection.
B-cubed: look at each record's companions
After pairs come groups, or clusters. The B-cubed score examines each record's proposed and reference groups. Common members are companions correctly kept together, including the record itself.
Local precision divides this common-member count by the proposed group size. It drops if outsiders are added. Local recall divides it by the expected group size. It drops if members of the same document are left out.
Teaching example: a record should be grouped with two others, making an expected group of three. The proposed group contains two correct members and one outsider. For this record, B-cubed precision and recall both equal 2/3. The calculation is repeated for each record, then precision and recall are averaged. We will compute F1 from those two averages: this convention reproduces the workflow's published values, as the four-record example will show.
A strategy that leaves everyone alone avoids oversized groups but misses expected links. Conversely, one huge group recovers every expected companion while adding many outsiders. That is why precision and recall must be read together.
Finally, a cannot-link is an explicit veto: in this project, two reliable, nonempty, differing references must not belong to the same group. A record alone forms a singleton; it is not deleted. We will separately measure group quality, the number of forbidden pairs they contain, and version preservation.
Prepare the GPU-based reader
What you need before starting
You need to be able to read Python, use a terminal, and run Docker. Basic PyTorch, supervised classification, and graph knowledge is useful; we will explain operations appearing in the snippets. A tensor here is a numerical array, and a graph is a set of records linked by proposed matches.
The reference profile is Linux with a 24 GiB RTX 3090, 24 GiB of RAM, and 30 GiB of working disk space, plus Docker storage. Graphics memory and machine RAM are distinct resources. These guidelines guarantee neither runtime nor the capacity of any other configuration. The presented results come from the r001 GPU workflow; they do not constitute CPU validation.
Unzip the course archive, available at the end of the article, and open a terminal at its root. All following commands run from this directory in the stated order. The archive's versions and settings are those to reproduce.
A GPU visible on the host is not enough: Docker must be able to access it. NVIDIA Container Toolkit provides the required integration. If it is missing, project diagnostics indicate the expected action without silently installing anything. GPU_DEVICE=1 selects another card before the workflow.
Check, build, and acquire assets
Start with preliminary checks, setup, and service builds.
sh scripts/preflight.sh
sh scripts/setup.sh
docker compose config --quiet
docker compose build acquisition tutorial learning inference
The preflight checks Docker, Compose, RAM, disk, and GPU. docker compose config --quiet succeeds without displaying content when configuration is valid. Each step must finish without error before the next.
Acquisition then retrieves the project's six intended CamemBERT assets.
docker compose run --rm acquisition python scripts/acquire.py --config configs/minimum.yaml
Their size and SHA-256 fingerprint are verified before atomic replacement: validated files take their intended place without exposing a partial copy as complete. The fingerprint checks the identity of expected bytes, not the relevance of future predictions. No private cache is distributed in the archive.
Now check the environment from inside the container, then run the project's tests.
docker compose run --rm tutorial python scripts/check_environment.py
docker compose run --rm tutorial python -m pytest -q
The environment check actually performs computation inside Docker. Tests examine conflicts, symmetry, and version preservation among other things. They verify program behaviors; they do not prove every business match will be correct. Check that your run finishes without failures and retain its report.
Generate records, candidates, and a symmetric score
Separate inputs from expected answers
Preparation places records and their fictional files in data/generated/input/. Gold identities remain in labels/. The learning container mounts only training and validation pairs; inference mounts no annotations.
The idea resembles an exercise whose answer key stays closed while you answer. The model must use observable fields, not read the gold identity of the pair it must classify.
docker compose run --rm tutorial python -m ged_rapprochement_fiches.cli prepare --config configs/minimum.yaml
docker compose run --rm tutorial python -m ged_rapprochement_fiches.cli baseline --config configs/minimum.yaml
docker compose run --rm inference python -m ged_rapprochement_fiches.cli inspect-mechanism --config configs/minimum.yaml
prepare creates the data. baseline produces simple reference methods for comparison with the model. Among them, the exact-reference method uses an equality signal; Jaccard measures lexical overlap. Jaccard's principle is “common elements / elements present in at least one of the two sets.” Exact set-preparation rules remain those of the ZIP.
inspect-mechanism shows serialized fields and symmetry. Serializing means turning a record's fields into an organized text input. The project passes the reference, its reliability, title, case, text, and version. The reliability field is not redundant: reading a reference and knowing it is trustworthy are different pieces of information.
Observe this representation before training. An educational inspection makes the computation readable; it must not be presented as an aggregate test score.
Train, interrupt, and resume
Training adjusts learned parameters using annotated pairs. Inference uses the resulting parameters to assign scores to new pairs.
docker compose run --rm learning python -m ged_rapprochement_fiches.cli train --config configs/minimum.yaml --mode interrupt
docker compose run --rm learning python -m ged_rapprochement_fiches.cli train --config configs/minimum.yaml --mode continuous
docker compose run --rm learning python -m ged_rapprochement_fiches.cli train --config configs/minimum.yaml --mode resume
docker compose run --rm learning python -m ged_rapprochement_fiches.cli verify-resume --config configs/minimum.yaml
The workflow distinguishes an interrupted run, a continuous run, and resumption. A checkpoint is a saved state for continuing computation; resumption is not arbitrary new training on the same files. verify-resume checks consistency between resumed and continuous runs. This reproducibility check does not replace evaluation on unfamiliar records.
To examine the training budget, optimizer, and restoration details, consult configs/minimum.yaml and the supplied code. Keep these settings during reference reproduction before experimenting in a separate copy.
From logit to decision
The model produces a logit, a raw score that can be positive, negative, or zero. Sigmoid converts it into a value between 0 and 1: sigmoid(z) = 1 / (1 + exp(-z)).
During training, the loss measures the gap between prediction and target. To understand unweighted binary cross-entropy, remember two cases: if the target is 1, the error is -log(p); if it is 0, the error is -log(1-p). High confidence in the wrong outcome is therefore costly.
Educational numerical example: for a genuinely positive pair, p = 0.9 yields loss of approximately 0.1054, versus 2.3026 with p = 0.1. Training uses this signal to adjust parameters; it does not write a textual rule from each example.
The BCEWithLogitsLoss documentation, also available in the PyTorch 2.8 reference, explains that this operation combines sigmoid and binary loss in a numerically stable way. It receives logits: do not apply an initial sigmoid “to help.”
An output between 0 and 1 remains a probabilistic model score. Its value does not prove an identical proportion of decisions will be correct on real documents.
Why calculate both A/B and B/A?
The identity sought is symmetric: if A and B represent the same document, so do B and A. Yet a model reading an ordered sequence can give two different logits depending on record order.
The project therefore encodes both orientations, then averages their logits. Here is the exact function:
def symmetric_logits(model, encoded_pairs, tokenizer):
if not encoded_pairs:
raise ValueError("Empty pair batch")
oriented = [row["orientations"][side] for side in [0, 1] for row in encoded_pairs]
batch = {
k: v.cuda()
for k, v in tokenizer.pad(oriented, padding=True, return_tensors="pt").items()
}
raw = model(**batch).squeeze(-1)
count = len(encoded_pairs)
return (raw[:count] + raw[count:]) / 2, raw[:count], raw[count:]
Start with oriented. The loop first visits side=0 for all pairs, then side=1 for all pairs. With pairs A/B and C/D, the order is A/B, C/D, B/A, D/C. It is not A/B, B/A, C/D, D/C. This organization explains the two slices in the return value.
tokenizer.pad(..., padding=True, return_tensors="pt") pads batch sequences to a common length and produces PyTorch tensors. Padding is technical filler, not document content; the attention mask distinguishes it from useful positions. The tokenizer contract details these inputs. Each tensor is then transferred to the GPU with .cuda().
The project's model provides one score per orientation. squeeze(-1) removes the final dimension of size 1: output with shape [2B, 1] becomes [2B], where B is the number of pairs. Do not confuse B pairs with 2B orientations.
Finally, raw[:count] contains direct logits and raw[count:] their reversed counterparts. Their elementwise mean returns one score per pair. Both raw series are also returned for inspection. An empty batch triggers an explicit error.
Averaging logits and averaging probabilities are not interchangeable
Illustrative example, unrelated to measured CamemBERT output: suppose a pair's logits are 4 in one direction and -2 in the other. Their mean is 1; its sigmoid is approximately 0.7311.
Applying both sigmoids first would give approximately 0.9820 and 0.1192. Their mean would be 0.5506, not 0.7311. The difference comes from the nonlinear transformation. To reproduce the project, retain the mean of logits, not a substituted mean of probabilities.
The final score is symmetric because swapping the two logits leaves their mean unchanged. In the r001 evidence, absolute symmetry error is zero for A-17/B-92. This verifies a computational property, not the identity of every record.
This property also assumes comparison of the same evaluations of both orientations. The snippet does not itself switch the model into evaluation mode. In PyTorch, eval() affects modules whose behavior differs during training, such as dropout, which randomly disables some values during learning. Calling context therefore remains important for a reproducible comparison.
Above all, symmetry is not transitivity. High scores for A/B and B/C do not guarantee a high score for A/C. This is precisely what grouping must handle.
Calibrate, group, and verify the veto
Choose a threshold before looking at the test
After training, inference produces scores without reading labels. Calibration then uses validation to choose the threshold, which is 0.5 in the r001 workflow. Testing measures the resulting fixed rule, rather than choosing afterward whichever value makes the result look best.
Here, calibration means choosing a decision threshold. This does not establish probability calibration: a score of 0.9 does not become a guarantee of 90% correct decisions through this operation alone.
docker compose run --rm inference python -m ged_rapprochement_fiches.cli predict --config configs/minimum.yaml
docker compose run --rm learning python -m ged_rapprochement_fiches.cli calibrate --config configs/minimum.yaml
docker compose run --rm inference python -m ged_rapprochement_fiches.cli cluster --config configs/minimum.yaml
docker compose run --rm tutorial python -m ged_rapprochement_fiches.cli evaluate --config configs/minimum.yaml
predict produces scores; calibrate fixes the decision rule on validation data; cluster builds groups; evaluate compares proposals with references. Keep the stages separate so errors can be located: wrong candidate, wrong score, or wrong group union.
The connected-components trap
The naive variant represents each record as a node. An accepted pair becomes an edge, an undirected link between two nodes. A connected component brings together records reachable by following these links, possibly through several intermediaries. The NetworkX documentation gives this definition; it does not mean that library is required by the course code.
Teaching example: with links A—B and B—C, all three records belong to the same component, even without an A—C link. If B was wrongly linked to C, the path alone is enough to bring C into A's group.
The problem is therefore not that the connected-components algorithm calculates incorrectly. It applies exactly the rule it was given. That rule may simply be too permissive for document identity.
Check every relationship between groups
The constrained variant asks a stricter question: can these two groups be joined without creating a forbidden or insufficiently justified relationship between their members?
Take groups {A, B} and {C, D}. Checking only the B/C link that suggested merging is insufficient. Cross-pairs are A/C, A/D, B/C, and B/D: four checks. More generally, joining groups of m and n members requires examining m × n cross-pairs.
The project's safeguard is shown below. Helper functions, including pair and cannot_link, belong to the complete ZIP code.
def can_merge_clusters(left, right, records, get_score, threshold):
crossed = sorted(pair(a, b) for a, b in product(left, right))
# Trusted administrative conflicts veto the entire merge, before neural work.
for a, b in crossed:
if cannot_link(records[a], records[b]):
return False, {"reason": "CANNOT_LINK", "pair": [a, b]}
for a, b in crossed:
value = get_score(a, b)
if value is None:
return False, {"reason": "REVIEW_UNSCORED", "pair": [a, b]}
if value < threshold:
return False, {
"reason": "BELOW_THRESHOLD",
"pair": [a, b],
"probability": value,
}
return True, {"reason": "ALL_CROSS_PAIRS_PASSED", "cross_pairs": len(crossed)}
The first line constructs every cross-pair with product(left, right), then places them in a stable order. It is limited neither to group representatives nor to the link that triggered the proposal.
The first loop looks for an administrative veto. If cannot_link finds two incompatible reliable references, the entire union is rejected with CANNOT_LINK. Because this loop precedes every get_score call, this function requests no neural score for that attempt, even if the conflict appears on the last examined pair.
A missing reference or one whose reliability is unconfirmed does not trigger this veto alone. But the absence of a veto is not proof of identity: score checks still have to pass.
The second loop requires a score for every cross-pair. get_score(a, b) supplies the available value; the workflow can compute additional scores for these checks. If the value remains None, the function returns REVIEW_UNSCORED. It does not convert “not examined” into “probably identical.”
If a score is strictly below the threshold, the union is rejected with BELOW_THRESHOLD. A value exactly equal to the threshold passes this check because the condition is value < threshold, not value <= threshold.
Finally, ALL_CROSS_PAIRS_PASSED means every cross-check succeeded. This is not yet a DMS modification: the function returns a boolean and a diagnostic. It deletes no files and itself merges no data.
What the check guarantees and what it assumes
Quick mental test: three of four cross-pairs score 0.9, but the fourth remains unscored. The union must be rejected even if the mean of known scores seems excellent. The same applies with a fourth value of 0.4 and threshold 0.5. These numbers illustrate the rule; they are not measured model scores.
Starting from already valid groups, checking every new cross-relationship preserves their constraints during a union. This still does not establish that each group corresponds to the correct document: distinct pairs can all receive wrongly high scores with no reliable reference to contradict them.
The snippet is not a complete input validator either. It assumes, among other things, nonempty groups, identifiers present in records, and finite numerical scores. It does not check these properties itself. For example, comparing NaN with the threshold using < does not trigger the rejection expected for an insufficient score. This describes a limitation of the isolated function, not the result of an audit of the project's upstream checks.
Finally, checking a local union does not necessarily find the best global partition. In an abstract example where A/B and B/C pass but A/C fails, joining A/B first or B/C first can lead to two different partitions. Sorting pairs within this function stabilizes their inspection; it does not prove the entire grouping process is independent of proposed-union order.
Read the test: the first loss occurs before the model
Recover the counts behind the scores
On the 648 test records, blocking retains 918 pairs out of 209,628 possible, approximately 0.44% of initial comparisons. This sharply reduces classifier work, but comes at a recall cost.
Of the 648 expected positive pairs, only 444 reach the candidates. That leaves 204 missing. Blocking recall is 444 / 648 = 0.685185.
On these 918 candidates, CamemBERT obtains 444 true positives, 4 false positives, and 0 false negatives. Its candidate recall is therefore 1. But zero false negatives refers only to received pairs, not those preselection excluded.
| CamemBERT pair metric | Calculation from published counts | Result |
|---|---|---|
| Precision | 444 / (444 + 4) |
0.991071 |
| Candidate recall | 444 / (444 + 0) |
1.00 |
| Candidate F1 | 2 × 444 / (2 × 444 + 4 + 0) |
0.995516 |
| End-to-end recall | 444 / (444 + 204) |
0.685185 |
| End-to-end F1 | 2 × 444 / (2 × 444 + 4 + 204) |
0.810219 |
The changed denominator explains the entire gap: the model recognizes the positives it receives, but the initial pair pipeline has already lost nearly a third. These 204 omissions cannot be fixed by training the same classifier longer on an unchanged candidate list.
The two reference methods yield end-to-end F1 of 0.568502 for Jaccard and 0.708373 for exact reference matching. CamemBERT therefore improves this metric in the supplied protocol without eliminating blocking losses.
Any additional grouping scores are counted at the next stage. They must not be retroactively included among the 918 candidates to claim a different initial recall: the two measurements concern different operations.
Preserving versions is not enough to form correct groups
Naive grouping preserves all 648 versions, but its groups contain 16 forbidden pairs and over-merge 6 logical documents. Its B-cubed F1 is 0.908720. A false pair link can therefore have consequences beyond its endpoints by connecting entire groups.
Constrained grouping also preserves all 648 versions, produces 295 groups, and contains no forbidden pairs or over-merged documents in this test. Its B-cubed precision is 1.00, recall 0.842593, and F1 0.914573.
Why does recall remain below 1 when precision is perfect? Because proposed groups no longer mix identities, but some remain incomplete: records for the same document stay separated. Conversely, preserving every version means only that none disappeared from outputs, not that each is with all its correct companions.
Constrained F1 is recovered with 2 × 1 × 0.842593 / (1 + 0.842593), approximately 0.914573. B-cubed recall is not pair recall: it averages recovered companions around each record and must not be confused with 0.685185.
Cost and limitations remain visible
The check requires 51 additional test pairs, against a planned budget of 20,000. These computations verify proposed unions; they are not equivalent to rerunning an exhaustive search for all missing pairs. Group fragmentation remains visible in B-cubed recall.
Observed grouping time includes inference on these extra pairs. The presented material gives no numerical duration to reproduce here, and this workflow justifies no throughput extrapolation. Doubling DMS size does not necessarily double this cost: candidate counts and group sizes also matter.
Results concern a synthetic corpus, with highly structured references and wording, one seed, and one machine. The seed fixes the experiment's pseudorandom draws; a single run defined this way does not measure variability across training runs. No business validation or human-reader study is provided.
The takeaway is therefore twofold: the model improves pair decisions in this protocol; the veto prevents the observed over-merges without making every group complete.
Exercise: predict a misleading transitive merge
Your prediction before the command
Return to the opening four records. A-17, B-92, and D-40 concern VC-204, with D-40 being version 2. C-11 concerns VC-205. Yet the model gives every pair high scores, including A-17/C-11.
Before running, predict how many groups naive connected components will produce. Then imagine the veto checked on every cross-pair: can C-11 be added to a group already containing A-17 simply because C-11 resembles B-92? Finally, what should happen to the version 2 file?
Write your answers, then run the exercise:
docker compose run --rm tutorial python -m ged_rapprochement_fiches.cli exercise --config configs/minimum.yaml
The command creates these four records and computes their actual model scores. The teaching numbers used earlier to explain logits and checks are not scores from this run. Likewise, any mathematical counterexamples marked illustrative in figures must not be confused with CamemBERT outputs.
What the r001 solution shows
The naive variant joins A-17, B-92, C-11, and D-40 into one group. It keeps all four versions but has 2 forbidden pairs and 2 over-merged logical documents, with B-cubed F1 of 0.769231.
The constrained variant forms two groups: {A-17, B-92, D-40} and {C-11}. It preserves all four versions, has no forbidden pairs, and achieves B-cubed F1 = 1.00.
The highlighted veto is A-17/C-11, whose reliable references conflict. B-92 does carry VC-204, but its reliability is unconfirmed. It must not become an intermediary allowing C-11 into A-17's group. That is the difference between checking the last added link and checking the union of groups.
Recalculate B-cubed by hand
To understand naive F1, set the network aside and examine only sets. Reference groups are {A-17, B-92, D-40} and {C-11}; the proposed group contains everyone.
For each of the three VC-204 records, all three correct members are present among four: local precision 3/4, local recall 3/3 = 1. For C-11, only one of four members is correct: local precision 1/4, local recall 1/1 = 1.
Mean precision is therefore (3/4 + 3/4 + 3/4 + 1/4) / 4 = 0.625. Mean recall is 1. F1 calculated from these two means is 2 × 0.625 × 1 / (0.625 + 1) = 0.769231 after rounding, reproducing the published value.
The naive group's perfect recall is not sufficient success: every correct companion is present, but so are outsiders. With the two correct groups, local precision and recall equal 1 for every record.
A variant to predict without changing the reference workflow
To check your understanding, imagine that C-11's reference reliability is no longer confirmed while high scores remain unchanged. Could a veto based on two reliable references still block A-17/C-11 by itself? No: its starting assumption would no longer hold. Score checks would remain necessary but would still not prove identity.
This is a reasoning variant, not executed in the published results. It shows why the definition of “reliable” must come from a business policy, not the model's confidence in its own score.
In the supplied exercise, as in the main workflow, output remains PROPOSITION_A_CONFIRMER. No original is deleted. A suggested group is a result to examine, not permission to erase a presumed duplicate.
Figures, milestones, and cleanup
Inspect before erasing
Read outputs/evaluation.json, outputs/groups.json, outputs/grouping.json, and outputs/exercise.json. Move from the overall score to a specific group: find its members, preserved versions, and reasons that prevented certain unions.
Then produce figures and check course milestones:
docker compose run --rm tutorial python lesson/figures/plot.py
docker compose run --rm tutorial python scripts/check_checkpoints.py
These milestones verify expected educational material. They complement evaluation without turning a proposal into administrative approval.
Save useful outputs before continuing. Cleanup follows this exact order:
docker compose down
docker compose run --rm tutorial python scripts/clean.py --outputs-only
docker compose down
The first down intentionally precedes cleanup; the second ends the workflow. clean --outputs-only displays and then deletes only outputs/, without touching records, files, corpus, or checkpoints. Do not run it before reading or copying results you want to keep.
If a problem occurs, return to the relevant stage
If the GPU check fails, revisit Docker environment diagnostics: the card's presence in Linux does not replace containerized computation. If acquisition fails, check downloads, sizes, and expected fingerprints rather than substituting a model found elsewhere.
If recall is low, first check whether the correct pairs appear among candidates. If a union is rejected, read its reason: CANNOT_LINK, REVIEW_UNSCORED, and BELOW_THRESHOLD represent three different situations. Raising a score does not resolve a reliable administrative conflict; inventing a score does not resolve an unexamined pair.
From tutorial to a real DMS
Before adapting it, define with business teams what constitutes the same logical document, what distinguishes a version from a neighboring document, and which references are genuinely reliable. The veto depends on these choices: incorrect information declared reliable can prevent a useful link.
Measure blocking recall on authorized, annotated data. Examine missing references, repetitive titles, and wording variations separately instead of settling for F1 on candidates alone. These are evaluation directions for a real application, not new populations already validated by this tutorial.
Version proposals and retain their members, scores, rules, and rejection reasons. Route unscored or ambiguous cases to review, separating that review from any file action. Document permissions and retention rules must continue to apply; membership in a proposed group grants no new rights.
The mechanism comes down to three distinctions: preselecting is not recognizing, recognizing a pair is not validating a group, and proposing a group is not merging its files. The veto blocks a predefined incompatibility; its absence does not certify document identity.
Download the course: r001 ZIP.
Documentation linked in the article explores the concepts further. Workflow snippets, commands, and results remain those of revision r001. Small calculations explicitly marked as educational explain these mechanisms without constituting new model measurements.