The evaluation harness can do more than score a model. Comparable stock and harnessed answers, citations, tool traces, and per-dimension grades can become candidate teaching material. DueCare turns that evidence into lineage-safe supervised fine-tuning (SFT) data first, Direct Preference Optimization (DPO) data second, and model-specific Low-Rank Adaptation (LoRA) experiments — without treating generated text as automatically fit to train.
The landing route starts with the two versioned datasets, then moves through a loading guide and a cross-dataset quality audit before any training plan.
Positive supervised targets, same-prompt preference pairs, reward labels, a raw-text-free response inventory, and an explicit quarantine lane. Release-manifest Secure Hash Algorithm 256-bit checksum (SHA-256): 56fa69c…22b74b.
Deterministic fictional scenarios across perspective, journey stage, evidence state, time, and jurisdiction. It contains visible decision scaffolds, not real worker cases or hidden reasoning. Release-manifest SHA-256: ea644df4…9211d.
Verifies both manifests, explains each file type in plain language, demonstrates standard Python, pandas, Hugging Face Datasets, Kagglehub, and Polars loaders, and saves three reviewer-facing charts.
Checks exact prompt and lineage-family isolation across train, validation, and test splits; profiles text and shard sizes; and writes an audit table, summary, report, and four charts.
The same prompt is run through controlled arms, graded against the same rubric, and retained with exact identity and provenance. Promotion is a sequence of explicit gates, not a score threshold alone.
Record stock and harnessed responses for an approved prompt, exact model revision, harness version, and knowledge-pack versions.
Retain citations, tool traces, deterministic checks, per-dimension grades, and intentionally visible reviewer rationales.
Reject unclear licensing, unsafe advice, sensitive data, unsupported citations, hidden-thought markup, duplicates, or incomplete grading.
Assign complete prompt and source lineages to train, validation, or held-out sets before trainer-specific formatting.
Train on approved final answers and reviewable response structure. Stable behavior belongs in the adapter; volatile facts stay in tools.
After SFT, prefer a grounded chosen answer over a traceable rejected failure using DPO or another compatible preference method.
Compare stock, stock + harness, trained, and trained + harness on the untouched holdout and on benign controls.
Send reviewed gaps back to prompt design and curation. Do not train on the evaluation holdout that exposed them.
Supervised fine-tuning and preference optimization answer different questions. DueCare records both stages separately so a skipped or failed preference stage cannot be reported as a completed training run.
An SFT row maps an approved instruction or chat history to a reviewed final answer. It can include citations and a deliberately authored, model-visible reasoning scaffold. The prompt portion can be masked so the loss trains the reply rather than memorizing the input.
A Direct Preference Optimization (DPO) row keeps one prompt with a chosen and rejected answer plus the reason for that preference. The rejected answer must represent a known failure — such as unsupported advice, missed evidence, or over-refusal — rather than an arbitrary weak sample.
Minimal-pair facts, controlled contract ablations, and reviewed reasoning repairs test whether the model follows evidence instead of keywords. These variants keep their source lineage and never silently replace the canonical row.
The export may contain complete final answers, citations, harness traces, concise explanations intentionally returned to the user, and visible judge or reviewer rationales. It does not retrieve, scrape, infer, or publish a provider’s private hidden chain-of-thought. Hidden-thought markup blocks a row from training.
Trainer formats can change. The canonical record stays tied to the prompt, source, model revision, review decision, and split assignment that made it eligible.
| contract field | what it records | why it blocks release when absent |
|---|---|---|
| Identity | Stable row identifier, prompt hash, response hash, generator version, artifact hash. | Prevents silent mutation and makes an exported row reproducible. |
| Lineage + split | Prompt lineage, source lineage, typology/corridor groups, train/validation/held-out assignment. | Prevents the same family or a near duplicate from leaking into evaluation. |
| Model provenance | Provider, exact base model identifier, immutable revision, harness/rubric version, pack versions. | An adapter and its result are meaningful only against the base revision actually used. |
| Source + license | Source references, allowed use, license metadata, curator decision, transformation history. | Public availability alone does not grant permission to train or redistribute. |
| Training content | Final answer, citations, optional visible rationale, SFT messages or chosen/rejected preference pair. | Separates reviewable model-visible material from prohibited hidden reasoning. |
| Safety evidence | Personally identifiable information scan, unsafe-advice check, citation verification, deterministic checks, grades, rejection reasons. | A high aggregate score cannot override a privacy, evidence, or safety failure. |
The flywheel is useful wherever a team needs to improve behavior without losing the ability to explain which data, model, and evaluation produced the change.
Turn recurring, reviewed failures into SFT targets, preference pairs, hard negatives, or counterfactual tests while preserving the original evaluation evidence.
Review provenance, redistribution rights, citations, privacy findings, and lineage assignments before creating a versioned public training-data candidate.
Run adversarial rewrite, protective answer generation, and strict judge review against approved seed prompts on a local Ollama host. The output is supervised and preference candidate JavaScript Object Notation Lines plus raw-text-free quarantine metadata; it remains safe_to_train=false until the full release gates pass.
Compare how much harness behavior different compatible base models internalize, then measure whether the harness still adds value after training.
Recompute hashes, inspect exclusions, verify untouched held-out lineages, and distinguish a dataset release from a notebook run or adapter release.
The active notebook can import an approved response bundle, export manifest-bound supervised and preference JavaScript Object Notation Lines, preflight a compatible NVIDIA Compute Unified Device Architecture runtime, run resumable Low-Rank Adaptation stages, and reload the exact base plus adapter for evaluation.
Start with a commit-pinned official Gemma 4 E2B or E4B model-variant preset, verify the dataset manifest and dependencies, run supervised fine-tuning, then run the requested preference stage. Download the adapter, logs, completion manifest, and four-arm results before the session ends. The larger 31-billion-parameter evaluation target is a workstation/server-class path, not a phone deployment claim.
The JavaScript Object Notation Lines and manifest contracts can feed controlled Unsloth, Hugging Face Transformer Reinforcement Learning, or Parameter-Efficient Fine-Tuning environments for another compatible base. Each model and immutable revision receives its own adapter and evaluation; an adapter trained for one base is never presented as portable to another.
A-00’s Already have a file? path can inspect controlled JavaScript Object Notation, JavaScript Object Notation Lines, or compressed archive exports and suggest a prompt, grading, comparison, or training workflow. A loose file stays inspection-only: it cannot train until its supervised/preference artifacts and manifest pass the same hashes, model-revision, provenance, license, privacy, lineage, holdout, and quality gates.
A notebook, dataset version, and adapter are separate releases. Each carries its own evidence, and later layers cannot borrow confidence from earlier smoke tests.
The website is an orientation layer. The repository owns the executable gates and the versioned methodology.