Panels and judge calibration
The engine is the finder of claims. [176] Each panel seat receives the claim, the lines, and the evaluation address. [176] No seat sees another seat's vote. [176]
A panel upholds a claim at a strict majority of its seats. [176] A runnable counterexample runs twice in a fresh container. [176] A counterexample that fails twice vetoes the claim. [176]
The calibration set
The record of real panels forms the labelled set. [176] The owner's rulings on those panels are the ground truth labels. [176] Each judge change is scored against that set before it ships. [176]
A judge change ships at a Cohen's kappa of 0.6 or better. [176] A judge runs without supervision at a kappa of 0.8 or better. [176] The judge model and its version stand beside each verdict. [176]
Failure classes
The steward counts the failure class of each decision. [176] The next design fix targets a class, not a single instance. [176] The table below comes from the current record. [119]
| Class | Count | Decisions |
|---|---|---|
| notification_only_path | 2 | 169, 156 addendum 5 |
| restart_race | 7 | 95, 148, 161, 170, 174, 168 addendum, 172 addendum |
| default_for_human | 13 | 97, 98, 102, 113, 132, 139, 153, 162, 165, 166, 139 addendum, 166 addendum, 166 addendum 2 |
| swallowed_failure | 5 | 90, 117, 131, 146, 154 |
| materials_or_version_drift | 0 | — |
| grader_mechanics | 2 | 111, 176 |
| no_failure_recorded | 194 | 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 91, 92, 93, 94, 96, 99, 100, 101, 103, 104, 105, 106, 107, 108, 109, 110, 112, 114, 115, 116, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 133, 134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 145, 147, 149, 150, 151, 152, 155, 156, 157, 158, 159, 160, 163, 164, 167, 168, 171, 172, 173, 175, 104 addendum, 105 addendum, 127 addendum, 128 addendum, 129 addendum, 132 addendum, 135 addendum, 123 addendum, 129 addendum 2, 119 addendum, 115 addendum, 138 addendum, 151 addendum, 150 addendum, 115 addendum 2, 115 addendum 3, 116 addendum, 154 addendum, 154 addendum 2, 154 addendum 3, 154 addendum 4, 156 addendum, 156 addendum 2, 156 addendum 3, 130 addendum, 154 addendum 5, 156 addendum 4, 156 addendum 6, 154 addendum 6, 156 addendum 7, 158 addendum, 158 addendum 2, 158 addendum 3, 160 addendum, 160 addendum 2, 155 addendum, 156 addendum 8, 155 addendum 2, 160 addendum 3, 156 addendum 9, 170 addendum |