Define the baseline and the evaluation separately.
The baseline is the observed performance of the current process under stated conditions. The evaluation defines the tasks, metrics, constraints and checks used to compare candidates. Record both before interpreting a proposed improvement.
A metric definition should specify the numerator, denominator, time window, exclusions and relevant groups. In the internal support test, choosing a knowledge card and meeting valid-closure conditions were different events. Correcting that distinction changed what was measured; it did not establish a higher resolution rate.
Write down competing explanations.
A delayed outcome can have several plausible causes. For each explanation, identify the supporting records, the evidence that would contradict it and the smallest useful experiment. Missing evidence should remain visible in the decision.
Polaris divides evidence gathering, analysis and verification into separate tasks. Failed hypotheses are retained, and a rejected explanation needs new evidence or changed conditions before it is proposed again. This keeps the investigation connected to observations across successive attempts.
Choose checks that match the claim.
For suitable executable tasks, acceptance rules are fixed outside the candidate workspace. The two-stage validation protocol separates the candidate run from scoring; the candidate cannot read the protected scoring scripts or labels. An independent review examines the outputs and supporting evidence.
Research diagnosis does not always have executable acceptance. The published R&D queue case includes replay and review records but no independent frozen acceptance. Its modeled estimate must therefore retain its assumptions and research scope instead of being described as a passed blind benchmark.
| Claim under evaluation | Evidence to examine |
|---|---|
| A metric was calculated correctly | Input definitions, recomputation and edge-case checks |
| A proposed artifact passed acceptance | Fixed criteria, isolated execution and recorded results |
| A simulation supports a hypothesis | Model assumptions, source fit and sensitivity to missing mechanisms |
| A business change improved operations | Actual pilot records and an agreed comparison method |
Record what changed between attempts.
The project history connects each hypothesis to a candidate, an executed check and the next decision. Include failed attempts, reviewer objections, source versions and unresolved questions so another reviewer can understand why the work changed.
This feedback-driven iteration is how Polaris describes recursive self-improvement within an evaluation. It refines hypotheses, investigation choices and candidate plans. These records do not by themselves demonstrate model-weight training or prove that an approach will transfer to a different organization.
Measure business impact in a separate stage.
A proposal that clears offline checks can inform a limited pilot. Before changing operations, agree the responsible team, intervention, comparison group or period, quality requirements and rollback conditions. Record concurrent changes that could affect interpretation.
Distinguish saved effort, shorter elapsed time and reduced spending. They answer different business questions and should not be added together as if they were the same benefit. The published internal tests establish bounded technical or research findings; field impact must be measured in the relevant customer setting.
Questions before an evaluation
Does every case have a held-out benchmark?
No. The validation scope depends on the case. Executable frozen acceptance, reviewed research and actual business pilots are different evidence levels and are described separately.
What does a passed check mean?
The candidate met the particular rules and inputs used by that check. Broader reliability and operational benefit require additional evidence.