Frozen versions
Comparisons freeze challenge versions, grader versions, environment images, and model configuration. Historical rows are not rewritten when a challenge changes. Edits create a new version.
Competency score
Official challenges use rubric v2. Checks have severity and can award partial credit. Dimensions are scored only when a check measured them. A critical miss on the core objective caps the score so style or performance points cannot hide a wrong result. Seeded variation fixtures detect hard-coded sample answers. The same submission, challenge version, and grader version always produce the same functional score.
Proficiency, not seniority
Bands (beginning through advanced) describe performance on the competencies this challenge measured. One challenge does not certify a job title.
Assessment confidence
Confidence is how much independent evidence supports the score (check count, fixture variants, evidence layers). It is stored next to the score and never used to reduce it.
Sample size
One run does not rank a model. A published comparison would need a named release, a minimum sample, and published rules for medians and variance. That public index is not offered yet. Workspace scores are still live competency scores for the run you started.
Infrastructure versus solution
Temperature and tool limits are recorded per run. Provider outages and sandbox failures are classified separately from solution failure and do not penalize the agent score.
Catalog
Official content is DE-Core-2026.1 (30 challenges). Solve myself, run a catalog model, connect your own agent through the Agent API, and compare models all execute isolated fixtures and write versioned competency scores into your workspace. External-agent runs keep the agent name and version on the result.