Validation guide

Validation guide — process & statistics audit

This is the auditor's manual. Open output/ and output/backtest_2024_25/ next to it; cross-check every stat to its source. Math lives in composite_score.py; pipeline in analysis.py; data in source-data/.

Design contract. Strength is anchored on finish-tier results, not on the same box stats we're scoring. The model adds adjustments on top of last year's finish — it does not re-rank by composite. Composite re-sort is a diagnostic only.

1. Run the pipeline yourself

cd glax-pipeline
python3 fetch_awards.py 2026                  # refresh awards_2026.csv (live data)
python3 run_glax_analysis.py --season both    # forward + backtest
python3 -m unittest discover -s tests -v      # 49 tests
python3 awards_tune.py                        # award weight grid (optional)

Reproducible artifacts:

2. Pipeline at a glance

finish_2026.tsv ─► tier / smooth SOS  ─┐
finish_2025.tsv ─► legacy prior ──────┐│
results.csv     ─► RPI + AGD ─────────┐││
                                       ▼▼▼
team_game_stats.csv ─► team_power (offense / defense / possession)
                            │
player_game_stats.csv ─► player_impact (SOS share × leverage)
                            │
roster.csv + class year ─► graduation projection (replacement-level)
awards_2026.csv         ─► award departure boost / returning relief
                            ▼
                projected_power_2027  ─► roster_rank_predictor (capped move)
                                       └► secondary_regression (flagged only)
                                       └► board_overrides (manual notes)
                                       └► schedule_risk_matrix (k × Δpower)

3. Domain math (validate every score in team_power_*.csv)

Each game produces three raw 0-100 scores; the season composite is the leverage × SOS-weighted average of those games.

3.1 Per-game raw scores (composite_score.py)

Offense — score_offense

possessions_floor = cal.offense_poss_floor      # 25 in 2025-26
rate = goals_scored / max(possessions_estimate, possessions_floor)
raw  = squash_high_is_better(rate, p50, p90)    # logistic on calibrated anchors

p50 → 55, p90 → 92, soft tail to 100. With the floor, scoring 6 goals on 30 possessions = rate 0.20; scoring 14 goals on 35 possessions = rate 0.40. Audit: open source-data/glax_2025_26_team_game_stats.csv, divide goals by possessions for any game, run domain_calibration.squash_high_is_better(rate, 0.40, 0.64) — the result must equal that game's offense_score in any debug log.

Defense — score_defense

Goalie block (when saves + goals_allowed present):

shots_faced = goals_allowed + saves
goalie = 0.45 * squash_low(shots_faced) + 0.55 * squash_high(save_pct)

Field block (GB + INT + CTO — no double-count in possession):

field_events = 0.70*GB + 0.80*INT + 0.85*CTO
field = squash_high(field_events, defense_field_p50, defense_field_p90)

Blend when both exist: 0.55*goalie + 0.45*field. Either block alone is used if the other is missing.

If goalie minutes are missing and goals_allowed = 0, the goalie block is None (not 100). Covered by tests/test_anchor.py.

Possession — score_possession

events = 1.00 * draw_controls
raw    = squash_high(events, possession_p50, possession_p90)

Possession baseline is the league p90 of draw-weighted events (run_meta.json → possession_baseline).

3.2 Game-to-season aggregation

context = sos_weight(opponent_rank) × leverage(margin)
weighted_avg = Σ(raw_g · context_g) / Σ(context_g)

• sos_weight — log-scaled smooth_finish_sos_weight (rank 1 → 1.00, rank 32 → 0.30). For OOS opponents it falls back to RPI strength (resolve_opponent_sos_rank); never bottom-tier-by-default.
• Top-20 boost — when both teams are finish ≤ 20, context × 1.08 (TOP20_SOS_BOOST). Records this in run_meta.json.
• leverage — 1.5 close (≤2 margin), 1.2 medium (3–5), 1.0 blowout (≥6). Mirrors Powerwise MOV cap of 10.

3.3 Composite

composite = 0.35 * offense + 0.35 * defense + 0.30 * possession

Domain weights live in DOMAIN_WEIGHTS. Sensitivity sweeps showed they don't move top-20 MAE meaningfully (within ±0.1). Composite is not the rank source — see §6.

4. Player impact (validate every row in player_impact_*.csv)

Per player, per domain, in compute_player_impacts:

share_raw = player_event / team_event                    # unweighted
share_sos = Σ(player_event · context) / Σ(team_event · context)
power_pts = share_sos · team_domain_score · domain_weight

• Position-aware (position_class): attack/midfield credit offense + possession; defense/goalie credit possession + defense; goalies credit defense via goalie minutes when present, else fall back to (ga_share + sv_share) / 2.
• stat_type='combined' rows are dropped from team aggregation (glax_data.py line ~338). Avoids double-counting.
• Roster matching uses normalized full name → jersey-anchored match → strict last-name equality (not substring). Unmatched seniors print to stderr; spot-check after every season.

Self-check. Sum offense_share_sos over a team across players matched to roster — should be ≤ 1.0 (some events are unattributed).

5. Graduation projection (validate every row in projected_power_*.csv)

For each departing senior in glax_*_roster.csv:

off_share = senior's SOS-weighted share of team goals (offense + utility positions only)
pos_share = senior's SOS-weighted share of weighted possession events
def_share = senior's SOS-weighted share of goalie minutes (or ga/sv if no minutes)

# Replacement-level subtraction (Opus #1 follow-up)
off_loss  = max(off_share - replacement_baseline[position]['offense'], 0)
pos_loss  = max(pos_share - replacement_baseline[position]['possession'], 0)
def_loss  = max(def_share - replacement_baseline[position]['defense'], 0)

Replacement baselines come from non-senior SOS shares pooled across 2024-25 + 2025-26 (roster_projection.get_replacement_baselines() and saved into run_meta.json → replacement_baselines). A bench attacker who steps in is not zero.

Concentration / depth / continuity multipliers (roster_projection.py):

Damping & caps (audit run_meta.json):

GRADUATION_DAMPING = 1.0      # no flat damping (Opus follow-up)
MAX_OFFENSE_LOSS   = 0.38
MAX_POSSESSION_LOSS = 0.42
MAX_DEFENSE_LOSS   = 0.45

After replacement-level subtraction, caps bind for fewer than 3 teams (was ~10 before the rewrite). When a cap binds it is reported in roster_stat_adjustments so the board can spot it.

Legacy prior (legacy_program_priors.csv): multi-year finish_*.tsv rolling strength → legacy_composite_adj (small ±2 nudge, weight = 0.12). MICDS, John Burroughs, Eureka, Cor Jesu sit > 0.80.

Manual sanity check. For any team in projected_power_*.csv:

projected_composite ≈ 0.35*off*(1−off_lost) + 0.35*def*(1−def_lost) + 0.30*pos*(1−pos_lost) + legacy_composite_adj

Reconstruct from team_power_*.csv + the loss percentages on the projected row + legacy_composite_adj. Should match within rounding.

6. Rank prediction (validate league_rerank_*.csv and ranking_recommendations_*.csv)

6.1 Roster-adjusted rank — the vote default

roster_rank_predictor.predict_ranks_from_graduation_rows (PROJECT.md design rule):

turnover_loss_pct_per_rank = 7.0 (% departure loss above median → ~1 rank slot)
proposed_move         = (departure_loss_pct - median_loss) / turnover_loss_pct_per_rank
stability_factor      = legacy + returning_share + low_loss_bonus + top8_bonus
move                  = proposed_move × (1 - stability_factor)
move                  = clamp(move, -MAX_RANK_SHIFT, +MAX_RANK_SHIFT)   # ±4 slots
projected_league_rank = prior_finish + move (then conflict-resolved to unique slots)

This is the primary predictor. Composite re-sort (projected_league_rank_composite) is shown only as a diagnostic — and the backtest below shows why we don't use it.

6.2 Secondary regression — flagged teams only

secondary_regression.apply_secondary_adjustments nudges the proposed slot when a team trips one of:

The output is recommended_rank_adjusted (board vote line) plus secondary_slot_delta and human-readable secondary_adjustment_reasons.

6.3 Board overrides

board_overrides_{label}.csv is a hand-edit template: program_notes, board_voted_rank, final_rank. The pipeline merges it back into the recommendations before the markdown brief is rendered.

7. Award point scale (validate awards_*.csv + awards_tune_results.csv)

Source: https://stl-lacrosse-awards.fly.dev/data/season-results-{year}.json (live for 2026; older years 404 until released).

Tuned weights (winner of the holdout grid in awards_tune_results.csv):

Graduation effect (per awards.apply_award_graduation_adjustments):

dep_boost  = min(0.14, departing_pts × 0.022)
total_off *= (1 + dep_boost)
total_pos *= (1 + dep_boost × 0.6)
total_def *= (1 + dep_boost × 0.8)

ret_relief = min(0.10, returning_pts × 0.018)
total_off *= (1 - ret_relief)
total_pos *= (1 - ret_relief × 0.5)

Board flag award_talent_loss triggers when departing points ≥ 10.

To re-tune: python3 awards_tune.py. The grid sweeps a dozen weight schemes against the 2025-26 holdout; the winner is recorded back in awards_weights.TUNED_AWARD_WEIGHTS.

8. Backtest validation (output/backtest_2024_25/)

Hold out 2024-25 stats + 2025 grads → predict 2026 finish. The numbers below are the current pipeline with all Opus fixes applied:

Reading guide:

• The roster-adjusted board line wins on every top-N cohort (top-8, top-16, top-20). Naive carry-forward is competitive on the bottom of the league, where movement is noise.
• Composite re-sort is included to show why we don't use it — re-sorting by box-score-driven composite triples the error in the top half. The model's value is in adjusting the prior, not in re-sorting.

Tier-conditioned MAE is in backtest_benchmark_by_tier.csv and backtest_team_rank_errors.csv (per-team residuals for every predictor).

Award holdout sweep: awards_tune_results.csv reports MAE for high-award-loss teams under each weight scheme.

9. Schedule risk (schedule_risk_matrix_*.csv)

k = least-squares fit of margin ≈ k × (composite_a − composite_b)  on MO-vs-MO games
expected_margin_a_minus_b = k × (projected_a − projected_b)
blowout = abs(expected_margin) ≥ 10

Two k values are written to run_meta.json: full-season (margin_k) and top-16-only (margin_k_top16). Both directions are written for every pair. No HFA yet — open backlog item.

10. Test coverage

49 unit tests across 14 files (tests/):

python3 -m unittest discover -s tests -v

11. Opus review — items shipped and items still open

Original review surfaced 5 P0/P1 weaknesses + ~30 section findings. Audit:

Conversation-level adds beyond Opus:

• #5 STL Lacrosse Awards — full ingest, roster matching, graduation effect, board flag, holdout-tuned weight grid.
• Award weights — All-American = 2× state 1st, conference-tier nuance.
• Per-team backtest residuals — backtest_team_rank_errors.csv.
• Forward benchmark file — benchmark_2027.md.

Open follow-ups (low priority unless you want them now):

1. Bootstrap rank confidence intervals (proj_rank_lo, proj_rank_hi).
2. Home-field advantage on schedule margins.
3. _int to distinguish missing vs zero in raw stat ingestion.
4. Backlog game-rooted predictors: goals per possession, SOS-weighted GA vs elite, H2H nudge inside tight clusters.

12. Auditor's checklist (one pass per season)

1. python3 run_glax_analysis.py --season both runs without errors.
2. python3 -m unittest discover -s tests — 49/49 pass.
3. output/run_meta.json shows damping=1.0, replacement_baseline=true, legacy_prior=true, oos_games_excluded_from_stats=true.
4. No team in team_power_*.csv has all three domain scores ≥ 95 (saturation guard). 4b. No team with games played has offense_score, defense_score, or possession_score exactly 0 (zero-floor guard — see sanity_report_*.md § Zero-domain guard).
5. composite_score == 100 ⇒ at least one game with all three domain raws hitting 100 (rare).
6. For every senior in roster.csv with grad_year == season_grad_year, a row exists in graduation_player_loss_*.csv or stderr logged the unmatched name.
7. projected_composite ≈ Σ domain × weight × (1 − loss) reconstructs from CSVs (within 0.05).
8. recommended_rank_adjusted for the prior #1 team is not the composite re-sort #1 unless they actually align (sanity: composite should never override the vote default by itself).
9. Backtest top-20 MAE for Roster-adjusted + secondary (board line) ≤ 2.5.
10. Award departure flagged teams (board award_talent_loss) have departing_awards_points ≥ 10 and a non-empty departing_awards string.

If any of those fail, do not ship the report — open output/sanity_report_*.md and the relevant CSV first.