Red-team follow-up: dedup config, PK validation, coverage warning

[nick-gorman]

Summary

Stacked on top of #98. Four small commits, each addressing one finding from a red-team pass against master (full notes in the companion nemosis-smoke repo's redteam/FINDINGS.md). All within the spirit of #98 — convert previously-silent footguns into clear errors or warnings, with regression tests — but slightly broader in scope, so kept on a separate branch.

What's in here

f5b947c — F5: install standard finalise pipeline for MARKET_PRICE_THRESHOLDS

processing_info_maps.finalise["MARKET_PRICE_THRESHOLDS"] was None, unique among search_type="all" tables. Every other one ships [most_recent_records_before_start_time, drop_duplicates_by_primary_key] — MARKET_PRICE_THRESHOLDS got neither. Effect:

• A 5-month live query returned 1269 rows for 16 distinct (EFFECTIVEDATE, VERSIONNO) pairs — the same byte-identical row repeating up to 114 times.
• Joining MARKET_PRICE_THRESHOLDS to DISPATCHPRICE on EFFECTIVEDATE produces cartesian-explosion bugs.
• Aggregates (mean VOLL etc.) are heavily biased toward older effective dates because more monthly archives ship them.

effective_date_group_col for this table is already [], which sends most_recent_records_before_start_time down the .tail(1) branch. PKs and primary_date_columns are also already set up. The fix is just to install the standard pipeline.

Strengthens the existing fixture test to assert the PK invariant — verified the new assertion FAILS without the fix (the 2-month fixture exhibits 2× duplication) and PASSES with it.

02b4ee2 — F3: validate PK columns in user select_columns up-front

For tables whose pipeline dedupes on PK, omitting a PK column from user-typed select_columns previously caused a bare pandas KeyError: Index(['VERSIONNO'], dtype='str') from inside the finalise step. Cryptic — the user has no clue the fix is "add VERSIONNO back" (or just use defaults).

New _validate_user_select_columns_includes_pk covers both:

• Dynamic: finalise pipeline contains query_wrappers.drop_duplicates_by_primary_key.
• Static: static_data_finaliser_map[table_name] contains _finalise_excel_data (which calls drop_duplicates(primary_keys) when the table is in defaults.table_primary_keys).

Raises a clear UserInputError naming the missing PK column(s) and suggesting either adding them back or passing select_columns=None. Two regression tests in test_errors.py.

46db129 — F1 (static): wire _check_loaded_select_columns into static_table

_check_loaded_select_columns was added in #98 for dynamic_data_compiler but not static_table. So a typo in user-typed select_columns (['Participant', 'duid'] instead of ['Participant', 'DUID']) silently shipped a stub DataFrame from static_table with just Participant.

Apply the same guard to static_table. Also fix the missing-raise in the post-load filter_cols check at the static_table call site, and in the matching spot in dynamic_data_compiler (was constructed-but-not-raised — same shape as the bugs fixed in #98 but two call sites further down). With _check_loaded_select_columns in place this path is mostly unreachable, but defence-in-depth matters for the defaults-inherited-but-vintage-missing edge case. Two regression tests.

896a56f — F15: partial-coverage WARNING for gaps in the user's window

A multi-period query that partly succeeds and partly 404s previously had no aggregate signal — only N separate per-file warnings that aggregating users tend not to read. So asking for 16 months of INTERMITTENT_GEN_SCADA against AEMO's ~6-7-week Reports/Current/ retention silently returned ~10% of the requested range, with all downstream means/aggregates correspondingly wrong.

Track per-period success through the fetch loop, but only count iterations whose data period actually overlaps the user's [start_time, end_time) window. After the loop, if some — but not all — of those in-window periods returned data, emit a single summary WARNING naming the table, missing count, coverage percentage, and the three common causes (retention window, not-yet-published month, pre-AEMO data).

The in-window check (_iteration_overlaps_window) translates each date_gen iteration into its [period_start, period_end) extent — monthly for MMS, daily for current/scraped, 5-min for FCAS — and runs the standard strict half-open overlap test:

period_start < user_end AND period_end > user_start

This naturally excludes the two kinds of "behind-the-scenes" iterations the fetch loop makes that the user didn't directly ask for:

• The 1-day buffer-back month/day before start_time — sits flush against user_start from the left, fails the strict period_end > user_start test.
• The ~120 historical months a search_type='all' table scans to compute "snapshot as of start_time" — entirely before user_start.

So a PARTICIPANT query for [2024-01-01, 2024-06-01) counts 5 in-window months (Jan–May) regardless of the ~115 historical-scan + buffer-back months the loop actually iterated. If all 5 return data → no warning. If one is missing → 4/5, warning fires for a real gap.

Three regression tests:

• partial coverage (DISPATCHPRICE 2018-04 → 2018-12 against a fixture with only 2018-04 and 2018-05; 2/8 in-window) emits the WARNING.
• full coverage (single fixtured month) emits NO warning — prevents the signal from being drowned in noise on every healthy query.
• out-of-window 404s for a search_type='all' table emit NO warning, even when scan months 404 — proves the in-window check correctly suppresses the historical-scan noise that a flat success-rate ratio would have false-positived on.

Test plan

• uv run pytest tests/ — 435 passed, 1 skipped (was 432 on PR Polish from DISPATCHPRICE user-testing: input validation, log clarity, index reset #98).
• Re-installed locally into the nemosis-smoke env and ran the three red-team smoke gates — all PASS:
  • 11_market_price_thresholds_dedup.py — confirms PK invariant on a 5-month live query (was FAIL before this branch)
  • 12_filter_cols_silent_skip.py — confirms filter_cols with a missing column raises (was FAIL)
  • 13_current_table_coverage.py — confirms the "Partial coverage" WARNING surfaces for a multi-month current-table query (was FAIL)
• Re-ran the existing smoke set (01, 03, 04, 06, 07, 09, 10) — all still PASS. (08 has a pre-existing smoke-script bug unrelated to NEMOSIS; 02 and 05 skipped only because they take ~3-5 min each.)
• Sanity-tested the partial-coverage warning verbiage live: INTERMITTENT_GEN_SCADA 4-month query shows 47/120 periods in the requested window returned data (39%) plus the cause hints.

Notes for review

• F5 is the only commit that changes data semantics for a previously-supported table. Anyone currently relying on "MARKET_PRICE_THRESHOLDS returns N copies of each row" (presumably nobody) would see a row-count change. The actual data values are unaffected.
• F3's static-table guard catches static_table("Generators and Scheduled Loads", select_columns=['Participant', 'Region']) (missing the DUID PK). If this is a known idiom anywhere, the error message points the user at select_columns=None.
• The F15 coverage check has no threshold/heuristic — it warns iff at least one in-window period is missing (and at least one succeeded; zero-data is NoDataToReturn's job). The strict half-open overlap test handles both the buffer-back month and search_type='all''s historical scan automatically.

🤖 Generated with Claude Code