fix(m5): eval pipeline — anomaly_regions schema + nested JSON + pure-LLM baseline

Critical eval-pipeline bugs found while running the stage-2 model:

1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training
   data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}.
   The model learned the correct format but scored 0 → VUS-PR=0. Now accepts
   both schemas + a balanced-brace JSON scanner (the flat regex could not
   match nested objects the model emits).
2. qa_judge._extract_json had the same nesting bug — reuse the scanner.
3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped
   to 192 in report + instruct_check.

Report enhancements:
- wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4
  'model vs pure LLM' is actually computed. Heavy predictors built/evaluated
  one at a time with GPU cleanup (single 12GB card).
- add QA-judge table across all 6 categories (design §6.5 问答均分).
- add --pure_llm / --max_per_cat flags.

Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm
0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither
model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
This commit is contained in:
张宗平
2026-06-30 10:02:59 +00:00
parent 8aeddebce9
commit 40ec960cf8
8 changed files with 430 additions and 55 deletions
+28
View File
@@ -45,6 +45,34 @@ class TestParseSegments:
# ok=False because no valid JSON, but segments extracted via fallback
assert segs == [(100, 150), (200, 250)]
def test_anomaly_regions_schema(self):
# the schema data/instruct.py actually emits (nested, start/end objects)
text = '{"anomaly_regions": [{"start": 10, "end": 93}, {"start": 245, "end": 286}], "n_regions": 2}'
segs, ok = parse_anomaly_segments(text)
assert ok is True
assert segs == [(10, 93), (245, 286)]
def test_anomaly_regions_in_prose(self):
text = 'Based on the series: {"anomaly_regions": [{"start": 5, "end": 8}]} done.'
segs, ok = parse_anomaly_segments(text)
assert ok is True
assert segs == [(5, 8)]
def test_segments_with_start_end_objects(self):
# also accept the {start,end} object form under the 'segments' key
text = '{"segments": [{"start": 1, "end": 4}]}'
segs, ok = parse_anomaly_segments(text)
assert ok is True
assert segs == [(1, 4)]
def test_balanced_json_handles_nested(self):
from tsmm.eval.parse_answer import _extract_balanced_json
text = 'x {"a": {"b": 1}, "c": [2,3]} y {"d": 4}'
blobs = _extract_balanced_json(text)
# outer nested object + the flat one, but NOT the inner {"b":1} alone
assert any('"a"' in b and '"b"' in b for b in blobs)
assert '{"d": 4}' in blobs
def test_empty_segments_list(self):
text = '{"segments": [], "type": "none"}'
segs, ok = parse_anomaly_segments(text)