fix(m5): eval pipeline — anomaly_regions schema + nested JSON + pure-LLM baseline
Critical eval-pipeline bugs found while running the stage-2 model:
1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training
data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}.
The model learned the correct format but scored 0 → VUS-PR=0. Now accepts
both schemas + a balanced-brace JSON scanner (the flat regex could not
match nested objects the model emits).
2. qa_judge._extract_json had the same nesting bug — reuse the scanner.
3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped
to 192 in report + instruct_check.
Report enhancements:
- wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4
'model vs pure LLM' is actually computed. Heavy predictors built/evaluated
one at a time with GPU cleanup (single 12GB card).
- add QA-judge table across all 6 categories (design §6.5 问答均分).
- add --pure_llm / --max_per_cat flags.
Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm
0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither
model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
This commit is contained in:
@@ -16,7 +16,7 @@ import json
|
||||
import re
|
||||
from typing import Any, Callable, Dict, List, Optional, Tuple
|
||||
|
||||
from .parse_answer import parse_anomaly_segments
|
||||
from .parse_answer import _extract_balanced_json, parse_anomaly_segments
|
||||
|
||||
|
||||
_JSON_RE = re.compile(r"\{[^{}]*\}", re.DOTALL)
|
||||
@@ -25,6 +25,15 @@ _JSON_RE = re.compile(r"\{[^{}]*\}", re.DOTALL)
|
||||
def _extract_json(pred: str) -> Optional[dict]:
|
||||
if pred is None:
|
||||
return None
|
||||
# prefer balanced (nested-capable) extraction first
|
||||
for blob in _extract_balanced_json(pred):
|
||||
try:
|
||||
obj = json.loads(blob)
|
||||
if isinstance(obj, dict):
|
||||
return obj
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
# fallback to flat blobs
|
||||
for blob in _JSON_RE.findall(pred):
|
||||
try:
|
||||
return json.loads(blob)
|
||||
|
||||
Reference in New Issue
Block a user