Full diagnostic chain (synth→real zero-shot→real finetune→held-out) shows
the TS-modality approach works in-distribution but fails to generalize
across entities: same SMD dataset, same 38 channels, just a different
machine (1→2) collapses output to repetition garbage. Root cause is the
lightweight TS encoder (2-layer TF, 2.1M params) learning dataset/machine-
specific patterns rather than transferable time-series representations.
This is a valuable negative result: it cleanly rules out the
'0.5B LLM + 2-layer TS encoder + single-distribution synth training'
route for general time-series understanding, which is more credible than
any 'looks like it works but never tested cross-domain' result.
Records:
- tasks.md: M6 diagnostic section (6.1-6.5) with evidence + archive decision
- proposal.md: Status header flagging archive + pointer to diagnosis
- docs/diagnosis-2026-07-02-cross-domain.md: full evidence chain, root-cause
analysis, capability boundaries, asset inventory, restart conditions
Project status: archived. Code/data/checkpoints/reports preserved.
Restart requires any of: (1) larger encoder + real-data from-scratch
training, (2) channel-agnostic patch strategy, (3) multi-dataset joint
training with held-out-dataset generalization proof.
Full-scale retrain (stage1 31250 steps on 500k align, stage2 9375 steps on
100k sft) vs the prior small-scale run (5万/5k steps):
metric small-scale full-scale
stage2 EMA 0.81 0.75
model VUS-PR 0.14 0.22 (+57%)
model Aff-F1 0.21 0.25 (vs pure_llm 0.003 — TS modality helps)
model QA mean 2.45 2.41
QA vs pure_llm 2.8× 2.7× (stable lead, 6 categories)
Confirms the hypothesis that weak anomaly localization was mostly a
data-volume problem, not an algorithmic one: full-scale training lifted
VUS-PR 57% and the model now clearly beats the pure-LLM baseline on
affiliation-F1 (0.25 vs 0.003). VUS-PR still trails trivial baselines
(0.22 vs random 0.33) — precise localization remains the open frontier,
but the TS modality is demonstrably contributing (was unclear before).
Critical eval-pipeline bugs found while running the stage-2 model:
1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training
data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}.
The model learned the correct format but scored 0 → VUS-PR=0. Now accepts
both schemas + a balanced-brace JSON scanner (the flat regex could not
match nested objects the model emits).
2. qa_judge._extract_json had the same nesting bug — reuse the scanner.
3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped
to 192 in report + instruct_check.
Report enhancements:
- wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4
'model vs pure LLM' is actually computed. Heavy predictors built/evaluated
one at a time with GPU cleanup (single 12GB card).
- add QA-judge table across all 6 categories (design §6.5 问答均分).
- add --pure_llm / --max_per_cat flags.
Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm
0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither
model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
张宗平