张宗平
40ec960cf8
Critical eval-pipeline bugs found while running the stage-2 model:
1. parse_anomaly_segments only recognized {"segments\:[[s,e]]}; the training
data (data/instruct.py) emits {"anomaly_regions":[{"start","end"}]}.
The model learned the correct format but scored 0 → VUS-PR=0. Now accepts
both schemas + a balanced-brace JSON scanner (the flat regex could not
match nested objects the model emits).
2. qa_judge._extract_json had the same nesting bug — reuse the scanner.
3. Generation truncated at 64-96 tokens (too short for stats JSON) → bumped
to 192 in report + instruct_check.
Report enhancements:
- wire the pure-LLM baseline (series-as-text, no TS modality) so design §5.4
'model vs pure LLM' is actually computed. Heavy predictors built/evaluated
one at a time with GPU cleanup (single 12GB card).
- add QA-judge table across all 6 categories (design §6.5 问答均分).
- add --pure_llm / --max_per_cat flags.
Results (reports/eval-2026-06-30.md): model QA-judge mean 2.45 vs pure_llm
0.88 (2.8x); anomaly localization (VUS-PR) is the weak point — neither
model nor pure_llm localize well; TS-modality ablation change_rate 0.98.
130 lines
5.7 KiB
JSON
130 lines
5.7 KiB
JSON
{
|
|
"summary": {
|
|
"anomaly_json_parse_rate": 1.0,
|
|
"anomaly_json_pass": true,
|
|
"overall_usability": 0.9166666666666666,
|
|
"overall_usability_pass": true
|
|
},
|
|
"per_category": {
|
|
"anomaly": {
|
|
"n": 10,
|
|
"parse_rate": 1.0,
|
|
"nonempty_rate": 1.0,
|
|
"rule_acc": 0.02923274723648774,
|
|
"judge_mean": 2.4099999999999997,
|
|
"usability": 1.0,
|
|
"examples": [
|
|
{
|
|
"pred": "{\"anomaly_regions\": [{\"start\": 10, \"end\": 93}, {\"start\": 245, \"end\": 286}], \"n_regions\": 2}",
|
|
"ref": "{\"anomaly_regions\": [{\"start\": 100, \"end\": 145}, {\"start\": 146, \"end\": 200}, {\"start\": 371, \"end\": 373}, {\"start\": 382, \"end\": 445}], \"n_regions\": 4}",
|
|
"parsed": true
|
|
},
|
|
{
|
|
"pred": "{\"anomaly_regions\": [{\"start\": 10, \"end\": 29}, {\"start\": 43, \"end\": 56}], \"n_regions\": 2}",
|
|
"ref": "{\"anomaly_regions\": [{\"start\": 130, \"end\": 202}, {\"start\": 274, \"end\": 328}, {\"start\": 344, \"end\": 346}], \"n_regions\": 3}",
|
|
"parsed": true
|
|
}
|
|
]
|
|
},
|
|
"describe": {
|
|
"n": 10,
|
|
"parse_rate": 1.0,
|
|
"nonempty_rate": 1.0,
|
|
"rule_acc": 0.0,
|
|
"judge_mean": 1.72,
|
|
"usability": 1.0,
|
|
"examples": [
|
|
{
|
|
"pred": "{\"stats\": [{\"name\": \"Active connections\", \"unit\": \"conn\", \"mean\": 43.0162578900111, \"std\": 10.0440401101011, \"min\": 25.0000091902004, \"max\": 61.204120208029}, {",
|
|
"ref": "{\"stats\": [{\"name\": \"Active connections\", \"unit\": \"conn\", \"mean\": 70.08136858780824, \"std\": 5.166230550292378, \"min\": 60.15172120024713, \"max\": 84.5416429780042",
|
|
"parsed": true
|
|
},
|
|
{
|
|
"pred": "{\"stats\": [{\"name\": \"Queue length\", \"unit\": \"tasks\", \"mean\": 42.0167398510011, \"std\": 5.04020002110111, \"min\": 31.0000000000001, \"max\": 55.0604040420208}, {\"sta",
|
|
"ref": "{\"stats\": [{\"name\": \"Queue length\", \"unit\": \"tasks\", \"mean\": 71.13851266805797, \"std\": 5.420230450566408, \"min\": 51.888456943983975, \"max\": 85.02043191625414}, ",
|
|
"parsed": true
|
|
}
|
|
]
|
|
},
|
|
"forecast": {
|
|
"n": 10,
|
|
"parse_rate": 0.5,
|
|
"nonempty_rate": 0.5,
|
|
"rule_acc": 0.0,
|
|
"judge_mean": 0.14,
|
|
"usability": 0.5,
|
|
"examples": [
|
|
{
|
|
"pred": "{\"stats\": [{\"name\": \"Disk I/O\", \"unit\": \"MB/s\", \"mean\": 43.0125678911111, \"std\": 10.0404029110121, \"min\": 27.0000001200001, \"max\": 61.912222028021}, {\"name\": \"M",
|
|
"ref": "{\"forecast\": [[61.36301953352816, 61.372689003454326, 61.382358473380485, 61.392027943306644, 61.40169741323281, 61.41136688315897, 61.42103635308513, 61.430705",
|
|
"parsed": true
|
|
},
|
|
{
|
|
"pred": "{\"stats\": [{\"name\": \"Disk I/O\", \"unit\": \"MB/s\", \"mean\": 43.0125678912111, \"std\": 10.0400219204111, \"min\": 25.0000000200004, \"max\": 61.821123021212}, {\"name\": \"B",
|
|
"ref": "{\"forecast\": [[35.87848293955361, 35.69287951372365, 35.5072760878937, 35.321672662063754, 35.1360692362338, 34.95046581040385, 34.764862384573895, 34.579258958",
|
|
"parsed": true
|
|
}
|
|
]
|
|
},
|
|
"root_cause": {
|
|
"n": 10,
|
|
"parse_rate": 1.0,
|
|
"nonempty_rate": 1.0,
|
|
"rule_acc": null,
|
|
"judge_mean": 3.75,
|
|
"usability": 1.0,
|
|
"examples": [
|
|
{
|
|
"pred": "The deviation in Active connections is most consistent with a sudden level shift. A plausible root cause is a deployment or configuration change that altered th",
|
|
"ref": "The deviation in Active connections is most consistent with an anomaly. A plausible root cause is a deployment or configuration change that altered the operatin",
|
|
"parsed": true
|
|
},
|
|
{
|
|
"pred": "The deviation in CPU usage is most consistent with a variance expansion. A plausible root cause is a deployment or configuration change that altered the operati",
|
|
"ref": "The deviation in CPU usage is most consistent with a gradual drift. A plausible root cause is a deployment or configuration change that altered the operating po",
|
|
"parsed": true
|
|
}
|
|
]
|
|
},
|
|
"event": {
|
|
"n": 10,
|
|
"parse_rate": 1.0,
|
|
"nonempty_rate": 1.0,
|
|
"rule_acc": null,
|
|
"judge_mean": 4.5,
|
|
"usability": 1.0,
|
|
"examples": [
|
|
{
|
|
"pred": "No known event corresponds to the observed change.",
|
|
"ref": "No known event corresponds to the observed change.",
|
|
"parsed": true
|
|
},
|
|
{
|
|
"pred": "No known event corresponds to the observed change.",
|
|
"ref": "No known event corresponds to the observed change.",
|
|
"parsed": true
|
|
}
|
|
]
|
|
},
|
|
"compare": {
|
|
"n": 10,
|
|
"parse_rate": 1.0,
|
|
"nonempty_rate": 1.0,
|
|
"rule_acc": null,
|
|
"judge_mean": 2.3899999999999997,
|
|
"usability": 1.0,
|
|
"examples": [
|
|
{
|
|
"pred": "Per-channel summary:\nActive connections: mean 30.26conn, range [15.47, 48.90];\nGPU utilization: mean 32.01%, range [19.00, 48.91];\nBattery level: mean 55.01%, r",
|
|
"ref": "Per-channel summary:\nActive connections: mean 57.19conn, range [28.39, 84.93]\nGPU utilization: mean 52.88%, range [17.81, 92.55]\nBattery level: mean 49.57%, ran",
|
|
"parsed": true
|
|
},
|
|
{
|
|
"pred": "Per-channel summary:\nCPU usage: mean 36.04%, range [12.57, 58.97]; Request latency p50: mean 32.04ms; Queue length: mean 31.05tasks; Disk I/O: mean 40.07MB/s; R",
|
|
"ref": "Per-channel summary:\nCPU usage: mean 33.10%, range [1.57, 54.50]\nRequest latency p50: mean 35.22ms, range [9.60, 59.28]\nQueue length: mean 26.72tasks, range [12",
|
|
"parsed": true
|
|
}
|
|
]
|
|
}
|
|
}
|
|
} |