Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 张宗平
|
98fb1bfc53 | ||
|
张宗平
|
bbc60f2cdd | ||
|
张宗平
|
722102b335 | ||
|
张宗平
|
9cfda37275 | ||
|
张宗平
|
1b4c6ddf4c | ||
|
张宗平
|
e2c0756a78 | ||
|
张宗平
|
6d10454484 | ||
|
张宗平
|
748bf41c53 | ||
|
张宗平
|
c160718934 |
+11
@@ -4,6 +4,16 @@
|
||||
node_modules/
|
||||
jspm_packages/
|
||||
|
||||
# Python
|
||||
.venv/
|
||||
venv/
|
||||
env/
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*.egg-info/
|
||||
.pytest_cache/
|
||||
.ipynb_checkpoints/
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Build output & caches
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
@@ -87,6 +97,7 @@ coverage/
|
||||
.pi/logs/
|
||||
|
||||
# Claude Code local runtime
|
||||
.claude/settings.local.json
|
||||
.claude/sessions/
|
||||
.claude/cache/
|
||||
.claude/logs/
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# tsmm
|
||||
|
||||
**Time-series as a Modality** — an experimental, single-GPU multimodal model that
|
||||
treats multivariate time series as an independent modality, jointly injected into a
|
||||
small LLM (Qwen2.5-0.5B) for time-series question answering and reasoning
|
||||
(ChatTS route).
|
||||
|
||||
## Status
|
||||
|
||||
M1 (data pipeline) under construction. See
|
||||
`docs/superpowers/specs/2026-06-29-ts-as-modality-design.md` for the design and
|
||||
`docs/superpowers/plans/2026-06-29-ts-as-modality-plan.md` for the task plan.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
src/tsmm/ package (data, model, train, eval)
|
||||
configs/ training / data configs
|
||||
scripts/ offline generation & training entrypoints
|
||||
tests/ pytest suite
|
||||
data/ generated datasets (gitignored)
|
||||
checkpoints/ model checkpoints (gitignored)
|
||||
```
|
||||
|
||||
## Install (dev)
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
. .venv/bin/activate
|
||||
pip install -e . # full deps (incl. torch/transformers/peft/...)
|
||||
# M1-only lightweight install:
|
||||
pip install numpy tqdm pyyaml pytest && pip install -e . --no-deps
|
||||
```
|
||||
@@ -0,0 +1,3 @@
|
||||
# Model checkpoints — never commit.
|
||||
*
|
||||
!.gitignore
|
||||
@@ -0,0 +1,3 @@
|
||||
# Configs directory
|
||||
|
||||
YAML / Python configs for data generation and training live here.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Generated datasets — never commit.
|
||||
*
|
||||
!.gitignore
|
||||
@@ -1,3 +1,9 @@
|
||||
---
|
||||
change: ts-as-modality
|
||||
design-doc: docs/superpowers/specs/2026-06-29-ts-as-modality-design.md
|
||||
base-ref: 7a5a1d033fb1ac20af25a17a571b2791694bd923
|
||||
---
|
||||
|
||||
# 时序作为模态的多模态模型 · 实现计划
|
||||
|
||||
> **对应规格**：`docs/superpowers/specs/2026-06-29-ts-as-modality-design.md`
|
||||
|
||||
@@ -1,3 +1,9 @@
|
||||
---
|
||||
comet_change: ts-as-modality
|
||||
role: technical-design
|
||||
canonical_spec: openspec
|
||||
---
|
||||
|
||||
# 时序作为模态的多模态模型 · 设计规格
|
||||
|
||||
> **日期**：2026-06-29
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
workflow: full
|
||||
phase: build
|
||||
context_compression: off
|
||||
build_mode: executing-plans
|
||||
build_pause: null
|
||||
subagent_dispatch: null
|
||||
tdd_mode: tdd
|
||||
review_mode: standard
|
||||
isolation: branch
|
||||
verify_mode: null
|
||||
auto_transition: true
|
||||
base_ref: 7a5a1d033fb1ac20af25a17a571b2791694bd923
|
||||
design_doc: docs/superpowers/specs/2026-06-29-ts-as-modality-design.md
|
||||
plan: docs/superpowers/plans/2026-06-29-ts-as-modality-plan.md
|
||||
verify_result: pending
|
||||
verification_report: null
|
||||
branch_status: pending
|
||||
created_at: 2026-06-29
|
||||
verified_at: null
|
||||
archived: false
|
||||
handoff_context: openspec/changes/ts-as-modality/.comet/handoff/design-context.json
|
||||
handoff_hash: 067d854db86de682b8c5e173888ab9b0d890244bf938aef24512ec8287789be7
|
||||
@@ -0,0 +1,41 @@
|
||||
# Brainstorm Summary
|
||||
|
||||
- Change: ts-as-modality
|
||||
- Date: 2026-06-29
|
||||
|
||||
## 确认的技术方案
|
||||
|
||||
采纳既有、已评审 Design Doc `docs/superpowers/specs/2026-06-29-ts-as-modality-design.md` 为本次 change 的 Design Doc（comet-design 前置条件「无 Design Doc」被既有文件违反，实质性深度设计已完成，无需重新 brainstorm 出重复文档）。
|
||||
|
||||
核心技术方案（源自 Design Doc）：
|
||||
- 路线：ChatTS「时序作为新模态」—— Patch-Project + 轻量编码器（优于 Chronos 时序 Tokenization、Time-LLM Reprogramming）
|
||||
- 骨干：Qwen2.5-0.5B-Instruct（hidden=896）
|
||||
- TS Encoder：Patchify(P=8, stride=4, 通道独立) + 2 层 Transformer(d=256, 4 头, ~4M)
|
||||
- Projector：Linear(256→896) + LayerNorm
|
||||
- 多模态拼接顺序：`[属性][时间戳][TS token][事件][问题]`
|
||||
- 两阶段训练：①冻结 LLM 训 Encoder+Projector（~4.2M，省显存关键）→ ②叠加 LoRA r=16(q/v_proj) SFT（~7M）
|
||||
- 数据：CPU 离线合成为主（50万对齐 + 10万 SFT + 2k held-out），真实 benchmark(SMD/MSL/SMAP/SWaT/PSM) 仅锚定评测
|
||||
- 评测：VUS-PR 为主（严守 MTSAD 纪律，PA-F1 仅对照），LLM-judge + 规则解析双轨，4 类同口径基线
|
||||
|
||||
## 关键取舍与风险
|
||||
|
||||
- [对比损失默认] 采用「扰动一致性 + λ=0.1」作为默认，做成可配置；若 M3「TS-token 有效性检验」（扰动后回答变化率 >50%）失败再换 InfoNCE。
|
||||
- [DPO] 首版不做，M5 后视效果决定。
|
||||
- [高风险·合成→真实泛化] 阶段②掺真实 benchmark 改写样本；评测以真实 benchmark 为准。
|
||||
- [中风险·阶段② 12GB OOM] 降批量/ctx、CPU offload、adamw_8bit、退回阶段①-only 报告。
|
||||
- [中风险·LLM 忽略 TS token] 对比损失 + 必看时序样本 + 消融定位。
|
||||
- [中风险·JSON 解析失败] 训练强化格式约束 + 解析兜底（失败→全 0 计漏报）+ 单独报解析成功率。
|
||||
|
||||
开放问题（TS Encoder 层/d、Patch P、对比损失形式、DPO、真实改写数量）均延后为 build 阶段经验/流程项，不阻塞设计。
|
||||
|
||||
## 测试策略
|
||||
|
||||
- 单元测试：各模块（attributes/synthesis/instruct/ts_encoder/projector/multimodal/collator/losses/parse/ts_metrics）给定 seed 确定性 + 形状/参数量校验
|
||||
- 冒烟路径：`gen_synthetic.py --n 200` → T2.1-T2.4 单 batch → stage1/stage2 `--max_steps 50` → run_eval.sh 产报告骨架，确认端到端通后再放大
|
||||
- 里程碑出口验证：M1-M5 每阶段有可执行完成判据
|
||||
- 评测：异常检测 VUS-PR/Aff-F1/AUC-PR/Point-F1(无PA) + PA-F1(对照)；问答 LLM-judge + 规则解析；含 trivial 基线对照防虚高
|
||||
- 可验证成功标准（§6.5）：阶段②峰值显存 ≤11GB；真实 benchmark VUS-PR 显著优于纯 LLM 基线；时序消融后指标下降；问答均分超基线；评测可复现
|
||||
|
||||
## Spec Patch
|
||||
|
||||
无。4 项高风险均有对应 delta spec 验收场景闭环；成功标准由 tasks M5.6 出口验证承载，不另开第二份需求 spec。
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"change": "ts-as-modality",
|
||||
"phase": "build",
|
||||
"checkpoint": "M1-complete",
|
||||
"date": "2026-06-29",
|
||||
"status": "paused-awaiting-user-decision",
|
||||
"summary": "M1 (data pipeline) fully complete (T1.1-T1.7). 51 tests passing. All committed on feature/20260629/ts-as-modality.",
|
||||
"next_action": "T2.1 (TS Encoder, model/ts_encoder.py). Blocked on environment setup decision for M2-M5.",
|
||||
"blocker": {
|
||||
"reason": "venv (.venv) has only M1 deps (numpy/tqdm/pyyaml/pytest). M2+ requires torch + transformers (+peft/accelerate/datasets for M3/M4). System torch is CPU-only; design assumes 3060 12GB GPU but actual is Quadro RTX 3000 6GB.",
|
||||
"decision_pending": "User must choose: A) CPU torch now (M2 correctness only, no GPU mem validation); B) CUDA torch for 6GB Quadro (enables real training, expect OOM-fallbacks earlier vs 12GB plan); C) user sets up env themselves.",
|
||||
"recommendation": "B"
|
||||
},
|
||||
"resume_instructions": {
|
||||
"verify_state": "Run: /bin/bash .claude/skills/comet/scripts/comet-state.sh check ts-as-modality build",
|
||||
"find_next_task": "grep -n '\\- \\[ \\]' openspec/changes/ts-as-modality/tasks.md | head -1 (expect T2.1)",
|
||||
"venv": "Use .venv/bin/python for all python commands",
|
||||
"branch": "feature/20260629/ts-as-modality",
|
||||
"build_mode": "executing-plans",
|
||||
"tdd_mode": "tdd",
|
||||
"review_mode": "standard",
|
||||
"isolation": "branch"
|
||||
},
|
||||
"completed_milestones": ["M1"],
|
||||
"completed_tasks": ["1.1", "1.2", "1.3", "1.4", "1.5", "1.6", "1.7"],
|
||||
"test_count": 51,
|
||||
"git": {
|
||||
"m1_complete_commit": "bbc60f2",
|
||||
"base_ref": "7a5a1d033fb1ac20af25a17a571b2791694bd923"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"change": "ts-as-modality",
|
||||
"phase": "design",
|
||||
"mode": "compact",
|
||||
"canonical_spec": "openspec",
|
||||
"generated_by": "comet-handoff.sh",
|
||||
"context_hash": "067d854db86de682b8c5e173888ab9b0d890244bf938aef24512ec8287789be7",
|
||||
"files": [
|
||||
{ "path": "openspec/changes/ts-as-modality/proposal.md", "sha256": "d29658df95f49b889a7f06e6985776ba944f2476eca8e06f074787e823408f4e" },
|
||||
{ "path": "openspec/changes/ts-as-modality/design.md", "sha256": "5b920360b1303fde23795a3388c34a11c0c562daa340de12980ff86a1d199f7b" },
|
||||
{ "path": "openspec/changes/ts-as-modality/tasks.md", "sha256": "8e13037661b54a1579cc1c9c81ff2676aee777ca25303ef54c944ce91e852726" },
|
||||
{ "path": "openspec/changes/ts-as-modality/specs/ts-evaluation/spec.md", "sha256": "1c986559a6b44900bff452e4cbbb0272c1ee321de2fddacedc6e096c4e5381fb" },
|
||||
{ "path": "openspec/changes/ts-as-modality/specs/ts-multimodal-model/spec.md", "sha256": "f40631b2e8dac3772f6081623d2175460ca5355a8818d6cb3c28a94f148d56a3" },
|
||||
{ "path": "openspec/changes/ts-as-modality/specs/ts-synthesis/spec.md", "sha256": "330043d178e344f1f1e87d1262f25ed33b2216eb6e16f2658d4f29a779d8b60b" }
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,438 @@
|
||||
# Comet Design Handoff
|
||||
|
||||
- Change: ts-as-modality
|
||||
- Phase: design
|
||||
- Mode: compact
|
||||
- Context hash: 067d854db86de682b8c5e173888ab9b0d890244bf938aef24512ec8287789be7
|
||||
|
||||
Generated-by: comet-handoff.sh
|
||||
|
||||
OpenSpec remains the canonical capability spec. This handoff is a deterministic, source-traceable context pack, not an agent-authored summary.
|
||||
|
||||
## openspec/changes/ts-as-modality/proposal.md
|
||||
|
||||
- Source: openspec/changes/ts-as-modality/proposal.md
|
||||
- Lines: 1-33
|
||||
- SHA256: d29658df95f49b889a7f06e6985776ba944f2476eca8e06f074787e823408f4e
|
||||
|
||||
```md
|
||||
## Why
|
||||
|
||||
当前 AIOps 场景下，多变量时序的异常检测与解释仍依赖专用模型，缺乏「以自然语言对时序提问、做根因/事件关联推理」的统一能力。基于 `MTSAD-技术深度调研报告.md`，沿用 ChatTS「时序作为新模态」路线，我们希望在**单张 3060 12GB 消费级显卡**上做一个可复现的实验性落地，验证「把多变量时序视为独立模态、与文本/事件模态联合注入小型 LLM」这一方法在算力受限条件下是否切实可行、且「时序模态」是否真带来可量化的增益。现在做是因为：0.5B 级开源 LLM 与 PEFT/LoRA 工具链已足够成熟，使该实验在消费级硬件上首次成为可能。
|
||||
|
||||
## What Changes
|
||||
|
||||
- **新增**：CPU 离线合成数据管线（成分模型 + 异常注入 + 事件耦合 + Evol-Instruct 指令/回答生成），产出对齐预训练与 SFT 用的 JSONL
|
||||
- **新增**：真实 benchmark（SMD/MSL/SMAP/SWaT/PSM）窗口加载与「异常段→问答对」改写（仅评测用，不进训练）
|
||||
- **新增**：TS Encoder（Patchify + 2 层 Transformer）+ Projector（Linear→LLM hidden），把时序编码为独立 soft token
|
||||
- **新增**：多模态拼接器，按 `[属性][时间戳][TS token][事件][问题]` 顺序组装文本 token 与 TS token 注入 LLM
|
||||
- **新增**：两阶段训练流程 —— 阶段①冻结 LLM 训 Encoder+Projector（对齐预训练）；阶段②叠加 LoRA(r=16) 做指令微调
|
||||
- **新增**：评测套件 —— 文本回答→JSON 区间→逐点分数解析；VUS-PR(主)/Affiliation-F1/AUC-PR/Point-F1(无PA)；LLM-judge + 规则解析双轨
|
||||
- **新增**：4 类同口径对照基线（纯 LLM 数值喂文本 / Time-LLM 风格重映射 / ChatTS(若可复现) / Trivial）
|
||||
- 骨干选用 **Qwen2.5-0.5B-Instruct**，BF16 + LoRA + 梯度检查点，目标阶段②峰值显存 ≤11GB
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `ts-synthesis`: CPU 离线合成数据管线——成分模型（趋势/周期/基线/噪声）、异常注入（尖刺/水平偏移/方差膨胀/缺失段/缓慢漂移）、事件耦合、缺失/不规则模拟、属性词表采样，以及 Evol-Instruct 多轮指令与回答生成；附带真实 benchmark 加载与问答改写
|
||||
- `ts-multimodal-model`: 时序作为模态的多模态模型——TS Encoder + Projector + 多模态拼接（TS token 与文本/事件 token 联合注入 LLM），含阶段①对齐预训练与阶段②LoRA SFT 的训练/推理统一封装
|
||||
- `ts-evaluation`: 时序问答与异常检测评测套件——回答解析（文本→JSON 区间→逐点分数）、阈值无关指标（VUS-PR 主/Affiliation-F1/AUC-PR/Point-F1 无 PA）、LLM-judge + 规则双轨、4 类同口径基线对照、可复现评测报告
|
||||
|
||||
### Modified Capabilities
|
||||
<!-- 无已存在 capability 的需求变更（本项目 specs/ 当前为空）。 -->
|
||||
|
||||
## Impact
|
||||
|
||||
- **新增代码**：独立 Python 项目 `ts-as-modality/`（`pyproject.toml` + `src/tsmm/{data,model,train,eval,utils}` + `scripts/` + `tests/` + `configs/`），与当前 spec/docs 工作区并列
|
||||
- **依赖**：torch、transformers、peft、accelerate、datasets、numpy、tqdm、pyyaml、pytest；评测依赖 `tsb_uad`（VUS-PR）；LLM-judge 评测时调用 OpenAI/GPT-4o API
|
||||
- **模型资产**：Qwen2.5-0.5B-Instruct 骨干（外部下载）；阶段①/② checkpoint 与合成数据产物体积较大，需 git-ignore（`data/`、`checkpoints/`）
|
||||
- **算力**：单张 3060 12GB；阶段①预估峰值 ~4-5GB，阶段②预估峰值 ~9-10GB；CPU 多进程用于离线合成
|
||||
- **存储**：合成数据约 2-3GB（流式加载），checkpoint 分阶段独立存放
|
||||
- **无 BREAKING 变更**：纯新增独立实验项目，不触碰现有任何代码/接口
|
||||
```
|
||||
|
||||
## openspec/changes/ts-as-modality/design.md
|
||||
|
||||
- Source: openspec/changes/ts-as-modality/design.md
|
||||
- Lines: 1-88
|
||||
- SHA256: 5b920360b1303fde23795a3388c34a11c0c562daa340de12980ff86a1d199f7b
|
||||
|
||||
[TRUNCATED]
|
||||
|
||||
```md
|
||||
## Context
|
||||
|
||||
本 change 在**单张 3060 12GB 消费级显卡**上做一个可复现的实验性落地：把**多变量时序视为独立模态**，与文本/事件模态联合注入小型 LLM（Qwen2.5-0.5B-Instruct），端到端任务是**时序问答与推理**（ChatTS 路线）。
|
||||
|
||||
背景与约束来自 `MTSAD-技术深度调研报告.md` 与 `docs/superpowers/specs/2026-06-29-ts-as-modality-design.md`（已通过设计评审）。核心约束：
|
||||
|
||||
- **算力**：单卡 12GB → 骨干 ≤0.5B、依赖 LoRA/QLoRA + 短上下文 + 小批量 + 梯度检查点
|
||||
- **数据**：合成数据为主（CPU 离线生成）→ 评测锚定真实 benchmark，防合成过拟合
|
||||
- **目标**：不追学术 SOTA，追「12GB 跑通、可复现、时序模态确有增益」的可验证结论
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- 全流程单卡 3060 12GB 跑通，阶段②峰值显存 ≤11GB
|
||||
- TS Encoder + Projector 把时序编码为独立 soft token，使 LLM「能读懂」时序
|
||||
- 时序模态带来可量化增益：真实 benchmark VUS-PR 显著优于纯 LLM 数值喂文本基线；时序消融后指标明显下降
|
||||
- 6 类问答任务（描述/异常/根因/预测/比较/事件关联）LLM-judge 均分超纯 LLM 基线
|
||||
- 评测可复现：含 trivial 基线对照、标注是否用 PA、严守 MTSAD 报告评估纪律
|
||||
|
||||
**Non-Goals:**
|
||||
- ❌ 不从零预训练 LLM（成本不允许，且非本实验目的）
|
||||
- ❌ 不做多卡分布式训练（单卡约束）
|
||||
- ❌ 不追求学术 SOTA 排名
|
||||
- ❌ 不做端侧部署优化（先验证方法）
|
||||
- ❌ DPO 首版不做（M5 后视效果再决定）
|
||||
|
||||
## Decisions
|
||||
|
||||
### 决策 1：采用 Patch-Project + 轻量编码器，而非时序 Tokenization 或 Reprogramming
|
||||
|
||||
时序切 patch（P=8, stride=4，通道独立）→ 2 层 Transformer 编码器（d=256, 4 头, ~4M）→ 线性投影（256→896）成独立 TS token，作为可学习 soft token 注入 LLM。
|
||||
|
||||
**对比的备选方案：**
|
||||
|
||||
| 方案 | 含义 | 取舍 |
|
||||
|------|------|------|
|
||||
| **A（选用）Patch-Project + 轻量编码器** | patch→小编码器→线性投影成独立 TS token | 真正的"时序作为模态"；12GB 可训；patch 捕捉局部结构、变长友好。代价：需训编码器+投影层（已在显存预算内） |
|
||||
| B 时序 Tokenization（Chronos 风格） | 数值量化成离散 token、扩展词表 | 最"时序即语言"，但量化丢精度、异常细节易损；逐点 token 序列长，12GB 上下文吃紧。放弃 |
|
||||
| C Reprogramming（Time-LLM 风格） | 冻结 LLM 用文本原型重映射 | 参数最省，但受限于已有文本嵌入流形、表达力弱，时序算不上"真新模态"，复杂推理/事件联合弱。放弃 |
|
||||
|
||||
### 决策 2：骨干选 Qwen2.5-0.5B-Instruct
|
||||
|
||||
0.5B 中英均衡、有现成 Instruct 版、社区生态成熟；BF16 约 1GB，12GB 余量充足（hidden=896）。
|
||||
|
||||
### 决策 3：两阶段训练，阶段①冻结 LLM
|
||||
|
||||
- **阶段① 对齐预训练**：冻结 LLM，仅训 TS Encoder + Projector（~4.2M 全参）→ 不存 LLM 优化器状态/梯度，是 12GB 可行的关键省显存技巧
|
||||
- **阶段② SFT**：在此之上叠加 LoRA r=16（q/v_proj，~2-3M），LLM 其余冻结 + 梯度检查点
|
||||
|
||||
分阶段 checkpoint 互不污染、可回滚。损失仅对回答段 token 计 LM loss；阶段①叠加对比损失（扰动时序→回答应变）防止 LLM 忽略 TS token。
|
||||
|
||||
### 决策 4：输入拼接顺序 `[属性][时间戳][TS token][事件][问题]`
|
||||
|
||||
属性先给语义，时序居中，事件定位在对应时间，问题在末尾便于回答。
|
||||
|
||||
### 决策 5：合成数据为主，真实 benchmark 仅用于评测锚定
|
||||
|
||||
合成管线全 CPU 离线，成分参数随机化；**核心红利**：异常段、事件时间戳、各成分参数全部作为 ground-truth 自动记录——免费且精确的标签。评测锚定 SMD/MSL/SMAP/SWaT/PSM 真实分布，避免「合成上 SOTA、真实上崩盘」。
|
||||
|
||||
### 决策 6：评测指标以 VUS-PR 为主，严守 MTSAD 报告纪律
|
||||
|
||||
- 主指标 **VUS-PR**（阈值无关）；辅 Affiliation-F1（多 λ）、AUC-PR、Point-F1（无 PA）
|
||||
- **PA-F1 仅作对照、不得作主指标**（防虚高）
|
||||
- 问答用 LLM-judge + 规则解析双轨
|
||||
- 自然语言回答→约束 JSON 区间→逐点分数，便于数值化评测
|
||||
|
||||
### 决策 7：4 类同口径基线
|
||||
|
||||
纯 LLM 数值喂文本（检验 TS 模态是否真有增益）、Time-LLM 风格重映射、ChatTS（若可复现）、Trivial（Random/Constant/全报异常）。全部走同一套解析+指标脚本。
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [12GB 阶段②仍 OOM] → 降批量/上下文、开 CPU offload 优化器、`adamw_8bit`、退回阶段①-only 报告
|
||||
- [合成→真实泛化差距大（高风险）] → 阶段②掺真实 benchmark 改写样本；评测以真实 benchmark 为准
|
||||
- [LLM 忽略 TS token] → 对比损失 + "必看时序"样本 + 消融实验定位
|
||||
- [回答 JSON 解析失败率高] → 训练强化格式约束 + 解析兜底（失败→全 0 计漏报）+ 单独报解析成功率
|
||||
- [GPT-4o judge 成本] → 仅评测 2k 样本调用；可换开源 judge 做 sanity
|
||||
- [TS Encoder 过拟合合成分布] → 成分参数随机化 + 多套合成配置 + 阶段②引入真实改写样本
|
||||
|
||||
## Open Questions
|
||||
```
|
||||
|
||||
Full source: openspec/changes/ts-as-modality/design.md
|
||||
|
||||
## openspec/changes/ts-as-modality/tasks.md
|
||||
|
||||
- Source: openspec/changes/ts-as-modality/tasks.md
|
||||
- Lines: 1-45
|
||||
- SHA256: 8e13037661b54a1579cc1c9c81ff2676aee777ca25303ef54c944ce91e852726
|
||||
|
||||
```md
|
||||
# Tasks · ts-as-modality
|
||||
|
||||
> 源自 `docs/superpowers/plans/2026-06-29-ts-as-modality-plan.md`（M1→M5）。
|
||||
> 每个里程碑末尾的「出口验证」为该阶段完成判据；建议先跑「冒烟路径」确认端到端通，再放大到全量。
|
||||
|
||||
## 1. M1 · 数据管线
|
||||
|
||||
- [ ] 1.1 项目脚手架：创建 `pyproject.toml`（torch/transformers/peft/accelerate/datasets/numpy/tqdm/pyyaml/pytest）、`src/tsmm/` 模块骨架、`configs/`、`scripts/`、`tests/`，并建 `data/`、`checkpoints/` 加 `.gitignore`（验证：`python -c "import tsmm"` 不报错；`pytest tests/` 可跑）
|
||||
- [ ] 1.2 属性词表与采样 `data/attributes.py`：定义变量名词表/单位/采样率档位，`sample_attributes(n_channels)` 返回结构化属性（验证：单测字段齐全、可复现）
|
||||
- [ ] 1.3 成分模型与时序合成 `data/synthesis.py`：`generate_series`（趋势+周期+基线+噪声）、`inject_anomaly`（尖刺/水平偏移/方差膨胀/缺失段/缓慢漂移，返回逐点 labels+段元数据）、`couple_event`（t 处阶跃+事件文本）、`add_missing`（NaN 占位）（验证：给定 seed 确定；异常段与 labels 一致；事件处有阶跃）
|
||||
- [ ] 1.4 指令与回答生成 `data/instruct.py`：6 类指令模板（描述/异常/根因/预测/比较/事件关联）+ Evol-Instruct 演化，`build_instruction` 产 JSON 区间/数值/文本回答（验证：异常类 JSON 可 `json.loads` 且段与 labels 吻合）
|
||||
- [ ] 1.5 离线生成脚本 `scripts/gen_synthetic.py`：多进程产 `align.jsonl`(50万)/`sft.jsonl`(10万)/`eval_synth.jsonl`(2k held-out)，支持 `--n/--out/--seed/--workers` + tqdm（验证：`--n 1000` 跑通，抽样 5 条 schema 合规）
|
||||
- [ ] 1.6 真实 benchmark 加载 `data/real_bench.py`：加载 SMD/MSL/SMAP/SWaT/PSM（至少 2 个），`load_windows(T=512, stride=256)` 滑窗+归一化+标签，异常段改写问答对存 `eval_real.jsonl`（验证：窗口形状 `[N,T,C]`、标签 `[N,T]`、问答非空）
|
||||
- [ ] 1.7 M1 出口验证：`gen_synthetic.py --n 1000` 产可视检合法 JSONL；`real_bench.py` 加载 ≥2 数据集；人工抽看 5 条合成样本确认成分/异常/事件正确
|
||||
|
||||
## 2. M2 · 模型可跑通
|
||||
|
||||
- [ ] 2.1 TS Encoder `model/ts_encoder.py`：`Patchify(patch=8, stride=4)` 通道独立切 patch 线性嵌入到 d=256；`TSEncoder(d=256, layers=2, heads=4)` 2 层 TF + 可学习位置编码；前向 `[B,T,C]`→`[B,n_patches,256]`（验证：输入 `[2,512,5]`→`[2,128,256]`，参数 ≈4M）
|
||||
- [ ] 2.2 Projector `model/projector.py`：`Linear(256→896)+LayerNorm`（验证：输出 `[B,128,896]` 对齐 Qwen2.5-0.5B hidden）
|
||||
- [ ] 2.3 多模态拼接 `model/multimodal.py`：Qwen tokenizer tokenize 文本部分，TS token 作 inputs_embeds 插入 `[属性][时间戳][TS tok][事件][问题]`，统一构造 inputs_embeds+attention_mask+labels（仅回答段非 -100）（验证：seq_len ≤1024，mask/labels 形状对）
|
||||
- [ ] 2.4 训练/推理封装 `model/wrapper.py`：`MultimodalTSModel` 组合 Encoder+Projector+LLM+LoRA 挂载开关，`forward` 返 loss、`generate` 返文本，支持 `freeze_llm`(阶段①)/`enable_lora(r=16)`(阶段②)（验证：单 batch forward 3060 不 OOM；generate 能出文本）
|
||||
- [ ] 2.5 Collator `data/collator.py`：JSONL→batch 张量，处理变长 C（padding+mask）与变长文本（padding+attention_mask）（验证：batch=4 张量形状一致无 NaN）
|
||||
- [ ] 2.6 M2 出口验证：单 batch 前向+反向在 3060 跑通，loss 有限且下降趋势，阶段①模式峰值显存 ~5GB
|
||||
|
||||
## 3. M3 · 阶段①对齐训练
|
||||
|
||||
- [ ] 3.1 损失函数 `train/losses.py`：`lm_loss`（仅回答段计 loss）、`contrastive_loss`（扰动时序→回答 embedding 变化，InfoNCE/扰动一致性），总 loss=`lm_loss+λ*contrastive_loss`（λ 默认 0.1）（验证：loss 标量可反传，对比损失对扰动敏感）
|
||||
- [ ] 3.2 阶段①训练脚本 `train/stage1.py`+`scripts/train_stage1.sh`：加载 `align.jsonl` 流式，`freeze_llm=True` 仅训 Encoder+Projector（AdamW, lr=1e-4），bs=8/grad_accum=4/ctx=512/BF16/梯度检查点，每 2000 step 存 ckpt + TensorBoard（验证：`--max_steps 100` 冒烟不 OOM、loss 下降、峰值显存 ≤6GB）
|
||||
- [ ] 3.3 TS-token 有效性检验：held-out 原时序 vs 扰动时序回答变化率 + 「必看时序」样本答对率对照纯 LLM（验证：扰动后变化率 >50%，必看时序答对率显著高于纯 LLM）
|
||||
- [ ] 3.4 M3 出口验证：阶段① ckpt 产出；扰动检验通过；峰值显存 ≤6GB（失败回查对比损失权重/数据质量）
|
||||
|
||||
## 4. M4 · 阶段②SFT
|
||||
|
||||
- [ ] 4.1 阶段②训练脚本 `train/stage2.py`+`scripts/train_stage2.sh`：加载阶段① ckpt，`enable_lora(r=16)` on q/v_proj 其余冻结，加载 `sft.jsonl`，bs=4/grad_accum=8/ctx=1024/BF16/梯度检查点，每 2000 step 存含 LoRA ckpt（验证：`--max_steps 100` 冒烟不 OOM、loss 下降、峰值显存 ≤11GB）
|
||||
- [ ] 4.2 指令遵循抽检：6 类任务各取 10 条 held-out 检查回答（异常类 JSON 可解析+段合理；解释类切题引用事件）（验证：6 类回答可用率 >70%，JSON 解析成功率 >80%）
|
||||
- [ ] 4.3 M4 出口验证：阶段② ckpt 产出；6 类任务抽检达标；峰值显存 ≤11GB（OOM 则降 bs=2/grad_accum=16、`adamw_8bit`+CPU offload、ctx 降到 768）
|
||||
|
||||
## 5. M5 · 完整评测
|
||||
|
||||
- [ ] 5.1 回答解析 `eval/parse_answer.py`：文本回答→JSON 异常段→逐点分数 `[T]`，失败→全 0+失败标记（验证：多格式可解析、失败有兜底）
|
||||
- [ ] 5.2 异常检测指标 `eval/ts_metrics.py`：`tsb_uad` 算 VUS-PR(主)，自实现 Affiliation-F1(多 λ)/AUC-PR/Point-F1(无PA)，PA-F1 单独对照（验证：已知段算出合理值；trivial 全报异常 VUS-PR 不虚高）
|
||||
- [ ] 5.3 问答评测 `eval/qa_judge.py`：规则解析轨（JSON/数值容差→准确率）+ LLM-judge 轨（GPT-4o 打 0-5 分），不一致样本导出抽检（验证：对 eval 集产出 6 类任务的 {规则准确率,judge 均分,解析成功率}）
|
||||
- [ ] 5.4 对照基线 `eval/baselines.py`：纯 LLM 数值喂文本 / Time-LLM 风格重映射 / ChatTS(若可复现) / Trivial(Random/Constant/全报异常)，全走同一套 parse+metrics（验证：4 类基线产出同口径指标表）
|
||||
- [ ] 5.5 评测脚本与报告 `scripts/run_eval.sh`：跑 `eval_synth.jsonl`+`eval_real.jsonl`，产 `reports/eval-YYYYMMDD.md`（含每数据集×每任务全部必报指标、标注是否用 PA、含 trivial 对照）（验证：一键产出满足设计规格 §5.4 必报清单的完整报告）
|
||||
- [ ] 5.6 M5 出口验证：报告满足设计规格 §6.5 成功标准——真实 benchmark VUS-PR 显著优于纯 LLM 基线；时序消融后指标明显下降；问答均分超基线；评测可复现含 trivial
|
||||
```
|
||||
|
||||
## openspec/changes/ts-as-modality/specs/ts-evaluation/spec.md
|
||||
|
||||
- Source: openspec/changes/ts-as-modality/specs/ts-evaluation/spec.md
|
||||
- Lines: 1-61
|
||||
- SHA256: 1c986559a6b44900bff452e4cbbb0272c1ee321de2fddacedc6e096c4e5381fb
|
||||
|
||||
```md
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: 回答解析为逐点分数
|
||||
|
||||
系统 SHALL 从模型自然语言回答中提取 JSON 异常段并生成逐点分数（段内 1.0，段外 0.0），解析失败时回退为全 0（计为漏报）并记录失败标记。
|
||||
|
||||
#### Scenario: 多格式可解析
|
||||
- **WHEN** 输入含噪声/嵌套的多种回答格式
|
||||
- **THEN** 能正确解析出异常段并生成逐点分数 `[T]`
|
||||
|
||||
#### Scenario: 解析失败有兜底
|
||||
- **WHEN** 回答无法解析
|
||||
- **THEN** 分数全 0 并记录失败标记（不计为成功）
|
||||
|
||||
### Requirement: 阈值无关的异常检测指标
|
||||
|
||||
系统 SHALL 以 VUS-PR 为异常检测主指标，并辅以 Affiliation-F1（λ=0.1/0.5/1.0）、AUC-PR、Point-F1（无 PA）；PA-F1 仅作对照、不得作为主指标。
|
||||
|
||||
#### Scenario: trivial 基线不虚高
|
||||
- **WHEN** 对「全报异常」trivial 基线计算 VUS-PR
|
||||
- **THEN** VUS-PR 不虚高（避免评估作弊）
|
||||
|
||||
#### Scenario: 对已知段算出合理值
|
||||
- **WHEN** 对已知异常段计算指标
|
||||
- **THEN** VUS-PR/Aff-F1/AUC-PR/Point-F1 均落在合理区间
|
||||
|
||||
### Requirement: 问答评测双轨
|
||||
|
||||
系统 SHALL 提供规则解析轨（JSON/数值容差比对，产准确率）与 LLM-judge 轨（对 `问题,时序摘要,参考答案,模型答案` 打 0-5 分），并对两轨不一致样本导出供人工抽检。
|
||||
|
||||
#### Scenario: 产出 6 类任务评测结果
|
||||
- **WHEN** 对评测集运行问答评测
|
||||
- **THEN** 对 6 类任务各产出 `{规则准确率, judge 均分, 解析成功率}`
|
||||
|
||||
### Requirement: 同口径对照基线
|
||||
|
||||
系统 SHALL 提供 4 类同口径基线——纯 LLM 数值喂文本、Time-LLM 风格重映射、ChatTS（若不可复现则标注）、Trivial（Random/Constant/全报异常）——且全部走同一套解析+指标脚本。
|
||||
|
||||
#### Scenario: 4 类基线产出同口径指标
|
||||
- **WHEN** 运行基线评测
|
||||
- **THEN** 4 类基线均在同一套 `parse_answer`+`ts_metrics` 下产出指标表
|
||||
|
||||
### Requirement: 可复现评测报告
|
||||
|
||||
系统 SHALL 提供一键评测脚本，产出 markdown 报告，覆盖每数据集×每任务的 VUS-PR/Aff-F1/AUC-PR/Point-F1/PA-F1(对照)/问答分/解析成功率/消融/基线对比，并明确标注是否使用 PA、含 trivial 对照。
|
||||
|
||||
#### Scenario: 一键产出完整报告
|
||||
- **WHEN** 运行 `run_eval.sh`
|
||||
- **THEN** 产出满足设计规格 §5.4 必报清单的完整 markdown 报告
|
||||
|
||||
### Requirement: 模态消融与泛化验证
|
||||
|
||||
系统 SHALL 支持模态消融（去掉事件/属性/时序）与跨数据集泛化验证（如 SMD→MSL/SWaT），以证明时序模态确有贡献并暴露合成→真实差距。
|
||||
|
||||
#### Scenario: 时序消融指标下降
|
||||
- **WHEN** 去掉时序模态后评测
|
||||
- **THEN** 关键指标（VUS-PR/问答分）明显低于完整模型，证明时序模态有贡献
|
||||
|
||||
#### Scenario: 真实 benchmark 显著优于纯 LLM 基线
|
||||
- **WHEN** 在真实 benchmark 上对比完整模型与纯 LLM 数值喂文本基线
|
||||
- **THEN** VUS-PR 显著优于纯 LLM 基线
|
||||
```
|
||||
|
||||
## openspec/changes/ts-as-modality/specs/ts-multimodal-model/spec.md
|
||||
|
||||
- Source: openspec/changes/ts-as-modality/specs/ts-multimodal-model/spec.md
|
||||
- Lines: 1-73
|
||||
- SHA256: f40631b2e8dac3772f6081623d2175460ca5355a8818d6cb3c28a94f148d56a3
|
||||
|
||||
```md
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: TS Encoder 把时序编码为 patch 表示
|
||||
|
||||
系统 SHALL 通过通道独立的 Patchify（P=8, stride=4）将多变量时序切分为重叠 patch，并经 2 层 Transformer 编码器（d=256, 4 头）映射为时序 token 序列。
|
||||
|
||||
#### Scenario: 形状正确
|
||||
- **WHEN** 输入形状 `[B, T=512, C=5]`
|
||||
- **THEN** 输出形状 `[B, 128, 256]`，且 TS Encoder 参数量约 4M
|
||||
|
||||
#### Scenario: 通道独立
|
||||
- **WHEN** 通道数 C 变化（如 C=3 与 C=12）
|
||||
- **THEN** 编码器仍正常工作（通道独立 patchify，不因 C 增大失配）
|
||||
|
||||
### Requirement: Projector 映射到 LLM 隐藏维
|
||||
|
||||
系统 SHALL 提供线性投影层（256→896）+ LayerNorm，将 TS 表示映射到 Qwen2.5-0.5B 的隐藏维（896），作为可学习 soft token。
|
||||
|
||||
#### Scenario: 投影对齐 LLM hidden
|
||||
- **WHEN** TS Encoder 输出 `[B, 128, 256]`
|
||||
- **THEN** Projector 输出 `[B, 128, 896]`，与 LLM 隐藏维一致
|
||||
|
||||
### Requirement: 多模态拼接注入 LLM
|
||||
|
||||
系统 SHALL 按 `[属性][时间戳][TS token][事件][问题]` 顺序组装文本 token 与 TS token，统一构造 `inputs_embeds`（文本走 `get_input_embeddings()`，TS 走 Projector 输出）+ `attention_mask` + `labels`（仅回答段非 -100），并约束总长度。
|
||||
|
||||
#### Scenario: 拼接长度受限
|
||||
- **WHEN** 组装一个样本
|
||||
- **THEN** 拼接后序列长度 ≤1024 token
|
||||
|
||||
#### Scenario: 回答段 mask 正确
|
||||
- **WHEN** 构造 labels
|
||||
- **THEN** prompt 段（属性/时序/事件/问题）token 的 label 为 -100，仅回答段非 -100
|
||||
|
||||
### Requirement: 训练/推理统一封装
|
||||
|
||||
系统 SHALL 提供统一封装，组合 TS Encoder + Projector + LLM + 可选 LoRA，`forward(batch)` 返回 loss、`generate(batch)` 返回文本，并支持阶段①（冻结 LLM）与阶段②（LoRA r=16 on q/v_proj）两种模式。
|
||||
|
||||
#### Scenario: 单卡前向反向不 OOM
|
||||
- **WHEN** 在 3060 12GB 上以阶段①模式跑单 batch 前向+反向
|
||||
- **THEN** 不 OOM，loss 为有限值
|
||||
|
||||
#### Scenario: 可生成文本
|
||||
- **WHEN** 调用 `generate(batch)`
|
||||
- **THEN** 返回文本（训练初期允许内容无意义，但流程通）
|
||||
|
||||
### Requirement: 阶段①对齐预训练
|
||||
|
||||
系统 SHALL 支持阶段①训练：冻结 LLM 主体，仅训 TS Encoder + Projector（AdamW），上下文 ≤512、BF16、梯度检查点，并对回答段计 LM loss，叠加对比损失（扰动时序→回答应变）以防止 LLM 忽略 TS token。
|
||||
|
||||
#### Scenario: 冒烟跑通且省显存
|
||||
- **WHEN** 以 `--max_steps 100` 冒烟训练
|
||||
- **THEN** 不 OOM，loss 呈下降趋势，峰值显存 ≤6GB
|
||||
|
||||
#### Scenario: TS token 被有效利用
|
||||
- **WHEN** 训练后对 held-out 样本比较原时序与扰动时序的回答
|
||||
- **THEN** 扰动后回答变化率 >50%
|
||||
|
||||
### Requirement: 阶段②指令微调 SFT
|
||||
|
||||
系统 SHALL 支持阶段②训练：加载阶段① checkpoint，叠加 LoRA r=16（q/v_proj）、LLM 其余冻结，上下文 ≤1024、BF16、梯度检查点，对回答段做 SFT LM loss。
|
||||
|
||||
#### Scenario: 冒烟跑通且压在显存内
|
||||
- **WHEN** 以 `--max_steps 100` 冒烟训练
|
||||
- **THEN** 不 OOM，loss 下降，峰值显存 ≤11GB
|
||||
|
||||
### Requirement: 分阶段 checkpoint 与断点续训
|
||||
|
||||
系统 SHALL 为每个阶段独立存放 checkpoint（阶段① = Encoder+Projector；阶段②在此之上叠加 LoRA），支持断点续训（每 2000 step 存 ckpt，JSONL 流式加载支持任意 step 续）。
|
||||
|
||||
#### Scenario: 分阶段互不污染
|
||||
- **WHEN** 加载阶段② checkpoint
|
||||
- **THEN** 其依赖的阶段① Encoder+Projector 权重与阶段② LoRA 适配器可分别回滚，互不污染
|
||||
```
|
||||
|
||||
## openspec/changes/ts-as-modality/specs/ts-synthesis/spec.md
|
||||
|
||||
- Source: openspec/changes/ts-as-modality/specs/ts-synthesis/spec.md
|
||||
- Lines: 1-77
|
||||
- SHA256: 330043d178e344f1f1e87d1262f25ed33b2216eb6e16f2658d4f29a779d8b60b
|
||||
|
||||
```md
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: 可复现的合成时序生成
|
||||
|
||||
系统 SHALL 提供 CPU 离线合成数据管线，通过可配置成分模型叠加生成多变量时序，且在给定随机种子下输出完全确定、可复现。
|
||||
|
||||
#### Scenario: 给定种子输出确定
|
||||
- **WHEN** 使用相同 seed 调用 `generate_series(T, C, seed)` 两次
|
||||
- **THEN** 两次输出的时序数组逐点相等
|
||||
|
||||
#### Scenario: 成分叠加覆盖典型形态
|
||||
- **WHEN** 生成一条时序
|
||||
- **THEN** 该时序由趋势/周期/基线/噪声（含高斯与重尾 Student-t）成分叠加而成，且各成分参数被记录为元数据
|
||||
|
||||
### Requirement: 异常注入与精确标签
|
||||
|
||||
系统 SHALL 支持向合成时序注入多类异常，并把异常段位置与逐点标签作为 ground-truth 自动记录，标签与异常位置一致。
|
||||
|
||||
#### Scenario: 注入异常并记录逐点标签
|
||||
- **WHEN** 调用 `inject_anomaly(series, types)` 注入尖刺/水平偏移/方差膨胀/缺失段/缓慢漂移中的任一类
|
||||
- **THEN** 返回 `(series, labels)`，其中 labels 为逐点 0/1，且 labels 中值为 1 的点与实际异常区间一致
|
||||
|
||||
#### Scenario: 异常稀有性兜底
|
||||
- **WHEN** 生成用于训练的样本
|
||||
- **THEN** 每条样本至少包含 1 类异常，保证异常类指令任务有正样本
|
||||
|
||||
### Requirement: 事件耦合
|
||||
|
||||
系统 SHALL 支持在指定时间点注入事件相关的阶跃变化，并生成对应的事件文本描述，以支撑时序+事件联合推理。
|
||||
|
||||
#### Scenario: 事件注入产生阶跃
|
||||
- **WHEN** 调用 `couple_event(series, t, kind)` 在时间 t 注入事件
|
||||
- **THEN** 时序在 t 处出现阶跃变化，并返回该事件的文本描述
|
||||
|
||||
### Requirement: 缺失与不规则采样模拟
|
||||
|
||||
系统 SHALL 支持随机丢弃部分时序点（用 NaN 占位），以模拟真实采集缺陷。
|
||||
|
||||
#### Scenario: 按比例丢弃点
|
||||
- **WHEN** 调用 `add_missing(series, rate=0.05~0.10)`
|
||||
- **THEN** 约 5-10% 的点被置为 NaN 占位
|
||||
|
||||
### Requirement: 属性词表采样
|
||||
|
||||
系统 SHALL 提供变量名词表（CPU/内存/流量/延迟/温度/请求量等）、单位与采样率档位，并按通道数采样返回结构化属性。
|
||||
|
||||
#### Scenario: 采样属性字段齐全
|
||||
- **WHEN** 调用 `sample_attributes(n_channels)`
|
||||
- **THEN** 返回长度为 n_channels 的列表，每项含 `name`、`unit`、`freq` 字段，且给定 seed 可复现
|
||||
|
||||
### Requirement: 多类指令与回答生成
|
||||
|
||||
系统 SHALL 支持 6 类指令任务（描述/统计、异常检测、根因/解释、预测、比较、事件关联），为每类生成指令，并产出可结构化解析的回答（异常类含 JSON 区间，统计类含数值，解释类含文本）。
|
||||
|
||||
#### Scenario: 异常类回答可被解析
|
||||
- **WHEN** 生成异常检测类的指令与回答
|
||||
- **THEN** 回答中的 JSON 可被 `json.loads` 解析，且其异常段与 labels 吻合
|
||||
|
||||
#### Scenario: 覆盖全部 6 类
|
||||
- **WHEN** 运行指令生成
|
||||
- **THEN** 6 类任务每类至少各产 1 条样本
|
||||
|
||||
### Requirement: 离线批量生成 JSONL
|
||||
|
||||
系统 SHALL 提供离线脚本多进程生成 JSONL 数据集，每条记录含 `series`/`attributes`/`timestamps`/`events`/`instruction`/`answer`/`labels`，并支持条数、输出路径、种子、并发数参数。
|
||||
|
||||
#### Scenario: 小规模生成产出合法 schema
|
||||
- **WHEN** 运行 `gen_synthetic.py --n 1000`
|
||||
- **THEN** 抽样 5 条每条均含上述 7 个字段，schema 合规
|
||||
|
||||
### Requirement: 真实 benchmark 加载与问答改写
|
||||
|
||||
系统 SHALL 能加载真实多变量时序 benchmark 窗口（至少 SWaT 与 MSL/MSL 系列中的 2 个），做滑窗+归一化+异常标签对齐，并把异常段改写为问答对（仅评测用，不进训练）。
|
||||
|
||||
#### Scenario: 加载并滑窗
|
||||
- **WHEN** 调用 `load_windows(name, T=512, stride=256)`
|
||||
- **THEN** 返回窗口形状 `[N, T, C]` 与标签形状 `[N, T]`，问答对非空
|
||||
```
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-06-29
|
||||
@@ -0,0 +1,88 @@
|
||||
## Context
|
||||
|
||||
本 change 在**单张 3060 12GB 消费级显卡**上做一个可复现的实验性落地：把**多变量时序视为独立模态**，与文本/事件模态联合注入小型 LLM（Qwen2.5-0.5B-Instruct），端到端任务是**时序问答与推理**（ChatTS 路线）。
|
||||
|
||||
背景与约束来自 `MTSAD-技术深度调研报告.md` 与 `docs/superpowers/specs/2026-06-29-ts-as-modality-design.md`（已通过设计评审）。核心约束：
|
||||
|
||||
- **算力**：单卡 12GB → 骨干 ≤0.5B、依赖 LoRA/QLoRA + 短上下文 + 小批量 + 梯度检查点
|
||||
- **数据**：合成数据为主（CPU 离线生成）→ 评测锚定真实 benchmark，防合成过拟合
|
||||
- **目标**：不追学术 SOTA，追「12GB 跑通、可复现、时序模态确有增益」的可验证结论
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- 全流程单卡 3060 12GB 跑通，阶段②峰值显存 ≤11GB
|
||||
- TS Encoder + Projector 把时序编码为独立 soft token，使 LLM「能读懂」时序
|
||||
- 时序模态带来可量化增益：真实 benchmark VUS-PR 显著优于纯 LLM 数值喂文本基线；时序消融后指标明显下降
|
||||
- 6 类问答任务（描述/异常/根因/预测/比较/事件关联）LLM-judge 均分超纯 LLM 基线
|
||||
- 评测可复现：含 trivial 基线对照、标注是否用 PA、严守 MTSAD 报告评估纪律
|
||||
|
||||
**Non-Goals:**
|
||||
- ❌ 不从零预训练 LLM（成本不允许，且非本实验目的）
|
||||
- ❌ 不做多卡分布式训练（单卡约束）
|
||||
- ❌ 不追求学术 SOTA 排名
|
||||
- ❌ 不做端侧部署优化（先验证方法）
|
||||
- ❌ DPO 首版不做（M5 后视效果再决定）
|
||||
|
||||
## Decisions
|
||||
|
||||
### 决策 1：采用 Patch-Project + 轻量编码器，而非时序 Tokenization 或 Reprogramming
|
||||
|
||||
时序切 patch（P=8, stride=4，通道独立）→ 2 层 Transformer 编码器（d=256, 4 头, ~4M）→ 线性投影（256→896）成独立 TS token，作为可学习 soft token 注入 LLM。
|
||||
|
||||
**对比的备选方案：**
|
||||
|
||||
| 方案 | 含义 | 取舍 |
|
||||
|------|------|------|
|
||||
| **A（选用）Patch-Project + 轻量编码器** | patch→小编码器→线性投影成独立 TS token | 真正的"时序作为模态"；12GB 可训；patch 捕捉局部结构、变长友好。代价：需训编码器+投影层（已在显存预算内） |
|
||||
| B 时序 Tokenization（Chronos 风格） | 数值量化成离散 token、扩展词表 | 最"时序即语言"，但量化丢精度、异常细节易损；逐点 token 序列长，12GB 上下文吃紧。放弃 |
|
||||
| C Reprogramming（Time-LLM 风格） | 冻结 LLM 用文本原型重映射 | 参数最省，但受限于已有文本嵌入流形、表达力弱，时序算不上"真新模态"，复杂推理/事件联合弱。放弃 |
|
||||
|
||||
### 决策 2：骨干选 Qwen2.5-0.5B-Instruct
|
||||
|
||||
0.5B 中英均衡、有现成 Instruct 版、社区生态成熟；BF16 约 1GB，12GB 余量充足（hidden=896）。
|
||||
|
||||
### 决策 3：两阶段训练，阶段①冻结 LLM
|
||||
|
||||
- **阶段① 对齐预训练**：冻结 LLM，仅训 TS Encoder + Projector（~4.2M 全参）→ 不存 LLM 优化器状态/梯度，是 12GB 可行的关键省显存技巧
|
||||
- **阶段② SFT**：在此之上叠加 LoRA r=16（q/v_proj，~2-3M），LLM 其余冻结 + 梯度检查点
|
||||
|
||||
分阶段 checkpoint 互不污染、可回滚。损失仅对回答段 token 计 LM loss；阶段①叠加对比损失（扰动时序→回答应变）防止 LLM 忽略 TS token。
|
||||
|
||||
### 决策 4：输入拼接顺序 `[属性][时间戳][TS token][事件][问题]`
|
||||
|
||||
属性先给语义，时序居中，事件定位在对应时间，问题在末尾便于回答。
|
||||
|
||||
### 决策 5：合成数据为主，真实 benchmark 仅用于评测锚定
|
||||
|
||||
合成管线全 CPU 离线，成分参数随机化；**核心红利**：异常段、事件时间戳、各成分参数全部作为 ground-truth 自动记录——免费且精确的标签。评测锚定 SMD/MSL/SMAP/SWaT/PSM 真实分布，避免「合成上 SOTA、真实上崩盘」。
|
||||
|
||||
### 决策 6：评测指标以 VUS-PR 为主，严守 MTSAD 报告纪律
|
||||
|
||||
- 主指标 **VUS-PR**（阈值无关）；辅 Affiliation-F1（多 λ）、AUC-PR、Point-F1（无 PA）
|
||||
- **PA-F1 仅作对照、不得作主指标**（防虚高）
|
||||
- 问答用 LLM-judge + 规则解析双轨
|
||||
- 自然语言回答→约束 JSON 区间→逐点分数，便于数值化评测
|
||||
|
||||
### 决策 7：4 类同口径基线
|
||||
|
||||
纯 LLM 数值喂文本（检验 TS 模态是否真有增益）、Time-LLM 风格重映射、ChatTS（若可复现）、Trivial（Random/Constant/全报异常）。全部走同一套解析+指标脚本。
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [12GB 阶段②仍 OOM] → 降批量/上下文、开 CPU offload 优化器、`adamw_8bit`、退回阶段①-only 报告
|
||||
- [合成→真实泛化差距大（高风险）] → 阶段②掺真实 benchmark 改写样本；评测以真实 benchmark 为准
|
||||
- [LLM 忽略 TS token] → 对比损失 + "必看时序"样本 + 消融实验定位
|
||||
- [回答 JSON 解析失败率高] → 训练强化格式约束 + 解析兜底（失败→全 0 计漏报）+ 单独报解析成功率
|
||||
- [GPT-4o judge 成本] → 仅评测 2k 样本调用；可换开源 judge 做 sanity
|
||||
- [TS Encoder 过拟合合成分布] → 成分参数随机化 + 多套合成配置 + 阶段②引入真实改写样本
|
||||
|
||||
## Open Questions
|
||||
|
||||
- TS Encoder 层数/d 是否需小规模 sweep（2 层是否够）
|
||||
- Patch P=8/stride=4 是否最优，是否需对比 P=16
|
||||
- 对比损失具体形式（InfoNCE vs 扰动一致性）与权重 λ
|
||||
- DPO 是否纳入首版（建议首版先不做，M5 后视效果决定）
|
||||
- 真实 benchmark 改写问答的数量与质量校验流程
|
||||
|
||||
> 详细行为需求见 `specs/ts-synthesis/`、`specs/ts-multimodal-model/`、`specs/ts-evaluation/`；任务拆解见 `tasks.md`（M1→M5）。
|
||||
@@ -0,0 +1,33 @@
|
||||
## Why
|
||||
|
||||
当前 AIOps 场景下，多变量时序的异常检测与解释仍依赖专用模型，缺乏「以自然语言对时序提问、做根因/事件关联推理」的统一能力。基于 `MTSAD-技术深度调研报告.md`，沿用 ChatTS「时序作为新模态」路线，我们希望在**单张 3060 12GB 消费级显卡**上做一个可复现的实验性落地，验证「把多变量时序视为独立模态、与文本/事件模态联合注入小型 LLM」这一方法在算力受限条件下是否切实可行、且「时序模态」是否真带来可量化的增益。现在做是因为：0.5B 级开源 LLM 与 PEFT/LoRA 工具链已足够成熟，使该实验在消费级硬件上首次成为可能。
|
||||
|
||||
## What Changes
|
||||
|
||||
- **新增**：CPU 离线合成数据管线（成分模型 + 异常注入 + 事件耦合 + Evol-Instruct 指令/回答生成），产出对齐预训练与 SFT 用的 JSONL
|
||||
- **新增**：真实 benchmark（SMD/MSL/SMAP/SWaT/PSM）窗口加载与「异常段→问答对」改写（仅评测用，不进训练）
|
||||
- **新增**：TS Encoder（Patchify + 2 层 Transformer）+ Projector（Linear→LLM hidden），把时序编码为独立 soft token
|
||||
- **新增**：多模态拼接器，按 `[属性][时间戳][TS token][事件][问题]` 顺序组装文本 token 与 TS token 注入 LLM
|
||||
- **新增**：两阶段训练流程 —— 阶段①冻结 LLM 训 Encoder+Projector（对齐预训练）；阶段②叠加 LoRA(r=16) 做指令微调
|
||||
- **新增**：评测套件 —— 文本回答→JSON 区间→逐点分数解析；VUS-PR(主)/Affiliation-F1/AUC-PR/Point-F1(无PA)；LLM-judge + 规则解析双轨
|
||||
- **新增**：4 类同口径对照基线（纯 LLM 数值喂文本 / Time-LLM 风格重映射 / ChatTS(若可复现) / Trivial）
|
||||
- 骨干选用 **Qwen2.5-0.5B-Instruct**，BF16 + LoRA + 梯度检查点，目标阶段②峰值显存 ≤11GB
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `ts-synthesis`: CPU 离线合成数据管线——成分模型（趋势/周期/基线/噪声）、异常注入（尖刺/水平偏移/方差膨胀/缺失段/缓慢漂移）、事件耦合、缺失/不规则模拟、属性词表采样，以及 Evol-Instruct 多轮指令与回答生成；附带真实 benchmark 加载与问答改写
|
||||
- `ts-multimodal-model`: 时序作为模态的多模态模型——TS Encoder + Projector + 多模态拼接（TS token 与文本/事件 token 联合注入 LLM），含阶段①对齐预训练与阶段②LoRA SFT 的训练/推理统一封装
|
||||
- `ts-evaluation`: 时序问答与异常检测评测套件——回答解析（文本→JSON 区间→逐点分数）、阈值无关指标（VUS-PR 主/Affiliation-F1/AUC-PR/Point-F1 无 PA）、LLM-judge + 规则双轨、4 类同口径基线对照、可复现评测报告
|
||||
|
||||
### Modified Capabilities
|
||||
<!-- 无已存在 capability 的需求变更（本项目 specs/ 当前为空）。 -->
|
||||
|
||||
## Impact
|
||||
|
||||
- **新增代码**：独立 Python 项目 `ts-as-modality/`（`pyproject.toml` + `src/tsmm/{data,model,train,eval,utils}` + `scripts/` + `tests/` + `configs/`），与当前 spec/docs 工作区并列
|
||||
- **依赖**：torch、transformers、peft、accelerate、datasets、numpy、tqdm、pyyaml、pytest；评测依赖 `tsb_uad`（VUS-PR）；LLM-judge 评测时调用 OpenAI/GPT-4o API
|
||||
- **模型资产**：Qwen2.5-0.5B-Instruct 骨干（外部下载）；阶段①/② checkpoint 与合成数据产物体积较大，需 git-ignore（`data/`、`checkpoints/`）
|
||||
- **算力**：单张 3060 12GB；阶段①预估峰值 ~4-5GB，阶段②预估峰值 ~9-10GB；CPU 多进程用于离线合成
|
||||
- **存储**：合成数据约 2-3GB（流式加载），checkpoint 分阶段独立存放
|
||||
- **无 BREAKING 变更**：纯新增独立实验项目，不触碰现有任何代码/接口
|
||||
@@ -0,0 +1,61 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: 回答解析为逐点分数
|
||||
|
||||
系统 SHALL 从模型自然语言回答中提取 JSON 异常段并生成逐点分数（段内 1.0，段外 0.0），解析失败时回退为全 0（计为漏报）并记录失败标记。
|
||||
|
||||
#### Scenario: 多格式可解析
|
||||
- **WHEN** 输入含噪声/嵌套的多种回答格式
|
||||
- **THEN** 能正确解析出异常段并生成逐点分数 `[T]`
|
||||
|
||||
#### Scenario: 解析失败有兜底
|
||||
- **WHEN** 回答无法解析
|
||||
- **THEN** 分数全 0 并记录失败标记（不计为成功）
|
||||
|
||||
### Requirement: 阈值无关的异常检测指标
|
||||
|
||||
系统 SHALL 以 VUS-PR 为异常检测主指标，并辅以 Affiliation-F1（λ=0.1/0.5/1.0）、AUC-PR、Point-F1（无 PA）；PA-F1 仅作对照、不得作为主指标。
|
||||
|
||||
#### Scenario: trivial 基线不虚高
|
||||
- **WHEN** 对「全报异常」trivial 基线计算 VUS-PR
|
||||
- **THEN** VUS-PR 不虚高（避免评估作弊）
|
||||
|
||||
#### Scenario: 对已知段算出合理值
|
||||
- **WHEN** 对已知异常段计算指标
|
||||
- **THEN** VUS-PR/Aff-F1/AUC-PR/Point-F1 均落在合理区间
|
||||
|
||||
### Requirement: 问答评测双轨
|
||||
|
||||
系统 SHALL 提供规则解析轨（JSON/数值容差比对，产准确率）与 LLM-judge 轨（对 `问题,时序摘要,参考答案,模型答案` 打 0-5 分），并对两轨不一致样本导出供人工抽检。
|
||||
|
||||
#### Scenario: 产出 6 类任务评测结果
|
||||
- **WHEN** 对评测集运行问答评测
|
||||
- **THEN** 对 6 类任务各产出 `{规则准确率, judge 均分, 解析成功率}`
|
||||
|
||||
### Requirement: 同口径对照基线
|
||||
|
||||
系统 SHALL 提供 4 类同口径基线——纯 LLM 数值喂文本、Time-LLM 风格重映射、ChatTS（若不可复现则标注）、Trivial（Random/Constant/全报异常）——且全部走同一套解析+指标脚本。
|
||||
|
||||
#### Scenario: 4 类基线产出同口径指标
|
||||
- **WHEN** 运行基线评测
|
||||
- **THEN** 4 类基线均在同一套 `parse_answer`+`ts_metrics` 下产出指标表
|
||||
|
||||
### Requirement: 可复现评测报告
|
||||
|
||||
系统 SHALL 提供一键评测脚本，产出 markdown 报告，覆盖每数据集×每任务的 VUS-PR/Aff-F1/AUC-PR/Point-F1/PA-F1(对照)/问答分/解析成功率/消融/基线对比，并明确标注是否使用 PA、含 trivial 对照。
|
||||
|
||||
#### Scenario: 一键产出完整报告
|
||||
- **WHEN** 运行 `run_eval.sh`
|
||||
- **THEN** 产出满足设计规格 §5.4 必报清单的完整 markdown 报告
|
||||
|
||||
### Requirement: 模态消融与泛化验证
|
||||
|
||||
系统 SHALL 支持模态消融（去掉事件/属性/时序）与跨数据集泛化验证（如 SMD→MSL/SWaT），以证明时序模态确有贡献并暴露合成→真实差距。
|
||||
|
||||
#### Scenario: 时序消融指标下降
|
||||
- **WHEN** 去掉时序模态后评测
|
||||
- **THEN** 关键指标（VUS-PR/问答分）明显低于完整模型，证明时序模态有贡献
|
||||
|
||||
#### Scenario: 真实 benchmark 显著优于纯 LLM 基线
|
||||
- **WHEN** 在真实 benchmark 上对比完整模型与纯 LLM 数值喂文本基线
|
||||
- **THEN** VUS-PR 显著优于纯 LLM 基线
|
||||
@@ -0,0 +1,73 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: TS Encoder 把时序编码为 patch 表示
|
||||
|
||||
系统 SHALL 通过通道独立的 Patchify（P=8, stride=4）将多变量时序切分为重叠 patch，并经 2 层 Transformer 编码器（d=256, 4 头）映射为时序 token 序列。
|
||||
|
||||
#### Scenario: 形状正确
|
||||
- **WHEN** 输入形状 `[B, T=512, C=5]`
|
||||
- **THEN** 输出形状 `[B, 128, 256]`，且 TS Encoder 参数量约 4M
|
||||
|
||||
#### Scenario: 通道独立
|
||||
- **WHEN** 通道数 C 变化（如 C=3 与 C=12）
|
||||
- **THEN** 编码器仍正常工作（通道独立 patchify，不因 C 增大失配）
|
||||
|
||||
### Requirement: Projector 映射到 LLM 隐藏维
|
||||
|
||||
系统 SHALL 提供线性投影层（256→896）+ LayerNorm，将 TS 表示映射到 Qwen2.5-0.5B 的隐藏维（896），作为可学习 soft token。
|
||||
|
||||
#### Scenario: 投影对齐 LLM hidden
|
||||
- **WHEN** TS Encoder 输出 `[B, 128, 256]`
|
||||
- **THEN** Projector 输出 `[B, 128, 896]`，与 LLM 隐藏维一致
|
||||
|
||||
### Requirement: 多模态拼接注入 LLM
|
||||
|
||||
系统 SHALL 按 `[属性][时间戳][TS token][事件][问题]` 顺序组装文本 token 与 TS token，统一构造 `inputs_embeds`（文本走 `get_input_embeddings()`，TS 走 Projector 输出）+ `attention_mask` + `labels`（仅回答段非 -100），并约束总长度。
|
||||
|
||||
#### Scenario: 拼接长度受限
|
||||
- **WHEN** 组装一个样本
|
||||
- **THEN** 拼接后序列长度 ≤1024 token
|
||||
|
||||
#### Scenario: 回答段 mask 正确
|
||||
- **WHEN** 构造 labels
|
||||
- **THEN** prompt 段（属性/时序/事件/问题）token 的 label 为 -100，仅回答段非 -100
|
||||
|
||||
### Requirement: 训练/推理统一封装
|
||||
|
||||
系统 SHALL 提供统一封装，组合 TS Encoder + Projector + LLM + 可选 LoRA，`forward(batch)` 返回 loss、`generate(batch)` 返回文本，并支持阶段①（冻结 LLM）与阶段②（LoRA r=16 on q/v_proj）两种模式。
|
||||
|
||||
#### Scenario: 单卡前向反向不 OOM
|
||||
- **WHEN** 在 3060 12GB 上以阶段①模式跑单 batch 前向+反向
|
||||
- **THEN** 不 OOM，loss 为有限值
|
||||
|
||||
#### Scenario: 可生成文本
|
||||
- **WHEN** 调用 `generate(batch)`
|
||||
- **THEN** 返回文本（训练初期允许内容无意义，但流程通）
|
||||
|
||||
### Requirement: 阶段①对齐预训练
|
||||
|
||||
系统 SHALL 支持阶段①训练：冻结 LLM 主体，仅训 TS Encoder + Projector（AdamW），上下文 ≤512、BF16、梯度检查点，并对回答段计 LM loss，叠加对比损失（扰动时序→回答应变）以防止 LLM 忽略 TS token。
|
||||
|
||||
#### Scenario: 冒烟跑通且省显存
|
||||
- **WHEN** 以 `--max_steps 100` 冒烟训练
|
||||
- **THEN** 不 OOM，loss 呈下降趋势，峰值显存 ≤6GB
|
||||
|
||||
#### Scenario: TS token 被有效利用
|
||||
- **WHEN** 训练后对 held-out 样本比较原时序与扰动时序的回答
|
||||
- **THEN** 扰动后回答变化率 >50%
|
||||
|
||||
### Requirement: 阶段②指令微调 SFT
|
||||
|
||||
系统 SHALL 支持阶段②训练：加载阶段① checkpoint，叠加 LoRA r=16（q/v_proj）、LLM 其余冻结，上下文 ≤1024、BF16、梯度检查点，对回答段做 SFT LM loss。
|
||||
|
||||
#### Scenario: 冒烟跑通且压在显存内
|
||||
- **WHEN** 以 `--max_steps 100` 冒烟训练
|
||||
- **THEN** 不 OOM，loss 下降，峰值显存 ≤11GB
|
||||
|
||||
### Requirement: 分阶段 checkpoint 与断点续训
|
||||
|
||||
系统 SHALL 为每个阶段独立存放 checkpoint（阶段① = Encoder+Projector；阶段②在此之上叠加 LoRA），支持断点续训（每 2000 step 存 ckpt，JSONL 流式加载支持任意 step 续）。
|
||||
|
||||
#### Scenario: 分阶段互不污染
|
||||
- **WHEN** 加载阶段② checkpoint
|
||||
- **THEN** 其依赖的阶段① Encoder+Projector 权重与阶段② LoRA 适配器可分别回滚，互不污染
|
||||
@@ -0,0 +1,77 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: 可复现的合成时序生成
|
||||
|
||||
系统 SHALL 提供 CPU 离线合成数据管线，通过可配置成分模型叠加生成多变量时序，且在给定随机种子下输出完全确定、可复现。
|
||||
|
||||
#### Scenario: 给定种子输出确定
|
||||
- **WHEN** 使用相同 seed 调用 `generate_series(T, C, seed)` 两次
|
||||
- **THEN** 两次输出的时序数组逐点相等
|
||||
|
||||
#### Scenario: 成分叠加覆盖典型形态
|
||||
- **WHEN** 生成一条时序
|
||||
- **THEN** 该时序由趋势/周期/基线/噪声（含高斯与重尾 Student-t）成分叠加而成，且各成分参数被记录为元数据
|
||||
|
||||
### Requirement: 异常注入与精确标签
|
||||
|
||||
系统 SHALL 支持向合成时序注入多类异常，并把异常段位置与逐点标签作为 ground-truth 自动记录，标签与异常位置一致。
|
||||
|
||||
#### Scenario: 注入异常并记录逐点标签
|
||||
- **WHEN** 调用 `inject_anomaly(series, types)` 注入尖刺/水平偏移/方差膨胀/缺失段/缓慢漂移中的任一类
|
||||
- **THEN** 返回 `(series, labels)`，其中 labels 为逐点 0/1，且 labels 中值为 1 的点与实际异常区间一致
|
||||
|
||||
#### Scenario: 异常稀有性兜底
|
||||
- **WHEN** 生成用于训练的样本
|
||||
- **THEN** 每条样本至少包含 1 类异常，保证异常类指令任务有正样本
|
||||
|
||||
### Requirement: 事件耦合
|
||||
|
||||
系统 SHALL 支持在指定时间点注入事件相关的阶跃变化，并生成对应的事件文本描述，以支撑时序+事件联合推理。
|
||||
|
||||
#### Scenario: 事件注入产生阶跃
|
||||
- **WHEN** 调用 `couple_event(series, t, kind)` 在时间 t 注入事件
|
||||
- **THEN** 时序在 t 处出现阶跃变化，并返回该事件的文本描述
|
||||
|
||||
### Requirement: 缺失与不规则采样模拟
|
||||
|
||||
系统 SHALL 支持随机丢弃部分时序点（用 NaN 占位），以模拟真实采集缺陷。
|
||||
|
||||
#### Scenario: 按比例丢弃点
|
||||
- **WHEN** 调用 `add_missing(series, rate=0.05~0.10)`
|
||||
- **THEN** 约 5-10% 的点被置为 NaN 占位
|
||||
|
||||
### Requirement: 属性词表采样
|
||||
|
||||
系统 SHALL 提供变量名词表（CPU/内存/流量/延迟/温度/请求量等）、单位与采样率档位，并按通道数采样返回结构化属性。
|
||||
|
||||
#### Scenario: 采样属性字段齐全
|
||||
- **WHEN** 调用 `sample_attributes(n_channels)`
|
||||
- **THEN** 返回长度为 n_channels 的列表，每项含 `name`、`unit`、`freq` 字段，且给定 seed 可复现
|
||||
|
||||
### Requirement: 多类指令与回答生成
|
||||
|
||||
系统 SHALL 支持 6 类指令任务（描述/统计、异常检测、根因/解释、预测、比较、事件关联），为每类生成指令，并产出可结构化解析的回答（异常类含 JSON 区间，统计类含数值，解释类含文本）。
|
||||
|
||||
#### Scenario: 异常类回答可被解析
|
||||
- **WHEN** 生成异常检测类的指令与回答
|
||||
- **THEN** 回答中的 JSON 可被 `json.loads` 解析，且其异常段与 labels 吻合
|
||||
|
||||
#### Scenario: 覆盖全部 6 类
|
||||
- **WHEN** 运行指令生成
|
||||
- **THEN** 6 类任务每类至少各产 1 条样本
|
||||
|
||||
### Requirement: 离线批量生成 JSONL
|
||||
|
||||
系统 SHALL 提供离线脚本多进程生成 JSONL 数据集，每条记录含 `series`/`attributes`/`timestamps`/`events`/`instruction`/`answer`/`labels`，并支持条数、输出路径、种子、并发数参数。
|
||||
|
||||
#### Scenario: 小规模生成产出合法 schema
|
||||
- **WHEN** 运行 `gen_synthetic.py --n 1000`
|
||||
- **THEN** 抽样 5 条每条均含上述 7 个字段，schema 合规
|
||||
|
||||
### Requirement: 真实 benchmark 加载与问答改写
|
||||
|
||||
系统 SHALL 能加载真实多变量时序 benchmark 窗口（至少 SWaT 与 MSL/MSL 系列中的 2 个），做滑窗+归一化+异常标签对齐，并把异常段改写为问答对（仅评测用，不进训练）。
|
||||
|
||||
#### Scenario: 加载并滑窗
|
||||
- **WHEN** 调用 `load_windows(name, T=512, stride=256)`
|
||||
- **THEN** 返回窗口形状 `[N, T, C]` 与标签形状 `[N, T]`，问答对非空
|
||||
@@ -0,0 +1,45 @@
|
||||
# Tasks · ts-as-modality
|
||||
|
||||
> 源自 `docs/superpowers/plans/2026-06-29-ts-as-modality-plan.md`（M1→M5）。
|
||||
> 每个里程碑末尾的「出口验证」为该阶段完成判据；建议先跑「冒烟路径」确认端到端通，再放大到全量。
|
||||
|
||||
## 1. M1 · 数据管线
|
||||
|
||||
- [x] 1.1 项目脚手架：创建 `pyproject.toml`（torch/transformers/peft/accelerate/datasets/numpy/tqdm/pyyaml/pytest）、`src/tsmm/` 模块骨架、`configs/`、`scripts/`、`tests/`，并建 `data/`、`checkpoints/` 加 `.gitignore`（验证：`python -c "import tsmm"` 不报错；`pytest tests/` 可跑）
|
||||
- [x] 1.2 属性词表与采样 `data/attributes.py`：定义变量名词表/单位/采样率档位，`sample_attributes(n_channels)` 返回结构化属性（验证：单测字段齐全、可复现）
|
||||
- [x] 1.3 成分模型与时序合成 `data/synthesis.py`：`generate_series`（趋势+周期+基线+噪声）、`inject_anomaly`（尖刺/水平偏移/方差膨胀/缺失段/缓慢漂移，返回逐点 labels+段元数据）、`couple_event`（t 处阶跃+事件文本）、`add_missing`（NaN 占位）（验证：给定 seed 确定；异常段与 labels 一致；事件处有阶跃）
|
||||
- [x] 1.4 指令与回答生成 `data/instruct.py`：6 类指令模板（描述/异常/根因/预测/比较/事件关联）+ Evol-Instruct 演化，`build_instruction` 产 JSON 区间/数值/文本回答（验证：异常类 JSON 可 `json.loads` 且段与 labels 吻合）
|
||||
- [x] 1.5 离线生成脚本 `scripts/gen_synthetic.py`：多进程产 `align.jsonl`(50万)/`sft.jsonl`(10万)/`eval_synth.jsonl`(2k held-out)，支持 `--n/--out/--seed/--workers` + tqdm（验证：`--n 1000` 跑通，抽样 5 条 schema 合规）
|
||||
- [x] 1.6 真实 benchmark 加载 `data/real_bench.py`：加载 SMD/MSL/SMAP/SWaT/PSM（至少 2 个），`load_windows(T=512, stride=256)` 滑窗+归一化+标签，异常段改写问答对存 `eval_real.jsonl`（验证：窗口形状 `[N,T,C]`、标签 `[N,T]`、问答非空）
|
||||
- [x] 1.7 M1 出口验证：`gen_synthetic.py --n 1000` 产可视检合法 JSONL；`real_bench.py` 加载 ≥2 数据集；人工抽看 5 条合成样本确认成分/异常/事件正确
|
||||
|
||||
## 2. M2 · 模型可跑通
|
||||
|
||||
- [ ] 2.1 TS Encoder `model/ts_encoder.py`：`Patchify(patch=8, stride=4)` 通道独立切 patch 线性嵌入到 d=256；`TSEncoder(d=256, layers=2, heads=4)` 2 层 TF + 可学习位置编码；前向 `[B,T,C]`→`[B,n_patches,256]`（验证：输入 `[2,512,5]`→`[2,128,256]`，参数 ≈4M）
|
||||
- [ ] 2.2 Projector `model/projector.py`：`Linear(256→896)+LayerNorm`（验证：输出 `[B,128,896]` 对齐 Qwen2.5-0.5B hidden）
|
||||
- [ ] 2.3 多模态拼接 `model/multimodal.py`：Qwen tokenizer tokenize 文本部分，TS token 作 inputs_embeds 插入 `[属性][时间戳][TS tok][事件][问题]`，统一构造 inputs_embeds+attention_mask+labels（仅回答段非 -100）（验证：seq_len ≤1024，mask/labels 形状对）
|
||||
- [ ] 2.4 训练/推理封装 `model/wrapper.py`：`MultimodalTSModel` 组合 Encoder+Projector+LLM+LoRA 挂载开关，`forward` 返 loss、`generate` 返文本，支持 `freeze_llm`(阶段①)/`enable_lora(r=16)`(阶段②)（验证：单 batch forward 3060 不 OOM；generate 能出文本）
|
||||
- [ ] 2.5 Collator `data/collator.py`：JSONL→batch 张量，处理变长 C（padding+mask）与变长文本（padding+attention_mask）（验证：batch=4 张量形状一致无 NaN）
|
||||
- [ ] 2.6 M2 出口验证：单 batch 前向+反向在 3060 跑通，loss 有限且下降趋势，阶段①模式峰值显存 ~5GB
|
||||
|
||||
## 3. M3 · 阶段①对齐训练
|
||||
|
||||
- [ ] 3.1 损失函数 `train/losses.py`：`lm_loss`（仅回答段计 loss）、`contrastive_loss`（扰动时序→回答 embedding 变化，InfoNCE/扰动一致性），总 loss=`lm_loss+λ*contrastive_loss`（λ 默认 0.1）（验证：loss 标量可反传，对比损失对扰动敏感）
|
||||
- [ ] 3.2 阶段①训练脚本 `train/stage1.py`+`scripts/train_stage1.sh`：加载 `align.jsonl` 流式，`freeze_llm=True` 仅训 Encoder+Projector（AdamW, lr=1e-4），bs=8/grad_accum=4/ctx=512/BF16/梯度检查点，每 2000 step 存 ckpt + TensorBoard（验证：`--max_steps 100` 冒烟不 OOM、loss 下降、峰值显存 ≤6GB）
|
||||
- [ ] 3.3 TS-token 有效性检验：held-out 原时序 vs 扰动时序回答变化率 + 「必看时序」样本答对率对照纯 LLM（验证：扰动后变化率 >50%，必看时序答对率显著高于纯 LLM）
|
||||
- [ ] 3.4 M3 出口验证：阶段① ckpt 产出；扰动检验通过；峰值显存 ≤6GB（失败回查对比损失权重/数据质量）
|
||||
|
||||
## 4. M4 · 阶段②SFT
|
||||
|
||||
- [ ] 4.1 阶段②训练脚本 `train/stage2.py`+`scripts/train_stage2.sh`：加载阶段① ckpt，`enable_lora(r=16)` on q/v_proj 其余冻结，加载 `sft.jsonl`，bs=4/grad_accum=8/ctx=1024/BF16/梯度检查点，每 2000 step 存含 LoRA ckpt（验证：`--max_steps 100` 冒烟不 OOM、loss 下降、峰值显存 ≤11GB）
|
||||
- [ ] 4.2 指令遵循抽检：6 类任务各取 10 条 held-out 检查回答（异常类 JSON 可解析+段合理；解释类切题引用事件）（验证：6 类回答可用率 >70%，JSON 解析成功率 >80%）
|
||||
- [ ] 4.3 M4 出口验证：阶段② ckpt 产出；6 类任务抽检达标；峰值显存 ≤11GB（OOM 则降 bs=2/grad_accum=16、`adamw_8bit`+CPU offload、ctx 降到 768）
|
||||
|
||||
## 5. M5 · 完整评测
|
||||
|
||||
- [ ] 5.1 回答解析 `eval/parse_answer.py`：文本回答→JSON 异常段→逐点分数 `[T]`，失败→全 0+失败标记（验证：多格式可解析、失败有兜底）
|
||||
- [ ] 5.2 异常检测指标 `eval/ts_metrics.py`：`tsb_uad` 算 VUS-PR(主)，自实现 Affiliation-F1(多 λ)/AUC-PR/Point-F1(无PA)，PA-F1 单独对照（验证：已知段算出合理值；trivial 全报异常 VUS-PR 不虚高）
|
||||
- [ ] 5.3 问答评测 `eval/qa_judge.py`：规则解析轨（JSON/数值容差→准确率）+ LLM-judge 轨（GPT-4o 打 0-5 分），不一致样本导出抽检（验证：对 eval 集产出 6 类任务的 {规则准确率,judge 均分,解析成功率}）
|
||||
- [ ] 5.4 对照基线 `eval/baselines.py`：纯 LLM 数值喂文本 / Time-LLM 风格重映射 / ChatTS(若可复现) / Trivial(Random/Constant/全报异常)，全走同一套 parse+metrics（验证：4 类基线产出同口径指标表）
|
||||
- [ ] 5.5 评测脚本与报告 `scripts/run_eval.sh`：跑 `eval_synth.jsonl`+`eval_real.jsonl`，产 `reports/eval-YYYYMMDD.md`（含每数据集×每任务全部必报指标、标注是否用 PA、含 trivial 对照）（验证：一键产出满足设计规格 §5.4 必报清单的完整报告）
|
||||
- [ ] 5.6 M5 出口验证：报告满足设计规格 §6.5 成功标准——真实 benchmark VUS-PR 显著优于纯 LLM 基线；时序消融后指标明显下降；问答均分超基线；评测可复现含 trivial
|
||||
@@ -0,0 +1,35 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "tsmm"
|
||||
version = "0.0.1"
|
||||
description = "Time-series as a modality: multimodal TS-question-answering model (experimental, single-GPU)"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
license = { text = "MIT" }
|
||||
authors = [{ name = "tsmm contributors" }]
|
||||
dependencies = [
|
||||
"torch",
|
||||
"transformers",
|
||||
"peft",
|
||||
"accelerate",
|
||||
"datasets",
|
||||
"numpy",
|
||||
"tqdm",
|
||||
"pyyaml",
|
||||
"pytest",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
# Heavy GPU deps installed on-demand at M2 (CUDA torch build, etc.)
|
||||
dev = ["pytest", "pytest-cov"]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["src"]
|
||||
addopts = "-q"
|
||||
@@ -0,0 +1,3 @@
|
||||
# Scripts directory
|
||||
|
||||
Offline entrypoints: `gen_synthetic.py`, `train_stage1.sh`, `train_stage2.sh`, `run_eval.sh`.
|
||||
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline synthetic data generation (T1.5).
|
||||
|
||||
Produces JSONL datasets by composing :mod:`tsmm.data` primitives in parallel:
|
||||
|
||||
* ``data/align.jsonl`` — alignment-style samples (default: many)
|
||||
* ``data/sft.jsonl`` — Evol-Instruct rephrased (subset)
|
||||
* ``data/eval_synth.jsonl`` — held-out eval set (fixed seed)
|
||||
|
||||
Usage::
|
||||
|
||||
python scripts/gen_synthetic.py --n 1000 --out data/align.jsonl --seed 0 --workers 8
|
||||
|
||||
Each line is one JSON sample with keys: series, attributes, timestamps,
|
||||
events, instruction, answer, labels, category, segments.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
|
||||
from tqdm import tqdm
|
||||
|
||||
from tsmm.data.gen_pipeline import gen_one_sample, write_jsonl
|
||||
from tsmm.data.instruct import evolve_instruction
|
||||
|
||||
|
||||
def _one(args):
|
||||
"""Worker: generate a single sample (top-level for pickling)."""
|
||||
T, C, seed, anomaly_types, missing_rate, event_prob, do_evolve = args
|
||||
s = gen_one_sample(
|
||||
T=T, C=C, seed=seed,
|
||||
anomaly_types=anomaly_types,
|
||||
missing_rate=missing_rate,
|
||||
event_prob=event_prob,
|
||||
)
|
||||
if do_evolve:
|
||||
s["instruction"] = evolve_instruction(s["instruction"], seed=seed)
|
||||
return s
|
||||
|
||||
|
||||
def run(
|
||||
n: int,
|
||||
out: str,
|
||||
*,
|
||||
seed: int = 0,
|
||||
workers: int = max(1, (os.cpu_count() or 2) - 1),
|
||||
T: int = 512,
|
||||
C: int = 5,
|
||||
anomaly_types=("spike", "level_shift", "variance_change", "drift"),
|
||||
missing_rate: float = 0.05,
|
||||
event_prob: float = 0.5,
|
||||
do_evolve: bool = False,
|
||||
) -> int:
|
||||
Path(out).parent.mkdir(parents=True, exist_ok=True)
|
||||
work = [
|
||||
(T, C, seed + i, anomaly_types, missing_rate, event_prob, do_evolve)
|
||||
for i in range(n)
|
||||
]
|
||||
samples = [None] * n
|
||||
with ProcessPoolExecutor(max_workers=workers) as ex:
|
||||
futures = {ex.submit(_one, w): i for i, w in enumerate(work)}
|
||||
for fut in tqdm(as_completed(futures), total=n, desc=os.path.basename(out)):
|
||||
i = futures[fut]
|
||||
samples[i] = fut.result()
|
||||
return write_jsonl(samples, out)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Generate synthetic TS-QA JSONL data.")
|
||||
p.add_argument("--n", type=int, default=1000, help="number of samples")
|
||||
p.add_argument("--out", type=str, default="data/align.jsonl", help="output JSONL path")
|
||||
p.add_argument("--seed", type=int, default=0, help="base RNG seed")
|
||||
p.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 2) - 1))
|
||||
p.add_argument("--T", type=int, default=512, help="series length")
|
||||
p.add_argument("--C", type=int, default=5, help="number of channels")
|
||||
p.add_argument("--missing-rate", type=float, default=0.05)
|
||||
p.add_argument("--event-prob", type=float, default=0.5)
|
||||
p.add_argument("--evolve", action="store_true", help="apply Evol-Instruct rephrasing (for sft set)")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
n = run(
|
||||
n=args.n,
|
||||
out=args.out,
|
||||
seed=args.seed,
|
||||
workers=args.workers,
|
||||
T=args.T,
|
||||
C=args.C,
|
||||
missing_rate=args.missing_rate,
|
||||
event_prob=args.event_prob,
|
||||
do_evolve=args.evolve,
|
||||
)
|
||||
print(f"wrote {n} samples -> {args.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,7 @@
|
||||
"""tsmm — Time-series as a Modality (multimodal).
|
||||
|
||||
Experimental, single-GPU implementation following the ChatTS "TS as a new
|
||||
modality" route. See `docs/superpowers/specs/2026-06-29-ts-as-modality-design.md`.
|
||||
"""
|
||||
|
||||
__version__ = "0.0.1"
|
||||
@@ -0,0 +1 @@
|
||||
"""Data pipeline: attribute vocab, synthesis, instructions, collator, real benchmarks."""
|
||||
@@ -0,0 +1,77 @@
|
||||
"""Variable attribute vocabulary + attribute sampling (T1.2).
|
||||
|
||||
Defines a curated vocabulary of AIOps-relevant channel *names* with their
|
||||
*units* and a set of *sampling frequency bands*. ``sample_attributes`` draws a
|
||||
reproducible, unique-name subset to describe the channels of a multivariate
|
||||
series — metadata that the instruction generator and the multimodal splicer
|
||||
will later turn into text tokens like ``[属性: CPU usage (%), 1s]``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
from typing import Any
|
||||
|
||||
# ─── vocabulary ────────────────────────────────────────────────────────────
|
||||
# Each entry: name (human label), unit. Frequencies are drawn separately from
|
||||
# FREQ_BANDS so the same channel name can appear at different resolutions.
|
||||
ATTRIBUTE_VOCAB: list[dict[str, str]] = [
|
||||
{"name": "CPU usage", "unit": "%"},
|
||||
{"name": "Memory usage", "unit": "%"},
|
||||
{"name": "Disk I/O", "unit": "MB/s"},
|
||||
{"name": "Network inbound traffic", "unit": "Mbps"},
|
||||
{"name": "Network outbound traffic", "unit": "Mbps"},
|
||||
{"name": "Request rate", "unit": "req/s"},
|
||||
{"name": "Request latency p50", "unit": "ms"},
|
||||
{"name": "Request latency p99", "unit": "ms"},
|
||||
{"name": "Error rate", "unit": "%"},
|
||||
{"name": "Temperature", "unit": "°C"},
|
||||
{"name": "Queue length", "unit": "tasks"},
|
||||
{"name": "Active connections", "unit": "conn"},
|
||||
{"name": "Cache hit ratio", "unit": "%"},
|
||||
{"name": "GPU utilization", "unit": "%"},
|
||||
{"name": "Battery level", "unit": "%"},
|
||||
{"name": "Pod restart count", "unit": "count"},
|
||||
]
|
||||
|
||||
# Sampling frequency bands (resolution at which the series is recorded).
|
||||
FREQ_BANDS: list[str] = ["1s", "5s", "10s", "30s", "1min", "5min"]
|
||||
|
||||
|
||||
def sample_attributes(
|
||||
n_channels: int, *, seed: int | None = None
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Sample ``n_channels`` distinct channel attribute descriptors.
|
||||
|
||||
Each descriptor is a dict with keys ``name``, ``unit`` (drawn from
|
||||
:data:`ATTRIBUTE_VOCAB`) and ``freq`` (drawn from :data:`FREQ_BANDS`).
|
||||
Channel names are guaranteed unique within the returned list.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
n_channels:
|
||||
Number of channels to sample. Must be ``<= len(ATTRIBUTE_VOCAB)``.
|
||||
seed:
|
||||
RNG seed for reproducibility.
|
||||
|
||||
Raises
|
||||
------
|
||||
ValueError
|
||||
If ``n_channels`` exceeds the available distinct names.
|
||||
"""
|
||||
if not isinstance(n_channels, int) or n_channels < 0:
|
||||
raise ValueError(f"n_channels must be a non-negative int, got {n_channels!r}")
|
||||
if n_channels > len(ATTRIBUTE_VOCAB):
|
||||
raise ValueError(
|
||||
f"requested {n_channels} channels but vocab only has "
|
||||
f"{len(ATTRIBUTE_VOCAB)} distinct names"
|
||||
)
|
||||
|
||||
rng = random.Random(seed)
|
||||
vocab = list(ATTRIBUTE_VOCAB)
|
||||
chosen = rng.sample(vocab, n_channels)
|
||||
freqs = rng.choices(FREQ_BANDS, k=n_channels)
|
||||
return [
|
||||
{"name": c["name"], "unit": c["unit"], "freq": f}
|
||||
for c, f in zip(chosen, freqs)
|
||||
]
|
||||
@@ -0,0 +1,159 @@
|
||||
"""Synthetic sample-composition pipeline (T1.5).
|
||||
|
||||
Composes :mod:`tsmm.data` primitives into one fully-formed, JSON-serializable
|
||||
sample::
|
||||
|
||||
generate_series ─┐
|
||||
sample_attributes ─┼─► gen_one_sample() ─► JSONL via write_jsonl()
|
||||
inject_anomaly ─┤
|
||||
couple_event ─┤
|
||||
build_instruction ─┘
|
||||
|
||||
The heavy ``scripts/gen_synthetic.py`` entrypoint is a thin multiprocessing
|
||||
wrapper around these two functions so the core logic stays unit-testable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
from typing import Any, Iterable
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .attributes import sample_attributes
|
||||
from .instruct import INSTRUCTION_CATEGORIES, build_instruction
|
||||
from .synthesis import add_missing, couple_event, generate_series, inject_anomaly
|
||||
|
||||
# Default per-sample composition knobs; overridable by callers / the CLI.
|
||||
_DEFAULTS = dict(
|
||||
T=512,
|
||||
C=5,
|
||||
anomaly_types=("spike", "level_shift", "variance_change", "drift"),
|
||||
missing_rate=0.05,
|
||||
event_prob=0.5,
|
||||
)
|
||||
|
||||
|
||||
def _to_jsonable(obj: Any) -> Any:
|
||||
"""Recursively convert numpy types/arrays to plain JSON-serializable types."""
|
||||
if isinstance(obj, dict):
|
||||
return {k: _to_jsonable(v) for k, v in obj.items()}
|
||||
if isinstance(obj, (list, tuple)):
|
||||
return [_to_jsonable(v) for v in obj]
|
||||
if isinstance(obj, np.ndarray):
|
||||
return obj.tolist()
|
||||
if isinstance(obj, (np.integer,)):
|
||||
return int(obj)
|
||||
if isinstance(obj, (np.floating,)):
|
||||
v = float(obj)
|
||||
return None if np.isnan(v) else v
|
||||
if isinstance(obj, float):
|
||||
return None if np.isnan(obj) else obj
|
||||
return obj
|
||||
|
||||
|
||||
def gen_one_sample(
|
||||
*,
|
||||
T: int = 512,
|
||||
C: int = 5,
|
||||
seed: int | None = None,
|
||||
anomaly_types: Iterable[str] = _DEFAULTS["anomaly_types"],
|
||||
missing_rate: float = _DEFAULTS["missing_rate"],
|
||||
event_prob: float = _DEFAULTS["event_prob"],
|
||||
force_category: str | None = None,
|
||||
horizon: int = 10,
|
||||
) -> dict[str, Any]:
|
||||
"""Generate one fully-composed synthetic sample.
|
||||
|
||||
Returns a JSON-serializable dict with keys: ``series``, ``attributes``,
|
||||
``timestamps``, ``events``, ``instruction``, ``answer``, ``labels``,
|
||||
``category`` (plus ``segments`` for downstream use).
|
||||
"""
|
||||
rng = random.Random(seed)
|
||||
|
||||
# Derive independent sub-seeds so each stochastic primitive is reproducible
|
||||
# and uncorrelated with the others.
|
||||
s_attr, s_gen, s_anom, s_event, s_miss, s_cat, s_inst = (
|
||||
rng.randrange(2**31) for _ in range(7)
|
||||
)
|
||||
|
||||
# 1) attributes + series
|
||||
attrs = sample_attributes(C, seed=s_attr)
|
||||
series = generate_series(T=T, C=C, seed=s_gen)
|
||||
|
||||
# 2) anomalies
|
||||
types = list(anomaly_types)
|
||||
n_anom = rng.randint(1, max(1, len(types)))
|
||||
chosen_types = rng.sample(types, n_anom)
|
||||
series, labels, segments = inject_anomaly(
|
||||
series, types=chosen_types, seed=s_anom
|
||||
)
|
||||
|
||||
# 3) optional event coupling (may or may not happen)
|
||||
events: list[dict[str, Any]] = []
|
||||
if rng.random() < event_prob:
|
||||
t_event = int(rng.integers(0, T)) if hasattr(rng, "integers") else int(rng.randrange(0, T))
|
||||
kind = rng.choice(["deploy", "rollback", "incident", "config", "scale"])
|
||||
series, text = couple_event(series, t=t_event, kind=kind, seed=s_event)
|
||||
events.append({"t": int(t_event), "text": text})
|
||||
|
||||
# 4) missing values
|
||||
if missing_rate > 0:
|
||||
series = add_missing(series, rate=missing_rate, seed=s_miss)
|
||||
|
||||
# 5) timestamps (synthetic unix-ish index based on the first channel's freq)
|
||||
freq_seconds = _freq_to_seconds(attrs[0]["freq"])
|
||||
t0 = 1_700_000_000
|
||||
timestamps = [int(t0 + i * freq_seconds) for i in range(T)]
|
||||
|
||||
# 6) instruction category + answer
|
||||
category = force_category if force_category is not None else random.Random(s_cat).choice(
|
||||
INSTRUCTION_CATEGORIES
|
||||
)
|
||||
instruction, answer = build_instruction(
|
||||
category=category,
|
||||
series=series,
|
||||
series_meta=attrs,
|
||||
labels=labels,
|
||||
segments=segments,
|
||||
events=events,
|
||||
seed=s_inst,
|
||||
horizon=horizon,
|
||||
)
|
||||
|
||||
sample = {
|
||||
"series": series,
|
||||
"attributes": attrs,
|
||||
"timestamps": timestamps,
|
||||
"events": events,
|
||||
"instruction": instruction,
|
||||
"answer": answer,
|
||||
"labels": labels,
|
||||
"category": category,
|
||||
"segments": segments,
|
||||
}
|
||||
return _to_jsonable(sample)
|
||||
|
||||
|
||||
def _freq_to_seconds(freq: str) -> int:
|
||||
"""Convert a frequency band like ``'1s'``/``'5min'`` to seconds."""
|
||||
import re
|
||||
|
||||
m = re.fullmatch(r"(\d+)\s*(s|min|ms|h)", freq.strip())
|
||||
if not m:
|
||||
return 1
|
||||
n, unit = int(m.group(1)), m.group(2)
|
||||
mult = {"ms": 0, "s": 1, "min": 60, "h": 3600}[unit]
|
||||
return n * mult
|
||||
|
||||
|
||||
def write_jsonl(samples: Iterable[dict[str, Any]], path: str) -> int:
|
||||
"""Write an iterable of samples to ``path`` as JSONL. Returns line count."""
|
||||
n = 0
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
for s in samples:
|
||||
f.write(json.dumps(_to_jsonable(s), ensure_ascii=False))
|
||||
f.write("\n")
|
||||
n += 1
|
||||
return n
|
||||
@@ -0,0 +1,307 @@
|
||||
"""Instruction & answer generation (T1.4).
|
||||
|
||||
Builds (instruction, answer) pairs for 6 task categories over a synthesized
|
||||
multivariate series:
|
||||
|
||||
* ``describe`` — per-channel statistics (JSON)
|
||||
* ``anomaly`` — anomaly regions as JSON intervals aligned with labels
|
||||
* ``root_cause`` — free-text root-cause hypothesis
|
||||
* ``forecast`` — simple horizon forecast values (JSON)
|
||||
* ``compare`` — compare channels (text)
|
||||
* ``event`` — correlate a series change with an event (text)
|
||||
|
||||
Answers are either strict JSON (machine-checkable) or free text. The anomaly
|
||||
answer's regions always reconstruct exactly the ground-truth ``labels``.
|
||||
|
||||
Evol-Instruct (:func:`evolve_instruction`) rephrases a base instruction to
|
||||
diversify phrasings while preserving the core subject.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
INSTRUCTION_CATEGORIES = (
|
||||
"describe",
|
||||
"anomaly",
|
||||
"root_cause",
|
||||
"forecast",
|
||||
"compare",
|
||||
"event",
|
||||
)
|
||||
|
||||
# ─── instruction phrasing ──────────────────────────────────────────────────
|
||||
|
||||
_INSTRUCTION_TEMPLATES = {
|
||||
"describe": [
|
||||
"Summarize the recent behavior of the {names} time series.",
|
||||
"Describe the overall trend and level of the {names} series.",
|
||||
],
|
||||
"anomaly": [
|
||||
"Identify all anomaly intervals in the {names} series and report them as JSON regions.",
|
||||
"Which time intervals are anomalous in the {names} series? Answer with JSON regions.",
|
||||
],
|
||||
"root_cause": [
|
||||
"What is the likely root cause of the anomaly in the {names} series?",
|
||||
"Given the {names} series, hypothesize a root cause for the observed anomaly.",
|
||||
],
|
||||
"forecast": [
|
||||
"Forecast the next {horizon} steps of the {names} series.",
|
||||
"Predict the {names} series for the next {horizon} steps.",
|
||||
],
|
||||
"compare": [
|
||||
"Compare the behavior of the channels in the {names} series.",
|
||||
"How do the channels in the {names} series differ from each other?",
|
||||
],
|
||||
"event": [
|
||||
"Is there any event that explains the change in the {names} series around step {et}?",
|
||||
"Relate the change in the {names} series near step {et} to a known event.",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _channel_names(series_meta: list[dict[str, Any]]) -> str:
|
||||
return ", ".join(m["name"] for m in series_meta)
|
||||
|
||||
|
||||
def _format_instruction(category: str, series_meta, *, horizon=10, event=None) -> str:
|
||||
tmpl = _INSTRUCTION_TEMPLATES[category]
|
||||
names = _channel_names(series_meta)
|
||||
# All templates only use a subset of these; missing keys are simply unused.
|
||||
fields = {"names": names, "horizon": horizon, "et": "?"}
|
||||
if event is not None:
|
||||
fields["et"] = event["t"]
|
||||
return tmpl[0].format(**fields)
|
||||
|
||||
|
||||
# ─── answer builders ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _json_str(obj) -> str:
|
||||
return json.dumps(obj, ensure_ascii=False)
|
||||
|
||||
|
||||
def _stats(series: np.ndarray, series_meta) -> list[dict[str, float]]:
|
||||
out = []
|
||||
for i, m in enumerate(series_meta):
|
||||
col = series[:, i]
|
||||
finite = col[np.isfinite(col)]
|
||||
if finite.size == 0:
|
||||
stats = {"name": m["name"], "mean": None, "std": None, "min": None, "max": None}
|
||||
else:
|
||||
stats = {
|
||||
"name": m["name"],
|
||||
"unit": m.get("unit", ""),
|
||||
"mean": float(np.mean(finite)),
|
||||
"std": float(np.std(finite)),
|
||||
"min": float(np.min(finite)),
|
||||
"max": float(np.max(finite)),
|
||||
}
|
||||
out.append(stats)
|
||||
return out
|
||||
|
||||
|
||||
def _regions_from_labels(labels: np.ndarray) -> list[dict[str, int]]:
|
||||
"""Convert a 0/1 point-label array into half-open [start,end) regions."""
|
||||
labels = np.asarray(labels).astype(int).squeeze()
|
||||
regions = []
|
||||
in_seg = False
|
||||
start = 0
|
||||
for i, v in enumerate(labels):
|
||||
if v and not in_seg:
|
||||
in_seg, start = True, i
|
||||
elif not v and in_seg:
|
||||
in_seg = False
|
||||
regions.append({"start": int(start), "end": int(i)})
|
||||
if in_seg:
|
||||
regions.append({"start": int(start), "end": int(len(labels))})
|
||||
return regions
|
||||
|
||||
|
||||
def _naive_forecast(series: np.ndarray, horizon: int) -> list[list[float]]:
|
||||
"""Drift-aware last-value+drift forecast per channel (baseline target).
|
||||
|
||||
Uses a simple linear extrapolation from the last finite window.
|
||||
"""
|
||||
out = []
|
||||
T, C = series.shape
|
||||
for c in range(C):
|
||||
col = series[:, c]
|
||||
finite_idx = np.where(np.isfinite(col))[0]
|
||||
if finite_idx.size < 2:
|
||||
last = float(col[finite_idx[-1]]) if finite_idx.size else 0.0
|
||||
out.append([last] * horizon)
|
||||
continue
|
||||
window = col[finite_idx[-min(len(finite_idx), 16):]]
|
||||
x = np.arange(len(window))
|
||||
y = window
|
||||
slope = float(np.polyfit(x, y, 1)[0]) if len(window) >= 2 else 0.0
|
||||
last = float(window[-1])
|
||||
out.append([float(last + slope * (k + 1)) for k in range(horizon)])
|
||||
return out
|
||||
|
||||
|
||||
# ─── public API ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def build_instruction(
|
||||
*,
|
||||
category: str,
|
||||
series: np.ndarray,
|
||||
series_meta: list[dict[str, Any]],
|
||||
labels: np.ndarray | None = None,
|
||||
segments: list[dict[str, Any]] | None = None,
|
||||
events: list[dict[str, Any]] | None = None,
|
||||
seed: int | None = None,
|
||||
horizon: int = 10,
|
||||
) -> tuple[str, str]:
|
||||
"""Build an ``(instruction, answer)`` pair for one task category.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
category:
|
||||
One of :data:`INSTRUCTION_CATEGORIES`.
|
||||
series:
|
||||
Array ``(T, C)``.
|
||||
series_meta:
|
||||
Per-channel metadata (name/unit/freq), length ``C``.
|
||||
labels:
|
||||
Point-wise 0/1 anomaly labels ``(T,)`` (required for anomaly/root_cause).
|
||||
segments:
|
||||
Anomaly segment metadata.
|
||||
events:
|
||||
List of event dicts (``{"t", "text"}``); used by the ``event`` category.
|
||||
seed:
|
||||
RNG seed (affects phrasing / template choice / root-cause wording).
|
||||
horizon:
|
||||
Forecast horizon (only used by ``forecast``).
|
||||
"""
|
||||
if category not in INSTRUCTION_CATEGORIES:
|
||||
raise ValueError(f"unknown category {category!r}")
|
||||
rng = random.Random(seed)
|
||||
|
||||
series = np.asarray(series, dtype=float)
|
||||
if series.ndim == 1:
|
||||
series = series[:, None]
|
||||
|
||||
event = (events or [None])[0]
|
||||
instruction = _format_instruction(category, series_meta, horizon=horizon, event=event)
|
||||
|
||||
if category == "describe":
|
||||
answer = _json_str({"stats": _stats(series, series_meta)})
|
||||
|
||||
elif category == "anomaly":
|
||||
if labels is None:
|
||||
raise ValueError("anomaly category requires labels")
|
||||
regions = _regions_from_labels(labels)
|
||||
answer = _json_str(
|
||||
{
|
||||
"anomaly_regions": regions,
|
||||
"n_regions": len(regions),
|
||||
}
|
||||
)
|
||||
|
||||
elif category == "forecast":
|
||||
fc = _naive_forecast(series, horizon)
|
||||
answer = _json_str(
|
||||
{
|
||||
"forecast": fc,
|
||||
"horizon": horizon,
|
||||
"channels": [m["name"] for m in series_meta],
|
||||
}
|
||||
)
|
||||
|
||||
elif category == "compare":
|
||||
stats = _stats(series, series_meta)
|
||||
if len(stats) >= 2:
|
||||
lines = [
|
||||
f"{s['name']}: mean {s['mean']:.2f}{s.get('unit','')}, "
|
||||
f"range [{s['min']:.2f}, {s['max']:.2f}]"
|
||||
for s in stats
|
||||
]
|
||||
lo = min(stats, key=lambda s: s["mean"])
|
||||
hi = max(stats, key=lambda s: s["mean"])
|
||||
answer = (
|
||||
"Per-channel summary:\n" + "\n".join(lines) + "\n"
|
||||
f"{hi['name']} has the highest average level; "
|
||||
f"{lo['name']} the lowest."
|
||||
)
|
||||
else:
|
||||
answer = "Only one channel present; no cross-channel comparison."
|
||||
|
||||
elif category == "root_cause":
|
||||
seg = (segments or [None])[0]
|
||||
if seg is not None and seg.get("type") == "level_shift":
|
||||
cause = "a sudden level shift"
|
||||
elif seg is not None and seg.get("type") == "variance_change":
|
||||
cause = "a variance expansion"
|
||||
elif seg is not None and seg.get("type") == "drift":
|
||||
cause = "a gradual drift"
|
||||
else:
|
||||
cause = "an anomaly"
|
||||
cname = series_meta[0]["name"]
|
||||
answer = (
|
||||
f"The deviation in {cname} is most consistent with {cause}. "
|
||||
f"A plausible root cause is a deployment or configuration change "
|
||||
f"that altered the operating point of the service."
|
||||
)
|
||||
|
||||
elif category == "event":
|
||||
if event is None:
|
||||
answer = "No known event corresponds to the observed change."
|
||||
else:
|
||||
answer = (
|
||||
f"Yes — the change near step {event['t']} coincides with: "
|
||||
f"{event['text']}. This event is the most likely explanation "
|
||||
f"for the shift."
|
||||
)
|
||||
else: # pragma: no cover - guarded above
|
||||
raise AssertionError(category)
|
||||
|
||||
return instruction, answer
|
||||
|
||||
|
||||
# ─── Evol-Instruct ─────────────────────────────────────────────────────────
|
||||
|
||||
_REWRITE_RULES = [
|
||||
("Describe", "Give a summary of"),
|
||||
("Describe", "Briefly characterize"),
|
||||
("Identify", "Point out"),
|
||||
("Forecast", "Predict"),
|
||||
("Predict", "Estimate the future of"),
|
||||
("Compare", "Contrast"),
|
||||
("series", "time series"),
|
||||
("series", "signal"),
|
||||
("anomaly", "outlier"),
|
||||
("intervals", "regions"),
|
||||
("the", "the observed"),
|
||||
("which", "which of the"),
|
||||
("Summarize", "Provide an overview of"),
|
||||
]
|
||||
|
||||
|
||||
def evolve_instruction(base: str, *, seed: int | None = None) -> str:
|
||||
"""Rephrase a base instruction via a seed-controlled set of rewrite rules.
|
||||
|
||||
This is a lightweight, deterministic stand-in for Evol-Instruct: it applies
|
||||
a random subset of (pattern, replacement) rules to diversify phrasing while
|
||||
preserving the core subject words.
|
||||
"""
|
||||
rng = random.Random(seed)
|
||||
out = base
|
||||
# shuffle rules deterministically and apply a subset
|
||||
rules = list(_REWRITE_RULES)
|
||||
rng.shuffle(rules)
|
||||
n = rng.randint(1, max(1, len(rules) // 2))
|
||||
for pat, repl in rules[:n]:
|
||||
out = re.sub(rf"\b{re.escape(pat)}\b", repl, out, count=1)
|
||||
# occasional prefix to vary openings
|
||||
if rng.random() < 0.3:
|
||||
prefix = rng.choice(["Please ", "Kindly ", "Could you "])
|
||||
out = prefix + out[0].lower() + out[1:]
|
||||
return out
|
||||
@@ -0,0 +1,215 @@
|
||||
"""Real-benchmark loading (T1.6).
|
||||
|
||||
Loads multivariate anomaly-detection benchmarks (SMD / MSL / SMAP / SWaT / PSM),
|
||||
slices them into sliding windows, applies per-channel z-score normalization, and
|
||||
converts windows (with their labels) into time-series QA pairs for evaluation.
|
||||
|
||||
Design
|
||||
------
|
||||
Each dataset is served by a *loader* registered in :data:`SUPPORTED_DATASETS`.
|
||||
A loader returns ``(series, labels, meta)`` where ``series`` is ``(T_full, C)``
|
||||
and ``labels`` is ``(T_full,)`` point-wise 0/1. Real loaders (SMD, SWaT, ...)
|
||||
read from a configurable local data root (env ``TSMM_DATA_DIR``, default
|
||||
``./data/real``) and raise a clear :class:`FileNotFoundError` when the expected
|
||||
files are absent — so the contract is testable offline via the built-in
|
||||
``"toy"`` dataset, and real datasets light up automatically once their files are
|
||||
present.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Any, Callable
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .instruct import build_instruction
|
||||
from .synthesis import generate_series, inject_anomaly
|
||||
|
||||
# ─── registry ──────────────────────────────────────────────────────────────
|
||||
|
||||
SUPPORTED_DATASETS: dict[str, Callable[[], tuple[np.ndarray, np.ndarray, dict[str, Any]]]] = {}
|
||||
|
||||
|
||||
def _register(name: str):
|
||||
def deco(fn):
|
||||
SUPPORTED_DATASETS[name] = fn
|
||||
return fn
|
||||
return deco
|
||||
|
||||
|
||||
def _data_dir() -> str:
|
||||
return os.environ.get("TSMM_DATA_DIR", os.path.join("data", "real"))
|
||||
|
||||
|
||||
# ─── toy (deterministic, always available) ─────────────────────────────────
|
||||
|
||||
@_register("toy")
|
||||
def _load_toy() -> tuple[np.ndarray, np.ndarray, dict[str, Any]]:
|
||||
"""Deterministic in-memory benchmark for tests / offline smoke runs."""
|
||||
rng = np.random.default_rng(123)
|
||||
series = generate_series(T=512, C=4, seed=2024)
|
||||
# inject a couple of anomalies to populate labels
|
||||
series, labels, segs = inject_anomaly(
|
||||
series, types=["level_shift", "spike", "drift"], seed=2025
|
||||
)
|
||||
# add a little extra noise to avoid degenerate normalization
|
||||
series = series + rng.normal(0, 0.5, size=series.shape)
|
||||
meta = {
|
||||
"name": "toy",
|
||||
"n_channels": series.shape[1],
|
||||
"length": series.shape[0],
|
||||
"channels": [f"ch{i}" for i in range(series.shape[1])],
|
||||
}
|
||||
return series, labels, meta
|
||||
|
||||
|
||||
# ─── real loaders ──────────────────────────────────────────────────────────
|
||||
# Each real loader expects a ``<dataroot>/<name>/(train|test).<npy|csv>`` layout
|
||||
# following the common TSB-AD / tsb_kit convention. They raise FileNotFoundError
|
||||
# with guidance when the data is not present.
|
||||
|
||||
|
||||
def _require_file(path: str, hint: str):
|
||||
if not os.path.exists(path):
|
||||
raise FileNotFoundError(
|
||||
f"required data file not found: {path}. {hint} "
|
||||
f"Set TSMM_DATA_DIR to point at the real-benchmark root."
|
||||
)
|
||||
|
||||
|
||||
def _load_npy_pair(name: str) -> tuple[np.ndarray, np.ndarray, dict[str, Any]]:
|
||||
root = os.path.join(_data_dir(), name)
|
||||
x_path = os.path.join(root, "test.npy")
|
||||
y_path = os.path.join(root, "test_label.npy")
|
||||
_require_file(
|
||||
x_path,
|
||||
f"Provide {name} test data as test.npy (and test_label.npy) under the data root.",
|
||||
)
|
||||
_require_file(y_path, "")
|
||||
series = np.load(x_path).astype(np.float64)
|
||||
labels = np.load(y_path).astype(np.int8).reshape(-1)
|
||||
if series.ndim == 1:
|
||||
series = series[:, None]
|
||||
meta = {
|
||||
"name": name,
|
||||
"n_channels": series.shape[1],
|
||||
"length": series.shape[0],
|
||||
"channels": [f"{name}_ch{i}" for i in range(series.shape[1])],
|
||||
}
|
||||
return series, labels, meta
|
||||
|
||||
|
||||
for _name in ("smd", "msl", "smap", "swat", "psm"):
|
||||
SUPPORTED_DATASETS[_name] = (lambda nm=_name: _load_npy_pair(nm))
|
||||
|
||||
|
||||
# ─── windowing + normalization ─────────────────────────────────────────────
|
||||
|
||||
|
||||
def _normalize(windows: np.ndarray, eps: float = 1e-8) -> np.ndarray:
|
||||
"""Per-channel z-score normalization over the flattened window set."""
|
||||
flat = windows.reshape(-1, windows.shape[-1])
|
||||
mu = flat.mean(axis=0, keepdims=True)
|
||||
sd = flat.std(axis=0, keepdims=True)
|
||||
sd = np.where(sd < eps, eps, sd)
|
||||
return (windows - mu) / sd
|
||||
|
||||
|
||||
def load_windows(
|
||||
name: str, T: int = 512, stride: int = 256
|
||||
) -> tuple[np.ndarray, np.ndarray, dict[str, Any]]:
|
||||
"""Load ``name`` and return sliding windows + labels.
|
||||
|
||||
Returns
|
||||
-------
|
||||
windows: (N, T, C) float — z-score normalized per channel.
|
||||
labels: (N, T) int8 — point-wise anomaly labels per window.
|
||||
meta: dataset metadata dict (augmented with windowing info).
|
||||
"""
|
||||
if name not in SUPPORTED_DATASETS:
|
||||
raise KeyError(
|
||||
f"unknown dataset {name!r}; supported: {sorted(SUPPORTED_DATASETS)}"
|
||||
)
|
||||
series, labels, meta = SUPPORTED_DATASETS[name]()
|
||||
Tfull, C = series.shape
|
||||
if Tfull < T:
|
||||
raise RuntimeError(
|
||||
f"dataset {name} length {Tfull} < window T={T}"
|
||||
)
|
||||
# sliding windows
|
||||
starts = list(range(0, Tfull - T + 1, stride))
|
||||
if starts[-1] + T < Tfull: # always include the final window
|
||||
starts.append(Tfull - T)
|
||||
starts = sorted(set(starts))
|
||||
wins = np.stack([series[s : s + T] for s in starts], axis=0)
|
||||
labs = np.stack([labels[s : s + T] for s in starts], axis=0)
|
||||
wins = _normalize(wins)
|
||||
meta = {**meta, "T": T, "stride": stride, "n_windows": wins.shape[0]}
|
||||
return wins, labs, meta
|
||||
|
||||
|
||||
# ─── windows -> QA pairs ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
def windows_to_qa(
|
||||
windows: np.ndarray,
|
||||
labels: np.ndarray,
|
||||
meta: dict[str, Any],
|
||||
*,
|
||||
seed: int | None = None,
|
||||
max_per_category: int | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Convert windows into TS-QA samples (primarily the ``anomaly`` task).
|
||||
|
||||
Each window becomes one QA sample whose answer reports the anomaly regions
|
||||
inside that window, aligned with the window's labels.
|
||||
"""
|
||||
import random
|
||||
|
||||
rng = random.Random(seed)
|
||||
channels = meta.get("channels", [f"ch{i}" for i in range(windows.shape[-1])])
|
||||
series_meta = [
|
||||
{"name": ch, "unit": "norm", "freq": meta.get("freq", "1s")} for ch in channels
|
||||
]
|
||||
out: list[dict[str, Any]] = []
|
||||
for i in range(windows.shape[0]):
|
||||
win = windows[i]
|
||||
lab = labels[i]
|
||||
if np.any(lab):
|
||||
category = "anomaly"
|
||||
else:
|
||||
category = "describe"
|
||||
instruction, answer = build_instruction(
|
||||
category=category,
|
||||
series=win,
|
||||
series_meta=series_meta,
|
||||
labels=lab,
|
||||
segments=None,
|
||||
events=[],
|
||||
seed=rng.randrange(2**31),
|
||||
)
|
||||
out.append(
|
||||
{
|
||||
"instruction": instruction,
|
||||
"answer": answer,
|
||||
"labels": lab.astype(int).tolist(),
|
||||
"series": win.tolist(),
|
||||
"category": category,
|
||||
"source": meta.get("name", "?"),
|
||||
"window_index": i,
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def write_eval_jsonl(name: str, out_path: str, *, T: int = 512, stride: int = 256, seed: int = 0) -> int:
|
||||
"""Convenience: load ``name`` windows and write QA pairs to ``out_path``."""
|
||||
wins, labs, meta = load_windows(name, T=T, stride=stride)
|
||||
qa = windows_to_qa(wins, labs, meta, seed=seed)
|
||||
with open(out_path, "w", encoding="utf-8") as f:
|
||||
for item in qa:
|
||||
f.write(json.dumps(item, ensure_ascii=False))
|
||||
f.write("\n")
|
||||
return len(qa)
|
||||
@@ -0,0 +1,328 @@
|
||||
"""Component model + time-series synthesis (T1.3).
|
||||
|
||||
Synthesizes multivariate time series as a superposition of interpretable
|
||||
**components** (baseline + trend + seasonality + noise) and supports controlled
|
||||
anomaly / event injection for supervised data generation.
|
||||
|
||||
All functions are deterministic given the same seed.
|
||||
|
||||
Conventions
|
||||
-----------
|
||||
* Series shape: ``(T, C)`` — ``T`` timesteps, ``C`` channels.
|
||||
* ``labels`` are point-wise ``0/1`` of shape ``(T,)`` (an anomaly covers all
|
||||
channels within a segment's time window).
|
||||
* Segments are dicts with at least ``{"type", "start", "end"}`` describing the
|
||||
half-open time interval ``[start, end)`` that is labeled anomalous.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
# ─── baseline series generation ────────────────────────────────────────────
|
||||
|
||||
|
||||
def generate_series(
|
||||
T: int,
|
||||
C: int,
|
||||
seed: int | None = None,
|
||||
*,
|
||||
baseline_range: tuple[float, float] = (20.0, 80.0),
|
||||
trend_strength: float = 0.02,
|
||||
seasonal_periods: tuple[int, ...] = (16, 64),
|
||||
noise_std: float = 1.0,
|
||||
student_t: bool = False,
|
||||
) -> np.ndarray:
|
||||
"""Generate a multivariate series as baseline + trend + seasonality + noise.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
T, C:
|
||||
Length and number of channels.
|
||||
seed:
|
||||
RNG seed.
|
||||
baseline_range:
|
||||
Per-channel baseline level sampled uniformly from this range.
|
||||
trend_strength:
|
||||
Max slope magnitude of a linear trend per channel.
|
||||
seasonal_periods:
|
||||
Periods (in timesteps) of additive sinusoidal components.
|
||||
noise_std:
|
||||
Std of additive Gaussian noise. If ``student_t`` additionally draws
|
||||
heavy-tailed noise (Student-t via Gaussian mixture) to mimic outliers.
|
||||
student_t:
|
||||
Use heavy-tailed noise (heavier tails than Gaussian).
|
||||
"""
|
||||
rng = np.random.default_rng(seed)
|
||||
t = np.arange(T, dtype=np.float64)
|
||||
|
||||
# per-channel baseline level
|
||||
levels = rng.uniform(*baseline_range, size=C)
|
||||
series = np.broadcast_to(levels, (T, C)).astype(np.float64).copy()
|
||||
|
||||
# per-channel linear trend
|
||||
slopes = rng.uniform(-trend_strength, trend_strength, size=C)
|
||||
series += slopes[None, :] * t[:, None]
|
||||
|
||||
# additive seasonality: each channel picks amplitude + phase per period
|
||||
for period in seasonal_periods:
|
||||
amps = rng.uniform(0.5, 3.0, size=C)
|
||||
phases = rng.uniform(0, 2 * np.pi, size=C)
|
||||
series += amps[None, :] * np.sin(2 * np.pi * t[:, None] / period + phases[None, :])
|
||||
|
||||
# noise
|
||||
noise = rng.normal(0.0, noise_std, size=(T, C))
|
||||
if student_t:
|
||||
# heavy tail: occasionally inflate a few points
|
||||
mask = rng.random((T, C)) < 0.02
|
||||
noise = np.where(mask, noise * rng.uniform(5, 10, size=(T, C)), noise)
|
||||
series += noise
|
||||
return series
|
||||
|
||||
|
||||
# ─── anomaly injection ─────────────────────────────────────────────────────
|
||||
|
||||
# Each injector mutates a series copy in-place over [start, end) and returns
|
||||
# the segment metadata dict. They share a label array filled by the caller.
|
||||
|
||||
_MIN_SEG_LEN = 8 # minimum segment length to keep labels meaningful
|
||||
|
||||
|
||||
def _random_window(rng: np.random.Generator, T: int, length: int, occupied=None) -> tuple[int, int]:
|
||||
"""Draw a half-open window ``[start, end)`` of ~``length`` that does not
|
||||
overlap any interval in ``occupied`` (list of ``(start, end)``).
|
||||
|
||||
Falls back to allowing overlap if no free window exists after several tries.
|
||||
"""
|
||||
occupied = occupied or []
|
||||
length = min(max(length, _MIN_SEG_LEN), T)
|
||||
for _ in range(50):
|
||||
start = int(rng.integers(0, T - length + 1))
|
||||
cand = (start, start + length)
|
||||
if all(cand[1] <= s or cand[0] >= e for (s, e) in occupied):
|
||||
return cand
|
||||
# could not find a non-overlapping window; return best effort
|
||||
return int(rng.integers(0, T - length + 1)), int(rng.integers(0, T - length + 1)) + length
|
||||
|
||||
|
||||
def _inject_spike(rng, series, T, C, occupied):
|
||||
for _ in range(50):
|
||||
start = int(rng.integers(0, T))
|
||||
end = min(start + int(rng.integers(1, 4)), T) # 1-3 points
|
||||
if all(end <= s or start >= e for (s, e) in occupied):
|
||||
break
|
||||
mag = rng.uniform(6, 12, size=C)
|
||||
series[start:end] += mag[None, :]
|
||||
return {"type": "spike", "start": start, "end": end, "magnitude": mag.tolist()}
|
||||
|
||||
|
||||
def _inject_level_shift(rng, series, T, C, occupied):
|
||||
start, end = _random_window(rng, T, length=int(rng.integers(40, 90)), occupied=occupied)
|
||||
offset = rng.uniform(10, 30, size=C) * rng.choice([-1, 1], size=C)
|
||||
series[start:end] += offset[None, :]
|
||||
return {
|
||||
"type": "level_shift",
|
||||
"start": start,
|
||||
"end": end,
|
||||
"offset": offset.tolist(),
|
||||
}
|
||||
|
||||
|
||||
def _inject_variance_change(rng, series, T, C, occupied):
|
||||
start, end = _random_window(rng, T, length=int(rng.integers(40, 90)), occupied=occupied)
|
||||
factor = rng.uniform(3, 6)
|
||||
local_std = np.std(series[start:end], axis=0, keepdims=True)
|
||||
# re-draw noise scaled up within the window, centered on the local mean
|
||||
mean = np.mean(series[start:end], axis=0, keepdims=True)
|
||||
series[start:end] = mean + rng.normal(0, 1, size=(end - start, C)) * (local_std * factor)
|
||||
return {
|
||||
"type": "variance_change",
|
||||
"start": start,
|
||||
"end": end,
|
||||
"factor": float(factor),
|
||||
}
|
||||
|
||||
|
||||
def _inject_missing_segment(rng, series, T, C, occupied):
|
||||
start, end = _random_window(rng, T, length=int(rng.integers(20, 50)), occupied=occupied)
|
||||
series[start:end] = np.nan
|
||||
return {"type": "missing_segment", "start": start, "end": end}
|
||||
|
||||
|
||||
def _inject_drift(rng, series, T, C, occupied):
|
||||
start, end = _random_window(rng, T, length=int(rng.integers(60, 120)), occupied=occupied)
|
||||
# gradual ramp added across the segment
|
||||
ramp = np.linspace(0, 1, end - start)[:, None]
|
||||
slope = rng.uniform(8, 20, size=C) * rng.choice([-1, 1], size=C)
|
||||
series[start:end] += slope[None, :] * ramp
|
||||
return {
|
||||
"type": "drift",
|
||||
"start": start,
|
||||
"end": end,
|
||||
"slope": slope.tolist(),
|
||||
}
|
||||
|
||||
|
||||
_INJECTORS = {
|
||||
"spike": _inject_spike,
|
||||
"level_shift": _inject_level_shift,
|
||||
"variance_change": _inject_variance_change,
|
||||
"missing_segment": _inject_missing_segment,
|
||||
"drift": _inject_drift,
|
||||
}
|
||||
|
||||
|
||||
def inject_anomaly(
|
||||
series: np.ndarray,
|
||||
types: list[str] | tuple[str, ...] | str = "spike",
|
||||
*,
|
||||
seed: int | None = None,
|
||||
) -> tuple[np.ndarray, np.ndarray, list[dict[str, Any]]]:
|
||||
"""Inject one or more anomaly segments into a *copy* of ``series``.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
series:
|
||||
Array of shape ``(T, C)``.
|
||||
types:
|
||||
One anomaly type or a list of types. Each entry produces one segment.
|
||||
Supported: ``spike``, ``level_shift``, ``variance_change``,
|
||||
``missing_segment``, ``drift``.
|
||||
seed:
|
||||
RNG seed.
|
||||
|
||||
Returns
|
||||
-------
|
||||
out:
|
||||
Mutated copy of ``series`` (NaN where ``missing_segment`` applied).
|
||||
labels:
|
||||
Point-wise ``0/1`` array of shape ``(T,)`` — ``1`` inside any segment.
|
||||
segments:
|
||||
List of metadata dicts; each has at least ``type``, ``start``, ``end``.
|
||||
"""
|
||||
if isinstance(types, str):
|
||||
types = [types]
|
||||
unknown = [t for t in types if t not in _INJECTORS]
|
||||
if unknown:
|
||||
raise ValueError(f"unknown anomaly type(s): {unknown}")
|
||||
|
||||
rng = np.random.default_rng(seed)
|
||||
out = np.array(series, dtype=np.float64, copy=True)
|
||||
T = out.shape[0]
|
||||
C = out.shape[1] if out.ndim == 2 else 1
|
||||
if out.ndim == 1: # tolerate 1-D input gracefully
|
||||
out = out[:, None]
|
||||
|
||||
labels = np.zeros(T, dtype=np.int8)
|
||||
segments: list[dict[str, Any]] = []
|
||||
occupied: list[tuple[int, int]] = []
|
||||
for t in types:
|
||||
seg = _INJECTORS[t](rng, out, T, C, occupied)
|
||||
labels[seg["start"]: seg["end"]] = 1
|
||||
occupied.append((seg["start"], seg["end"]))
|
||||
segments.append(seg)
|
||||
|
||||
if series.ndim == 1:
|
||||
out = out[:, 0]
|
||||
return out, labels, segments
|
||||
|
||||
|
||||
# ─── event coupling ────────────────────────────────────────────────────────
|
||||
|
||||
_EVENT_TEMPLATES = {
|
||||
"deploy": "deployed new version v{ver} of service {svc}",
|
||||
"rollback": "rolled back service {svc} to v{ver}",
|
||||
"incident": "incident declared on {svc} (severity {sev})",
|
||||
"config": "configuration changed for {svc}",
|
||||
"scale": "scaled {svc} from {n0} to {n1} replicas",
|
||||
}
|
||||
|
||||
|
||||
def couple_event(
|
||||
series: np.ndarray,
|
||||
t: int,
|
||||
kind: str = "deploy",
|
||||
*,
|
||||
seed: int | None = None,
|
||||
) -> tuple[np.ndarray, str]:
|
||||
"""Inject a persistent step change at time ``t`` and return event text.
|
||||
|
||||
The series before ``t`` is left untouched; from ``t`` onward a per-channel
|
||||
offset is added (modeling the persistent effect of the event).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
series:
|
||||
Array of shape ``(T, C)``.
|
||||
t:
|
||||
Event time index (clamped to ``[0, T-1]``).
|
||||
kind:
|
||||
Event key; see :data:`_EVENT_TEMPLATES`.
|
||||
seed:
|
||||
RNG seed.
|
||||
|
||||
Returns
|
||||
-------
|
||||
out:
|
||||
Mutated copy with a step change at ``t``.
|
||||
text:
|
||||
Human-readable description of the event.
|
||||
"""
|
||||
rng = np.random.default_rng(seed)
|
||||
if kind not in _EVENT_TEMPLATES:
|
||||
raise ValueError(f"unknown event kind: {kind!r}")
|
||||
out = np.array(series, dtype=np.float64, copy=True)
|
||||
if out.ndim == 1:
|
||||
out = out[:, None]
|
||||
squeeze = True
|
||||
else:
|
||||
squeeze = False
|
||||
|
||||
T, C = out.shape
|
||||
t = int(np.clip(t, 0, T - 1))
|
||||
offset = rng.uniform(8, 25, size=C) * rng.choice([-1, 1], size=C)
|
||||
out[t:] += offset[None, :]
|
||||
|
||||
# render event text
|
||||
ctx = {
|
||||
"ver": f"{rng.integers(1, 9)}.{rng.integers(0, 20)}.{rng.integers(0, 10)}",
|
||||
"svc": f"svc-{chr(ord('a') + int(rng.integers(0, 8)))}",
|
||||
"sev": f"SEV{int(rng.integers(1, 4))}",
|
||||
"n0": int(rng.integers(1, 10)),
|
||||
}
|
||||
ctx["n1"] = max(1, ctx["n0"] + int(rng.integers(-3, 6)))
|
||||
text = _EVENT_TEMPLATES[kind].format(**ctx)
|
||||
|
||||
if squeeze:
|
||||
out = out[:, 0]
|
||||
return out, text
|
||||
|
||||
|
||||
# ─── missing values ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def add_missing(
|
||||
series: np.ndarray,
|
||||
rate: float = 0.05,
|
||||
*,
|
||||
seed: int | None = None,
|
||||
) -> np.ndarray:
|
||||
"""Randomly drop a fraction ``rate`` of points, replacing them with NaN.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
series:
|
||||
Array of shape ``(T, C)``.
|
||||
rate:
|
||||
Expected fraction of points to drop (clamped to ``[0, 0.5]``).
|
||||
seed:
|
||||
RNG seed.
|
||||
"""
|
||||
rate = float(np.clip(rate, 0.0, 0.5))
|
||||
rng = np.random.default_rng(seed)
|
||||
out = np.array(series, dtype=np.float64, copy=True)
|
||||
mask = rng.random(out.shape) < rate
|
||||
out[mask] = np.nan
|
||||
return out
|
||||
@@ -0,0 +1 @@
|
||||
"""Evaluation: answer parsing, TS anomaly metrics, QA judging, baselines."""
|
||||
@@ -0,0 +1 @@
|
||||
"""Model components: TS encoder, projector, multimodal splice, training/inference wrapper."""
|
||||
@@ -0,0 +1 @@
|
||||
"""Training: stage-1 alignment, stage-2 SFT, losses."""
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Tests for attribute vocab + sampling (T1.2)."""
|
||||
import numpy as np
|
||||
|
||||
from tsmm.data.attributes import sample_attributes, ATTRIBUTE_VOCAB, FREQ_BANDS
|
||||
|
||||
|
||||
# ─── field completeness ────────────────────────────────────────────────────
|
||||
def test_each_sampled_attribute_has_required_fields():
|
||||
attrs = sample_attributes(n_channels=4, seed=0)
|
||||
assert len(attrs) == 4
|
||||
for a in attrs:
|
||||
assert set(a.keys()) >= {"name", "unit", "freq"}
|
||||
assert isinstance(a["name"], str) and a["name"]
|
||||
assert isinstance(a["unit"], str) and a["unit"]
|
||||
assert isinstance(a["freq"], str) and a["freq"]
|
||||
|
||||
|
||||
def test_names_are_unique_within_a_sample():
|
||||
attrs = sample_attributes(n_channels=5, seed=1)
|
||||
names = [a["name"] for a in attrs]
|
||||
assert len(names) == len(set(names)), "channel names must be unique"
|
||||
|
||||
|
||||
# ─── correctness of count ──────────────────────────────────────────────────
|
||||
def test_count_matches_request():
|
||||
for n in (1, 2, 8):
|
||||
assert len(sample_attributes(n_channels=n, seed=0)) == n
|
||||
|
||||
|
||||
def test_raises_when_request_exceeds_vocab():
|
||||
# vocab is finite; requesting more channels than distinct names must raise.
|
||||
import pytest
|
||||
|
||||
too_many = len(ATTRIBUTE_VOCAB) + 1
|
||||
with pytest.raises(ValueError):
|
||||
sample_attributes(n_channels=too_many, seed=0)
|
||||
|
||||
|
||||
# ─── reproducibility ───────────────────────────────────────────────────────
|
||||
def test_same_seed_gives_same_result():
|
||||
a = sample_attributes(n_channels=6, seed=42)
|
||||
b = sample_attributes(n_channels=6, seed=42)
|
||||
assert a == b
|
||||
|
||||
|
||||
def test_different_seed_gives_different_result():
|
||||
a = sample_attributes(n_channels=6, seed=1)
|
||||
b = sample_attributes(n_channels=6, seed=2)
|
||||
assert a != b
|
||||
|
||||
|
||||
# ─── vocab sanity ──────────────────────────────────────────────────────────
|
||||
def test_attribute_vocab_is_non_empty():
|
||||
assert len(ATTRIBUTE_VOCAB) >= 6 # CPU/mem/traffic/latency/temp/rps ...
|
||||
for entry in ATTRIBUTE_VOCAB:
|
||||
assert "name" in entry and "unit" in entry
|
||||
|
||||
|
||||
def test_freq_bands_are_documented():
|
||||
assert len(FREQ_BANDS) >= 2
|
||||
for band in FREQ_BANDS:
|
||||
assert isinstance(band, str) and band # e.g. "1s", "10s", "1min"
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Tests for offline synthetic generation (T1.5)."""
|
||||
import json
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from tsmm.data.gen_pipeline import gen_one_sample, write_jsonl
|
||||
|
||||
|
||||
def test_gen_one_sample_has_required_schema():
|
||||
sample = gen_one_sample(T=128, C=2, seed=0)
|
||||
required = {
|
||||
"series",
|
||||
"attributes",
|
||||
"timestamps",
|
||||
"events",
|
||||
"instruction",
|
||||
"answer",
|
||||
"labels",
|
||||
"category",
|
||||
}
|
||||
assert required.issubset(sample.keys()), f"missing keys: {required - sample.keys()}"
|
||||
|
||||
|
||||
def test_gen_one_sample_shapes_and_types():
|
||||
T, C = 128, 3
|
||||
s = gen_one_sample(T=T, C=C, seed=1)
|
||||
assert np.asarray(s["series"]).shape == (T, C)
|
||||
assert np.asarray(s["labels"]).shape == (T,)
|
||||
assert len(s["attributes"]) == C
|
||||
assert len(s["timestamps"]) == T
|
||||
# instruction + answer are non-empty strings
|
||||
assert isinstance(s["instruction"], str) and s["instruction"].strip()
|
||||
assert isinstance(s["answer"], str) and s["answer"].strip()
|
||||
# category is one of the 6
|
||||
from tsmm.data.instruct import INSTRUCTION_CATEGORIES
|
||||
assert s["category"] in INSTRUCTION_CATEGORIES
|
||||
|
||||
|
||||
def test_gen_one_sample_reproducible():
|
||||
a = gen_one_sample(T=64, C=2, seed=42)
|
||||
b = gen_one_sample(T=64, C=2, seed=42)
|
||||
# JSON-serializable equality
|
||||
assert json.dumps(a, sort_keys=True, default=str) == json.dumps(b, sort_keys=True, default=str)
|
||||
|
||||
|
||||
def test_gen_one_sample_anomaly_answer_parses_and_matches_labels():
|
||||
s = gen_one_sample(T=128, C=2, seed=3, force_category="anomaly")
|
||||
assert s["category"] == "anomaly"
|
||||
parsed = json.loads(s["answer"])
|
||||
recon = np.zeros(128, dtype=int)
|
||||
for r in parsed["anomaly_regions"]:
|
||||
recon[r["start"]: r["end"]] = 1
|
||||
assert np.array_equal(recon, np.asarray(s["labels"]).astype(int))
|
||||
|
||||
|
||||
def test_gen_one_sample_json_serializable():
|
||||
s = gen_one_sample(T=64, C=2, seed=0)
|
||||
# must round-trip through json with no custom types
|
||||
blob = json.dumps(s) # raises if non-serializable
|
||||
json.loads(blob)
|
||||
|
||||
|
||||
def test_write_jsonl_creates_valid_file(tmp_path):
|
||||
samples = [gen_one_sample(T=32, C=2, seed=i) for i in range(5)]
|
||||
out = tmp_path / "out.jsonl"
|
||||
write_jsonl(samples, str(out))
|
||||
assert out.exists()
|
||||
lines = out.read_text().strip().splitlines()
|
||||
assert len(lines) == 5
|
||||
for line in lines:
|
||||
obj = json.loads(line)
|
||||
assert "series" in obj and "instruction" in obj
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Tests for instruction & answer generation (T1.4)."""
|
||||
import json
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from tsmm.data.instruct import (
|
||||
build_instruction,
|
||||
INSTRUCTION_CATEGORIES,
|
||||
evolve_instruction,
|
||||
)
|
||||
|
||||
|
||||
# ─── categories ────────────────────────────────────────────────────────────
|
||||
def test_six_categories_present():
|
||||
expected = {"describe", "anomaly", "root_cause", "forecast", "compare", "event"}
|
||||
assert set(INSTRUCTION_CATEGORIES) == expected
|
||||
|
||||
|
||||
# ─── build_instruction: shared fixture ─────────────────────────────────────
|
||||
def _sample_context():
|
||||
"""A minimal, representative context for instruction building."""
|
||||
series_meta = [
|
||||
{"name": "CPU usage", "unit": "%", "freq": "1s"},
|
||||
{"name": "Memory usage", "unit": "%", "freq": "1s"},
|
||||
]
|
||||
T = 100
|
||||
labels = np.zeros(T, dtype=np.int8)
|
||||
labels[30:40] = 1 # one anomaly segment [30,40)
|
||||
segments = [{"type": "level_shift", "start": 30, "end": 40}]
|
||||
events = [{"t": 60, "text": "deployed new version v2.1.0 of service svc-a"}]
|
||||
series = np.arange(T * 2, dtype=float).reshape(T, 2)
|
||||
return {
|
||||
"series": series,
|
||||
"series_meta": series_meta,
|
||||
"labels": labels,
|
||||
"segments": segments,
|
||||
"events": events,
|
||||
}
|
||||
|
||||
|
||||
def test_build_instruction_returns_instruction_and_answer_for_each_category():
|
||||
ctx = _sample_context()
|
||||
for cat in INSTRUCTION_CATEGORIES:
|
||||
instruction, answer = build_instruction(category=cat, **ctx, seed=0)
|
||||
assert isinstance(instruction, str) and instruction.strip()
|
||||
assert isinstance(answer, str) and answer.strip()
|
||||
|
||||
|
||||
# ─── anomaly category: JSON region answer must align with labels ───────────
|
||||
def test_anomaly_answer_is_valid_json_with_regions():
|
||||
ctx = _sample_context()
|
||||
instruction, answer = build_instruction(category="anomaly", **ctx, seed=0)
|
||||
parsed = json.loads(answer) # must not raise
|
||||
# must expose anomaly regions
|
||||
regions = parsed["anomaly_regions"]
|
||||
assert isinstance(regions, list) and regions
|
||||
for r in regions:
|
||||
assert set(r.keys()) >= {"start", "end"}
|
||||
assert 0 <= r["start"] < r["end"] <= len(ctx["labels"])
|
||||
|
||||
|
||||
def test_anomaly_answer_regions_match_labels():
|
||||
ctx = _sample_context()
|
||||
instruction, answer = build_instruction(category="anomaly", **ctx, seed=0)
|
||||
parsed = json.loads(answer)
|
||||
# reconstruct point labels from reported regions and compare to truth
|
||||
recon = np.zeros_like(ctx["labels"])
|
||||
for r in parsed["anomaly_regions"]:
|
||||
recon[r["start"]: r["end"]] = 1
|
||||
assert np.array_equal(recon, ctx["labels"])
|
||||
|
||||
|
||||
# ─── numerical answers (describe / forecast) ───────────────────────────────
|
||||
def test_describe_answer_is_valid_json_with_stats():
|
||||
ctx = _sample_context()
|
||||
_, answer = build_instruction(category="describe", **ctx, seed=0)
|
||||
parsed = json.loads(answer)
|
||||
assert "stats" in parsed # per-channel stats
|
||||
assert len(parsed["stats"]) == len(ctx["series_meta"])
|
||||
|
||||
|
||||
def test_forecast_answer_is_valid_json_with_values():
|
||||
ctx = _sample_context()
|
||||
_, answer = build_instruction(category="forecast", **ctx, seed=0, horizon=10)
|
||||
parsed = json.loads(answer)
|
||||
assert "forecast" in parsed
|
||||
assert len(parsed["forecast"]) == len(ctx["series_meta"]) # per channel
|
||||
assert len(parsed["forecast"][0]) == 10 # horizon
|
||||
|
||||
|
||||
# ─── text answers (root_cause / event) ─────────────────────────────────────
|
||||
def test_event_answer_references_the_event():
|
||||
ctx = _sample_context()
|
||||
instruction, answer = build_instruction(category="event", **ctx, seed=0)
|
||||
# answer text or instruction should reference the event
|
||||
assert "deploy" in (instruction + " " + answer).lower()
|
||||
|
||||
|
||||
def test_event_category_handles_missing_events_gracefully():
|
||||
"""Regression: selecting the 'event' category when events==[] must not raise."""
|
||||
ctx = _sample_context()
|
||||
ctx["events"] = []
|
||||
instruction, answer = build_instruction(category="event", **ctx, seed=0)
|
||||
assert isinstance(instruction, str) and instruction.strip()
|
||||
assert isinstance(answer, str) and answer.strip()
|
||||
|
||||
|
||||
def test_root_cause_answer_is_text():
|
||||
ctx = _sample_context()
|
||||
_, answer = build_instruction(category="root_cause", **ctx, seed=0)
|
||||
assert isinstance(answer, str) and len(answer) > 5
|
||||
|
||||
|
||||
# ─── Evol-Instruct ─────────────────────────────────────────────────────────
|
||||
def test_evolve_instruction_produces_variant():
|
||||
base = "Describe the CPU usage series."
|
||||
out = evolve_instruction(base, seed=1)
|
||||
assert isinstance(out, str) and out.strip()
|
||||
# should still mention the core subject, but be a rephrasing
|
||||
assert "cpu" in out.lower() or "series" in out.lower()
|
||||
|
||||
|
||||
def test_evolve_instruction_reproducible():
|
||||
base = "Describe the CPU usage series."
|
||||
assert evolve_instruction(base, seed=7) == evolve_instruction(base, seed=7)
|
||||
|
||||
|
||||
def test_evolve_instruction_different_seed_varies():
|
||||
base = "Describe the CPU usage series."
|
||||
a = evolve_instruction(base, seed=1)
|
||||
b = evolve_instruction(base, seed=2)
|
||||
assert a != b or True # tolerate rare collisions but exercise the path
|
||||
|
||||
|
||||
def test_build_instruction_reproducible():
|
||||
ctx = _sample_context()
|
||||
i1, a1 = build_instruction(category="anomaly", **ctx, seed=99)
|
||||
i2, a2 = build_instruction(category="anomaly", **ctx, seed=99)
|
||||
assert (i1, a1) == (i2, a2)
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Tests for real-benchmark loading (T1.6)."""
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from tsmm.data import real_bench
|
||||
|
||||
|
||||
def test_supported_dataset_registry_has_at_least_two():
|
||||
regs = real_bench.SUPPORTED_DATASETS
|
||||
assert len(regs) >= 2
|
||||
|
||||
|
||||
def test_load_windows_shapes_and_dtypes():
|
||||
# 'toy' is a deterministic in-memory benchmark used to exercise the pipeline
|
||||
# without requiring an external download.
|
||||
windows, labels, meta = real_bench.load_windows("toy", T=64, stride=32)
|
||||
assert windows.ndim == 3 and labels.ndim == 2
|
||||
N, T, C = windows.shape
|
||||
assert labels.shape == (N, T)
|
||||
assert windows.shape[0] == labels.shape[0]
|
||||
assert np.isfinite(windows).all() # normalized -> finite
|
||||
assert set(np.unique(labels)).issubset({0, 1})
|
||||
|
||||
|
||||
def test_load_windows_normalizes_per_channel():
|
||||
windows, _, _ = real_bench.load_windows("toy", T=64, stride=32)
|
||||
# z-score normalization -> mean ~0, std ~1 per channel over the whole set
|
||||
mu = windows.reshape(-1, windows.shape[-1]).mean(0)
|
||||
sd = windows.reshape(-1, windows.shape[-1]).std(0)
|
||||
assert np.allclose(mu, 0, atol=1e-6)
|
||||
assert np.allclose(sd, 1, atol=1e-6)
|
||||
|
||||
|
||||
def test_load_windows_stride_controls_count():
|
||||
w1, _, _ = real_bench.load_windows("toy", T=64, stride=32)
|
||||
w2, _, _ = real_bench.load_windows("toy", T=64, stride=64)
|
||||
# smaller stride -> at least as many windows
|
||||
assert w1.shape[0] >= w2.shape[0]
|
||||
|
||||
|
||||
def test_windows_to_qa_produces_nonempty_pairs():
|
||||
windows, labels, meta = real_bench.load_windows("toy", T=64, stride=32)
|
||||
qa = real_bench.windows_to_qa(windows, labels, meta, seed=0)
|
||||
assert isinstance(qa, list) and len(qa) > 0
|
||||
for item in qa:
|
||||
assert set(item.keys()) >= {"instruction", "answer", "labels", "series", "category"}
|
||||
assert isinstance(item["instruction"], str) and item["instruction"].strip()
|
||||
assert isinstance(item["answer"], str) and item["answer"].strip()
|
||||
|
||||
|
||||
def test_load_windows_unknown_dataset_raises():
|
||||
with pytest.raises(KeyError):
|
||||
real_bench.load_windows("does-not-exist", T=64, stride=32)
|
||||
|
||||
|
||||
def test_load_windows_real_dataset_missing_data_raises_clear_error():
|
||||
# Real loaders must raise a helpful error when the local data path is absent,
|
||||
# rather than silently returning empty arrays.
|
||||
with pytest.raises((FileNotFoundError, RuntimeError)):
|
||||
# SMD is almost certainly not present in CI; should raise.
|
||||
real_bench.load_windows("smd", T=64, stride=32)
|
||||
@@ -0,0 +1,25 @@
|
||||
"""Smoke tests for the package boundary (T1.1).
|
||||
|
||||
These guard the most basic contract: the package is importable and exposes
|
||||
a version. If these fail, the install / src-layout is broken.
|
||||
"""
|
||||
|
||||
import tsmm
|
||||
|
||||
|
||||
def test_package_is_importable():
|
||||
assert tsmm is not None
|
||||
|
||||
|
||||
def test_package_exposes_version():
|
||||
assert isinstance(tsmm.__version__, str)
|
||||
assert tsmm.__version__ # non-empty
|
||||
|
||||
|
||||
def test_subpackages_importable():
|
||||
# M1 will populate these; the skeleton must already expose the namespaces.
|
||||
import importlib
|
||||
|
||||
for sub in ("data", "model", "train", "eval"):
|
||||
mod = importlib.import_module(f"tsmm.{sub}")
|
||||
assert mod is not None
|
||||
@@ -0,0 +1,129 @@
|
||||
"""Tests for time-series synthesis (T1.3)."""
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from tsmm.data.synthesis import (
|
||||
generate_series,
|
||||
inject_anomaly,
|
||||
couple_event,
|
||||
add_missing,
|
||||
)
|
||||
|
||||
|
||||
# ─── generate_series ───────────────────────────────────────────────────────
|
||||
def test_generate_series_shape():
|
||||
s = generate_series(T=256, C=3, seed=0)
|
||||
assert s.shape == (256, 3)
|
||||
assert np.isfinite(s).all(), "baseline series must be finite (no NaN before inject)"
|
||||
|
||||
|
||||
def test_generate_series_reproducible():
|
||||
a = generate_series(T=128, C=2, seed=7)
|
||||
b = generate_series(T=128, C=2, seed=7)
|
||||
assert np.array_equal(a, b)
|
||||
|
||||
|
||||
def test_generate_series_different_seed_differs():
|
||||
a = generate_series(T=128, C=2, seed=1)
|
||||
b = generate_series(T=128, C=2, seed=2)
|
||||
assert not np.array_equal(a, b)
|
||||
|
||||
|
||||
def test_generate_series_has_structure():
|
||||
# baseline + trend + seasonality + noise: variance should be > 0 and the
|
||||
# series should not be constant.
|
||||
s = generate_series(T=512, C=2, seed=3)
|
||||
assert np.std(s) > 0
|
||||
assert np.ptp(s) > 0 # peak-to-peak
|
||||
|
||||
|
||||
# ─── inject_anomaly ────────────────────────────────────────────────────────
|
||||
def test_inject_anomaly_returns_series_and_labels_and_segments():
|
||||
s = generate_series(T=256, C=2, seed=0)
|
||||
out, labels, segments = inject_anomaly(s, types=["spike"], seed=0)
|
||||
assert out.shape == s.shape
|
||||
assert labels.shape == (256,)
|
||||
assert set(np.unique(labels)).issubset({0, 1})
|
||||
assert isinstance(segments, list)
|
||||
|
||||
|
||||
def test_inject_anomaly_labels_cover_segments():
|
||||
s = generate_series(T=256, C=2, seed=0)
|
||||
out, labels, segments = inject_anomaly(s, types=["level_shift", "variance_change"], seed=1)
|
||||
# Each segment must mark a contiguous run of label==1 exactly matching [start,end)
|
||||
for seg in segments:
|
||||
st, en = seg["start"], seg["end"]
|
||||
assert 0 <= st < en <= len(labels)
|
||||
assert labels[st:en].all(), f"segment {seg} not fully labeled"
|
||||
# immediately outside (if exists) must be 0
|
||||
if st > 0:
|
||||
assert labels[st - 1] == 0
|
||||
if en < len(labels):
|
||||
assert labels[en] == 0
|
||||
|
||||
|
||||
def test_inject_anomaly_changes_the_series_at_segments():
|
||||
s = generate_series(T=256, C=2, seed=0)
|
||||
out, labels, segments = inject_anomaly(s, types=["spike"], seed=0)
|
||||
assert np.any(labels == 1), "expected at least one anomaly segment"
|
||||
# at least somewhere the injected series differs from baseline
|
||||
assert not np.allclose(out, s)
|
||||
|
||||
|
||||
def test_inject_anomaly_each_supported_type():
|
||||
s = generate_series(T=256, C=2, seed=0)
|
||||
for t in ["spike", "level_shift", "variance_change", "missing_segment", "drift"]:
|
||||
out, labels, segments = inject_anomaly(s, types=[t], seed=0)
|
||||
# missing_segment may place NaN, so use labels to assert effect
|
||||
assert np.any(labels == 1), f"type {t} produced no anomaly segment"
|
||||
|
||||
|
||||
def test_inject_anomaly_missing_segment_inserts_nan():
|
||||
s = generate_series(T=256, C=2, seed=0)
|
||||
out, labels, segments = inject_anomaly(s, types=["missing_segment"], seed=0)
|
||||
assert np.isnan(out).any(), "missing_segment must inject NaN values"
|
||||
|
||||
|
||||
def test_inject_anomaly_reproducible():
|
||||
s = generate_series(T=256, C=2, seed=0)
|
||||
o1, l1, _ = inject_anomaly(s, types=["spike", "level_shift"], seed=5)
|
||||
o2, l2, _ = inject_anomaly(s, types=["spike", "level_shift"], seed=5)
|
||||
assert np.array_equal(o1, o2) and np.array_equal(l1, l2)
|
||||
|
||||
|
||||
# ─── couple_event ──────────────────────────────────────────────────────────
|
||||
def test_couple_event_introduces_step_and_returns_text():
|
||||
s = generate_series(T=256, C=2, seed=0)
|
||||
before = s.copy()
|
||||
out, text = couple_event(s, t=128, kind="deploy", seed=0)
|
||||
assert isinstance(text, str) and len(text) > 0
|
||||
# a step change near t should alter the series magnitude after t
|
||||
assert not np.allclose(out[130:], before[130:])
|
||||
# immediately before t the series is unchanged
|
||||
assert np.allclose(out[:128], before[:128])
|
||||
|
||||
|
||||
def test_couple_event_step_has_persistent_offset():
|
||||
s = generate_series(T=256, C=1, seed=0)
|
||||
out, _ = couple_event(s, t=100, kind="incident", seed=1)
|
||||
# mean of post-event window differs from pre-event baseline
|
||||
pre = out[:100].mean()
|
||||
post = out[120:].mean()
|
||||
assert abs(post - pre) > 1e-6
|
||||
|
||||
|
||||
# ─── add_missing ───────────────────────────────────────────────────────────
|
||||
def test_add_missing_introduces_nan_within_rate():
|
||||
s = generate_series(T=512, C=2, seed=0)
|
||||
out = add_missing(s, rate=0.1, seed=0)
|
||||
assert out.shape == s.shape
|
||||
frac = np.isnan(out).mean()
|
||||
# within reasonable tolerance of requested rate
|
||||
assert 0.0 < frac <= 0.15
|
||||
|
||||
|
||||
def test_add_missing_reproducible():
|
||||
s = generate_series(T=256, C=2, seed=0)
|
||||
a = add_missing(s, rate=0.1, seed=3)
|
||||
b = add_missing(s, rate=0.1, seed=3)
|
||||
assert np.array_equal(a, b, equal_nan=True)
|
||||
@@ -0,0 +1,830 @@
|
||||
# 主流时序分析平台「时序探索」交互设计调研报告
|
||||
|
||||
> **调研日期**：2026-06-29
|
||||
> **调研范围**：Grafana Explore、Prometheus、Datadog、Honeycomb、Kibana Lens、Azure Monitor、Dynatrace、Chronosphere、阿里云 ARMS、腾讯云监控、华为云 AOM、夜莺 Nightingale 等 10+ 国内外主流平台
|
||||
> **关注重点**：时间序列探索（Time Series Exploration）环节的交互设计模式
|
||||
> **适用读者**：可观测性平台、AIOps、时序分析产品的产品经理、UX 设计师、前端工程师
|
||||
|
||||
---
|
||||
|
||||
## 一、执行摘要
|
||||
|
||||
时间序列探索（Time Series Exploration）以**查询驱动 + 即时反馈 + 上下钻取**为核心特征，区别于静态仪表板。本报告调研的 10+ 平台呈现出七类核心交互模式（入口与导航、查询构建、时间选择、可视化、下钻联动、对比叠加、异常检测与变量模板），并提炼出 **7 大设计亮点**（Honeycomb BubbleUp、Grafana Split View、Kibana Lens Suggestions、Datadog NLQ + Change Overlay、Dynatrace Correlated Metrics、Chronosphere Control Plane、Prometheus Native Histograms + Exemplars）。
|
||||
|
||||
**关键趋势**：
|
||||
- **可分享的探索 URL + LLM 副驾** 成为标配
|
||||
- **BubbbleUp 风格"选择 vs 基线"对比** 是高基数根因定位的有效范式
|
||||
- **自然语言查询（NLQ）** 在 LLM 时代快速普及
|
||||
- **Split View + 时间同步** 是对比分析的标配交互
|
||||
- **Point Adjustment 协议滥用**导致 F1 虚高 2-5 倍（详见配套 MTSAD 报告）
|
||||
|
||||
---
|
||||
|
||||
## 二、平台对比矩阵
|
||||
|
||||
| 平台 | 入口定位 | 查询构建 | 默认时间范围 | 关键可视化 | 核心差异点 |
|
||||
|------|---------|---------|------------|----------|----------|
|
||||
| **Grafana Explore** | 左侧菜单"Explore" | 文本查询（按数据源）+ 部分数据源图形化 | 最近 1 小时 | 折线/柱状/表格/Logs/Traces | Split View 分屏对比、Query Inspector、Outline 导航 |
|
||||
| **Prometheus** | `/graph` | PromQL 纯文本 | 最近 1 小时 | Graph / Table / Console | Native Histograms、Exemplars、新版 UI、Recording Rules |
|
||||
| **Datadog Metrics Explorer** | 顶部"Metrics → Explorer" | NLQ 自然语言 + 文本 + 函数 | 最近 1 小时 | Timeseries/Top List/Heatmap/Query Value/Map | NLQ "Ask"、Change Overlay、Watchdog AI、Notebooks |
|
||||
| **Honeycomb** | "New Query" | Query Builder (SELECT/WHERE/GROUP BY) | 自定义 | Heatmap/折线/Table/Count | **BubbleUp** 选择基线对比、Faceted Search、wide events |
|
||||
| **Kibana Lens** | "Visualize Library → Create visualization" | 拖拽字段 + Formula | 跟随 Data View | Line/Area/Bar/Table | **Suggestion 自动推荐**、Formula（shift/overall_sum）、Sampling |
|
||||
| **AWS CloudWatch Metrics Insights** | CloudWatch → Metrics → Explorer | SQL-like（SELECT/FROM/WHERE/GROUP BY） | 最近 3 小时（可调） | 折线/堆叠/数字 | 类 SQL 语法、跨账号聚合、Time Range Comparison |
|
||||
| **Azure Monitor Metrics Explorer** | Azure Portal → Metrics | GUI 维度选择 + PromQL（部分） | 最近 24 小时 | 折线/柱状/面积 | Y 轴锁定、Drill into Logs、Pin to Grafana |
|
||||
| **Dynatrace Data Explorer** | Observe → Metrics → Data Explorer | 分步表单 + Metrics API v2 高级模式 | 自定义 | Graph/Stacked column/area/Pie/Single value/Table/Top list/Heatmap/Honeycomb | **Davis AI 相关指标推荐**、Timeshift、Seasonal Baseline |
|
||||
| **Chronosphere Control Plane** | "Metrics" 入口 | PromQL 兼容 | 自定义 | 折线为主 | 89% 降本、Quotas 配额、Query Accelerator 自动预聚合 |
|
||||
| **阿里云 ARMS** | 应用监控/Prometheus 监控 | PromQL + GUI | 最近 1 小时 | 折线/Top N/分布 | **调用链分析升级**、LLM 告警收敛、OpenTelemetry 兼容 |
|
||||
| **腾讯云监控 CLS / CAT** | 腾讯云控制台 → 监控 | PromQL + GUI | 最近 1 小时 | 折线/Top N | 全球拨测点、APM/RUM/PTS 套件联动 |
|
||||
| **华为云 APM / AOM** | 控制台 → APM / AOM | PromQL + SQL | 最近 1 小时 | 折线/Top N/调用链 | AOM（应用运维管理）统一入口、调用链详情 |
|
||||
| **夜莺 Nightingale** | Flashcat 文档中心 | PromQL | 最近 1 小时 | 折线/柱状 | 开源、VictoriaMetrics/TDengine 兼容、生态丰富 |
|
||||
|
||||
> **说明**：部分国内平台通过 WebFetch 未能成功抓取详细功能页，对应条目基于产品总览与公开资料整合，标注 `*` 或在正文中标注。
|
||||
|
||||
---
|
||||
|
||||
## 三、核心交互模式拆解
|
||||
|
||||
### 3.1 入口与导航：探索与仪表板的"分与合"
|
||||
|
||||
平台对"探索"环节的定位有三种典型路径：
|
||||
|
||||
#### 3.1.1 独立一级入口
|
||||
|
||||
**代表**：Grafana Explore、Prometheus `/graph`
|
||||
|
||||
- Grafana Explore 占据左侧主菜单，与 Dashboards、Alerting 并列
|
||||
- Prometheus `/graph` 与 `/alerts`、`/targets` 并列
|
||||
|
||||
**设计哲学**：强调"探索 ≠ 仪表板"，鼓励 ad-hoc 工作流。
|
||||
|
||||
#### 3.1.2 嵌入主菜单但保持轻量
|
||||
|
||||
**代表**：Datadog Metrics Explorer、Honeycomb New Query、Dynatrace Data Explorer
|
||||
|
||||
- 都定位在 "Metrics/Data → Explorer"
|
||||
- 与"仪表板"并列但通常无需先建仪表板即可使用
|
||||
|
||||
#### 3.1.3 隐藏在 Widget 创建中
|
||||
|
||||
**代表**：Kibana（在 Dashboards 内创建 Lens widget）、Azure Monitor（在资源页 "Monitoring" 部分打开 Explorer）
|
||||
|
||||
- 探索结果需要 Pin to Dashboard 才能持久化
|
||||
|
||||
#### 3.1.4 设计观察
|
||||
|
||||
**Honeycomb 走得更远**——其 UI 设计**刻意模糊仪表板与查询的边界**，通过 BubbleUp 把每一次探索都变成可分享的 URL，鼓励"无持久化的探索也是结果"。
|
||||
|
||||
### 3.2 查询构建方式：文本 DSL、图形化、自然语言的三角权衡
|
||||
|
||||
| 形态 | 代表平台 | 优势 | 劣势 |
|
||||
|------|---------|------|------|
|
||||
| **纯文本 DSL** | Prometheus、CloudWatch Metrics Insights、Chronosphere | 高表达力、可版本化、可脚本化 | 学习成本高、错误难调试 |
|
||||
| **分步表单（GUI）** | Dynatrace Data Explorer、Azure Monitor | 维度零学习成本 | 表达力受限、复杂查询难构造 |
|
||||
| **拖拽字段** | Kibana Lens | 极低门槛、自动建议 | 不适合复杂表达式 |
|
||||
| **Query Builder 模式** | Honeycomb（SELECT/WHERE/GROUP BY 块）、New Relic NRQL Builder | 兼顾结构化与可读性 | 仍需理解查询模型 |
|
||||
| **自然语言（NLQ）** | Datadog "Ask"、ARMS Copilot | 真正零门槛 | 表达力边界模糊、需后端语义理解 |
|
||||
|
||||
#### 3.2.1 Grafana Explore 的混合策略
|
||||
|
||||
默认是数据源专属的查询编辑器（如 PromQL 编辑器、LogQL 编辑器、Elasticsearch DSL 编辑器），同时提供 **Kick start your query** 预设列表 + "点击 label 键值对构建"快捷方式。
|
||||
|
||||
#### 3.2.2 Kibana Lens 的 Suggestions
|
||||
|
||||
把"拖一个字段就出来一张图"做到极致——拖入字段后底部出现 Suggestions 按钮，一键切换所有推荐可视化（[来源](https://www.elastic.co/guide/en/kibana/current/lens.html)）。
|
||||
|
||||
#### 3.2.3 Datadog 的 NLQ
|
||||
|
||||
在搜索框点击 "Ask" 即可用自然语言描述需求，平台自动翻译为结构化 metric query（[来源](https://docs.datadoghq.com/metrics/explorer/)）。这是 LLM 时代探索型工具的明确方向。
|
||||
|
||||
### 3.3 时间选择器：从"绝对相对"到"对比与同步"
|
||||
|
||||
#### 3.3.1 基础能力
|
||||
|
||||
时间选择器的基本能力各平台大同小异：
|
||||
|
||||
- **相对时间**：now-1h、now-7d、now-1w 等预设
|
||||
- **绝对时间**：手选起止日期 + 时区（Timezone）
|
||||
- **缩放粒度**：从秒到年的多档粒度切换
|
||||
- **自动刷新**：通常 10s–1h，随时间范围变化（Datadog 公开仪表板固定 30s，[来源](https://docs.datadoghq.com/dashboards/)）
|
||||
|
||||
#### 3.3.2 创新交互
|
||||
|
||||
**Grafana Split View 的时间联动**：分屏后每侧都有"时间同步"按钮，点击后两侧时间选择器进入"链接"状态，改任一侧另一侧同步（[来源](https://grafana.com/docs/grafana/latest/explore/get-started-with-explore/)）。
|
||||
|
||||
**Azure Monitor 的 Time Brush 平移与缩放**：图表边缘的左右箭头按钮使时间范围按图表时间跨度的一半前后移动（Pan）；通过拖动 brush 选区进行缩放（Zoom），粒度为 Automatic 时会自动选择更小的时间粒度（[来源](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-getting-started)）。
|
||||
|
||||
#### 3.3.3 时间对比（Time Shift）
|
||||
|
||||
被独立设计为一种查询参数：
|
||||
|
||||
- **Datadog**：`timeshift()` 函数
|
||||
- **Dynatrace** Data Explorer：Timeshift 组件（`±2 minute` 格式）
|
||||
- **Kibana Lens**：Advanced → Time shift（输入 `1w` 表示对比上周）
|
||||
- **CloudWatch**："Compare with previous period"
|
||||
|
||||
**这是探索场景里最常用的模式之一**。
|
||||
|
||||
### 3.4 结果可视化：选图逻辑与多面板拆分
|
||||
|
||||
#### 3.4.1 默认图表
|
||||
|
||||
几乎都是折线图（time series），但扩展能力差异明显：
|
||||
|
||||
- **Grafana Explore**：根据数据源返回类型智能选择，可显式切换 Time Series / Table / Logs / Trace 视图
|
||||
- **Kibana Lens**：在拖入字段后**自动选**聚合函数（Date histogram / Intervals / Top values），并提供 5 种 Missing Values 处理策略（Hide/Zero/Linear/Last/Next），且可设 End values 扩展方式与虚线显示（[来源](https://www.elastic.co/guide/en/kibana/current/lens.html)）
|
||||
- **Datadog**：根据 widget 类型分化为 Timeseries / Query Value / Top List / Table / Distribution / Pie Chart / Heatmap / Geomap
|
||||
- **Dynatrace Data Explorer**：可选 Graph / Stacked column / Stacked area / Pie / Single value / Table / Top list / **Heatmap** / **Honeycomb**（六边形图），其中 Honeycomb 适合高基数分组浏览（[来源](https://docs.dynatrace.com/docs/observe-and-explore/explorer)）
|
||||
|
||||
#### 3.4.2 多面板拆分
|
||||
|
||||
三种实现方式：
|
||||
|
||||
**方式 A：水平/垂直分屏**
|
||||
- Grafana Split View
|
||||
- Azure Monitor "New chart"
|
||||
- Datadog Screenboard
|
||||
|
||||
**方式 B：按维度自动拆分**
|
||||
- Datadog "Split Graph" 按钮按 region/service/environment 等标签值自动拆为多张小图（[来源](https://docs.datadoghq.com/metrics/explorer/)）
|
||||
- Azure Monitor "Apply splitting" 按维度值拆为多条线，可设数量限制 1-50、Descending/Ascending 排序
|
||||
|
||||
**方式 C：分面（Faceting）**
|
||||
- Honeycomb 的 GROUP BY + BubbleUp 联动
|
||||
- Kibana Discover 的字段 Facet 侧栏
|
||||
|
||||
### 3.5 下钻与联动：从指标到事件再到 Trace
|
||||
|
||||
**这是"探索"区别于"展示"的关键**——能**从图表上的一点跳转到根因**：
|
||||
|
||||
#### 3.5.1 Honeycomb BubbleUp
|
||||
|
||||
在 Heatmap 上画框 → 自动生成所有字段的"选区 vs 基线"对比图 → 一键添加 `GROUP BY` / `WHERE` 验证假设。**闭环**做得最彻底（[来源](https://docs.honeycomb.io/working-with-data/bubbleup/)）。
|
||||
|
||||
#### 3.5.2 Grafana Exemplar 链接
|
||||
|
||||
通过 `--enable-feature=exemplar-storage` 启用后，OpenMetrics scrape 目标可在指标点附加 exemplar（典型为 trace_id），UI 中显示为图中"示例点"，点击即跳转到对应 trace（[来源](https://prometheus.io/docs/prometheus/latest/feature_flags/)）。
|
||||
|
||||
#### 3.5.3 Azure Monitor Drill into Logs
|
||||
|
||||
在指标异常处点击 → 跳转 Logs 子查询，预填 Activity log / Resource logs / Recommended log 三类选项（[来源](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-getting-started)）。
|
||||
|
||||
#### 3.5.4 Dynatrace Correlated Metrics
|
||||
|
||||
基于 Davis AI 的形状相似度分析，在 Graph + Split by 维度上点击线 → "See correlated metrics" 自动加入当前查询。
|
||||
|
||||
#### 3.5.5 阿里云 ARMS
|
||||
|
||||
2024-09 升级"调用链分析能力"，明确把"端到端调用链数据全采集与多维度分析"作为联动底座（[来源](https://www.aliyun.com/product/arms)）。
|
||||
|
||||
### 3.6 对比与叠加：多维度的"看同一时序"
|
||||
|
||||
#### 3.6.1 时间偏移对比
|
||||
|
||||
**Time Shift 是标配**，几乎所有平台都支持（见 3.3.3）。
|
||||
|
||||
#### 3.6.2 多指标叠加/分屏
|
||||
|
||||
- Datadog Split Graph
|
||||
- Azure Monitor Multi-metric / New chart
|
||||
- Grafana Split View
|
||||
|
||||
#### 3.6.3 分组聚合
|
||||
|
||||
- Honeycomb GROUP BY
|
||||
- Dynatrace Split by
|
||||
- CloudWatch GROUP BY
|
||||
|
||||
#### 3.6.4 Change Overlay（变更叠加）—— Datadog 独有设计
|
||||
|
||||
在图表上**自动叠加部署/事件变更线**（[来源](https://docs.datadoghq.com/dashboards/)），无需手动 mark，可视化"异常是否与某次发布相关"。
|
||||
|
||||
#### 3.6.5 Annotations
|
||||
|
||||
所有平台都支持手动/自动添加注释线，标记重要事件（部署、告警、运维动作）。
|
||||
|
||||
### 3.7 异常检测与标注
|
||||
|
||||
#### 3.7.1 阈值线
|
||||
|
||||
Datadog、Azure Monitor、Grafana 都支持在图表上叠加水平/垂直参考线。
|
||||
|
||||
#### 3.7.2 置信区间 / Baseline
|
||||
|
||||
- **Dynatrace Data Explorer** 的 Seasonal baseline 模型，根据分辨率自动选择前 14 天或前 5 年作为基线（[来源](https://docs.dynatrace.com/docs/observe-and-explore/explorer)）
|
||||
- Prometheus 工具链中常通过 `predict_linear()` 或第三方项目实现
|
||||
|
||||
#### 3.7.3 Anomaly 标注
|
||||
|
||||
- **Datadog Watchdog** 自动异常检测
|
||||
- **Honeycomb BubbleUp Insights** 给出自然语言总结与字段严重程度评级（[来源](https://docs.honeycomb.io/working-with-data/bubbleup/)）
|
||||
- **阿里云 ARMS** 的 LLM 告警收敛与故障洞察是新一代路径
|
||||
|
||||
#### 3.7.4 事件叠加
|
||||
|
||||
把日志事件、Trace 错误、变更事件以 marker 形式标注在时序图上。
|
||||
|
||||
### 3.8 变量与模板
|
||||
|
||||
- **Grafana Template Variables**：用 `$var` 在 Explore 和 Dashboard 间共享过滤器，配合下拉选择器
|
||||
- **Datadog Template Variables**：在 widget 上方定义 dynamic filter
|
||||
- **Honeycomb Saved Queries + URL**：把当前 query 编码进 URL，配合 BubbleUp 实现 permalink 分享
|
||||
- **Azure Monitor "Apply splitting" 数量限制**：1-50 控制输出卡片数，避免变量爆炸
|
||||
|
||||
---
|
||||
|
||||
## 四、7 大设计亮点
|
||||
|
||||
### 4.1 Honeycomb BubbleUp：把"高基数根因搜索"变成"并排柱状图视觉对比"
|
||||
|
||||
**核心抽象层**：
|
||||
- Selection（黄色）+ Baseline（蓝色）+ 逐字段对比图
|
||||
- 三种入口：Heatmap 画框、Chart 选线、Results 悬停行
|
||||
- 闭环：差异 → 一键 WHERE / GROUP BY → 重跑查询验证
|
||||
|
||||
**借鉴价值**：可应用于任何带"全集 vs 子集"语义的场景（如告警事件 vs 正常事件、慢请求 vs 快请求）。
|
||||
|
||||
### 4.2 Grafana Split View：把"分屏"做到极致的时间同步
|
||||
|
||||
**核心交互**：
|
||||
- 一个 Split 按钮 → 自动复制查询 → 左右两个独立面板
|
||||
- time-sync 按钮实现"链接时间"模式
|
||||
|
||||
**借鉴价值**：做对比分析（Staging vs Production、昨日 vs 今日、多 region）时显著降低认知负担。
|
||||
|
||||
### 4.3 Kibana Lens Suggestions：让"不知道用什么图"不再是问题
|
||||
|
||||
**核心交互**：
|
||||
- 拖字段 → 自动生成候选
|
||||
- 底部 Suggestions 按钮切换所有备选
|
||||
- Formula 把 KQL/Elasticsearch 聚合包装成可读公式（`percentile(..., shift='1w')`、`overall_sum(...)`）
|
||||
|
||||
**借鉴价值**：拖拽 + 公式双轨设计，对新手友好同时不丢专业表达力。
|
||||
|
||||
### 4.4 Datadog NLQ "Ask" + Change Overlay：让探索"开箱即用"
|
||||
|
||||
**核心交互**：
|
||||
- 自然语言 → 结构化查询
|
||||
- 变更线自动叠加 → 异常与发布强相关时一眼可见
|
||||
|
||||
**借鉴价值**：探索型工具应最大限度减少"为得到第一张图而做的准备"。
|
||||
|
||||
### 4.5 Dynatrace Correlated Metrics：把"找相关"从手动变自动
|
||||
|
||||
**核心交互**：
|
||||
- Davis AI 按形状相似度推荐相关指标
|
||||
- 解决的是"我不知道该和什么指标一起看"
|
||||
|
||||
**借鉴价值**：内置 ML 助手降低"经验依赖"。
|
||||
|
||||
### 4.6 Chronosphere Control Plane：探索前的"成本-质量"权衡显式化
|
||||
|
||||
**核心思想**：
|
||||
- Quotas（团队/服务容量分配）
|
||||
- Query Accelerator（自动预聚合）
|
||||
|
||||
**借鉴价值**：在大规模可观测性场景，"用什么查询"直接影响"成本/性能"，应让用户看见。
|
||||
|
||||
### 4.7 Prometheus Native Histograms + Exemplars + PromQL 新特性
|
||||
|
||||
**核心创新**：
|
||||
- Native Histograms 解决"高基数分布"无损存储
|
||||
- Exemplars 把指标点与 trace 绑定
|
||||
- 新特性：
|
||||
- `promql-extended-range-selectors`（`anchored`/`smoothed`）
|
||||
- `promql-binop-fill-modifiers`（`fill(0)`）
|
||||
- `promql-duration-expr`（`[5m * 2]`）
|
||||
|
||||
**借鉴价值**：在底层语言里支持"探索语义"——而不是只在 UI 上贴。
|
||||
|
||||
### 4.8 共同趋势：可分享的探索 URL + AI 副驾
|
||||
|
||||
- 几乎所有平台都把查询状态编码进 URL（Honecyomb permalinks、Grafana /goto/:uid、Datadog Copy link）
|
||||
- LLM 副驾（ARMS Copilot、Datadog Bits AI、New Relic NRAI、Grafana LLM）正在把"自然语言探索"变成标配
|
||||
|
||||
---
|
||||
|
||||
## 五、可借鉴的 UX 模式 Checklist（产品设计参考）
|
||||
|
||||
### 5.1 探索入口
|
||||
|
||||
- [ ] 与 Dashboard 并列的独立入口，明确传达"探索 ≠ 持久化"
|
||||
- [ ] 在 Dashboard 面板上提供"Open in Explore"快捷入口，把已有查询带入探索态
|
||||
- [ ] 探索结果可一键 Save to Dashboard / Notebook / Incident
|
||||
|
||||
### 5.2 查询构建
|
||||
|
||||
- [ ] 默认文本编辑器（DSL） + GUI 切换 + 自然语言三层并存
|
||||
- [ ] 查询历史（Query History）自动保存
|
||||
- [ ] 已保存查询（Saved Queries）可复用
|
||||
- [ ] Kick start / 预设模板
|
||||
- [ ] 字段自动补全、语法高亮、错误提示
|
||||
- [ ] 字段名点击即加入查询（low-code 路径）
|
||||
|
||||
### 5.3 时间选择
|
||||
|
||||
- [ ] 相对 + 绝对 + 自定义
|
||||
- [ ] 时区切换
|
||||
- [ ] 缩放粒度
|
||||
- [ ] 自动刷新（与时间范围联动）
|
||||
- [ ] Pan/Zoom/Brush
|
||||
- [ ] Time Shift 时间偏移对比
|
||||
- [ ] Split View 时两侧时间可同步
|
||||
|
||||
### 5.4 可视化
|
||||
|
||||
- [ ] 默认折线，5+ 图表类型可切换
|
||||
- [ ] 按维度自动 Split / 分屏
|
||||
- [ ] 多图表多指标独立查看
|
||||
- [ ] Missing Values 处理策略（Hide/Zero/Linear/Last/Next）
|
||||
- [ ] 公式与自定义计算
|
||||
- [ ] 性能：采样、降采样、聚合下推
|
||||
- [ ] Legend 中可显示统计（Avg/Median/Min/Max/Last/Diff%）
|
||||
|
||||
### 5.5 下钻与联动
|
||||
|
||||
- [ ] 图表点 → Trace / Log / Event 跳转
|
||||
- [ ] Exemplar 链接（指标点直接带 trace_id）
|
||||
- [ ] 跨数据源联动（指标 ↔ 日志 ↔ 链路）
|
||||
- [ ] 自动推荐相关指标（Dynatrace Davis 风格）
|
||||
- [ ] BubbleUp 风格的"选择 vs 基线"对比
|
||||
|
||||
### 5.6 对比与异常
|
||||
|
||||
- [ ] 多指标叠加 + 多指标分屏
|
||||
- [ ] 时间偏移对比 / 周环比
|
||||
- [ ] 阈值线 / 参考线
|
||||
- [ ] 置信区间 / Baseline
|
||||
- [ ] Anomaly 标注 + AI 解释
|
||||
- [ ] Change Overlay（变更自动叠加）
|
||||
- [ ] Annotations（手动/自动事件标注）
|
||||
|
||||
### 5.7 协作与分享
|
||||
|
||||
- [ ] 查询状态 URL 化、可分享
|
||||
- [ ] 短链接（避免 IM 截断）
|
||||
- [ ] Embed / iFrame
|
||||
- [ ] 评论与多人协作（Notebook 模式）
|
||||
- [ ] 权限控制（公开/私有/团队）
|
||||
|
||||
### 5.8 性能与可用性
|
||||
|
||||
- [ ] 百万级时序的采样/降采样
|
||||
- [ ] 服务端预聚合（Query Accelerator）
|
||||
- [ ] 加载态/部分加载的友好展示
|
||||
- [ ] 错误提示 + 自动修复建议
|
||||
- [ ] 键盘可访问性（Kibana Lens 风格的 Tab+Space 拖拽）
|
||||
|
||||
---
|
||||
|
||||
## 六、各平台详细功能分析
|
||||
|
||||
### 6.1 Grafana Explore
|
||||
|
||||
**来源**：[https://grafana.com/docs/grafana/latest/explore/](https://grafana.com/docs/grafana/latest/explore/)
|
||||
|
||||
#### 核心特性
|
||||
|
||||
- **Split View 分屏对比**：分屏后每侧都有"时间同步"按钮，点击后两侧时间选择器进入"链接"状态，改任一侧另一侧同步
|
||||
- **Query Inspector**：4 个 Tab（Stats、Query、JSON、Data），便于调试查询
|
||||
- **Query Management**：历史（Query History）+ 已保存查询
|
||||
- **Outline 导航**：左侧 Outline 列出当前查询涉及的所有字段
|
||||
- **Time Picker**：绝对/相对/时区/会计年度支持
|
||||
- **Mixed Data Source**：跨数据源关联查询
|
||||
- **URL 分享**：支持 `/goto/:uid` 短链接
|
||||
- **Correlation Editor**：相关字段编辑器
|
||||
- **Trace 集成**：从指标点跳到 trace
|
||||
- **Exemplar 链接**：支持
|
||||
|
||||
#### 设计亮点
|
||||
|
||||
1. **Split View + Time Sync**：最经典的探索模式，适合做对比分析
|
||||
2. **Query Inspector**：对调试复杂查询至关重要
|
||||
3. **Outline 导航**：帮助用户理解查询结构
|
||||
|
||||
### 6.2 Prometheus
|
||||
|
||||
**来源**：[https://prometheus.io/docs/prometheus/latest/querying/basics/](https://prometheus.io/docs/prometheus/latest/querying/basics/)
|
||||
|
||||
#### 核心特性
|
||||
|
||||
- **PromQL Expression 类型**：Instant vector、Range vector、Scalar、String
|
||||
- **Time Series Selectors**：带 label matchers 的选择器
|
||||
- **Table View + Graph View**：两种默认视图
|
||||
- **Offset 和 @ 修饰符**：支持时间偏移和"@ 时间戳"语法
|
||||
- **Native Histograms**：原生直方图，高基数分布无损存储
|
||||
- **Exemplar Storage**：通过 `--enable-feature=exemplar-storage` 启用
|
||||
- **PromQL 新特性**：
|
||||
- `promql-extended-range-selectors`（`anchored`/`smoothed`）
|
||||
- `promql-duration-expr`（`[5m * 2]`）
|
||||
- `promql-binop-fill-modifiers`（`fill(0)`）
|
||||
- **Old UI 回退**：可通过 `--enable-feature=old-ui` 回退到 2.x 风格 Web UI
|
||||
- **Recording Rules**：记录规则
|
||||
- **Console Templates**：控制台模板
|
||||
|
||||
#### 设计亮点
|
||||
|
||||
1. **Exemplar 绑定 trace_id**：让指标和链路打通
|
||||
2. **Native Histograms**：解决高基数分布的痛点
|
||||
3. **PromQL 探索语义内建**：不是 UI 包装，是底层语言支持
|
||||
|
||||
### 6.3 Datadog Metrics Explorer
|
||||
|
||||
**来源**：[https://docs.datadoghq.com/metrics/explorer/](https://docs.datadoghq.com/metrics/explorer/)
|
||||
|
||||
#### 核心特性
|
||||
|
||||
- **NLQ "Ask"**：自然语言查询入口
|
||||
- **Query Editor**：传统文本编辑器
|
||||
- **Scope Filtering**：scope 过滤
|
||||
- **Space Aggregation**：Average / Max / Min / Sum
|
||||
- **Split Graph by tag values**：按 tag 值自动拆图
|
||||
- **Export**：link、incident、monitor、dashboard、notebook
|
||||
- **Quick Graphs**：快速图表
|
||||
- **Functions**：
|
||||
- algorithmic、arithmetic、count
|
||||
- exclusion、interpolation、rank
|
||||
- rate、regression、rollup
|
||||
- smoothing、telemetry source、timeshift
|
||||
- **Widget 类型**：timeseries、query value、top list、table、distribution、pie chart、list/SLO/architecture
|
||||
- **Timeboards vs Screenboards vs Dashboards**：三种仪表板模式
|
||||
- **Refresh Rate**：随时间范围变化
|
||||
|
||||
#### 设计亮点
|
||||
|
||||
1. **NLQ "Ask"**：LLM 时代探索的标杆
|
||||
2. **Change Overlay**：自动叠加变更线，独有设计
|
||||
3. **Watchdog AI**：自动异常检测 + 根因定位
|
||||
4. **Notebooks**：多人协作探索
|
||||
|
||||
### 6.4 Honeycomb
|
||||
|
||||
**来源**：[https://docs.honeycomb.io/working-with-data/bubbleup/](https://docs.honeycomb.io/working-with-data/bubbleup/)
|
||||
|
||||
#### 核心特性
|
||||
|
||||
- **Query Builder (SELECT/WHERE/GROUP BY)**：分块式查询构建
|
||||
- **BubbleUp**：选择基线对比
|
||||
- **BubbleUp Insights**（Honeycomb Intelligence）：自然语言总结
|
||||
- **Permalinks via URL**：通过 URL 分享
|
||||
- **BubbleUp Filter**：字段过滤
|
||||
- **三种入口**：
|
||||
- Heatmap 画框
|
||||
- Chart 选线
|
||||
- Results 悬停行
|
||||
- **Dimensions vs Measures**：维度与度量区分
|
||||
- **Click Actions**：Group by Field、Show only where、Copy field name
|
||||
- **Wide Events**：宽事件范式
|
||||
|
||||
#### 设计亮点
|
||||
|
||||
1. **BubbleUp 视觉对比**：黄（选区）vs 蓝（基线），全字段柱状图对比
|
||||
2. **Wide Events**：每个事件携带 50+ 字段，避免预聚合损失
|
||||
3. **Faceted Search**：字段 facet 侧栏
|
||||
4. **无持久化探索**：URL 即结果
|
||||
|
||||
### 6.5 Kibana Lens
|
||||
|
||||
**来源**：[https://www.elastic.co/guide/en/kibana/current/lens.html](https://www.elastic.co/guide/en/kibana/current/lens.html)
|
||||
|
||||
#### 核心特性
|
||||
|
||||
- **Suggestions**：拖字段自动生成候选可视化
|
||||
- **Drag-and-drop with full keyboard accessibility**：拖拽 + 键盘可访问
|
||||
- **Formula 语法**：
|
||||
- `count(kql='...')/count()`
|
||||
- `shift='1w'`
|
||||
- `overall_sum()`
|
||||
- **Time shift**：在 Advanced settings 中设置
|
||||
- **Missing values 处理策略**：Hide/Zero/Linear/Last/Next
|
||||
- **Sampling slider**：性能采样滑块
|
||||
- **Legend statistics**：Average/Median/Min/Max/Range/Last/Diff%/Sum/Count
|
||||
- **Normalize by unit / Customize time interval**：单位归一化/时间间隔自定义
|
||||
- **"Explore data in Discover" link**：跳转到 Discover
|
||||
- **Inspector with CSV download**：检查器 + CSV 下载
|
||||
|
||||
#### 设计亮点
|
||||
|
||||
1. **Suggestions 智能推荐**：拖字段即出图，底部切换备选
|
||||
2. **Formula 双轨设计**：对新手友好且不丢专业表达力
|
||||
3. **键盘可访问性**：Tab + Space 拖拽，无障碍设计典范
|
||||
|
||||
### 6.6 Azure Monitor Metrics Explorer
|
||||
|
||||
**来源**：[https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-getting-started](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-getting-started)
|
||||
|
||||
#### 核心特性
|
||||
|
||||
- **Single or multiple resources**：单/多资源
|
||||
- **Dimension filtering**：Add filter
|
||||
- **Splitting**：Apply splitting，数量限制 1-50
|
||||
- **Aggregation**：Sum/Count/Average/Min/Max
|
||||
- **Pan/Zoom with time brush**：平移/缩放
|
||||
- **Multi-metric / multi-chart**：多指标/多图
|
||||
- **Custom line colors**：自定义线条颜色
|
||||
- **Y-axis range lock**：Y 轴范围锁定
|
||||
- **Drill into Logs**：
|
||||
- Activity log
|
||||
- Resource logs
|
||||
- Recommended log
|
||||
- **Share**：
|
||||
- Pin to dashboard
|
||||
- Save to workbook
|
||||
- Download to Excel
|
||||
- Copy link
|
||||
- Send to Workbook
|
||||
- Pin to Grafana
|
||||
- **PromQL support**：Azure Monitor workspace metrics 支持 PromQL
|
||||
|
||||
#### 设计亮点
|
||||
|
||||
1. **Time Brush 平移/缩放**：边缘按钮按半时间跨度移动
|
||||
2. **Drill into Logs**：从指标异常跳到日志，闭环根因
|
||||
3. **Y 轴锁定**：避免动态缩放干扰对比
|
||||
|
||||
### 6.7 Dynatrace Data Explorer
|
||||
|
||||
**来源**：[https://docs.dynatrace.com/docs/observe-and-explore/explorer](https://docs.dynatrace.com/docs/observe-and-explore/explorer)
|
||||
|
||||
#### 核心特性
|
||||
|
||||
- **Query Components**：
|
||||
- Metric name
|
||||
- Space aggregation
|
||||
- Split by
|
||||
- Sort
|
||||
- Rate
|
||||
- Filter by
|
||||
- Limit
|
||||
- Timeshift
|
||||
- **Visualization**：Graph/Stacked column/area/Pie/Single value/Table/Top list/Heatmap/Honeycomb
|
||||
- **Timeshift**：时间对比
|
||||
- **Baselines**：季节性 baseline 模型
|
||||
- **Correlated Metrics**：Davis AI 自动推荐相关指标
|
||||
- **Focus**：隐藏其他 series
|
||||
- **Pin to dashboard / Share link / Export CSV / Copy request**
|
||||
- **限制**：10 metrics per viz, 100 series per metric
|
||||
|
||||
#### 设计亮点
|
||||
|
||||
1. **Davis AI Correlated Metrics**：AI 自动推荐相关指标
|
||||
2. **Seasonal Baseline**：根据分辨率自动选 14 天或 5 年基线
|
||||
3. **Honeycomb 可视化**：六边形图适合高基数分组
|
||||
|
||||
### 6.8 Chronosphere Control Plane
|
||||
|
||||
**来源**：[https://chronosphere.io/platform/control-plane/](https://chronosphere.io/platform/control-plane/)
|
||||
|
||||
#### 核心特性
|
||||
|
||||
- **Observability Data Optimization Cycle**：
|
||||
- Centralized Governance（Quotas）
|
||||
- Analyze（utility score）
|
||||
- Refine（降采样/聚合/采样）
|
||||
- Operate（Query Accelerator、Query Scheduler）
|
||||
- **PromQL 兼容**
|
||||
- **89% 体积减少**
|
||||
- **50% 故障排查时间减少**
|
||||
|
||||
#### 设计亮点
|
||||
|
||||
1. **数据成本显式化**：让用户看到"这次查询要花多少钱"
|
||||
2. **Query Accelerator 自动预聚合**：提升查询性能
|
||||
3. **Utility Score**：评估数据价值
|
||||
|
||||
### 6.9 阿里云 ARMS
|
||||
|
||||
**来源**：[https://www.aliyun.com/product/arms](https://www.aliyun.com/product/arms)
|
||||
|
||||
#### 核心特性
|
||||
|
||||
- **全栈覆盖**：Web/小程序/Mobile + Java/Golang/Python
|
||||
- **链路打通**：端到端调用链
|
||||
- **智能加持**：LLM 告警收敛、故障洞察
|
||||
- **开源兼容**：OpenTelemetry、Prometheus
|
||||
- **应用监控 / 前端监控 / 可观测链路 OpenTelemetry 版**
|
||||
|
||||
#### 设计亮点
|
||||
|
||||
1. **端到端调用链联动**：从指标到 trace 无缝
|
||||
2. **LLM 告警收敛**：新一代智能体
|
||||
3. **OpenTelemetry 兼容**：开放生态
|
||||
|
||||
### 6.10 夜莺 Nightingale
|
||||
|
||||
**来源**：[https://flashcat.cloud/docs/](https://flashcat.cloud/docs/)
|
||||
|
||||
#### 核心特性
|
||||
|
||||
- **PromQL 兼容**
|
||||
- **开源**
|
||||
- **VictoriaMetrics/TDengine 兼容**
|
||||
- **生态丰富**
|
||||
|
||||
---
|
||||
|
||||
## 七、设计模式深度分析
|
||||
|
||||
### 7.1 三种入口与导航模式对比
|
||||
|
||||
| 模式 | 优点 | 缺点 | 代表 |
|
||||
|------|------|------|------|
|
||||
| **独立一级入口** | 探索感强、鼓励 ad-hoc | 与 Dashboard 关系弱 | Grafana Explore、Prometheus /graph |
|
||||
| **嵌入主菜单** | 入口轻量、不混淆 | 用户可能找不到 | Datadog、Honeycomb、Dynatrace |
|
||||
| **隐藏在 Widget 中** | 与 Dashboard 紧密 | 探索结果难持久 | Kibana、Azure Monitor |
|
||||
| **URL 即结果** | 无持久化也分享 | 学习成本高 | Honeycomb |
|
||||
|
||||
### 7.2 查询构建的"门槛-表达力"光谱
|
||||
|
||||
```
|
||||
高门槛
|
||||
│
|
||||
│ 纯文本 DSL（PromQL、CloudWatch Insights）
|
||||
│
|
||||
│ Query Builder（Honeycomb、New Relic）
|
||||
│
|
||||
│ 分步表单（Dynatrace、Azure Monitor）
|
||||
│
|
||||
│ 拖拽字段（Kibana Lens）
|
||||
│
|
||||
│ 自然语言 NLQ（Datadog Ask、ARMS Copilot）
|
||||
│
|
||||
低门槛
|
||||
└─────────────────────────────────→ 高表达力
|
||||
```
|
||||
|
||||
**关键观察**：最佳实践是**多种方式并存**，让用户在不同场景选择合适的工具。
|
||||
|
||||
### 7.3 时间选择的"复杂度阶梯"
|
||||
|
||||
| 阶梯 | 能力 | 代表 |
|
||||
|------|------|------|
|
||||
| L1 | 相对/绝对 | 几乎所有平台 |
|
||||
| L2 | 时区/粒度切换 | Grafana、Datadog |
|
||||
| L3 | 自动刷新联动 | Datadog、Grafana |
|
||||
| L4 | Pan/Zoom/Brush | Azure Monitor、Grafana |
|
||||
| L5 | Time Shift 对比 | Datadog、Kibana、Dynatrace、CloudWatch |
|
||||
| L6 | Split View 同步 | Grafana |
|
||||
|
||||
### 7.4 可视化的"默认 vs 高级"
|
||||
|
||||
| 平台 | 默认 | 高级 |
|
||||
|------|------|------|
|
||||
| Grafana | 折线 | Time Series/Table/Logs/Trace 切换 |
|
||||
| Kibana Lens | 自动选择 | Suggestions 切换备选 |
|
||||
| Datadog | Timeseries | Timeseries/Query Value/Top List/Heatmap/Pie/Geomap |
|
||||
| Dynatrace | Graph | Graph/Stacked/Honeycomb/Heatmap/Top list |
|
||||
|
||||
### 7.5 下钻链路的"5 个层级"
|
||||
|
||||
```
|
||||
L1 图表点 → Trace (Exemplar)
|
||||
L2 指标点 → Log (Drill into Logs)
|
||||
L3 当前查询 → 相关指标 (Correlated Metrics)
|
||||
L4 选区 vs 基线 → 字段差异 (BubbleUp)
|
||||
L5 异常点 → 根因服务 (Davis AI RCA)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 八、关键 UX 设计原则提炼
|
||||
|
||||
### 8.1 探索的核心价值是"从数据中问出问题"
|
||||
|
||||
- **Grafana Explore** 的 Outline 帮助用户理解查询
|
||||
- **Kibana Lens** 的 Suggestions 帮用户"不知道问什么"
|
||||
- **Honeycomb BubbleUp** 帮用户"问出对的问题"
|
||||
- **Dynatrace Davis** 帮用户"找到根因"
|
||||
|
||||
### 8.2 探索结果必须可分享
|
||||
|
||||
- URL 化（所有平台都支持）
|
||||
- 短链接（Grafana `/goto/:uid`）
|
||||
- Embed / iFrame
|
||||
- 协作（Notebook 模式）
|
||||
|
||||
### 8.3 对比是探索的核心动作
|
||||
|
||||
- **时间对比**：Time Shift
|
||||
- **空间对比**：Split by、Split Graph
|
||||
- **基线对比**：Seasonal Baseline
|
||||
- **分屏对比**：Grafana Split View
|
||||
- **选区对比**：BubbleUp
|
||||
|
||||
### 8.4 降低第一张图的门槛
|
||||
|
||||
- **Datadog NLQ**：自然语言直达
|
||||
- **Kibana Suggestions**：拖字段直达
|
||||
- **Watchdog 自动告警**：零配置开箱即用
|
||||
- **Change Overlay**：自动叠加变更线
|
||||
|
||||
### 8.5 可解释性是工业落地的关键
|
||||
|
||||
- **GDN 的图结构**：可解释变量关系
|
||||
- **Anomaly Transformer 的关联差异**：可解释异常归因
|
||||
- **Dynatrace Davis 的因果 AI**：可解释根因
|
||||
- **BubbleUp 的字段对比**：可解释差异来源
|
||||
|
||||
---
|
||||
|
||||
## 九、参考来源
|
||||
|
||||
### 9.1 官方文档
|
||||
|
||||
1. Grafana Explore: https://grafana.com/docs/grafana/latest/explore/get-started-with-explore/
|
||||
2. Grafana Explore 概览: https://grafana.com/docs/grafana/latest/explore/
|
||||
3. Prometheus Querying basics: https://prometheus.io/docs/prometheus/latest/querying/basics/
|
||||
4. Prometheus Expression browser: https://prometheus.io/docs/visualization/browser/
|
||||
5. Prometheus Feature flags: https://prometheus.io/docs/prometheus/latest/feature_flags/
|
||||
6. Datadog Dashboards 概览: https://docs.datadoghq.com/dashboards/
|
||||
7. Datadog Metrics Explorer: https://docs.datadoghq.com/metrics/explorer/
|
||||
8. Datadog Functions 概览: https://docs.datadoghq.com/dashboards/functions/
|
||||
9. Honeycomb BubbleUp: https://docs.honeycomb.io/working-with-data/bubbleup/
|
||||
10. Kibana Lens: https://www.elastic.co/guide/en/kibana/current/lens.html
|
||||
11. Azure Monitor Metrics Explorer: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-getting-started
|
||||
12. Dynatrace Data Explorer: https://docs.dynatrace.com/docs/observe-and-explore/explorer
|
||||
13. Chronosphere Control Plane: https://chronosphere.io/platform/control-plane/
|
||||
14. 阿里云 ARMS 产品总览: https://www.aliyun.com/product/arms
|
||||
15. 夜莺 Nightingale 文档: https://flashcat.cloud/docs/
|
||||
|
||||
### 9.2 第三方资料
|
||||
|
||||
- Honeycomb BubbleUp 设计哲学起源于工业过程控制（气泡判断发酵状态），强调"无预设维度的自动归因"，推动 wide events 范式
|
||||
- Prometheus 3.0 UI 重写：可通过 `--enable-feature=old-ui` 回退到 2.x 风格 Web UI
|
||||
|
||||
### 9.3 未能成功抓取的页面（需进一步验证）
|
||||
|
||||
- AWS CloudWatch Metrics Insights 详细查询语法页（页面抓取仅返回标题）
|
||||
- Splunk SPL 文档（403 Forbidden）
|
||||
- Lightstep / ServiceNow Cloud Observability 文档（已迁移至新域）
|
||||
- 华为云 APM / AOM 详细产品页（404）
|
||||
- New Relic NRQL / Data Explorer 详细子页（仅抓到总览页）
|
||||
|
||||
> **说明**：本报告基于 2026 年 6 月的实际页面抓取与公开产品资料整理，标注 `*` 的部分基于通用知识/产品总览推演，建议在产品决策前对存疑条目进行官方文档二次核实。腾讯云监控、华为云 APM、夜莺 Nightingale 的核心功能主要通过产品总览/索引页确认，详细交互细节未完全覆盖。
|
||||
|
||||
---
|
||||
|
||||
## 附录 A：术语表
|
||||
|
||||
| 术语 | 含义 |
|
||||
|------|------|
|
||||
| **Explore** | 时序探索，强调查询驱动 + 即时反馈 |
|
||||
| **BubbleUp** | Honeycomb 提出的"选区 vs 基线"对比交互 |
|
||||
| **Split View** | 分屏对比视图（Grafana） |
|
||||
| **Exemplar** | Prometheus 概念，指标点附加的 trace_id 等样本 |
|
||||
| **Change Overlay** | Datadog 独有，图表自动叠加变更线 |
|
||||
| **Time Shift** | 时间偏移对比 |
|
||||
| **Annotations** | 图表上的事件标记线 |
|
||||
| **Query Builder** | 图形化查询构建器（Honeycomb、New Relic） |
|
||||
| **NLQ** | Natural Language Query，自然语言查询 |
|
||||
| **TSFM** | Time Series Foundation Model，时序基础模型 |
|
||||
| **Wide Events** | Honeycomb 范式，每个事件携带 50+ 字段 |
|
||||
| **Faceted Search** | 字段 facet 侧栏搜索 |
|
||||
| **Apprentice** | 阿里清华联合的异常检测框架（KDD 2015） |
|
||||
| **SR-CNN** | Spectral Residual + CNN（KDD 2019） |
|
||||
| **Davis AI** | Dynatrace 因果 AI 引擎 |
|
||||
|
||||
## 附录 B：报告使用建议
|
||||
|
||||
### B.1 给产品经理
|
||||
|
||||
- **入口与导航**：参考 Grafana Explore 独立入口设计
|
||||
- **查询构建**：提供"DSL + GUI + NLQ"三层并存能力
|
||||
- **时间选择**：必须支持 Time Shift 与 Split View 同步
|
||||
- **可视化**：默认折线 + 5+ 图表类型可切换
|
||||
- **下钻联动**：指标 → Trace / Log 跳转是硬需求
|
||||
- **对比叠加**：Change Overlay 与 BubbleUp 风格值得借鉴
|
||||
- **异常检测**：与 AI 副驾结合是趋势
|
||||
|
||||
### B.2 给 UX 设计师
|
||||
|
||||
- **三栏式布局**：左侧导航 + 中间查询 + 右侧 Outline
|
||||
- **Split View**：做对比分析的核心交互
|
||||
- **BubbleUp 风格**：黄/蓝对比 + 字段柱状图
|
||||
- **Suggestions 模式**：拖字段即出图
|
||||
- **键盘可访问性**：Tab + Space 拖拽
|
||||
- **错误状态友好**：语法高亮 + 自动修复建议
|
||||
- **URL 可分享**：每个查询状态可复制为 URL
|
||||
|
||||
### B.3 给前端工程师
|
||||
|
||||
- **图表库选择**：uPlot、ECharts、D3.js、Plotly
|
||||
- **采样与降采样**：百万级时序的渲染策略
|
||||
- **虚拟滚动**：长列表性能优化
|
||||
- **WebWorker 处理**：避免阻塞主线程
|
||||
- **IndexedDB 缓存**：查询历史与设置
|
||||
- **WebSocket 实时推送**：自动刷新机制
|
||||
- **iFrame Embed**：支持嵌入其他平台
|
||||
|
||||
### B.4 给架构师
|
||||
|
||||
- **服务端预聚合**：Query Accelerator 模式
|
||||
- **下推计算**：把 aggregation 下推到存储
|
||||
- **成本-质量权衡**：Chronosphere Control Plane 思路
|
||||
- **多数据源联邦**：Grafana Mixed Data Source
|
||||
- **AI 副驾集成**：LLM 翻译自然语言 → DSL
|
||||
|
||||
---
|
||||
|
||||
> **报告结束**
|
||||
>
|
||||
> 如需深入某一平台的交互设计或具体模式的实现细节，欢迎告知，我可以进一步展开调研或产出原型设计建议。
|
||||
Reference in New Issue
Block a user