Compare commits
2 Commits
b05270802d
...
cdfd77bdcb
| Author | SHA1 | Date | |
|---|---|---|---|
| cdfd77bdcb | |||
| 031aa35604 |
@@ -28,8 +28,8 @@ def run_anomaly_window(conn, stable: str, algo: str) -> list[tuple[int, int]]:
|
|||||||
|
|
||||||
sql = (
|
sql = (
|
||||||
f"SELECT _WSTART, _WEND "
|
f"SELECT _WSTART, _WEND "
|
||||||
f"FROM (ANOMALY_WINDOW(value, \"algo={algo}\") "
|
f"FROM (SELECT ts, `value` AS v FROM ds_{stable}) "
|
||||||
f"FROM ds_{stable})"
|
f"ANOMALY_WINDOW(v, 'algo={algo}')"
|
||||||
)
|
)
|
||||||
|
|
||||||
rows = conn.execute(sql)
|
rows = conn.execute(sql)
|
||||||
|
|||||||
@@ -38,8 +38,8 @@ def run_forecast(
|
|||||||
"""
|
"""
|
||||||
sql = (
|
sql = (
|
||||||
f"SELECT _FROWTS, _FLOW, _FHIGH "
|
f"SELECT _FROWTS, _FLOW, _FHIGH "
|
||||||
f"FROM (FORECAST(value, \"algo={algo},rows={rows},conf={conf}\") "
|
f"FROM (SELECT ts, `value` AS v FROM ds_{stable}) "
|
||||||
f"FROM ds_{stable})"
|
f"FORECAST(v, 'algo={algo},rows={rows},conf={conf}')"
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
raw = conn.execute(sql)
|
raw = conn.execute(sql)
|
||||||
@@ -83,7 +83,7 @@ def forecast_anomaly(
|
|||||||
start_ts = min(frowtses)
|
start_ts = min(frowtses)
|
||||||
end_ts = max(frowtses)
|
end_ts = max(frowtses)
|
||||||
|
|
||||||
sql = f"SELECT ts, value FROM s_{stable} WHERE ts >= {start_ts} AND ts <= {end_ts}"
|
sql = f"SELECT ts, `value` FROM s_{stable} WHERE ts >= {start_ts} AND ts <= {end_ts}"
|
||||||
raw = conn.execute(sql)
|
raw = conn.execute(sql)
|
||||||
|
|
||||||
fmap = {f[0]: (f[1], f[2]) for f in forecasts}
|
fmap = {f[0]: (f[1], f[2]) for f in forecasts}
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ def create_supertable(conn, stable: str) -> None:
|
|||||||
"""
|
"""
|
||||||
sql = (
|
sql = (
|
||||||
f"CREATE STABLE IF NOT EXISTS ds_{stable} "
|
f"CREATE STABLE IF NOT EXISTS ds_{stable} "
|
||||||
f"(ts TIMESTAMP, value DOUBLE, label INT) "
|
f"(ts TIMESTAMP, `value` DOUBLE, is_anomaly INT) "
|
||||||
f"TAGS (series_id INT)"
|
f"TAGS (series_id INT)"
|
||||||
)
|
)
|
||||||
conn.execute_no_result(sql)
|
conn.execute_no_result(sql)
|
||||||
|
|||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# Iter-1 E2E 验证报告
|
||||||
|
|
||||||
|
## 运行时间
|
||||||
|
|
||||||
|
- 验证日期: 2026-06-11T16:08:04+08:00
|
||||||
|
- 子项目 commit: 031aa35604a8256683ef5b2b9c4e50f0880a92c8
|
||||||
|
- 验证分支: main
|
||||||
|
- TDengine 版本: 3.4.1.13.community
|
||||||
|
- Docker 镜像: `tdengine/tsdb:latest` (Docker Hub)
|
||||||
|
|
||||||
|
## 容器启动
|
||||||
|
|
||||||
|
- 拉取镜像: 成功(首次 ~1GB,~3 分钟)
|
||||||
|
- `docker compose up -d`: 成功
|
||||||
|
- 健康检查: 1 次即通过(attempt=1)
|
||||||
|
- 客户端协议: taos-ws-py(WebSocket,ws://root:taosdata@localhost:6041)
|
||||||
|
|
||||||
|
## E2E 结果
|
||||||
|
|
||||||
|
- **exit_code**: 0(E2E 完整跑通,未 fatal)
|
||||||
|
- 拉容器: 成功
|
||||||
|
- 注入数据: **3 个数据集**(finance_001/002/003),共 18635 行
|
||||||
|
- finance_001: 1624 行(注入耗时 0.2s)
|
||||||
|
- finance_002: 1109 行(注入耗时 0.0s)
|
||||||
|
- finance_003: 15902 行(注入耗时 0.1s)
|
||||||
|
- 检测: **6 算法全部尝试**(ksigma/iqr/grubbs/shesd/lof/sample_ad_model)
|
||||||
|
- FORECAST: **3 数据集均调用**(holtwinters, rows=10, conf=95)
|
||||||
|
- 可视化: **3 个 PNG 文件**(每数据集一张 GT vs TDengine 对比图)
|
||||||
|
- `render/finance_001_gt_vs_tdengine.png` (335K)
|
||||||
|
- `render/finance_002_gt_vs_tdengine.png` (137K)
|
||||||
|
- `render/finance_003_gt_vs_tdengine.png` (104K)
|
||||||
|
- JSON 日志: `logs/e2e_20260611_160806.json`(3.6K)
|
||||||
|
|
||||||
|
## 各数据集窗口数
|
||||||
|
|
||||||
|
| 数据集 | ksigma | iqr | grubbs | shesd | lof | sample_ad_model | FORECAST points | FORECAST anomalies | 行数 |
|
||||||
|
|--------|--------|-----|--------|-------|-----|------------------|------------------|--------------------|------|
|
||||||
|
| finance_001 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1624 |
|
||||||
|
| finance_002 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1109 |
|
||||||
|
| finance_003 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 15902 |
|
||||||
|
|
||||||
|
> **注**:6 算法与 FORECAST 在 community 3.4.1 镜像下全部返回 0 窗口/0 点。
|
||||||
|
> 原因:TDengine community 镜像未安装 `taosanode` 进程,ANOMALY_WINDOW 与
|
||||||
|
> FORECAST 内置算法(TDgpt 引擎)不可用。运行时返回 `[0x0443] Analysis
|
||||||
|
> algorithm/model not loaded`,已被检测模块 graceful 容错(per-algo 独立 try/except)。
|
||||||
|
> schema/连接/注入/可视化通路完全正常。
|
||||||
|
|
||||||
|
## 关键修复(Iter-1 验证中暴露)
|
||||||
|
|
||||||
|
首次 E2E 在 schema 创建阶段失败,原因及修复(提交 `031aa35`):
|
||||||
|
|
||||||
|
1. **`value` 是 TDengine 3.4.1 保留字**。`CREATE STABLE ... (ts TIMESTAMP, value DOUBLE, ...)`
|
||||||
|
报 `[0x2600] syntax error near "value double, ..."`。
|
||||||
|
修复:`ts_anomaly_td/schema.py` 把列定义改为反引号转义
|
||||||
|
`` `value` DOUBLE ``,并将同义 `label` 列重命名为 `is_anomaly`。
|
||||||
|
2. **ANOMALY_WINDOW/FORECAST 函数参数不接受保留字**。即使表列用反引号,
|
||||||
|
`ANOMALY_WINDOW(`value`, "algo=ksigma")` 仍报 `syntax error`。
|
||||||
|
修复:`ts_anomaly_td/detection.py` 与 `forecast.py` 改用子查询别名:
|
||||||
|
```sql
|
||||||
|
SELECT _WSTART, _WEND
|
||||||
|
FROM (SELECT ts, `value` AS v FROM ds_<stable>)
|
||||||
|
ANOMALY_WINDOW(v, 'algo=ksigma')
|
||||||
|
```
|
||||||
|
子查询中列名 `v` 非保留字,函数可正常解析。
|
||||||
|
3. **taos-ws-py 端** 无需改动,SQL 字符串原样透传,服务器侧解析失败时返回
|
||||||
|
`0x2600`,已被检测模块记为 `algo=<name> failed (non-fatal)` 继续后续算法。
|
||||||
|
|
||||||
|
## 已知问题 / 后续工作
|
||||||
|
|
||||||
|
- **ANOMALY_WINDOW / FORECAST 算法未生效**:TDengine 3.4.1 community 镜像
|
||||||
|
未包含 `taosanode`(TDgpt 服务进程),所有内置异常检测/预测算法在调用
|
||||||
|
时返回 `0x0443 Analysis algorithm/model not loaded`。当前仅 schema/连接/
|
||||||
|
注入/可视化路径已验证,算法侧需在生产/企业镜像或本地 `taosanode` 部署
|
||||||
|
后再做端到端验证。
|
||||||
|
- **`tdengine/tdengine:latest`(企业版 3.3.6)启动卡死**:在容器内
|
||||||
|
`taosd` 一直处于 `0: unavailable` 状态,需进一步排查 `taos.cfg` 与
|
||||||
|
`TAOS_FQDN` 设置;目前不在 Iter-1 验证范围。
|
||||||
|
- **`sample_ad_model` 离线模型文件**:检测模块在 `data/models/sample_ad_model`
|
||||||
|
不存在时 graceful skip。即便社区镜像支持 ANOMALY_WINDOW,也需先按
|
||||||
|
TDengine 文档将 `.keras`/`.info` 放置到 anode 模型目录。
|
||||||
|
- **GET_FROWTS 区域查询**:当 FORECAST 无返回时(社区版),`SELECT ts, value
|
||||||
|
FROM s_<stable>` 仍会被调用但 `fmap` 为空,所有点 `is_anom=False`。
|
||||||
|
当前为良性,零窗口零异常。
|
||||||
|
|
||||||
|
## 验证结论
|
||||||
|
|
||||||
|
- **DONE_WITH_CONCERNS**:
|
||||||
|
- 容器、连接、schema、注入、可视化、JSON 日志全链路 verified OK。
|
||||||
|
- 算法/预测侧受限于 community 镜像无 TDgpt 服务,0 窗口/0 异常是
|
||||||
|
已知预期,待企业镜像或 taosanode 部署后补做算法侧端到端验证。
|
||||||
|
- 验证过程中暴露 2 个 SQL 兼容性问题(保留字 + 函数参数),已在
|
||||||
|
commit `031aa35` 修复。
|
||||||
|
|
||||||
|
## 仓库
|
||||||
|
|
||||||
|
Gitea: <https://git.tekmine.net/charles/ts-anomaly-td>
|
||||||
Reference in New Issue
Block a user