feat(m2): MultimodalTSModel wrapper — end-to-end fwd/generate + LoRA (T2.4)

- src/tsmm/model/wrapper.py: MultimodalTSModel combining TS Encoder + Projector
  + Qwen2.5-0.5B LLM + LoRA. Two modes: freeze_llm() (stage ①) and
  enable_lora(r=16) (stage ②, peft on q/v_proj). forward() takes a raw batch
  {series, attributes, timestamps, events, question, answer} end-to-end
  (encoder→projector→splice→LLM) OR pre-spliced inputs_embeds.
- src/tsmm/model/multimodal.py: device-aware splice (move ids/ts_embeds to
  the bound embedding layer's device).
- tests/test_wrapper.py: 6 tests — finite loss, grad flows into
  encoder/projector under freeze_llm, generate returns text, freeze_llm and
  enable_lora mode invariants, GPU memory budget. 80 tests passing.
- Measured: stage ① fwd+bwd peak 1.87 GB (design budget ~5 GB).

Spec clarifications (small tier, comet-build Step 4):
- Wrapper batch contract = {series, attributes, timestamps, events, question,
  answer} (lists of str + series tensor). Collator (T2.5) will produce this.
- Encoder/Projector kept fp32; their output cast to LLM dtype (bf16) before
  splice, so inputs_embeds matches LLM weights.
This commit is contained in:
张宗平
2026-06-30 02:11:36 +00:00
parent f509d47878
commit aec8fd5ef1
4 changed files with 333 additions and 3 deletions
+1 -1
View File
@@ -18,7 +18,7 @@
- [x] 2.1 TS Encoder `model/ts_encoder.py``Patchify(patch=8, stride=4)` 通道独立切 patch 线性嵌入到 d=256`TSEncoder(d=256, layers=2, heads=4)` 2 层 TF + 可学习位置编码;前向 `[B,T,C]``[B,n_patches,256]`(验证:输入 `[2,512,5]``[2,127,256]`spec 澄清:n_patches=(T-P)/S+1=127 非 128;通道独立=每通道共享 patch 线性后对通道 mean-pool;参数 ≈2.1M 非 4M
- [x] 2.2 Projector `model/projector.py``Linear(256→896)+LayerNorm`(验证:输出 `[B,n_patches,896]` 对齐 Qwen2.5-0.5B hidden
- [x] 2.3 多模态拼接 `model/multimodal.py`Qwen tokenizer tokenize 文本部分,TS token 作 inputs_embeds 插入 `[属性][时间戳][TS tok][事件][问题][回答][EOS]`,统一构造 inputs_embeds+attention_mask+labels(仅回答段非 -100)(验证:8 项单测过——seq_len ≤1024、mask 全 1、回答段 label 非 -100、TS token 计入序列、无 NaN、超长左截断)。spec 澄清:训练拼回答+EOS,用 `len(tokenizer)` 构建嵌入表以容纳 special tokens。
- [ ] 2.4 训练/推理封装 `model/wrapper.py``MultimodalTSModel` 组合 Encoder+Projector+LLM+LoRA 挂载开关,`forward` 返 loss、`generate` 返文本,支持 `freeze_llm`(阶段①)/`enable_lora(r=16)`(阶段②)(验证:单 batch forward 3060 不 OOMgenerate 出文本
- [x] 2.4 训练/推理封装 `model/wrapper.py``MultimodalTSModel` 组合 Encoder+Projector+LLM+LoRA 挂载开关,`forward` 返 loss、`generate` 返文本,支持 `freeze_llm`(阶段①)/`enable_lora(r=16)`(阶段②)(验证:端到端 forward+backward 跑通、grad 流入 Encoder/Projector、generate 出文本、阶段①峰值 1.87GB ≪5GB 设计预算)。spec 澄清:wrapper 既接收原始 batch(series+文本列表)走端到端,也兼容预拼接 inputs_embedsencoder/projector fp32,输出投影到 LLM dtype(bf16)。
- [ ] 2.5 Collator `data/collator.py`JSONL→batch 张量,处理变长 Cpadding+mask)与变长文本(padding+attention_mask)(验证:batch=4 张量形状一致无 NaN
- [ ] 2.6 M2 出口验证:单 batch 前向+反向在 3060 跑通,loss 有限且下降趋势,阶段①模式峰值显存 ~5GB