feat(m2): multimodal splice [属性][时间戳][TS tok][事件][问题][回答] (T2.3)
- src/tsmm/model/multimodal.py: MultimodalSplicer — binds LLM input embedding layer, assembles text embeds (via tokenizer) + TS token embeds (from Projector) into inputs_embeds/attention_mask/labels; labels = -100 everywhere except answer+EOS (training); left-truncation to max_len=1024. - tests/test_multimodal.py: 8 tests against real Qwen2.5-0.5B tokenizer + stub embedding (shapes, all-ones mask, answer-only labels, TS token count in sequence, no-NaN, truncation). 74 tests passing. - configs/llm.yaml: offline model path (downloaded via proxy to ~/models/Qwen2.5-0.5B-Instruct; HF hub unreachable on host). Spec clarifications (small tier, comet-build Step 4): - Training appends answer + EOS to the prompt layout. - Build embedding layer with len(tokenizer) (incl. added special tokens), not tokenizer.vocab_size which excludes them (eos=151645).
This commit is contained in:
@@ -17,7 +17,7 @@
|
||||
|
||||
- [x] 2.1 TS Encoder `model/ts_encoder.py`:`Patchify(patch=8, stride=4)` 通道独立切 patch 线性嵌入到 d=256;`TSEncoder(d=256, layers=2, heads=4)` 2 层 TF + 可学习位置编码;前向 `[B,T,C]`→`[B,n_patches,256]`(验证:输入 `[2,512,5]`→`[2,127,256]`;spec 澄清:n_patches=(T-P)/S+1=127 非 128;通道独立=每通道共享 patch 线性后对通道 mean-pool;参数 ≈2.1M 非 4M)
|
||||
- [x] 2.2 Projector `model/projector.py`:`Linear(256→896)+LayerNorm`(验证:输出 `[B,n_patches,896]` 对齐 Qwen2.5-0.5B hidden)
|
||||
- [ ] 2.3 多模态拼接 `model/multimodal.py`:Qwen tokenizer tokenize 文本部分,TS token 作 inputs_embeds 插入 `[属性][时间戳][TS tok][事件][问题]`,统一构造 inputs_embeds+attention_mask+labels(仅回答段非 -100)(验证:seq_len ≤1024,mask/labels 形状对)
|
||||
- [x] 2.3 多模态拼接 `model/multimodal.py`:Qwen tokenizer tokenize 文本部分,TS token 作 inputs_embeds 插入 `[属性][时间戳][TS tok][事件][问题][回答][EOS]`,统一构造 inputs_embeds+attention_mask+labels(仅回答段非 -100)(验证:8 项单测过——seq_len ≤1024、mask 全 1、回答段 label 非 -100、TS token 计入序列、无 NaN、超长左截断)。spec 澄清:训练拼回答+EOS,用 `len(tokenizer)` 构建嵌入表以容纳 special tokens。
|
||||
- [ ] 2.4 训练/推理封装 `model/wrapper.py`:`MultimodalTSModel` 组合 Encoder+Projector+LLM+LoRA 挂载开关,`forward` 返 loss、`generate` 返文本,支持 `freeze_llm`(阶段①)/`enable_lora(r=16)`(阶段②)(验证:单 batch forward 3060 不 OOM;generate 能出文本)
|
||||
- [ ] 2.5 Collator `data/collator.py`:JSONL→batch 张量,处理变长 C(padding+mask)与变长文本(padding+attention_mask)(验证:batch=4 张量形状一致无 NaN)
|
||||
- [ ] 2.6 M2 出口验证:单 batch 前向+反向在 3060 跑通,loss 有限且下降趋势,阶段①模式峰值显存 ~5GB
|
||||
|
||||
Reference in New Issue
Block a user