feat(m2): Projector Linear(256→896)+LayerNorm (T2.2)
- src/tsmm/model/projector.py: Projector (Linear + LayerNorm). - tests/test_projector.py: 5 tests (shape, Qwen hidden alignment, LayerNorm present, gradient flow, no-NaN). - 66 tests passing.
This commit is contained in:
@@ -16,7 +16,7 @@
|
|||||||
## 2. M2 · 模型可跑通
|
## 2. M2 · 模型可跑通
|
||||||
|
|
||||||
- [x] 2.1 TS Encoder `model/ts_encoder.py`:`Patchify(patch=8, stride=4)` 通道独立切 patch 线性嵌入到 d=256;`TSEncoder(d=256, layers=2, heads=4)` 2 层 TF + 可学习位置编码;前向 `[B,T,C]`→`[B,n_patches,256]`(验证:输入 `[2,512,5]`→`[2,127,256]`;spec 澄清:n_patches=(T-P)/S+1=127 非 128;通道独立=每通道共享 patch 线性后对通道 mean-pool;参数 ≈2.1M 非 4M)
|
- [x] 2.1 TS Encoder `model/ts_encoder.py`:`Patchify(patch=8, stride=4)` 通道独立切 patch 线性嵌入到 d=256;`TSEncoder(d=256, layers=2, heads=4)` 2 层 TF + 可学习位置编码;前向 `[B,T,C]`→`[B,n_patches,256]`(验证:输入 `[2,512,5]`→`[2,127,256]`;spec 澄清:n_patches=(T-P)/S+1=127 非 128;通道独立=每通道共享 patch 线性后对通道 mean-pool;参数 ≈2.1M 非 4M)
|
||||||
- [ ] 2.2 Projector `model/projector.py`:`Linear(256→896)+LayerNorm`(验证:输出 `[B,128,896]` 对齐 Qwen2.5-0.5B hidden)
|
- [x] 2.2 Projector `model/projector.py`:`Linear(256→896)+LayerNorm`(验证:输出 `[B,n_patches,896]` 对齐 Qwen2.5-0.5B hidden)
|
||||||
- [ ] 2.3 多模态拼接 `model/multimodal.py`:Qwen tokenizer tokenize 文本部分,TS token 作 inputs_embeds 插入 `[属性][时间戳][TS tok][事件][问题]`,统一构造 inputs_embeds+attention_mask+labels(仅回答段非 -100)(验证:seq_len ≤1024,mask/labels 形状对)
|
- [ ] 2.3 多模态拼接 `model/multimodal.py`:Qwen tokenizer tokenize 文本部分,TS token 作 inputs_embeds 插入 `[属性][时间戳][TS tok][事件][问题]`,统一构造 inputs_embeds+attention_mask+labels(仅回答段非 -100)(验证:seq_len ≤1024,mask/labels 形状对)
|
||||||
- [ ] 2.4 训练/推理封装 `model/wrapper.py`:`MultimodalTSModel` 组合 Encoder+Projector+LLM+LoRA 挂载开关,`forward` 返 loss、`generate` 返文本,支持 `freeze_llm`(阶段①)/`enable_lora(r=16)`(阶段②)(验证:单 batch forward 3060 不 OOM;generate 能出文本)
|
- [ ] 2.4 训练/推理封装 `model/wrapper.py`:`MultimodalTSModel` 组合 Encoder+Projector+LLM+LoRA 挂载开关,`forward` 返 loss、`generate` 返文本,支持 `freeze_llm`(阶段①)/`enable_lora(r=16)`(阶段②)(验证:单 batch forward 3060 不 OOM;generate 能出文本)
|
||||||
- [ ] 2.5 Collator `data/collator.py`:JSONL→batch 张量,处理变长 C(padding+mask)与变长文本(padding+attention_mask)(验证:batch=4 张量形状一致无 NaN)
|
- [ ] 2.5 Collator `data/collator.py`:JSONL→batch 张量,处理变长 C(padding+mask)与变长文本(padding+attention_mask)(验证:batch=4 张量形状一致无 NaN)
|
||||||
|
|||||||
@@ -0,0 +1,22 @@
|
|||||||
|
"""Projector (T2.2): maps TS tokens from encoder dim to LLM hidden dim.
|
||||||
|
|
||||||
|
``Linear(d → h)`` + ``LayerNorm``. Output feeds as soft tokens into the LLM
|
||||||
|
``inputs_embeds`` (see ``model/multimodal.py``, T2.3).
|
||||||
|
|
||||||
|
Design ref: §2.2 — Projector = Linear(256→896) + LayerNorm.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import torch
|
||||||
|
from torch import nn
|
||||||
|
|
||||||
|
|
||||||
|
class Projector(nn.Module):
|
||||||
|
def __init__(self, in_dim: int = 256, out_dim: int = 896) -> None:
|
||||||
|
super().__init__()
|
||||||
|
self.linear = nn.Linear(in_dim, out_dim)
|
||||||
|
self.norm = nn.LayerNorm(out_dim)
|
||||||
|
|
||||||
|
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||||
|
# x: [..., in_dim] → [..., out_dim]
|
||||||
|
return self.norm(self.linear(x))
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
"""Tests for Projector (T2.2): TS token d → LLM hidden."""
|
||||||
|
import torch
|
||||||
|
|
||||||
|
from tsmm.model.projector import Projector
|
||||||
|
|
||||||
|
|
||||||
|
class TestProjector:
|
||||||
|
def test_output_shape(self):
|
||||||
|
proj = Projector(in_dim=256, out_dim=896)
|
||||||
|
x = torch.randn(2, 127, 256)
|
||||||
|
out = proj(x)
|
||||||
|
assert out.shape == (2, 127, 896)
|
||||||
|
|
||||||
|
def test_aligns_qwen_hidden(self):
|
||||||
|
# Qwen2.5-0.5B hidden = 896
|
||||||
|
proj = Projector(in_dim=256, out_dim=896)
|
||||||
|
out = proj(torch.randn(1, 127, 256))
|
||||||
|
assert out.shape[-1] == 896
|
||||||
|
|
||||||
|
def test_has_layernorm(self):
|
||||||
|
proj = Projector(in_dim=256, out_dim=896)
|
||||||
|
assert any(isinstance(m, torch.nn.LayerNorm) for m in proj.modules())
|
||||||
|
|
||||||
|
def test_gradient_flows(self):
|
||||||
|
proj = Projector(in_dim=256, out_dim=896)
|
||||||
|
x = torch.randn(1, 31, 256)
|
||||||
|
out = proj(x)
|
||||||
|
out.sum().backward()
|
||||||
|
assert proj.linear.weight.grad is not None
|
||||||
|
assert proj.linear.weight.grad.abs().sum() > 0
|
||||||
|
|
||||||
|
def test_no_nan(self):
|
||||||
|
proj = Projector(in_dim=256, out_dim=896)
|
||||||
|
out = proj(torch.randn(4, 128, 256))
|
||||||
|
assert not torch.isnan(out).any()
|
||||||
Reference in New Issue
Block a user