GitHub - FeSens/openTPU：由 AI 开发的开源 AI 加速器——集 RTL、ISA、模拟器、编译器和剖析器于一体。在 Kintex-7 PCIe 卡上运行 Qwen3、LFM2.5 和 Qwen3.5。

一种由 AI 开发的开源 AI 加速器。An open-source AI accelerator, developed by AI.

openTPU 将 auto-arch-tournament 的经验应用于 AI 加速器。它提出了两个问题：AI 智能体在硬件设计方面能走多远？它们能否构建出运行自身推理的芯片？openTPU brings the lessons of auto-arch-tournament to AI accelerators. It asks two questions: how far can AI agents go at hardware design, and can they build the chip that runs their own inference?

otpu-chat 正在 FPGA 卡上运行 LFM2.5-230M（左），otpu-smi 显示该卡的利用率和 DRAM 带宽（右）。otpu-chat running LFM2.5-230M on the FPGA card (left), with otpu-smi showing the card's utilization and DRAM bandwidth (right).

openTPU 也是一个学习项目。整个加速器都包含在一个小型单体仓库中，你可以从头到尾通读其中的内容：硬件设计（SystemVerilog）、指令集、位精确模拟器、内核语言及其编译器，以及驱动真实 PCIe 卡的主机软件。如果你想了解 AI 加速器如何工作，从 Python 中的矩阵乘法一直深入到硬件连线，这里是个不错的起点。openTPU is also a learning project. The whole accelerator lives in one small monorepo that you can read end to end: the hardware design (SystemVerilog), the instruction set, a bit-exact simulator, a kernel language and its compiler, and the host software that drives a real PCIe card. If you want to understand how an AI accelerator works, from a matmul in Python down to the wires, this is a good place to start.

该设计在 Inspur YPCB-00338 卡（Xilinx Kintex-7 xc7k480t、两个 DDR3 通道）上运行了十个采用真实权重的现代模型，并且该卡生成的 token 与模拟器完全一致，逐位匹配。The design runs ten modern models with their real weights on an Inspur YPCB-00338 card (Xilinx Kintex-7 xc7k480t, two DDR3 channels), and the card produces the same tokens as the simulator, bit for bit.

Measured on the card: the first three models on 2026-09-29 with the production image deploy_champ_e698dcd7. LFM2-2.6B, SmolLM3-3B and Phi-4-mini on 2026-09-30, and Qwen3.5-2B and 4B and Gemma 4 on 2026-10-01, with build B, deploy_fused133c_79c5707a, production since then. Build B decodes LFM2-2.6B, SmolLM3 and Phi-4-mini 8-9% faster than e698dcd7 (Gemma 4 E2B 10%), at 91-94% of the DRAM peak instead of 82-87%. Qwen3.5-4B's int8 image is over 4 GiB.

• 镜像：主线 e698dcd，所有型号共用一个比特流。它配备由内存核心内部的一颗小型 CPU 完成校准的 LiteDRAM 控制器、四列脉动矩阵单元，以及流引擎（docs/stream.md）。DDR3-1066，峰值带宽为 17.1 GB/s。
  The image: main e698dcd at 133.33 MHz, one bitstream for all models. It has LiteDRAM controllers calibrated by a small CPU inside the memory core, a four-column systolic matrix unit and the stream engine (docs/stream.md). DDR3-1066, with a 17.1 GB/s peak.
• The host: the card sits in opentpu (Intel Core i7-4790).
• 方法，tools/qual/perf.py：decode 是在 512 个 token 的提示词之后进行 64 个贪心 token 解码，循环中由主机执行 argmax（非流式）。“Device”仅统计加速器运行的周期；“wall”还包括主机耗时。Prefill 是 512 个 token 的提示词，在设备上运行。
  Method, tools/qual/perf.py: decode is 64 greedy tokens after a 512-token prompt, with the host's argmax in the loop (not streamed). "Device" counts only the cycles the accelerator runs; "wall" adds the host. Prefill is the 512-token prompt, on the device.
• DRAM 流量来自显卡运行时自身的计数器。
  DRAM traffic comes from the card's own counters while it runs.
• Gemma 4 E2B 将其逐层嵌入表保留在卡上（镜像大小为 3.5–3.6 GiB；见 docs/gemma4.md）；使用 int8 时无法装下。使用任一输出头时，它在三个提示词上的贪心生成 token 均与 Hugging Face 一致。E4B 的嵌入表（2.95 GB）留在主机上，每生成一个 token，就将一行 11 KB 的数据复制到卡上；其镜像大小为 3.96 GiB，使用 int8，输出头和前 24 层的下投影则采用 4-bit（见 docs/gemma4_e4b.md）。在卡自身的解码循环中（由卡选出每个 token），Gemma 4 解码更快：E2B 为 11.01 / 12.73 tok/s（int8 / 4-bit 输出头），E4B 为 3.83 tok/s，均在设备上测得。
  Gemma 4 E2B keeps its per-layer embedding tables on the card (3.5-3.6 GiB images; docs/gemma4.md); in int8 it does not fit. It matches Hugging Face's greedy tokens on three prompts with either head. E4B's table (2.95 GB) stays on the host, which copies one 11 KB row into the card per token; its image is 3.96 GiB, int8 with the head and the first 24 layers' down projections in 4-bit (docs/gemma4_e4b.md). In the card's own decode loop (the card picking every token) Gemma 4 decodes faster: E2B 11.01 / 12.73 tok/s (int8 / 4-bit head), E4B 3.83 tok/s, on the device.
• 每种配置都与模拟器逐个 token、按位置完全匹配，并与驻留解码程序相符。更多详情请参阅 docs/board.md。
  Every configuration matches the simulator token for token, per-position and with the resident decode program. More detail in docs/board.md.

在运行设备的同时流式传回 logits（tools/decode_profile.py，96 个 token）时，4-bit 解码速度更快，设备端/墙钟 tok/s：With the logits streamed back while the card runs (tools/decode_profile.py, 96 tokens), 4-bit decode is faster, in device / wall tok/s:

• LFM2：89.5 / 84.5；
  LFM2: 89.5 / 84.5;
• Qwen3：33.7 / 33.3；
  Qwen3: 33.7 / 33.3;
• Qwen3.5: 24.6 / 24.2;
• LFM2-2.6B: 11.07 / 11.02 (build B);
• SmolLM3-3B: 8.92 / 8.89 (build B);
• Phi-4-mini: 6.69 / 6.67 (build B).

之前的生产映像 se-cand3 使用 Xilinx MIG、双列矩阵单元和 120.755 MHz 时钟构建。按照相同方式测量，新映像：The previous production image, se-cand3, was built with the Xilinx MIG, a two-column matrix unit and a 120.755 MHz clock. Measured the same way, the new image:

• 解码：在所有配置下均与 se-cand3 相差不超过 2.3%。解码受 DRAM 限制，LiteDRAM 的读取速度达到 DDR3 峰值的 82-85%，与 MIG 一样。
  decode: within 2.3% of se-cand3's in every configuration. Decode is bound by DRAM, and LiteDRAM reads at 82-85% of the DDR3 peak, as the MIG did.
• prefill: 1.3x (Qwen3.5) to 2.0x (LFM2 4-bit) faster.
• 校准：图像启动时，核心的 CPU 会在 12 秒内完成两个 DDR3 通道的校准，无需主机参与。
  calibration: when the image starts, the core's CPU calibrates both DDR3 channels in 12 s, with no host involvement.

之前的图像及其编号位于 docs/board.md 的第 5 节。The earlier images and their numbers are in docs/board.md, section 5.

大于显卡 4 GiB 容量的混合专家模型，会将专家从主机存储流式传入（docs/offload.md，第 10 节）。显卡负责为每个 token 路由并计算所有专家，并将专家保存在其 DRAM 中按层划分的槽位里。主机只会以链路速率将缺失的专家从池文件复制到这些槽位中（第 10.1 节）。以下数据于 2026-10-01 使用构建版本 B（79c5707a）测得：由显卡自身的解码循环为每个 token 选择专家，专家使用 4 位量化，输出头使用 int8：Mixture-of-experts models bigger than the card's 4 GiB run with their experts streamed from host storage (docs/offload.md, section 10). The card routes each token and computes every expert, and it keeps the experts in per-layer slots in its DRAM. The host only copies missing experts from a pool file into those slots, at the link's rate (section 10.1). Measured on 2026-10-01 with build B (79c5707a), the card's own decode loop picking every token, 4-bit experts, int8 head:

• LFM2.5-8B-A1B（85 亿参数，17 亿激活参数）：处理 160 个 token 时速度为 10.6 tok/s。98.5% 的专家调用命中了槽位，每个 token 流式传输了 5.2 MB。
  LFM2.5-8B-A1B (8.5B parameters, 1.7B active): 10.6 tok/s over 160 tokens. 98.5% of expert uses hit the slots, and 5.2 MB streamed per token.
• Qwen3.5-35B-A3B（34.7B 参数，3.0B 激活参数）：使用 Hugging Face 的 16 个贪心 token 时，速度为 3.95 tok/s。专家使用命中率为 62%，每个 token 通过 PCIe 以 1.41 GB/s 的速率传输 153 MB（第 10.3 节）。
  Qwen3.5-35B-A3B (34.7B parameters, 3.0B active): 3.95 tok/s, with Hugging Face's 16 greedy tokens. 62% of expert uses hit, and 153 MB streamed per token at 1.41 GB/s over PCIe (section 10.3).
• 两者都与模拟器逐位匹配。
  Both match the simulator bit for bit.

4 位权重（docs/quant.md）使用 FP4 值和两级块缩放，每个权重占 4.25 位，并将 LM head 保持为 int8 以确保准确性。它们将每个 token 的字节数减少约三分之一，并将解码速度提升 40%（Qwen3.5）至 45%（Qwen3、LFM2），但困惑度会出现可测量的上升；docs/quant.md 中报告了各模型的具体情况。4-bit weights (docs/quant.md) use FP4 values with two-level block scales, 4.25 bits per weight, and keep the LM head in int8 for accuracy. They cut the bytes per token by about a third and raise decode speed by 40% (Qwen3.5) to 45% (Qwen3, LFM2), at a measurable cost in perplexity that docs/quant.md reports per model.

主机几乎已经不再构成瓶颈。对于 LFM2 和 Qwen3，卡上运行的是一个只编译一次的解码程序：它从寄存器读取位置，并查找对应的 embedding 和 RoPE 行；卡仍在运行时，logits 就会流回主机。在 omarchy 上，主机每个 token 会增加 0.17 到 0.30 毫秒的开销（在 opentpu 上则为 0.45 到 1.3 毫秒）。Qwen3.5 的解码也采用相同方式；但其 prefill 仍由主机提前为每个分块编译程序，再交由卡执行。The host is nearly out of the way. For LFM2 and Qwen3 the card runs one decode program compiled once, which reads the position from a register and looks up its own embedding and RoPE rows, and the logits stream back while the card is still running: the host adds 0.17 to 0.30 ms per token on omarchy (0.45 to 1.3 ms on opentpu). Qwen3.5 runs the same way for decode; its prefill still compiles each chunk's program on the host, ahead of the card.

  Kernels in ol              mlp, attention, full model layers
        |  @ol.jit
  Language + compiler        layouts, affine loop addressing, fusion
        |
  ISA                        8 x 32-bit words per instruction
        |
  ISA simulator  <======>  RTL          same bits, checked by the tests
  (Python)                 (SystemVerilog)
                            |  Vivado bitstream
                           FPGA card    Kintex-7 xc7k480t
                            |  PCIe
                           Host         otpu-chat, otpu-smi, otpu-lens

这台机器刻意设计得很简单。一个序列器每个周期向几个单元发出一条指令：DMA 负责移动数据，矩阵单元对从 DRAM 流入的 int8 权重进行乘法运算，向量单元执行 fp32 运算，量化器则将结果转换回 int8。这里没有缓存，也没有隐藏的调度机制：每次数据移动都是一条指令，因此跟踪记录会准确显示周期都花在了哪里。docs/isa.md 描述了完整的指令集。The machine is deliberately simple. A sequencer issues one instruction per cycle to a few units: DMA moves data, the matrix unit multiplies int8 weights streamed from DRAM, the vector unit does fp32 math, and a quantizer turns results back into int8. There is no cache and no hidden scheduling: every data movement is an instruction, so a trace shows exactly where the cycles go. docs/isa.md describes the whole instruction set.

内核如下所示：A kernel looks like this:

from opentpu import language as ol

@ol.jit
def mlp(h, gamma, w_gate, w_up, w_down, out, eps):   # simplified; see kernels/mlp.py
    x = ol.load(h)
    xs = ol.quantize(rmsnorm(x, ol.load(gamma), eps))
    g = ol.dot(xs, w_gate)
    u = ol.dot(xs, w_up)
    a = ol.all_gather(silu(g) * u)
    y = ol.all_gather(ol.dot(a, w_down))
    if ol.program_id() == 0:
        ol.store(out, x + y)

由于每一次数据移动都是一条指令，因此运行跟踪记录可以解释其速度。性能分析器 Lens 可从 RTL、模拟器或加速卡记录一次运行，并在浏览器中打开记录，展示屋顶线图、时间线和逐条指令表格（docs/lens.md）。Because every data movement is an instruction, a trace of a run explains its speed. Lens, the profiler, records a run from the RTL, the simulator or the card and opens it in the browser, with a roofline, a timeline and per-instruction tables (docs/lens.md).

Lens 正在重放 Qwen3 解码步骤的一部分。颜色表示每个周期中各个单元的状态：忙碌、等待 DRAM，或等待另一条指令。Lens replaying part of a Qwen3 decode step. Colours show what each unit is doing in each cycle: busy, waiting on DRAM, or waiting on another instruction.

除了卡片之外，所有内容都在笔记本电脑上运行。Everything except the card runs on a laptop.

pip install -e .
pip install pytest torch transformers
python3 -m pytest -q          # RTL tests also need Verilator 5

hf download LiquidAI/LFM2.5-230M --local-dir models/LFM2.5-230M
otpu-chat --model lfm2 --backend isa    # chat on the simulator

使用卡片构建比特流（在 boards/ypcb-00338 中执行 make bit），通过 JTAG 加载该比特流，然后运行 sudo otpu-setup 和 otpu-chat --backend board。docs/board.md 详细介绍了启动流程。With a card, build the bitstream (make bit in boards/ypcb-00338), load it over JTAG, then run sudo otpu-setup and otpu-chat --backend board. docs/board.md walks through the bring-up.

1. docs/isa.md：指令集。其他一切都以此为基础。
   docs/isa.md: the instruction set. Everything else is built on it.
2. opentpu/kernels and docs/compiler.md: how a kernel becomes instructions.
3. opentpu/isasim.py：模拟器，也就是规范。
   opentpu/isasim.py: the simulator, which is the spec.
4. rtl/: the hardware, starting from rtl/top/otpu_top.sv.
5. docs/lfm2.md, docs/qwen35.md, docs/llama.md, docs/benchmarks.md: whole models and where their cycles go.
6. docs/board.md: the physical card, from clocks to PCIe.

• DRAM 性能的最后几个百分点。解码受 DRAM 效率限制：读取速度达到 DDR3-1066 峰值的 82% 至 85%。提高 LiteDRAM 路径效率的工作正在进行中。
  The last few percent of DRAM. Decode is bound by DRAM efficiency: it reads 82 to 85% of the DDR3-1066 peak. Work on the LiteDRAM path's efficiency is under way.
• 时序裕量与面积。设计达到 133.33 MHz；在这一时钟频率下，128 字节端口恰好与两个 DDR3 通道匹配，但余量很小（WNS +0.032 ns）。多轮 Vivado 运行持续优化时序裕量和面积。解码受 DRAM 限制，因此提高时钟频率主要有助于预填充。
  Timing margin and area. The design closes 133.33 MHz, the clock at which the 128-byte port matches the two DDR3 channels, but only just (WNS +0.032 ns). A tournament of Vivado runs keeps working on its margin and area. Decode is bound by DRAM, so a faster clock mostly helps prefill.
• 更快的预填充。四列脉动矩阵单元已包含在生产镜像中；预填充仍受矩阵单元乘法速率的限制。
  Faster prefill. The four-column systolic matrix unit is in the production image; prefill is still limited by the matrix unit's multiply rate.

欢迎提交问题反馈和拉取请求。大部分工作只需要 Python 和 Verilator，无需 FPGA。对 ISA、模拟器或 RTL 的更改必须确保 `python3 -m pytest -q` 测试通过；关于性能的说法应说明测量方法。Issues and pull requests are welcome, and most of the work needs only Python and Verilator, not an FPGA. Changes to the ISA, the simulator or the RTL must keep python3 -m pytest -q passing, and performance claims should say how they were measured.

Apache 2.0 许可证。请参阅 LICENSE。Apache License 2.0. See LICENSE.