In the fiercely contested race to break NVIDIA’s datacenter monopoly on AI inference, two revolutionary silicon startups led by legendary semiconductor architects have abandoned traditional GPU architectures entirely: Tenstorrent, spearheaded by microprocessor icon Jim Keller, and Groq, founded by former Google TPU architect Jonathan Ross.

Both companies argue that GPUs—which were originally engineered to render 3D polygon graphics for video games—are fundamentally ill-suited for transformer inference workloads. GPUs waste immense silicon die area on graphics texture mappers, complex dynamic schedulers, and high-latency DRAM memory controllers. To eliminate these bottlenecks, both Tenstorrent and Groq engineered custom silicon around a radical architectural principle: massive on-chip Static RAM (SRAM) and completely deterministic hardware scheduling.

Yet while both architectures eliminate external DRAM latency, their execution models represent two opposing hardware philosophies. Groq’s Language Processing Unit (LPU) executes pure, single-instruction synchronous tensor pipelines across a massive 230MB on-chip SRAM pool, completely eliminating runtime instruction caches and branch predictors. Tenstorrent’s Tensix Core architecture (Wormhole and Blackhole) combines custom RISC-V compute cores with Network-on-Chip (NoC) mesh interconnects, providing open-source compiler flexibility and modular scaling. Analyzing the TCO of these next-generation accelerators reveals the future of non-NVIDIA enterprise AI.

What Is the Difference Between Tenstorrent and Groq LPU?

The SRAM Advantage: 80 Terabytes/sec Bandwidth vs HBM3 Latency

To appreciate why Groq achieves 500+ tokens per second on open-weights language models, we must compare the physical access latency of High Bandwidth Memory (HBM3) versus On-Chip Static RAM (SRAM):

• NVIDIA H100 (HBM3 Architecture): An H100 packages 80GB of HBM3 memory stacked adjacent to the GPU die via a silicon interposer, delivering approximately 3.35 TB/s of memory bandwidth. However, fetching data out of external HBM3 requires approximately 150 to 200 clock cycles of memory controller latency.
• Groq LPU (Pure On-Die SRAM): The Groq LPU contains zero external memory controllers. It features 230 megabytes of true on-die SRAM interwoven directly between matrix execution units. Because data resides on the physical silicon die less than a millimeter from the ALUs, memory bandwidth reaches a staggering 80 terabytes per second with an access latency of just 1 to 3 clock cycles.

The Costly Trap: The Groq Chip Count Explosion

The Achilles’ heel of Groq’s pure SRAM architecture is density. SRAM cells are physically massive; a single SRAM bit requires six transistors (6T), compared to a single transistor and capacitor (1T-1C) in standard DRAM. As a result, fitting 230 megabytes of SRAM onto a Groq chip consumes almost the entire silicon die area.

Consider what this means when deploying a 70-billion parameter language model (such as Llama 3 70B in 8-bit precision = ~75 gigabytes):

1. Because each Groq LPU holds only 230 MB of SRAM, a 75 GB model cannot fit onto one chip, nor can it fit onto ten chips.
2. It requires a minimum cluster of 300 to 576 interconnected Groq LPUs across multiple server racks, connected via high-speed optical links, purely to hold the model weights in memory!
3. While that 500-chip cluster generates text at an astonishing 300 tokens per second, the capital hardware acquisition cost exceeds $1.5 million.

Tenstorrent takes a pragmatic hybrid approach: the Wormhole n300 pairs 104 MB of ultra-fast on-chip SRAM with 24 gigabytes of high-density GDDR6 memory on a standard dual-slot PCIe card for just $1,399. You can run 7B and 13B models locally inside a standard workstation without requiring a dedicated multi-rack data center.

Compiler Co-Design: Why Open Toolchains Beat Proprietary Stacks

The history of computing is littered with brilliant, high-performance silicon architectures that failed commercially because their software compilers were proprietary, fragile, and abandoned. Silicon is only as good as the software toolchain that translates PyTorch code into hardware instructions.

This is where Tenstorrent’s open-source philosophy stands in stark contrast to Groq:

• Tenstorrent TT-Metalium Open-Source Toolchain: Tenstorrent has open-sourced its entire low-level software stack on GitHub. TT-Metalium provides direct C++ APIs allowing developers to write custom kernels for the Tensix cores and RISC-V processors, bypassing high-level abstraction layers. Higher up the stack, TT-Buda and TT-Forge ingest standard PyTorch and ONNX models, automatically partitioning tensor graphs across the 2D Network-on-Chip. Because the toolchain is open, researchers can inspect, modify, and optimize compiler passes for novel model architectures.
• Groq Deterministic Compiler Complexity: Groq’s hardware has zero runtime dynamic schedulers; the Groq Compiler must determine the exact clock cycle when every single byte travels across the chip. While this delivers unparalleled latency predictability, writing compiler passes that map arbitrary, dynamic neural network layers into Groq’s synchronous spatial pipeline is an immensely complex engineering task, creating software friction for non-standard transformer architectures.

The 4:00 PM Resolution: Choosing Your Next-Gen Accelerator

Related analysis: For next-generation PC NPU comparisons, explore our benchmark test of Intel Lunar Lake NPU vs AMD Strix Point XDNA 2.