The Von Neumann Bottleneck: Arithmetic Logic vs Memory Movement Energy

For more than half a century, microprocessor computing was governed by the classic Von Neumann architecture: a central processing unit (CPU) containing arithmetic logic units (ALUs) and registers, connected via a shared bus to memory. In general-purpose computation, this architecture is brilliantly flexible. However, when applied to deep learning neural networks, the Von Neumann architecture collapses into an energetic catastrophe.

Deep neural network inference—whether convolutional vision networks or transformer language models—is essentially an astronomical quantity of dense matrix-matrix multiplications ($C = A imes B$) composed of repetitive Multiply-Accumulate ($MAC$) operations ($y = a \cdot b + c$). In an 8-bit integer ($INT8$) computation, the physical arithmetic operation of multiplying two numbers and adding the result requires approximately 0.1 to 0.2 picojoules ($pJ$) of electrical energy on an advanced 5nm semiconductor process node.

However, fetching those two 8-bit numbers from an on-chip SRAM cache costs approximately 5 picojoules—twenty-five times more energy than the arithmetic itself. Worse, fetching those operands from external LPDDR5 or HBM DRAM memory consumes between 100 and 200 picojoules—a thousand-fold energy penalty! When silicon designers attempt to scale AI performance simply by packing wider vector SIMD execution units into a traditional CPU or GPU pipeline, the chip spends virtually all its thermal budget moving bytes through wires rather than performing mathematical inference. This is the Memory Wall.

Architectural Foundations: How 1D SIMD Vector Engines Work

Single Instruction, Multiple Data (SIMD) has been the dominant paradigm for parallel compute since the 1990s, evolving through Intel AVX-512, Arm Neon, and modern GPU Streaming Multiprocessors (SMs). In a SIMD architecture, a single control unit decodes one instruction and dispatches it simultaneously to a 1-dimensional array of parallel functional units.

Consider calculating a 16-element dot product on a SIMD vector pipeline:

• The vector controller issues a load instruction to read Vector $A$ from cache into a vector register (e.g., a 512-bit `zmm` register).
• A second load instruction reads Vector $B$ from cache into another vector register.
• The SIMD ALU array executes sixteen parallel multiply operations.
• The resulting vector is written back to a destination register file.
• An accumulation reduction tree adds the elements together, requiring successive internal register transfers.

While SIMD provides massive speedups over scalar execution for 1D signal processing (such as audio filtering or video pixel color adjustments), it fails during 2D matrix multiplication because operands are not spatially reused across adjacent execution units. In matrix multiplication, every row of Matrix $A$ must be multiplied by every column of Matrix $B$. In a SIMD engine, identical weight values must be read from the register file over and over again for each column calculation, saturating register file read ports and dissipating massive dynamic capacitance energy.

The Systolic Principle: The Cardiovascular Analogy in Silicon

The term “systolic” was coined by computer scientists H.T. Kung and Charles Leiserson in their seminal 1978 paper. The name is a direct biological analogy to the human cardiovascular system: just as the biological heart rhythmically pumps blood cells through arterial networks where tissues continuously extract oxygen at each capillary stage, a systolic array rhythmically pumps data words through a 2D grid of Processing Elements (PEs) where each cell performs computation and passes the data forward on each clock tick.

In a 2D systolic array, each Processing Element is a compact, autonomous hardware cell containing an INT8 or FP16 multiplier, an accumulator register, and pipeline registers linked directly to its immediate north, south, east, and west neighbors. There is no complex decoding logic, no branch prediction, and no centralized register file access during computation.

Data enters exclusively through the boundary edges of the grid (typically activations enter from the left, while weights or partial sums enter from the top). On every clock pulse, data steps forward by exactly one cell. As activations and weights traverse the grid, they interact inside each PE, multiplying and accumulating on-the-fly. By the time data flows out the opposite edge of the array, the complete matrix product has been computed—without a single intermediate read or write to an external register file!

Dataflow Paradigms: Weight Stationary vs Output Stationary vs Input Stationary

The physical efficiency of a systolic array depends heavily on its programmed dataflow architecture. Silicon architects categorize systolic engines based on which matrix operand remains frozen inside the PE registers while the other operands move:

• Weight Stationary (WS) — The Google TPU Architecture: In a Weight Stationary array (utilized in Google TPU v1 through v5p), model weights are loaded once from memory and pre-loaded directly into local registers inside each Processing Element, where they remain frozen (stationary). Streaming activation inputs flow horizontally from left to right, while partial sums accumulate vertically from top to bottom. WS is extraordinarily efficient for large convolutional layers and batch processing where millions of input activations reuse the exact same static weights.
• Output Stationary (OS) — Hailo & Tenstorrent Implementations: In an Output Stationary dataflow, the accumulator register inside each PE remains stationary, holding the running partial sum for a specific coordinate of output Matrix $C$. Activations stream continuously from the west, while weights stream continuously from the north. The PE multiplies the passing inputs and accumulates locally until the final dot product is complete, at which point the finished result is shifted out of the array. OS eliminates the energy required to constantly shift partial sums across the array, drastically reducing wire capacitance.
• Input / Activation Stationary (IS): Activations remain stationary inside PEs while weights and partial sums stream through. This is widely used in streaming audio processing and recurrent neural networks where activation vectors are updated iteratively.

Silicon Energy Forensics: Register File Wire Capacitance & Thermal Footprint

To quantify why systolic arrays dominate edge AI accelerators like the Hailo-8 (26 TOPS at 2.5W) or Google Coral (4 TOPS at 2W), we must analyze the physics of wire capacitance.

In standard CMOS silicon, dynamic energy consumption is governed by the classic equation:

Where $C$ is physical wire and gate capacitance, and $V$ is supply voltage. In a conventional GPU or SIMD CPU, routing an operand from a central register file to a functional unit requires driving electrical signals across long metal interconnect traces (often measuring several millimeters across the die). These long wires possess high parasitic capacitance ($C$), dissipating significant energy on every clock cycle.

In a 2D systolic array, the wire connecting Processing Element $(i, j)$ to its adjacent neighbor $(i+1, j)$ is microscopic—measuring only a few micrometers in length. By confining data movement to ultra-short local interconnects, the physical capacitance $C$ drops by two orders of magnitude. This enables systolic arrays to achieve energy efficiencies between 0.15 and 0.40 pJ/MAC, compared to 1.5 to 4.0 pJ/MAC for high-end desktop GPU tensor cores.

The Structural Trade-Off: Matrix Density vs Algorithmic Rigidity

If systolic arrays are so extraordinarily energy-efficient, why haven’t they completely replaced general-purpose CPUs and GPUs? The answer lies in algorithmic rigidity and utilization efficiency.

A 2D systolic array is a specialized mathematical ASIC. It excels when the dimensions of the input matrices perfectly match or evenly divide into the dimensions of the physical silicon array (e.g., multiplying a 256×256 matrix on a 128×128 systolic grid). However, when modern neural networks deploy sparse matrices (where 70% of weights are zero), dynamic attention masks with variable sequence lengths, or non-matrix operations (such as FlashAttention online softmax reductions), large portions of the systolic array sit completely unutilized—consuming static leakage power while contributing zero compute.

Modern heterogeneous edge processors—such as the Apple M4 Neural Engine, Tenstorrent Wormhole, and Snapdragon X Elite—resolve this dilemma by combining both paradigms: pairing a dense 2D systolic array for raw matrix-matrix multiplication ($GEMM$) with a flexible wide SIMD vector engine (HVX/Neon) to execute non-linear activations, layer normalizations, and dynamic softmax calculations.

The Roofline Model: Arithmetic Intensity & SRAM Caching Thresholds

To mathematically determine whether a deep learning model executes at peak silicon throughput or stalls on memory access, semiconductor architects plot performance on the Williams Roofline Model. The Roofline model graphs computational throughput ($TOPS$) against Arithmetic Intensity ($I$), defined as the ratio of arithmetic operations performed per byte of data transferred from memory:

Every edge silicon accelerator has a critical operational saturation point, $I_{sat} = ext{Peak Compute} / ext{Memory Bandwidth}$. For example, an edge NPU rated at 26 TOPS with a modest 16 GB/s LPDDR4x interface requires an arithmetic intensity of $26,000 / 16 pprox 1,625 ext{ Operations per Byte}$ to achieve full compute utilization. If a workload’s arithmetic intensity falls below this threshold—which is universally the case during single-batch autoregressive LLM decoding where $I pprox 1 ext{ to }2 ext{ OPs/Byte}$—the silicon is 100% memory-bandwidth bound.

Systolic arrays break free of the memory-bound flatline during dense matrix multiplications by caching intermediate tiles in tightly coupled SRAM. Because the arithmetic intensity of matrix multiplication scales with the tile dimension ($O(N)$ operations for every $O(1)$ memory access), systolic arrays push operational intensity past the knee of the roofline curve, achieving 90%+ hardware execution efficiency where traditional SIMD vector pipelines stall waiting for cache line fills.

To inspect physical single-board implementations of edge NPUs, study our comprehensive benchmark on Tenstorrent Wormhole: RISC-V Tensix Core Guide, and review our teardown on Hailo-8 vs Google Coral Edge TPU.