What actually happens on the wires between FPGA and DRAM

A Hacker News post highlighted a chapter that walks through what happens on the wires between an FPGA and its DRAM, and why a DDR3+ interface has to be tuned at boot. According to the article, the interface has two halves: the command and address half runs one way from the controller to the DRAM and is shared by every chip of a rank, while the data half is bidirectional and split into byte lanes, each with its own strobe.

The report says SDR memory moves one word per clock, whereas DDR memories move two, one on each edge. At those rates the clock sent by the controller cannot be trusted to sample the data, because board and chip delays are too large. Data therefore travels with its own clock, the DQS strobe. The article calls this a source-synchronous interface: both signals follow the same path, so their delays cancel out.

On writes the controller sends DQS centre aligned, its edges in the middle of each bit; on reads the DRAM sends it edge aligned and the receiver delays it by a quarter of a clock. Before and after the burst, DQS is driven without carrying data, as a preamble and a postamble. According to the article, DDR3 and DDR4 route clock, command and address pins as one bus that passes by every chip in turn and ends on a termination, the fly-by topology. That keeps heavily loaded signals clean at high rates, but each chip receives the clock a little later than the previous one.

The report notes that data lanes are point to point, so their delays do not follow the clock, and with fly-by routing the skew between the first and last chip can exceed a clock period. Each lane thus needs its own DQS delay. Write leveling measures that delay: the controller puts the DRAM into a special mode and sends DQS pulses until the sampled value flips, and LiteDRAM exposes the necessary registers and BIOS scan. The article adds that the Artix-7 has no output delays, so with one static DQS phase only some lanes meet the timing requirement.