NVDIMMs are gaining interest lately, so The SSD Guy thought it might be worthwhile to explain both what they are and how NVDIMM nomenclature works.
As I was writing it I noticed that the post got pretty long, so I have split it into two parts. The first part explains what an NVDIMM is and defines the names for today’s three kinds of NVDIMM. The second part tells about software changes used to support NVDIMMs in BIOS, operating systems, and even processor instruction sets. It also discusses the problem of security.
In case the name is unfamiliar, NVDIMM stands for “Nonvolatile Dual-Inline Memory Module.” Standard computer memory – DRAM – is inserted into the system in the DIMM form factor, but DRAM loses its data when power is removed. The NVDIMM is nonvolatile, or persistent, so its data remains intact despite a loss of power. This takes some effort and always costs more for reasons that will be explained shortly.
Although might seem a little odd to discuss memory in a forum devoted to SSDs, which are clearly storage, the NVDIMM is a storage device, so it rightly belongs on The SSD Guy blog. The NVDIMM is helping the industry prepare for the introduction of Storage Class Memory (SCM) a term introduced over a decade ago by IBM when the company realized that the distinction between memory and storage would inevitably blur over time. That time is now.
An NVDIMM, or nonvolatile DIMM, differs from a standard DRAM-based DIMM by maintaining its data even when power is lost. This is one of the roles that storage plays in today’s computing systems. An NVDIMM fits into a standard memory DIMM socket and communicates using a standard DRAM hardware interface like DDR3 or DDR4.
The Origins of the NVDIMM
The concept of an NVDIMM first came to my attention in 2008 with Simtek’s spin-off AgigA Tech (both of which are now owned by Cypress Semiconductor) although there may have been earlier examples. Certainly there were battery-backed DRAM modules (DIMMs) before that, but the idea of backing the data up to flash was new.
AgigA’s concept was to start with a standard DIMM and add NAND, a small controller, and a supercapacitor. During a power failure then the energy stored in the supercapacitor would keep the DIMM alive, allowing the DRAM to maintain its data long enough that the controller could copy the DRAM’s entire contents into the NAND flash. After that the module would be powered down. When power was restored then the controller would move data from the NAND flash back into the DRAM and the supercapacitor would be recharged to prepare for the next outage.
The use of two memory types significantly increases the cost of the module, so this approach is only adopted in systems where the additional cost is considered worthwhile. The majority of today’s applications are in storage systems, where the NVDIMM serves as a write journal that stores data that may not have already been updated within the disk array. Should the NVDIMM data be more current than the HDD/SSD data when power is restored, then the system will read from the NVDIMM rather than from the drives.
New Kinds of NVDIMM
The idea of putting nonvolatile memory (also called “persistent” memory) into a DIMM gained momentum, and other forms of NVDIMM appeared, some of which will be described shortly. When JEDEC defined an NVDIMM nomenclature It bestowed the name NVDIMM-N upon the approach described above.
Another type, later given the name NVDIMM-F, is essentially an SSD that uses the DRAM’s DDR3 or DDR4 bus instead of SATA, SAS, or PCIe/NVMe. This increases the maximum bandwidth between the flash and the processor and, like an SSD, uses a block access protocol. The concept was championed in 2014 by Diablo Technologies who asked Objective Analysis to write a white paper to explain the approach. Diablo’s key customer for its flash DIMM chipset was SMART Storage Systems, which was acquired by SanDisk (which, in turn was acquired by Western Digital or WDC). SanDisk branded the product the ULLtraDIMM, with the “ULL” part standing for “Ultra-Low-Latency. The DRAM bus and its software protocol certainly do provide significantly lower latency than any standard hard drive interface protocol like SATA. IBM picked this product up, reselling under the IBM brand eXFlash DIMM. Both SanDisk and IBM later abandoned the technology, which I have heard was due to performance issues stemming from the lack of an interrupt pin on the DDR3 bus.
Diablo has since introduced its Memory1 product, which uses flash to mimic a very large DRAM on a single DIMM, resulting in a device which does not use block access. Netlist has a similar product called the HybriDIMM. Objective Analysis wrote a white paper for Netlist to explain how this product helps.
Standard Names Become Necessary
So far I have written about three kinds of NVDIMMs:
- DRAM backed up by NAND
- An SSD on the DRAM bus
- DIMMS that appear to have a large DRAM but are largely made of NAND flash
How do customers know that the NVDIMM they buy is the one they really want?
Chip standards body JEDEC defined a standard nomenclature to categorize different NVDIMM types, a daunting task given that the approach is new and still very much in flux. (JEDEC is the agency that defines DRAM standards like DDR3 and DDR4.) So far there are only three NVDIMM types which are defined by the table below:
Certain protocols have also been added to the DDR4 bus and the internal registers and device identification (“SPD”) to allow greater processor control over the NVDIMM. Pins have also been added to provide a charging current for the backup power supply (a supercapacitor or battery) and to notify the module that power loss is imminent.
The NVDIMM-P specification is being worked out to support NAND flash as well as all types of emerging memory technologies, including MRAM, PCM, ReRAM, 3D XPoint, etc. right on the host memory interface. It’s not hard to understand why it’s not yet fully defined—that’s a big job! The final specification is expected to include a DDR5 interface and retrofits to DDR4.
I’ll stop here, having gone over mainly hardware definitions and issues. Part 2 explains the software & firmware support requirements of NVDIMMs, the new processor instructions Intel has developed to support their use, and it will touch upon new security issues that NVDIMMs pose.
