Assise: Performance and Availability via NVM Colocation in a Distributed File System

Disaggregated, or non-local, file storage has become a common design pattern in cloud systems, offering benefits of resource pooling and server specialization, where the inherent overhead of separating compute and storage is mostly hidden by storage device latency. We take an alternate approach, motivated by the commercial availability of very low latency non-volatile memory (NVM). By colocating computation and NVM storage, we can provide applications much higher I/O performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated cache coherence protocol for managing a set of colocated NVM storage devices as a layer. Unlike disaggregated file stores, Assise avoids the read and write amplification of page granularity operations. Instead, remote NVM serves as an intermediate, byte-addressable cache between colocated NVM and slower storage, such as SSDs. We compare Assise to Ceph/Bluestore, NFS, and Octopus on a cluster with Intel Optane DC persistent memory modules and SSDs for common cloud applications and benchmarks, such as LevelDB, Postfix, MinuteSort, and FileBench. We find that Assise improves write latency up to 22x, throughput up to 56x, fail-over time up to 103x, and scales up to 6x better than Ceph, while providing stronger consistency semantics.