Design Choices

Table 1 describes the current design limits within the OpenFAM reference implementation. Since the maximum FAM size in the reference implementation is bound by the amount of memory available at the memory server, these parameters should not pose undue limitations on memory size, while enabling developers to build reasonable sized applications using the reference implementation.

Table 1: Data set limits per memory server in the reference implementation
Item	Limit
Maximum FAM size	128 TB
Minimum region size	8 GB
Maximum region size	1 TB
Maximum number of regions	16 K regions
Minimum data Item size	128 bytes
Maximum data Item size	512 GB
Maximum number of data items within a region	~8 B data Items

As mentioned in The OpenFAM API: A Programming Model for Disaggregated Persistent Memory, the reference implementation uses libfabric to handle the underlying memory traffic. Libfabric uses 64 bit keys to manage memory regions and associated access permissions. Within OpenFAM, every data item requires independent access permissions, so OpenFAM data items are mapped to libfabric regions for access from remote PEs. Rather than assigning random keys for data items within libfabric, access keys are explicitly constructed using a combination of OpenFAM region Id, the data item id and OpenFAM permissions using the sequence Key = <Region Id ><DataItem Id><R|RW>. This enables the library to dynamically construct the required libfabric keys from data structures maintained within the implementation. Corresponding to the limits shown in Table 1, the field widths in the libfabric key are shown in Table 2.

Table 2: Construction of libfabric access keys
Item	Limit
Region Id	14 bits
Data Item Id	33 bits
Permission	1 bit (0 = R \| 1 = RW)
Total key length	48 bits
Reserved for future use	16 bits

The implementation supports the complete API with exceptions highlighted in Table 3.

Table 3: OpenFAM APIs currently not available in the reference implementation
API	Memory Server	Shared Memory	Comment
fam_map	NO	YES	Libfabric currently does not support mmap APIs
fam_unmap	NO	YES	Libfabric currently does not support mmap APIs
fam_fence	YES (only with sockets provider)	NO	Fence operation is not supported by PSM2 and Verbs libfabric provider. Support for other providers and Shared Memory model will be added in future

Since the underlying libfabric providers do not support 128-bit compare and swap (CAS) operations, CAS operations have been implemented in software.