Back-Invalidate in CXL 3.0
Updated: Oct 3rd, 2023
3.3.3 in CXL Specification
“To enable a device to implement an inclusive Snoop Filter for tracking host caching, a Back-Invalidation snoop (BISnp) is initiated from the device to change the cache state of the host. The flows related to this channel are captured in Section 3.5.1. The definition of “inclusive Snoop Filter” for the purpose of CXL is a device structure that tracks cacheline granular host caching and is a limited size that is a small subset of the total Host Physical Address space supported by the device. In 68B flits, only the CXL.cache D2H Request flows can be used for device-attached memory to manage coherence with the host as shown in Section 3.5.2.3. This flow is used for addresses with the HDM-D memory attribute. A major constraint with this flow is that the D2H Req channel can be blocked waiting on forward progress of the M2S Request channel which disallows an inclusive Snoop Filter architecture. For the HDM-DB memory region, the BISnp channel (instead of CXL.cache) is used to resolve coherence. CXL host implementations may have a mix of devices with HDM-DB and HDM-D below a Root Port.” (Intel, 2022, p. 112) (pdf) [1]
“The BI-mechanism also enables the implementation of shared and hardware-enforced coherent memory across multiple hosts, as illustrated in Figure 18b. Here the memory device (GFD/MLD) maintains a directory (or a snoop filter) to track ownership of cache lines in the coherent shared memory. Thus, when Host 1 obtains ownership of cache line X as a shared copy, it updates the directory from “I” state to “S” state with Host 1 as the sharer. When Host 3 asks for the same cache line X in shared state, it provides the data and updates the directory for X to indicate that both Host 1 and Host 3 have it Shared. When Host 4 requests an exclusive copy, it issues a Back Invalidate to both Host H1 and Host H3, waits for the response from both to ensure that Host 1 and Host 3 have invalidated their copies of X and updates its directory to mark X as “E” by Host 4, prior to sending the data and ownership to Host 4. This memory device can be a Multi-Logical Device (MLD) or a GFAM Device (GFD).” (Sharma et al., 2023, p. 20) (pdf). [2]
CXL 3.0 Back Invalidate enables inclusive snoop filters and peer-to-peer memory requests
CXL3 Fabric – Introduction and use cases
Compute Express LinkTM (CXL™): An Open Industry Standard for Composable Computing
FMS 2023 Tutorial - Presentation
Existing Bias – Flip mechanism needed HDM to be tracked fully since device could not back snoop the host. Back Invalidate with CXL 3.0 enables snoop filter implementation resulting in large memory that can be mapped to HDM