VexRiscv/doc/smp/smp.md

Coherent interface specification

Features :

• 3 interface (write, read, probe) composed of 7 streams
• Two data paths (read + write), but allow dirty/clean sharing by reusing the write data path
• Allow multi level coherent interconnect
• No ordering, but provide barrier
• Allow cache-full and cache-less agents

A few hint to help reading the spec

In order to make the spec more readable, there is some definitions :

Stream

A stream is a primitive interface which carry transactions using a valid/ready handshake.

Memory copy

To talk in a non abstract way, in a system with multiple caches, a given memory address can potentialy be loaded in multiple caches at the same time. So let's define that :

• The DDR memory is named main memory
• Each cache line can be loaded with a part of the main memory, let's name that a memory copy

Master / Interconnect / Slave

A master could be for instance a CPU cache, the side of the interconnect toward the main memory or toward a more general interconnect.

A slave could be main memory, the side of the interconnect toward a CPU cache or toward a less general interconnect.

The spec will try to stay abstract and define the coherent interface as something which can be used between two agents (cpu, interconnect, main memory)

Memory copy status

Memory copy, in other words, cache line, have more states than non coherent systems :

Name	Description
Valid/Invalid	Line loaded or not
Shared/Unique	shared => multiple copy of the cache line in different caches, unique => no other caches has a copy of the line
Owner/Lodger	lodger => copy of the line, but no other responsibility, owner => the given cache is responsible to write back dirty data and answer probes with the data
Clean/Dirty	clean => match main memory, dirty => main memory need updates

All combination of those cache flag are valid. But if a cache line is invalid, the other status have no meaning.

Later in the spec, memory copy state can be described for example as :

• VSOC for (Valid, Shared, Owner, Clean)
• V-OC for (Valid, Shared or Unique, Owner, Clean)
• !V-OC for NOT (Valid, Shared or Unique, Owner, Clean)
• ...

Coherent interface

One full coherent interface is composed of 3 inner interfaces, them-self composed of 7 stream described bellow as interfaceName (Side -> StreamName -> Side -> StreamName -> ...)

• write (M -> writeCmd -> S -> writeRsp -> M)
• read (M -> readCmd- > S -> readRsp -> M -> readAck -> S)
• probe (S -> probeCmd -> M -> probeRsp -> S)

The following streams could physically be merges in order to reduce the number of arbitration :

• writeCmd, probeRsp, readAck
• writeRsp, readRsp

Read interface

Used by masters to obtain new memory copies and make copies unique (used to write them).

Composed of 3 stream :

Name	Direction	Description
readCmd	M -> S	Emit memory read and cache management commands
readRsp	M <- S	Return some data and/or a status from readCmd
readAck	M -> S	Return ACK from readRsp to synchronize the interconnect status

Write interface

Used by masters to write data back to the memory and notify the interconnect of memory copies eviction (used to keep potential directories updated).

Composed of 2 stream :

Name	Direction	Description
writeCmd	M -> S	Emit memory writes and cache management commands
writeRsp	M <- S	Return a status from writeCmd

Probe interface

Used by the interconnect to order master to change their memory copies status and get memory copies owners data.

Composed of 2 stream :

Name	Direction	Description
probeCmd	M <- S	Used for cache management
probeRsp	M -> S	Acknowledgment

Transactions

This chapter define transactions moving over the 3 previously defined interface (read/write/probe).

Read commands

Emitted on the readCmd channel (master -> slave)

Command	Initial state	Description	Usage example
readShared	I---	Get a memory copy as V---	Want to read a uncached address
readUnique	I---	Get a memory copy as VUO-	Want to write a uncached address
readOnce	I---	Get a memory copy without coherency tracking	Instruction cache read
makeUnique	VS--	Make other memory copy as I--- and make yourself VUO-	Want to write into a shared line
readBarrier	N/A	Ensure that the visibility of the memory operations of this channel do not cross the barrier	ISA fence

makeUnique should be designed with care. There is a few corner cases :

• While a master has a inflight makeUnique, a probe can change its state, in such case, the makeUnique become weak and invalidation is canceled. This is usefull for multi level coherent interconnects.
• Multi level coherent interconnect should be careful to properly move the ownership and not lose dirty data

I'm not sure yet if we should add some barrier transactions to enforce

Read responses

Emitted on the readRsp channel (master <- slave)

readSuccess, readError, data shared/unique clean/dirty owner/notOwner

Responses	From command	Description
readSuccess	makeUnique, readBarrier	-
readError	readShared, readUnique, readOnce	Bad address
readData	readShared, readUnique, readOnce	Data + coherency status (V---)

Read ack

Emitted on the readAck channel (master -> slave), it carry no information, just a notification that the master received the read response

Name	From command	Description
readSuccess	*	-

Write commands

Write commands can be emitted on the writeCmd channel (master -> slave)

Name	Initial state	Description	Usage example
writeInvalid	V-O-	Write the memory copy and update it status to I---	Need to free the dirty cache line
writeShare	V-O-	Write the memory copy but keep it as VSO-	A probe makeShared asked it
writeUnique	VUO-	Write the memory copy but keep it as VUO-	A probe probeOnce need to read the data
evict	V---, !V-OD	Notify the interconnect that the cache line is now I---	Need to free a clean cache line
writeBarrier	N/A	Ensure that the visibility of the memory operations of this channel do not cross the barrier	ISA fence

Write responses

Emitted on the writeRsp channel (master <- slave), it carry no information, just a notification that the corresponding command is done.

Name	From command	Description
writeSuccess	*	-

Probe commands

Probe commands can be emitted on the probeCmd channel (slave -> master)

Name	Description	Usage example
makeInvalid	Make the memory copy I---	Another cache want to make his shared copy unique to write it
makeShared	Make the memory copy VS--	Another cache want to read a memory block, so unique copy need to be shared
probeOnce	Read the V-O- memory copy	A non coherent agent did a readOnce

makeInvalid and makeShared could result into one of the following probeSuccess, writeInvalid, writeShare

probeOnce can result into one of the following probeSuccess, writeShare, writeUnique

To help the slave matching the writeInvalid and writeShare generated from a probe, those request are tagged with a matching ID.

Probe responses

Emitted on the probeRsp channel (master -> slave), it carry no information, just a notification that the corresponding command is done.

Name	From command	Description
probeSuccess	*	-

Channel interlocking

This is a delicate subject as if everything was permited, it would be easy to end up with deadlocks.

There is the streams priority (top => high priority, bottom => low priority) A lower priority stream should not block a higher priority stream in order to avoid deadlocks.

• writeCmd, writeRsp, readRsp, readAck, probeRsp. Nothing should realy block them excepted bandwidth
• probeCmd. Can be blocked by inflight/generated writes
• readCmd. Can be blocked by inflight/generated probes

In other words :

Masters can emit writeCmd and wait their writeRsp completion before answering probes commands. Slaves can emit probeCmd and wait their proveRsp completion before answering reads. Slaves can emit readRsp and wait on their readAck completion before doing anything else

Interface subsets

There is a few cases where you could need a specific subset of the coherent interface :

• Instruction caches do not necessarily need to maintain the coherency with the memory system.
• DMA need to read and write the memory system, but are cache-less (no probe)

ReadOnly interface without maintained coherency

Such interface is only composed of the read bus on which the readCmd stream can only use readOnce requests

WriteOnly interface

In such interface, there is no read/probe buses, but only a writeCmd and a writeRsp stream. The writeCmd will invalidate other memory copies, then write into the memory while the writeRsp will return a writeSuccess/writeError status.