diff options
| -rw-r--r-- | content.tex | 4187 |
1 files changed, 4187 insertions, 0 deletions
diff --git a/content.tex b/content.tex new file mode 100644 index 0000000..1d7cc91 --- /dev/null +++ b/content.tex @@ -0,0 +1,4187 @@ +\chapter{INTRODUCTION}\label{sec:INTRODUCTION} + +This document describes the specifications of the “virtio” family of +devices. These are devices are found in virtual environments, yet by +design they are not all that different from physical devices, and this +document treats them as such. This allows the guest to use standard +drivers and discovery mechanisms. + +The purpose of virtio and this specification is that virtual +environments and guests should have a straightforward, efficient, +standard and extensible mechanism for virtual devices, rather +than boutique per-environment or per-OS mechanisms. + + Straightforward: Virtio devices use normal bus mechanisms of + interrupts and DMA which should be familiar to any device driver + author. There is no exotic page-flipping or COW mechanism: it's just + a normal device. +\footnote{This lack of page-sharing implies that the implementation of the +device (e.g. the hypervisor or host) needs full access to the +guest memory. Communication with untrusted parties (i.e. +inter-guest communication) requires copying. +} + + Efficient: Virtio devices consist of rings of descriptors + for input and output, which are neatly separated to avoid cache + effects from both guest and device writing to the same cache + lines. + + Standard: Virtio makes no assumptions about the environment in which + it operates, beyond supporting the bus attaching the device. Virtio + devices are implemented over PCI and other buses, and earlier drafts + been implemented on other buses not included in this spec. +\footnote{The Linux implementation further separates the PCI virtio code +from the specific virtio drivers: these drivers are shared with +the non-PCI implementations (currently lguest and S/390). +} + + Extensible: Virtio PCI devices contain feature bits which are + acknowledged by the guest operating system during device setup. + This allows forwards and backwards compatibility: the device + offers all the features it knows about, and the driver + acknowledges those it understands and wishes to use. + +\section{Key words}\label{sec:Key words} + +The key words must, must not, required, shall, shall not, should, +should not, recommended, may, and optional are to be interpreted as +described in [RFC 2119]. Note that for reasons of style, these words +are not capitalized in this document. + +\section{Definitions}\label{sec:Definitions} + +term + Definition + +\section{Key concepts}\label{sec:Key concepts} + +Guest + Definition... + +Host + Definition + +Device + Definition + +Driver + Definition + +\chapter{Normative References}\label{sec:Normative References} + +[RFC 2119] S. Bradner, Key words for use in RFCs to Indicate Requirement Levels, http://www.ietf.org/rfc/rfc2119.txt IETF (Internet Engineering Task Force) RFC 2119, March 1997. + +[S390 PoP] z/Architecture Principles of Operation, IBM Publication SA22-7832 + +[S390 Common I/O] ESA/390 Common I/O-Device and Self-Description, IBM Publication SA22-7204 + +\chapter{Non-Normative References}\label{sec:Non-Normative References} + + + +\chapter{The Virtio Standard}\label{sec:The Virtio Standard} + +\chapter{Basic Facilities of a Virtio Device}\label{sec:Basic Facilities of a Virtio Device} + +A virtio device is discovered and identified by a bus-specific method +(see the bus specific sections: \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus}, +\ref{sec:Virtio Transport Options / Virtio Over MMIO}~\nameref{sec:Virtio Transport Options / Virtio Over MMIO} and \ref{sec:Virtio Transport Options / Virtio Over MMIO}~\nameref{sec:Virtio Transport Options / Virtio Over MMIO}). Each +device consists of the following parts: + +o Device Status field +o Feature bits +o Configuration space +o One or more virtqueues + +Unless explicitly specified otherwise, all multi-byte fields are little-endian. +To reinforce this the examples use typenames like "le16" instead of "uint16_t". + +\section{Device Status Field}\label{sec:Basic Facilities of a Virtio Device / Device Status Field} + +The Device Status field is updated by the guest to indicate its +progress. This provides a simple low-level diagnostic: it's most +useful to imagine them hooked up to traffic lights on the console +indicating the status of each device. + +This field is 0 upon reset, otherwise at least one bit should be set: + + ACKNOWLEDGE (1) Indicates that the guest OS has found the + device and recognized it as a valid virtio device. + + DRIVER (2) Indicates that the guest OS knows how to drive the + device. Under Linux, drivers can be loadable modules so there + may be a significant (or infinite) delay before setting this + bit. + + FEATURES_OK (8) Indicates that the driver has acknowledged all the + features it understands, and feature negotiation is complete. + + DRIVER_OK (4) Indicates that the driver is set up and ready to + drive the device. + + FAILED (128) Indicates that something went wrong in the guest, + and it has given up on the device. This could be an internal + error, or the driver didn't like the device for some reason, or + even a fatal error during device operation. The device must be + reset before attempting to re-initialize. + +\section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature Bits} + +Each virtio device lists all the features it understands. During +device initialization, the guest reads this and tells the device the +subset that it understands. The only way to renegotiate is to reset +the device. + +This allows for forwards and backwards compatibility: if the device is +enhanced with a new feature bit, older guests will not write that +feature bit back to the device and it can go into backwards +compatibility mode. Similarly, if a guest is enhanced with a feature +that the device doesn't support, it see the new feature is not offered +and can go into backwards compatibility mode (or, for poor +implementations, set the FAILED Device Status bit). + +Feature bits are allocated as follows: + + 0 to 23: Feature bits for the specific device type + + 24 to 32: Feature bits reserved for extensions to the queue and + feature negotiation mechanisms + + 33 and above: Feature bits reserved for future extensions. + +For example, feature bit 0 for a network device (i.e. Subsystem +Device ID 1) indicates that the device supports checksumming of +packets. + +In particular, new fields in the device configuration space are +indicated by offering a feature bit, so the guest can check +before accessing that part of the configuration space. + +\subsection{Legacy Interface: A Note on transitions from earlier drafts}\label{sec:Basic Facilities of a Virtio Device / Feature Bits / Legacy Interface: A Note on transitions from earlier drafts} + +Earlier drafts of this specification (up to 0.9.X) defined a similar, but +different interface between the hypervisor and the guest. +Since these are widely deployed, this specification +accommodates optional features to simplify transition +from these earlier draft interfaces. Specifically: + +Legacy Interface + is an interface specified by an earlier draft of this specification + (up to 0.9.X) +Legacy Device + is a device implemented before this specification was released, + and implementing a legacy interface on the host side +Legacy Driver + is a driver implemented before this specification was released, + and implementing a legacy interface on the guest side + +Legacy devices and legacy drivers are not compliant with this +specification. + +To simplify transition from these earlier draft interfaces, +it is possible to implement: + +Transitional Device + a device supporting both drivers conforming to this + specification, and allowing legacy drivers. + +Transitional Driver + a driver supporting both devices conforming to this + specification, and legacy devices. + +Transitional devices and transitional drivers can be compliant with +this specification (ie. when not operating in legacy mode). + +Devices or drivers with no legacy compatibility are referred to as +non-transitional devices and drivers, respectively. + +Transitional Drivers can detect Legacy Devices by detecting that +the feature bit VIRTIO_F_VERSION_1 is not offered. +Transitional devices can detect Legacy drivers by detecting that +VIRTIO_F_VERSION_1 has not been acknowledged by the driver. +In this case device is used through the legacy interface. + +To make them easier to locate, specification sections documenting +these transitional features are explicitly marked with 'Legacy +Interface' in the section title. + +\section{Configuration Space}\label{sec:Basic Facilities of a Virtio Device / Configuration Space} + +Configuration space is generally used for rarely-changing or +initialization-time parameters. Drivers must not assume reads from +fields greater than 32 bits wide are atomic, nor or reads from +multiple fields. + +Each transport provides a generation count for the configuration +space, which must change whenever there is a possibility that two +accesses to the configuration space can see different versions of that +space. + +Thus drivers should read configuration space fields like so: + +\begin{lstlisting} + u32 before, after; + do { + before = get_config_generation(device); + // read config entry/entries. + after = get_config_generation(device); + } while (after != before); +\end{lstlisting} + +Note that configuration space generally uses the little-endian format +for multi-byte fields. + +Note that future versions of this specification will likely +extend the configuration space for devices by adding extra fields +at the tail end of some structures in configuration space. + +To allow forward compatibility with such extensions, drivers must +not limit structure size and configuration space size. Instead, +drivers should only check that configuration space is *large enough* to +contain the fields required for device operation. + +For example, if the specification states that configuration +space 'includes a single 8-bit field' drivers should understand this to mean that +the configuration space can also include an arbitrary amount of +tail padding, and accept any configuration space size equal to or +greater than the specified 8-bit size. + +\subsection{Legacy Interface: A Note on Configuration Space endian-ness}\label{sec:Basic Facilities of a Virtio Device / Configuration Space / Legacy Interface: A Note on Configuration Space endian-ness} + +Note that for legacy interfaces, configuration space is generally the +guest's native endian, rather than PCI's little-endian. + +\subsection{Legacy Interface: Configuration Space}\label{sec:Basic Facilities of a Virtio Device / Configuration Space / Legacy Interface: Configuration Space} + +Legacy devices did not have a configuration generation field, thus are +susceptible to race conditions if configuration is updated. This +effects the block capacity and network mac fields; best practice is to +read these fields multiple times until two reads generate a consistent +result. + +\section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} + +The mechanism for bulk data transport on virtio devices is +pretentiously called a virtqueue. Each device can have zero or more +virtqueues: for example, the simplest network device has one for +transmit and one for receive. Each queue has a 16-bit queue size +parameter, which sets the number of entries and implies the total size +of the queue. + +Each virtqueue consists of three parts: + + Descriptor Table + + Available Ring + + Used Ring + +where each part is physically-contiguous in guest memory, +and has different alignment requirements. + +The memory aligment and size requirements, in bytes, of each part of the +virtqueue are summarized in the following table: + +\begin{verbatim} ++------------+-----------------------------------------+ +| Virtqueue Part | Alignment | Size | ++------------+-----------------------------------------+ ++------------+-----------------------------------------+ +| Descriptor Table | 16 | 16 * (Queue Size) | ++------------+-----------------------------------------+ +| Available Ring | 2 | 6 + 2 * (Queue Size) | ++------------+-----------------------------------------+ +| Used Ring | 4 | 6 + 4 * (Queue Size) | ++------------+-----------------------------------------+ +\end{verbatim} + +The Alignment column gives the miminum alignment: for each part +of the virtqueue, the physical address of the first byte of it +must be a multiple of the specified alignment value. + +The Size column gives the total number of bytes required for each +part of the virtqueue. + +Queue Size corresponds to the maximum number of buffers in the +virtqueue. For example, if Queue Size is 4 then at most 4 buffers +can be queued at any given time. Queue Size value is always a +power of 2. The maximum Queue Size value is 32768. This value +is specified in a bus-specific way. + +When the driver wants to send a buffer to the device, it fills in +a slot in the descriptor table (or chains several together), and +writes the descriptor index into the available ring. It then +notifies the device. When the device has finished a buffer, it +writes the descriptor into the used ring, and sends an interrupt. + + +\subsection{Legacy Interfaces: A Note on Virtqueue Layout}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout} + +For Legacy Interfaces, several additional +restrictions are placed on the virtqueue layout: + +Each virtqueue occupies two or more physically-contiguous pages +(usually defined as 4096 bytes, but depending on the transport) +and consists of three parts: + +\begin{verbatim} ++-------------------+-----------------------------------+-----------+ +| Descriptor Table | Available Ring (padding) | Used Ring | ++-------------------+-----------------------------------+-----------+ +\end{verbatim} + +The bus-specific Queue Size field controls the total number of bytes +required for the virtqueue according to the following formula: + +\begin{lstlisting} + #define ALIGN(x) (((x) + PAGE_SIZE) & ~PAGE_SIZE) + static inline unsigned vring_size(unsigned int qsz) + { + return ALIGN(sizeof(struct vring_desc)*qsz + sizeof(u16)*(3 + qsz)) + + ALIGN(sizeof(u16)*3 + sizeof(struct vring_used_elem)*qsz); + } +\end{lstlisting} + +This wastes some space with padding. +The legacy virtqueue layout structure therefore looks like this: + +\begin{lstlisting} + struct vring { + // The actual descriptors (16 bytes each) + struct vring_desc desc[ Queue Size ]; + + // A ring of available descriptor heads with free-running index. + struct vring_avail avail; + + // Padding to the next PAGE_SIZE boundary. + char pad[ Padding ]; + + // A ring of used descriptor heads with free-running index. + struct vring_used used; + }; +\end{lstlisting} + +\subsection{Legacy Interfaces: A Note on Virtqueue Endianness}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness} + +Note that the endian of fields and in the virtqueue is the native +endian of the guest, not little-endian as specified by this standard. +It is assumed that the host is already aware of the guest endian. + +\subsection{Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing} +The message framing (the particular layout of descriptors) is +independent of the contents of the buffers. For example, a network +transmit buffer consists of a 12 byte header followed by the network +packet. This could be most simply placed in the descriptor table as a +12 byte output descriptor followed by a 1514 byte output descriptor, +but it could also consist of a single 1526 byte output descriptor in +the case where the header and packet are adjacent, or even three or +more descriptors (possibly with loss of efficiency in that case). + +Note that, some implementations may have large-but-reasonable +restrictions on total descriptor size (such as based on IOV_MAX in the +host OS). This has not been a problem in practice: little sympathy +will be given to drivers which create unreasonably-sized descriptors +such as by dividing a network packet into 1500 single-byte +descriptors! + +\subsubsection{Legacy Interface: Message Framing}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing} + +Regrettably, initial driver implementations used simple layouts, and +devices came to rely on it, despite this specification wording. In +addition, the specification for virtio_blk SCSI commands required +intuiting field lengths from frame boundaries (see + \ref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation}~\nameref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation}) + +It is thus recommended that when using legacy interfaces, transitional +drivers be conservative in their assumptions, unless the +VIRTIO_F_ANY_LAYOUT feature is accepted. + +\subsection{The Virtqueue Descriptor Table}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table} + +The descriptor table refers to the buffers the guest is using for +the device. The addresses are physical addresses, and the buffers +can be chained via the next field. Each descriptor describes a +buffer which is read-only or write-only, but a chain of +descriptors can contain both read-only and write-only buffers. + +The actual contents of the memory offered to the device depends on the +device type. Most common is to begin the data with a header +(containing little-endian fields) for the device to read, and postfix +it with a status tailer for the device to write. + +No descriptor chain may be more than $2^{32}$ bytes long in total. + +\begin{lstlisting} + struct vring_desc { + /* Address (guest-physical). */ + le64 addr; + /* Length. */ + le32 len; + + /* This marks a buffer as continuing via the next field. */ + #define VRING_DESC_F_NEXT 1 + /* This marks a buffer as write-only (otherwise read-only). */ + #define VRING_DESC_F_WRITE 2 + /* This means the buffer contains a list of buffer descriptors. */ + #define VRING_DESC_F_INDIRECT 4 + /* The flags as indicated above. */ + le16 flags; + /* Next field if flags & NEXT */ + le16 next; + }; +\end{lstlisting} + +The number of descriptors in the table is defined by the queue size +for this virtqueue. + +\subsubsection{Indirect Descriptors}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors} + +Some devices benefit by concurrently dispatching a large number +of large requests. The VIRTIO_RING_F_INDIRECT_DESC feature can be +used to allow this (see \ref{sec:virtio-ring.h}~\nameref{sec:virtio-ring.h}). To increase +ring capacity it is possible to store a table of indirect +descriptors anywhere in memory, and insert a descriptor in main +virtqueue (with flags\&VRING_DESC_F_INDIRECT on) that refers to memory buffer +containing this indirect descriptor table; fields addr and len +refer to the indirect table address and length in bytes, +respectively. The indirect table layout structure looks like this +(len is the length of the descriptor that refers to this table, +which is a variable, so this code won't compile): + +\begin{lstlisting} + struct indirect_descriptor_table { + /* The actual descriptors (16 bytes each) */ + struct vring_desc desc[len / 16]; + }; +\end{lstlisting} + +The first indirect descriptor is located at start of the indirect +descriptor table (index 0), additional indirect descriptors are +chained by next field. An indirect descriptor without next field +(with flags\&VRING_DESC_F_NEXT off) signals the end of the descriptor. +An +indirect descriptor can not refer to another indirect descriptor +table (flags\&VRING_DESC_F_INDIRECT must be off). A single indirect descriptor +table can include both read-only and write-only descriptors; +write-only flag (flags\&VRING_DESC_F_WRITE) in the descriptor that refers to it +is ignored. + +\subsection{The Virtqueue Available Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Available Ring} + +The available ring refers to what descriptor chains we are offering the +device: each entry refers to the head of a descriptor chain. The “flags” field +is currently 0 or 1: 1 indicating that we do not need an interrupt +when the device consumes a descriptor chain from the available +ring. Alternatively, the guest can ask the device to delay interrupts +until an entry with an index specified by the “used_event” field is +written in the used ring (equivalently, until the idx field in the +used ring will reach the value used_event + 1). The method employed by +the device is controlled by the VIRTIO_RING_F_EVENT_IDX feature bit +(see \ref{sec:virtio-ring.h}~\nameref{sec:virtio-ring.h}). This interrupt suppression is +merely an optimization; it may not suppress interrupts entirely. + +The “idx” field indicates where we would put the next descriptor +entry (modulo the queue size). This starts at 0, and increases. + +\begin{lstlisting} + struct vring_avail { + #define VRING_AVAIL_F_NO_INTERRUPT 1 + le16 flags; + le16 idx; + le16 ring[ /* Queue Size */ ]; + le16 used_event; /* Only if VIRTIO_RING_F_EVENT_IDX */ + }; +\end{lstlisting} + +\subsection{The Virtqueue Used Ring}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring} + +The used ring is where the device returns buffers once it is done +with them. The flags field can be used by the device to hint that +no notification is necessary when the guest adds to the available +ring. Alternatively, the “avail_event” field can be used by the +device to hint that no notification is necessary until an entry +with an index specified by the “avail_event” is written in the +available ring (equivalently, until the idx field in the +available ring will reach the value avail_event + 1). The method +employed by the device is controlled by the guest through the +VIRTIO_RING_F_EVENT_IDX feature bit ( +see \ref{sec:virtio-ring.h}~\nameref{sec:virtio-ring.h}). +\footnote{These fields are kept here because this is the only part of the +virtqueue written by the device +} + +Each entry in the ring is a pair: the head entry of the +descriptor chain describing the buffer (this matches an entry +placed in the available ring by the guest earlier), and the total +of bytes written into the buffer. The latter is extremely useful +for guests using untrusted buffers: if you do not know exactly +how much has been written by the device, you usually have to zero +the buffer to ensure no data leakage occurs. + +\begin{lstlisting} + /* le32 is used here for ids for padding reasons. */ + struct vring_used_elem { + /* Index of start of used descriptor chain. */ + le32 id; + /* Total length of the descriptor chain which was used (written to) */ + le32 len; + }; + + struct vring_used { + #define VRING_USED_F_NO_NOTIFY 1 + le16 flags; + le16 idx; + struct vring_used_elem ring[ /* Queue Size */]; + le16 avail_event; /* Only if VIRTIO_RING_F_EVENT_IDX */ + }; +\end{lstlisting} + +\subsection{Helpers for Operating Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Helpers for Operating Virtqueues} + +The Linux Kernel Source code contains the definitions above and +helper routines in a more usable form, in +include/linux/virtio_ring.h. This was explicitly licensed by IBM +and Red Hat under the (3-clause) BSD license so that it can be +freely used by all other projects, and is reproduced (with slight +variation to remove Linux assumptions) in \ref{sec:virtio-ring.h}~\nameref{sec:virtio-ring.h}. + +\chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} + +We start with an overview of device initialization, then expand on the +details of the device and how each step is preformed. This section +should be read along with the bus-specific section which describes +how to communicate with the specific device. + +\section{Device Initialization}\label{sec:General Initialization And Device Operation / Device Initialization} + +1. Reset the device. + +2. The ACKNOWLEDGE status bit is set: we have noticed the device. + +3. The DRIVER status bit is set: we know how to drive the device. + +4. Device feature bits are read, and the the subset of feature bits + understood by the OS and driver is written to the device. + +5. The FEATURES_OK status bit is set. + +6. The status byte is re-read to ensure the FEATURES_OK bit is still + set: otherwise, the device does not support our subset of features + and the device is unusable. + +7. Device-specific setup, including discovery of virtqueues for the + device, optional per-bus setup, reading and possibly writing the + device's virtio configuration space, and population of virtqueues. + +8. The DRIVER_OK status bit is set. At this point the device is + "live". + +If any of these steps go irrecoverably wrong, the guest should +set the FAILED status bit to indicate that it has given up on the +device (it can reset the device later to restart if desired). + +The device must not consume buffers before DRIVER_OK, and the driver +must not notify the device before it sets DRIVER_OK. + +Devices should support all valid combinations of features, but we know +that implementations may well make assuptions that they will only be +used by fully-optimized drivers. The resetting of the FEATURES_OK flag +provides a semi-graceful failure mode for this case. + +\subsection{Legacy Interface: Device Initialization}\label{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization} +Legacy devices do not support the FEATURES_OK status bit, and thus did +not have a graceful way for the device to indicate unsupported feature +combinations. It also did not provide a clear mechanism to end +feature negotiation, which meant that devices finalized features on +first-use, and no features could be introduced which radically changed +the initial operation of the device. + +Legacy device implementations often used the device before setting the +DRIVER_OK bit. + +The result was the steps 5 and 6 were omitted, and steps 7 and 8 +were conflated. + +\section{Device Operation}\label{sec:General Initialization And Device Operation / Device Operation} + +There are two parts to device operation: supplying new buffers to +the device, and processing used buffers from the device. As an +example, the simplest virtio network device has two virtqueues: the +transmit virtqueue and the receive virtqueue. The driver adds +outgoing (read-only) packets to the transmit virtqueue, and then +frees them after they are used. Similarly, incoming (write-only) +buffers are added to the receive virtqueue, and processed after +they are used. + +\subsection{Supplying Buffers to The Device}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device} + +Actual transfer of buffers from the guest OS to the device +operates as follows: + +1. Place the buffer(s) into free descriptor(s). + +2. Place the id of the buffer in the next ring entry of the + available ring. + +3. The steps (1) and (2) may be performed repeatedly if batching + is possible. + +4. A memory barrier should be executed to ensure the device sees + the updated descriptor table and available ring before the next + step. + +5. The available “idx” field should be increased by the number of + entries added to the available ring. + +6. A memory barrier should be executed to ensure that we update + the idx field before checking for notification suppression. + +7. If notifications are not suppressed, the device should be + notified of the new buffers. + +Note that the above code does not take precautions against the +available ring buffer wrapping around: this is not possible since +the ring buffer is the same size as the descriptor table, so step +(1) will prevent such a condition. + +In addition, the maximum queue size is 32768 (it must be a power +of 2 which fits in 16 bits), so the 16-bit “idx” value can always +distinguish between a full and empty buffer. + +Here is a description of each stage in more detail. + +\subsubsection{Placing Buffers Into The Descriptor Table}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Placing Buffers Into The Descriptor Table} + +A buffer consists of zero or more read-only physically-contiguous +elements followed by zero or more physically-contiguous +write-only elements (it must have at least one element). This +algorithm maps it into the descriptor table: + +for each buffer element, b: + + (a) Get the next free descriptor table entry, d + + (b) Set d.addr to the physical address of the start of b + + (c) Set d.len to the length of b. + + (d) If b is write-only, set d.flags to VRING_DESC_F_WRITE, + otherwise 0. + + (e) If there is a buffer element after this: + + i. Set d.next to the index of the next free descriptor + element. + + ii. Set the VRING_DESC_F_NEXT bit in d.flags. + +In practice, the d.next fields are usually used to chain free +descriptors, and a separate count kept to check there are enough +free descriptors before beginning the mappings. + +\subsubsection{Updating The Available Ring}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating The Available Ring} + +The head of the buffer we mapped is the first d in the algorithm +above. A naive implementation would do the following (with the +appropriate conversion to-and-from little-endian assumed): + +\begin{lstlisting} + avail->ring[avail->idx % qsz] = head; +\end{lstlisting} + +However, in general we can add many descriptor chains before we update +the “idx” field (at which point they become visible to the +device), so we keep a counter of how many we've added: + +\begin{lstlisting} + avail->ring[(avail->idx + added++) % qsz] = head; +\end{lstlisting} + +\subsubsection{Updating The Index Field}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Updating The Index Field} + +Once the index field of the virtqueue is updated, the device will +be able to access the descriptor chains we've created and the +memory they refer to. This is why a memory barrier is generally +used before the index update, to ensure it sees the most up-to-date +copy. + +The index field always increments, and we let it wrap naturally at +65536: + +\begin{lstlisting} + avail->idx += added; +\end{lstlisting} + +\subsubsection{Notifying The Device}\label{sec:General Initialization And Device Operation / Device Operation / Supplying Buffers to The Device / Notifying The Device} + +The actual method of device notification is bus-specific, but generally +it can be expensive. So the device can suppress such notifications if it +doesn't need them. We have to be careful to expose the new index +value before checking if notifications are suppressed: it's OK to notify +gratuitously, but not to omit a required notification. So again, +we use a memory barrier here before reading the flags or the +avail_event field. + +If the VIRTIO_F_RING_EVENT_IDX feature is not negotiated, and if the +VRING_USED_F_NOTIFY flag is not set, we go ahead and notify the +device. + +If the VIRTIO_F_RING_EVENT_IDX feature is negotiated, we read the +avail_event field in the available ring structure. If the +available index crossed_the avail_event field value since the +last notification, we go ahead and write to the PCI configuration +space. The avail_event field wraps naturally at 65536 as well, +iving the following algorithm for calculating whether a device needs +notification: + +\begin{lstlisting} + (u16)(new_idx - avail_event - 1) < (u16)(new_idx - old_idx) +\end{lstlisting} + +\subsection{Receiving Used Buffers From The Device}\label{sec:General Initialization And Device Operation / Device Operation / Receiving Used Buffers From The Device} + +Once the device has used a buffer (read from or written to it, or +parts of both, depending on the nature of the virtqueue and the +device), it sends an interrupt, following an algorithm very +similar to the algorithm used for the driver to send the device a +buffer: + +1. Write the head descriptor number to the next field in the used + ring. + +2. Update the used ring index. + +3. Deliver an interrupt if necessary: + + (a) If the VIRTIO_F_RING_EVENT_IDX feature is not negotiated: + check if the VRING_AVAIL_F_NO_INTERRUPT flag is not set in + avail->flags. + + (b) If the VIRTIO_F_RING_EVENT_IDX feature is negotiated: check + whether the used index crossed the used_event field value + since the last update. The used_event field wraps naturally + at 65536 as well: +\begin{lstlisting} + (u16)(new_idx - used_event - 1) < (u16)(new_idx - old_idx) +\end{lstlisting} + +For each ring, guest should then disable interrupts by writing +VRING_AVAIL_F_NO_INTERRUPT flag in avail structure, if required. +It can then process used ring entries finally enabling interrupts +by clearing the VRING_AVAIL_F_NO_INTERRUPT flag or updating the +EVENT_IDX field in the available structure. The guest should then +execute a memory barrier, and then recheck the ring empty +condition. This is necessary to handle the case where after the +last check and before enabling interrupts, an interrupt has been +suppressed by the device: + +\begin{lstlisting} + vring_disable_interrupts(vq); + + for (;;) { + if (vq->last_seen_used != le16_to_cpu(vring->used.idx)) { + vring_enable_interrupts(vq); + mb(); + + if (vq->last_seen_used != le16_to_cpu(vring->used.idx)) + break; + } + + struct vring_used_elem *e = vring.used->ring[vq->last_seen_used%vsz]; + process_buffer(e); + vq->last_seen_used++; + } +\end{lstlisting} + +\subsection{Notification of Device Configuration Changes}\label{sec:General Initialization And Device Operation / Device Operation / Notification of Device Configuration Changes} + +For devices where the configuration information can be changed, an +interrupt is delivered when a configuration change occurs. + + + +\chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} + +Virtio can use various different busses, thus the standard is split +into virtio general and bus-specific sections. + +\section{Virtio Over PCI Bus}\label{sec:Virtio Transport Options / Virtio Over PCI Bus} + +Virtio devices are commonly implemented as PCI devices. + +\subsection{PCI Device Discovery}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery} + +Any PCI device with Vendor ID 0x1AF4, and Device ID 0x1000 through +0x103F inclusive is a virtio device +\footnote{The actual value within this range is ignored +}. + +The Subsystem Device ID indicates which virtio device is +supported by the device. The Subsystem Vendor ID should reflect +the PCI Vendor ID of the environment (it's currently only used +for informational purposes by the guest). + +All Drivers must match devices with any Revision ID, this +is to allow devices to be versioned without breaking drivers. + +\subsubsection{Legacy Interfaces: A Note on PCI Device Discovery}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery / Legacy Interfaces: A Note on PCI Device Discovery} +Transitional devices must have a Revision ID of 0 to match +legacy drivers. + +Non-transitional devices must have a Revision ID of 1 or higher. + +Both transitional and non-transitional drivers must match +any Revision ID value. + +\subsection{PCI Device Layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout} + +To configure the device, +use I/O and/or memory regions and/or PCI configuration space of the PCI device. +These contain the virtio header registers, the notification register, the +ISR status register and device specific registers, as specified by Virtio ++ Structure PCI Capabilities + +There may be different widths of accesses to the I/O region; the +“natural” access method for each field must be +used (i.e. 32-bit accesses for 32-bit fields, etc). + +PCI Device Configuration Layout includes the common configuration, +ISR, notification and device specific configuration +structures. + +Unless explicitly specified otherwise, all multi-byte fields are little-endian. + +\subsubsection{Common configuration structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout} +Common configuration structure layout is documented below: + +\begin{lstlisting} + struct virtio_pci_common_cfg { + /* About the whole device. */ + le32 device_feature_select; /* read-write */ + le32 device_feature; /* read-only */ + le32 guest_feature_select; /* read-write */ + le32 guest_feature; /* read-write */ + le16 msix_config; /* read-write */ + le16 num_queues; /* read-only */ + u8 device_status; /* read-write */ + u8 config_generation; /* read-only */ + + /* About a specific virtqueue. */ + le16 queue_select; /* read-write */ + le16 queue_size; /* read-write, power of 2, or 0. */ + le16 queue_msix_vector; /* read-write */ + le16 queue_enable; /* read-write */ + le16 queue_notify_off; /* read-only */ + le64 queue_desc; /* read-write */ + le64 queue_avail; /* read-write */ + le64 queue_used; /* read-write */ + }; +\end{lstlisting} + +device_feature_select + + Selects which Feature Bits does device_feature field refer to. + Value 0x0 selects Feature Bits 0 to 31 + Value 0x1 selects Feature Bits 32 to 63 + All other values cause reads from device_feature to return 0. + +device_feature + + Used by Device to report Feature Bits to Driver. + Device Feature Bits selected by device_feature_select. + +guest_feature_select + + Selects which Feature Bits does guest_feature field refer to. + Value 0x0 selects Feature Bits 0 to 31 + Value 0x1 selects Feature Bits 32 to 63 + When set to any other value, reads from guest_feature + return 0, writing 0 into guest_feature has no effect, and + writing any other value into guest_feature is an error. + +guest_feature + + Used by Driver to acknowledge Feature Bits to Device. + Guest Feature Bits selected by guest_feature_select. + +msix_config + + Configuration Vector for MSI-X. + +num_queues + + Specifies the maximum number of virtqueues supported by device. + +device_status + + Device Status field. Writing 0 into this field resets the + device. + +config_generation + + Configuration atomicity value. Changes every time the + configuration noticeably changes. This means the device may + only change the value after a configuration read operation, + but it must change if there is any risk of a device seeing an + inconsistent configuration state. + +queue_select + + Queue Select. Selects which virtqueue do other fields refer to. + +queue_size + + Queue Size. On reset, specifies the maximum queue size supported by + the hypervisor. This can be modified by driver to reduce memory requirements. + Set to 0 if this virtqueue is unused. + +queue_msix_vector + + Queue Vector for MSI-X. + +queue_enable + + Used to selectively prevent host from executing requests from this virtqueue. + 1 - enabled; 0 - disabled + +queue_notify_off + + Used to calculate the offset from start of Notification structure at + which this virtqueue is located. + Note: this is *not* an offset in bytes. See notify_off_multiplier below. + +queue_desc + + Physical address of Descriptor Table. + +queue_avail + + Physical address of Available Ring. + +queue_used + + Physical address of Used Ring. + +\subsubsection{ISR status structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / ISR status structure layout} +ISR status structure includes a single 8-bit ISR status field. + +\subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification structure layout} +Notification structure is always a multiple of 2 bytes in size. +It includes 2-byte Queue Notify fields for each virtqueue of +the device. Note that multiple virtqueues can use the same +Queue Notify field, if necessary. + +\subsubsection{Device specific structure}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device specific structure} + +Device specific structure is optional. + +\subsubsection{Legacy Interfaces: A Note on PCI Device Layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout} + +Transitional devices should present part of configuration +registers in a legacy configuration structure in BAR0 in the first I/O +region of the PCI device, as documented below. + +There may be different widths of accesses to the I/O region; the +“natural” access method for each field in the virtio header must be +used (i.e. 32-bit accesses for 32-bit fields, etc), but +when accessed through the legacy interface the +device-specific region can be accessed using any width accesses, and +should obtain the same results. + +Note that this is possible because while the virtio header is PCI +(i.e. little) endian, when using the legacy interface the device-specific +region is encoded in the native endian of the guest (where such distinction is +applicable). + +When used through the legacy interface, the virtio header looks as follows: + +\begin{verbatim} ++------------++---------------------+---------------------+----------+--------+---------+---------+---------+--------+ +| Bits || 32 | 32 | 32 | 16 | 16 | 16 | 8 | 8 | ++------------++---------------------+---------------------+----------+--------+---------+---------+---------+--------+ +| Read/Write || R | R+W | R+W | R | R+W | R+W | R+W | R | ++------------++---------------------+---------------------+----------+--------+---------+---------+---------+--------+ +| Purpose || Device | Guest | Queue | Queue | Queue | Queue | Device | ISR | +| || Features bits 0:31 | Features bits 0:31 | Address | Size | Select | Notify | Status | Status | ++------------++---------------------+---------------------+----------+--------+---------+---------+---------+--------+ +\end{verbatim} + + +If MSI-X is enabled for the device, two additional fields +immediately follow this header: + + +\begin{verbatim} ++------------++----------------+--------+ +| Bits || 16 | 16 | +\end{verbatim} + +----------------+--------+ +\begin{verbatim} ++------------++----------------+--------+ +| Read/Write || R+W | R+W | ++------------++----------------+--------+ +| Purpose || Configuration | Queue | +| (MSI-X) || Vector | Vector | ++------------++----------------+--------+ +\end{verbatim} + +Note: When MSI-X capability is enabled, device specific configuration starts at +byte offset 24 in virtio header structure. When MSI-X capability is not +enabled, device specific configuration starts at byte offset 20 in virtio +header. ie. once you enable MSI-X on the device, the other fields move. +If you turn it off again, they move back! + +Immediately following these general headers, there may be +device-specific headers: + +\begin{verbatim} ++------------++--------------------+ +| Bits || Device Specific | +\end{verbatim} + +--------------------+ +\begin{verbatim} ++------------++--------------------+ +| Read/Write || Device Specific | ++------------++--------------------+ +| Purpose || Device Specific... | +| || | ++------------++--------------------+ +\end{verbatim} + +Note that only Feature Bits 0 to 31 are accessible through the +Legacy Interface. When used through the Legacy Interface, +Transitional Devices must assume that Feature Bits 32 to 63 +are not acknowledged by Driver. + +As legacy devices had no configuration generation field, +see \ref{sec:Basic Facilities of a Virtio Device / Configuration Space / Legacy Interface: Configuration Space}~\nameref{sec:Basic Facilities of a Virtio Device / Configuration Space / Legacy Interface: Configuration Space} for workarounds. + +\subsection{PCI-specific Initialization And Device Operation}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation} + +\subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization} + +This documents PCI-specific steps executed during Device Initialization. +As the first step, driver must detect device configuration layout +to locate configuration fields in memory,I/O or configuration space of the +device. + +\paragraph{Virtio Device Configuration Layout Detection}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtio Device Configuration Layout Detection} + +As a prerequisite to device initialization, driver executes a +PCI capability list scan, detecting virtio configuration layout using Virtio +Structure PCI capabilities. + +Virtio Device Configuration Layout includes virtio configuration header, Notification +and ISR Status and device configuration structures. +Each structure can be mapped by a Base Address register (BAR) belonging to +the function, located beginning at 10h in Configuration Space, +or accessed though PCI configuration space. + +Actual location of each structure is specified using vendor-specific PCI capability located +on capability list in PCI configuration space of the device. +This virtio structure capability uses little-endian format; all bits are +read-only: + +\begin{lstlisting} + struct virtio_pci_cap { + u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ + u8 cap_next; /* Generic PCI field: next ptr. */ + u8 cap_len; /* Generic PCI field: capability length */ + u8 cfg_type; /* Identifies the structure. */ + u8 bar; /* Where to find it. */ + u8 padding[3]; /* Pad to full dword. */ + le32 offset; /* Offset within bar. */ + le32 length; /* Length of the structure, in bytes. */ + }; +\end{lstlisting} + +This structure can optionally followed by extra data, depending on +other fields, as documented below. + +Note that future versions of this specification will likely +extend devices by adding extra fields at the tail end of some structures. + +To allow forward compatibility with such extensions, drivers must +not limit structure size. Instead, drivers should only +check that structures are *large enough* to contain the fields +required for device operation. + +For example, if the specification states 'structure includes a +single 8-bit field' drivers should understand this to mean that +the structure can also include an arbitrary amount of tail padding, +and accept any structure size equal to or greater than the +specified 8-bit size. + +The fields are interpreted as follows: + +cap_vndr + 0x09; Identifies a vendor-specific capability. + +cap_next + Link to next capability in the capability list in the configuration space. + +cap_len + Length of the capability structure, including the whole of + struct virtio_pci_cap, and extra data if any. + This length might include padding, or fields unused by the driver. + +cfg_type + identifies the structure, according to the following table. + +\begin{lstlisting} + /* Common configuration */ + #define VIRTIO_PCI_CAP_COMMON_CFG 1 + /* Notifications */ + #define VIRTIO_PCI_CAP_NOTIFY_CFG 2 + /* ISR Status */ + #define VIRTIO_PCI_CAP_ISR_CFG 3 + /* Device specific configuration */ + #define VIRTIO_PCI_CAP_DEVICE_CFG 4 + /* PCI configuration access */ + #define VIRTIO_PCI_CAP_PCI_CFG 5 +\end{lstlisting} + + Any other value - reserved for future use. Drivers must + ignore any vendor-specific capability structure which has + a reserved cfg_type value. + + More than one capability can identify the same structure - this makes it + possible for the device to expose multiple interfaces to drivers. The order of + the capabilities in the capability list specifies the order of preference + suggested by the device; drivers should use the first interface that they can + support. For example, on some hypervisors, notifications using IO accesses are + faster than memory accesses. In this case, hypervisor can expose two + capabilities with cfg_type set to VIRTIO_PCI_CAP_NOTIFY_CFG: + the first one addressing an I/O BAR, the second one addressing a memory BAR. + Driver will use the I/O BAR if I/O resources are available, and fall back on + memory BAR when I/O resources are unavailable. + +bar + values 0x0 to 0x5 specify a Base Address register (BAR) belonging to + the function located beginning at 10h in Configuration Space + and used to map the structure into Memory or I/O Space. + The BAR is permitted to be either 32-bit or 64-bit, it can map Memory Space + or I/O Space. + + Any other value - reserved for future use. Drivers must + ignore any vendor-specific capability structure which has + a reserved bar value. + +offset + indicates where the structure begins relative to the base address associated + with the BAR. + +length + indicates the length of the structure. + This size might include padding, or fields unused by the driver. + Drivers are also recommended to only map part of configuration structure + large enough for device operation. + For example, a future device might present a large structure size of several + MBytes. + As current devices never utilize structures larger than 4KBytes in size, + driver can limit the mapped structure size to e.g. + 4KBytes to allow forward compatibility with such devices without loss of + functionality and without wasting resources. + + +If cfg_type is VIRTIO_PCI_CAP_NOTIFY_CFG this structure is immediately followed +by additional fields: + +\begin{lstlisting} + struct virtio_pci_notify_cap { + struct virtio_pci_cap cap; + le32 notify_off_multiplier; /* Multiplier for queue_notify_off. */ + }; +\end{lstlisting} + +notify_off_multiplier + + Virtqueue offset multiplier, in bytes. Must be even and either a power of two, or 0. + Value 0x1 is reserved. + For a given virtqueue, the address to use for notifications is calculated as follows: + + queue_notify_off * notify_off_multiplier + offset + + If notify_off_multiplier is 0, all virtqueues use the same address in + the Notifications structure! + +If cfg_type is VIRTIO_PCI_CAP_PCI_CFG the fields bar, offset and length are RW +and this structure is immediately followed by an additional field: + +\begin{lstlisting} + struct virtio_pci_cfg_cap { + __u8 pci_cfg_data[4]; /* Data for BAR access. */ + }; +\end{lstlisting} + +pci_cfg_data + + This RW field allows an indirect access to any BAR on the + device using PCI configuration accesses. + + The BAR to access is selected using the bar field. + The length of the access is specified by the length + field, which can be set to 1, 2 and 4. + The offset within the BAR is specified by the offset + field, which must be aligned to length bytes. + + After this field is written by driver, the first length + bytes in pci_cfg_data are written at the selected + offset in the selected BAR. + + When this field is read by driver, length bytes at the + selected offset in the selected BAR are read into pci_cfg_data. + +\subparagraph{Legacy Interface: A Note on Device Layout Detection}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtio Device Configuration Layout Detection / Legacy Interface: A Note on Device Layout Detection} + +Legacy drivers skipped Device Layout Detection step, assuming legacy +configuration space in BAR0 in I/O space unconditionally. + +Legacy devices did not have the Virtio PCI Capability in their +capability list. + +Therefore: + +Transitional devices should expose the Legacy Interface in I/O +space in BAR0. + +Transitional drivers should look for the Virtio PCI +Capabilities on the capability list. +If these are not present, driver should assume a legacy device. + +Non-transitional drivers should look for the Virtio PCI +Capabilities on the capability list. +If these are not present, driver should assume a legacy device, +and fail gracefully. + +Non-transitional devices, on a platform where a legacy driver for +a legacy device with the same ID might have previously existed, +must take the following steps to fail gracefully when a legacy +driver attempts to drive them: + +1) Present an I/O BAR in BAR0, and +2) Respond to a single-byte zero write to offset 18 + (corresponding to Device Status register in the legacy layout) + of BAR0 by presenting zeroes on every BAR and ignoring writes. + +\paragraph{Queue Vector Configuration}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Queue Vector Configuration} + +When MSI-X capability is present and enabled in the device +(through standard PCI configuration space) Configuration/Queue +MSI-X Vector registers are used to map configuration change and queue +interrupts to MSI-X vectors. In this case, the ISR Status is unused. + +Writing a valid MSI-X Table entry number, 0 to 0x7FF, to one of +Configuration/Queue Vector registers, maps interrupts triggered +by the configuration change/selected queue events respectively to +the corresponding MSI-X vector. To disable interrupts for a +specific event type, unmap it by writing a special NO_VECTOR +value: + +\begin{lstlisting} + /* Vector value used to disable MSI for queue */ + #define VIRTIO_MSI_NO_VECTOR 0xffff +\end{lstlisting} + +Reading these registers returns vector mapped to a given event, +or NO_VECTOR if unmapped. All queue and configuration change +events are unmapped by default. + +Note that mapping an event to vector might require allocating +internal device resources, and might fail. Devices report such +failures by returning the NO_VECTOR value when the relevant +Vector field is read. After mapping an event to vector, the +driver must verify success by reading the Vector field value: on +success, the previously written value is returned, and on +failure, NO_VECTOR is returned. If a mapping failure is detected, +the driver can retry mapping with fewervectors, or disable MSI-X. + +\paragraph{Virtqueue Configuration}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtqueue Configuration} + +As a device can have zero or more virtqueues for bulk data +transport (for example, the simplest network device has two), the driver +needs to configure them as part of the device-specific +configuration. + +This is done as follows, for each virtqueue a device has: + +1. Write the virtqueue index (first queue is 0) to the Queue + Select field. + +2. Read the virtqueue size from the Queue Size field, which is + always a power of 2. This controls how big the virtqueue is + (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues}). If this field is 0, the virtqueue does not exist. + +3. Optionally, select a smaller virtqueue size and write it in the Queue Size + field. + +4. Allocate and zero Descriptor Table, Available and Used rings for the + virtqueue in contiguous physical memory. + +5. Optionally, if MSI-X capability is present and enabled on the + device, select a vector to use to request interrupts triggered + by virtqueue events. Write the MSI-X Table entry number + corresponding to this vector in Queue Vector field. Read the + Queue Vector field: on success, previously written value is + returned; on failure, NO_VECTOR value is returned. + +\subparagraph{Legacy Interface: A Note on Virtqueue Configuration}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtqueue Configuration / Legacy Interface: A Note on Virtqueue Configuration} +When using the legacy interface, the page size for a virtqueue on a PCI virtio +device is defined as 4096 bytes. Driver writes the physical address, divided +by 4096 to the Queue Address field +\footnote{The 4096 is based on the x86 page size, but it's also large +enough to ensure that the separate parts of the virtqueue are on +separate cache lines. +}. + +\subsubsection{Notifying The Device}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notifying The Device} + +Device notification occurs by writing the 16-bit virtqueue index +of this virtqueue to the Queue Notify field. + +\subsubsection{Virtqueue Interrupts From The Device}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Virtqueue Interrupts From The Device} + +If an interrupt is necessary: + + (a) If MSI-X capability is disabled: + + i. Set the lower bit of the ISR Status field for the device. + + ii. Send the appropriate PCI interrupt for the device. + + (b) If MSI-X capability is enabled: + + i. Request the appropriate MSI-X interrupt message for the + device, Queue Vector field sets the MSI-X Table entry + number. + + ii. If Queue Vector field value is NO_VECTOR, no interrupt + message is requested for this event. + +The guest interrupt handler should: + +1. If MSI-X capability is disabled: read the ISR Status field, + which will reset it to zero. If the lower bit is zero, the + interrupt was not for this device. Otherwise, the guest driver + should look through the used rings of each virtqueue for the + device, to see if any progress has been made by the device + which requires servicing. + +2. If MSI-X capability is enabled: look through the used rings of + each virtqueue mapped to the specific MSI-X vector for the + device, to see if any progress has been made by the device + which requires servicing. + +\subsubsection{Notification of Device Configuration Changes}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notification of Device Configuration Changes} + +Some virtio PCI devices can change the device configuration +state, as reflected in the virtio header in the PCI configuration +space. In this case: + +1. If MSI-X capability is disabled: an interrupt is delivered and + the second highest bit is set in the ISR Status field to + indicate that the driver should re-examine the configuration + space. Note that a single interrupt can indicate both that one + or more virtqueue has been used and that the configuration + space has changed: even if the config bit is set, virtqueues + must be scanned. + +2. If MSI-X capability is enabled: an interrupt message is + requested. The Configuration Vector field sets the MSI-X Table + entry number to use. If Configuration Vector field value is + NO_VECTOR, no interrupt message is requested for this event. + +\section{Virtio Over MMIO}\label{sec:Virtio Transport Options / Virtio Over MMIO} + +Virtual environments without PCI support (a common situation in +embedded devices models) might use simple memory mapped device +("virtio-mmio") instead of the PCI device. + +The memory mapped virtio device behaviour is based on the PCI +device specification. Therefore most of operations like device +initialization, queues configuration and buffer transfers are +nearly identical. Existing differences are described in the +following sections. + +\subsection{MMIO Device Discovery}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO Device Discovery} + +Unlike PCI, MMIO provides no generic device discovery. For +systems using Flattened Device Trees the suggested format is: + +\begin{lstlisting} + virtio_block@1e000 { + compatible = "virtio,mmio"; + reg = <0x1e000 0x100>; + interrupts = <42>; + } +\end{lstlisting} + +\subsection{MMIO Device Layout}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO Device Layout} + +MMIO virtio devices provides a set of memory mapped control +registers, all 32 bits wide, followed by device-specific +configuration space. The following list presents their layout: + +* Offset from the device base address | Direction | Name + Description + +* 0x000 | R | MagicValue + Magic value. Must be 0x74726976 (a Little Endian equivalent + of a "virt" string). + +* 0x004 | R | Version + Device version number. Devices compliant with this specification + must return value 0x2. + +* 0x008 | R | DeviceID + Virtio Subsystem Device ID. + See \ref{sec:Device Types}~\nameref{sec:Device Types} for possible values. Value zero (0x0) + is invalid and devices returning this ID must be ignored + by the guest. + +* 0x00c | R | VendorID + Virtio Subsystem Vendor ID. + +* 0x010 | R | HostFeatures + Flags representing features the device supports. + Reading from this register returns 32 consecutive flag bits, + first bit depending on the last value written to the + HostFeaturesSel register. Access to this register returns + bits HostFeaturesSel*32 to (HostFeaturesSel*32)+31, eg. + feature bits 0 to 31 if HostFeaturesSel is set to 0 and + features bits 32 to 63 if HostFeaturesSel is set to 1. + Also see \ref{sec:Basic Facilities of a Virtio Device / Feature Bits}~\nameref{sec:Basic Facilities of a Virtio Device / Feature Bits}. + +* 0x014 | W | HostFeaturesSel + Device (Host) features word selection. + Writing to this register selects a set of 32 device feature bits + accessible by reading from the HostFeatures register. Device driver + must write a value to the HostFeaturesSel register before + reading from the HostFeatures register. + +* 0x020 | W | GuestFeatures + Flags representing device features understood and activated by + the driver. + Writing to this register sets 32 consecutive flag bits, first + bit depending on the last value written to the GuestFeaturesSel + register. Access to this register sets bits GuestFeaturesSel*32 + to (GuestFeaturesSel*32)+31, eg. feature bits 0 to 31 if + GuestFeaturesSel is set to 0 and features bits 32 to 63 if + GuestFeaturesSel is set to 1. Also see \ref{sec:Basic Facilities of a Virtio Device / Feature Bits}~\nameref{sec:Basic Facilities of a Virtio Device / Feature Bits}. + +* 0x024 | W | GuestFeaturesSel + Activated (Guest) features word selection. + Writing to this register selects a set of 32 activated feature + bits accessible by writing to the GuestFeatures register. + Device driver must write a value to the GuestFeaturesSel + register before writing to the GuestFeatures register. + +* 0x030 | W | QueueSel + Virtual queue index (first queue is 0). + Writing to this register selects the virtual queue that the + following operations on the QueueNumMax, QueueNum, QueueReady, + QueueDescLow, QueueDescHigh, QueueAvailLow, QueueAvailHigh, + QueueUsedLow and QueueUsedHigh registers apply to. + +* 0x034 | R | QueueNumMax + Maximum virtual queue size. + Reading from the register returns the maximum size of the queue + the Host is ready to process or zero (0x0) if the queue is not + available. This applies to the queue selected by writing to + QueueSel and is allowed only when QueueReady is set to zero + (0x0), so when the queue is not in use. + +* 0x038 | W | QueueNum + Virtual queue size. + Queue size is the number of elements in the queue, therefore size + of the Descriptor Table and both Available and Used rings. + Writing to this register notifies the Host what size of the + queue the Guest will use. This applies to the queue selected by + writing to QueueSel and is allowed only when QueueReady is set + to zero (0x0), so when the queue is not in use. + +* 0x03c | RW | QueueReady + Virtual queue ready bit. + Writing one (0x1) to this register notifies the Host that the + virtual queue is ready to be used. Reading from this register + returns the last value written to it. Both read and write + accesses apply to the queue selected by writing to QueueSel. + When the Guest wants to stop using the queue it must write + zero (0x0) to this register and read the value back to + ensure synchronisation. + +* 0x050 | W | QueueNotify + Queue notifier. + Writing a queue index to this register notifies the Host that + there are new buffers to process in the queue. + +* 0x60 | R | InterruptStatus + Interrupt status. + Reading from this register returns a bit mask of interrupts + asserted by the device. An interrupt is asserted if the + corresponding bit is set, ie. equals one (1). + + – Bit 0 | Used Ring Update + This interrupt is asserted when the Host has updated the Used + Ring in at least one of the active virtual queues. + + – Bit 1 | Configuration change + This interrupt is asserted when configuration of the device has + changed. + +* 0x064 | W | InterruptACK + Interrupt acknowledge. + Writing to this register notifies the Host that the Guest + finished handling interrupts. Set bits in the value clear + the corresponding bits of the InterruptStatus register. + +* 0x070 | RW | Status + Device status. + Reading from this register returns the current device status + flags. + Writing non-zero values to this register sets the status flags, + indicating the Guest progress. Writing zero (0x0) to this + register triggers a device reset, including clearing all + bits in the InterruptStatus register and ready bits in the + QueueReady register for all queues in the device. + See also p. \ref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization}~\nameref{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization}. + +* 0x080 | W | QueueDescLow + 0x084 | W | QueueDescHigh + Virtual queue's Descriptor Table 64 bit long physical address. + Writing to these two registers (lower 32 bits of the address + to QueueDescLow, higher 32 bits to QueueDescHigh) notifies + the host about location of the Descriptor Table of the queue + selected by writing to the QueueSel register. It is allowed + only when QueueReady is set to zero (0x0), so when the queue + is not in use. + +* 0x090 | W | QueueAvailLow + 0x094 | W | QueueAvailHigh + Virtual queue's Available Ring 64 bit long physical address. + Writing to these two registers (lower 32 bits of the address + to QueueAvailLow, higher 32 bits to QueueAvailHigh) notifies + the host about location of the Available Ring of the queue + selected by writing to the QueueSel register. It is allowed + only when QueueReady is set to zero (0x0), so when the queue + is not in use. + +* 0x0a0 | W | QueueUsedLow + 0x0a4 | W | QueueUsedHigh + Virtual queue's Used Ring 64 bit long physical address. + Writing to these two registers (lower 32 bits of the address + to QueueUsedLow, higher 32 bits to QueueUsedHigh) notifies + the host about location of the Used Ring of the queue + selected by writing to the QueueSel register. It is allowed + only when QueueReady is set to zero (0x0), so when the queue + is not in use. + +* 0x0fc | R | ConfigGeneration + Configuration atomicity value. + Changes every time the configuration noticeably changes. This + means the device may only change the value after a configuration + read operation, but it must change if there is any risk of a + device seeing an inconsistent configuration state. + +* 0x100+ | RW | Config + Device-specific configuration space starts at an offset 0x100 + and is accessed with byte alignment. Its meaning and size + depends on the device and the driver. + +All register values are organized as Little Endian. + +Accessing memory locations not explicitly described above (or +- in case of the configuration space - described in the device +specification), writing to the registers described as "R" and +reading from registers described as "W" is not permitted and +can cause undefined behavior. + +\subsection{MMIO-specific Initialization And Device Operation}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation} + +\subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Device Initialization} + +The guest must start the device initialization by reading and +checking values from the MagicValue and the Version registers. +If both values are valid, it must read the DeviceID register +and if its value is zero (0x0) must abort initialization and +must not access any other register. + +Further initialization must follow the procedure described in +p. \ref{sec:General Initialization And Device Operation / Device Initialization}~\nameref{sec:General Initialization And Device Operation / Device Initialization}. + +\subsubsection{Virtqueue Configuration}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Virtqueue Configuration} + +1. Select the queue writing its index (first queue is 0) to the + QueueSel register. + +2. Check if the queue is not already in use: read the QueueReady + register, returned value should be zero (0x0). + +3. Read maximum queue size (number of elements) from the + QueueNumMax register. If the returned value is zero (0x0) the + queue is not available. + +4. Allocate and zero the queue pages, making sure the memory + is physically contiguous. It is recommended to align the + Used Ring to an optimal boundary (usually page size). + Size of the allocated queue may be smaller than or equal to + the maximum size returned by the Host. + +5. Notify the Host about the queue size by writing the size to + the QueueNum register. + +6. Write physical addresses of the queue's Descriptor Table, + Available Ring and Used Ring to (respectively) the QueueDescLow/ + QueueDescHigh, QueueAvailLow/QueueAvailHigh and QueueUsedLow/ + QueueUsedHigh register pairs. + +7. Write 0x1 to the QueueReady register. + +\subsubsection{Notifying The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifying The Device} + +The device is notified about new buffers available in a queue by +writing the queue index to the QueueNum register. + +\subsubsection{Notifications From The Device}\label{sec:Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device} + +The memory mapped virtio device is using single, dedicated +interrupt signal, which is raised when at least one of the +interrupts described in the InterruptStatus register +description is asserted. After receiving an interrupt, the +driver must read the InterruptStatus register to check what +caused the interrupt (see the register description). After the +interrupt is handled, the driver must acknowledge it by writing +a bit mask corresponding to the serviced interrupt to the +InterruptACK register. + +As documented in the InterruptStatus register description, +the device may notify the driver about a new used buffer being +available in the queue or about a change in the device +configuration. + +\subsection{Legacy interface}\label{sec:Virtio Transport Options / Virtio Over MMIO / Legacy interface} + +The legacy MMIO transport used page-based addressing, resulting +in a slightly different control register layout, the device +initialization and the virtual queue configuration procedure. + +The following list presents control registers layout, omitting +descriptions of registers which did not change their function +nor behaviour: + +* Offset from the device base address | Direction | Name + Description + +* 0x000 | R | MagicValue + +* 0x004 | R | Version + Device version number. Legacy devices must return value 0x1. + +* 0x008 | R | DeviceID + +* 0x00c | R | VendorID + +* 0x010 | R | HostFeatures + +* 0x014 | W | HostFeaturesSel + +* 0x020 | W | GuestFeatures + +* 0x024 | W | GuestFeaturesSel + +* 0x028 | W | GuestPageSize + Guest page size. + Device driver must write the guest page size in bytes to the + register during initialization, before any queues are used. + This value must be a power of 2 and is used by the Host to + calculate the Guest address of the first queue page + (see QueuePFN). + +* 0x030 | W | QueueSel + Virtual queue index (first queue is 0). + Writing to this register selects the virtual queue that the + following operations on the QueueNumMAx, QueueNum, QueueAlign + and QueuePFN registers apply to. + +* 0x034 | R | QueueNumMax + Maximum virtual queue size. + Reading from the register returns the maximum size of the queue + the Host is ready to process or zero (0x0) if the queue is not + available. This applies to the queue selected by writing to the + QueueSel and is allowed only when the QueuePFN is set to zero + (0x0), so when the queue is not actively used. + +* 0x038 | W | QueueNum + Virtual queue size. + Queue size is the number of elements in the queue, therefore size + of the descriptor table and both available and used rings. + Writing to this register notifies the Host what size of the + queue the Guest will use. This applies to the queue selected by + writing to the QueueSel register. + +* 0x03c | W | QueueAlign + Used Ring alignment in the virtual queue. + Writing to this register notifies the Host about alignment + boundary of the Used Ring in bytes. This value must be a power + of 2 and applies to the queue selected by writing to the QueueSel + register. + +* 0x040 | RW | QueuePFN + Guest physical page number of the virtual queue. + Writing to this register notifies the host about location of the + virtual queue in the Guest's physical address space. This value + is the index number of a page starting with the queue + Descriptor Table. Value zero (0x0) means physical address zero + (0x00000000) and is illegal. When the Guest stops using the + queue it must write zero (0x0) to this register. + Reading from this register returns the currently used page + number of the queue, therefore a value other than zero (0x0) + means that the queue is in use. + Both read and write accesses apply to the queue selected by + writing to the QueueSel register. + +* 0x050 | W | QueueNotify + +* 0x060 | R | InterruptStatus + +* 0x064 | W | InterruptACK + +* 0x070 | RW | Status + Device status. + Reading from this register returns the current device status + flags. + Writing non-zero values to this register sets the status flags, + indicating the Guest progress. Writing zero (0x0) to this + register triggers a device reset. This should include + setting QueuePFN to zero (0x0) for all queues in the device. + Also see \ref{sec:General Initialization And Device Operation / Device Initialization}~\nameref{sec:General Initialization And Device Operation / Device Initialization}. + +* 0x100+ | RW | Config + +The virtual queue page size is defined by writing to the GuestPageSize +register, as written by the guest. This must be done before the +virtual queues are configured. + +The virtual queue layout follows +p. \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}, +with the alignment defined in the QueueAlign register. + +The virtual queue is configured as follows: + +1. Select the queue writing its index (first queue is 0) to the + QueueSel register. + +2. Check if the queue is not already in use: read the QueuePFN + register, returned value should be zero (0x0). + +3. Read maximum queue size (number of elements) from the + QueueNumMax register. If the returned value is zero (0x0) the + queue is not available. + +4. Allocate and zero the queue pages in contiguous virtual + memory, aligning the Used Ring to an optimal boundary (usually + page size). Size of the allocated queue may be smaller than or + equal to the maximum size returned by the Host. + +5. Notify the Host about the queue size by writing the size to + the QueueNum register. + +6. Notify the Host about the used alignment by writing its value + in bytes to the QueueAlign register. + +7. Write the physical number of the first page of the queue to + the QueuePFN register. + +Notification mechanisms did not change. + +\section{Virtio over channel I/O}\label{sec:Virtio Transport Options / Virtio over channel I/O} + +S/390 based virtual machines support neither PCI nor MMIO, so a +different transport is needed there. + +virtio-ccw uses the standard channel I/O based mechanism used for +the majority of devices on S/390. A virtual channel device with a +special control unit type acts as proxy to the virtio device +(similar to the way virtio-pci uses a PCI device) and +configuration and operation of the virtio device is accomplished +(mostly) via channel commands. This means virtio devices are +discoverable via standard operating system algorithms, and adding +virtio support is mainly a question of supporting a new control +unit type. + +As the S/390 is a big endian machine, the data structures transmitted +via channel commands are big-endian: this is made clear by use of +the types be16, be32 and be64. + +\subsection{Basic Concepts}\label{sec:Virtio Transport Options / Virtio over channel I/O / Basic Concepts} + +As a proxy device, virtio-ccw uses a channel-attached I/O control +unit with a special control unit type (0x3832) and a control unit +model corresponding to the attached virtio device's subsystem +device ID, accessed via a virtual I/O subchannel and a virtual +channel path of type 0x32. This proxy device is discoverable via +normal channel subsystem device discovery (usually a STORE +SUBCHANNEL loop) and answers to the basic channel commands, most +importantly SENSE ID. + +For a virtio-ccw proxy device, SENSE ID will return the following +information: + +\begin{verbatim} ++-------+--------------------------------------------+ +| Bytes | Contents | +|-------|--------------------------------------------| +| 0 | reserved | 0xff | +|-------|-----------------------|--------------------| +| 1-2 | control unit type | 0x3832 | +|-------|-----------------------|--------------------| +| 3 | control unit model | <virtio device id> | +|-------|-----------------------|--------------------| +| 4-5 | device type | zeroes (unset) | +|-------|-----------------------|--------------------| +| 6 | device model | zeroes (unset) | +|-------|-----------------------|--------------------| +| 7-255 | extended SenseId data | zeroes (unset) | ++-------+--------------------------------------------+ +\end{verbatim} + +A driver for virtio-ccw devices MUST check for a control unit +type of 0x3832 and MUST ignore the device type and model. + +In addition to the basic channel commands, virtio-ccw defines a +set of channel commands related to configuration and operation of +virtio: + +\begin{lstlisting} + #define CCW_CMD_SET_VQ 0x13 + #define CCW_CMD_VDEV_RESET 0x33 + #define CCW_CMD_SET_IND 0x43 + #define CCW_CMD_SET_CONF_IND 0x53 + #define CCW_CMD_SET_IND_ADAPTER 0x73 + #define CCW_CMD_READ_FEAT 0x12 + #define CCW_CMD_WRITE_FEAT 0x11 + #define CCW_CMD_READ_CONF 0x22 + #define CCW_CMD_WRITE_CONF 0x21 + #define CCW_CMD_WRITE_STATUS 0x31 + #define CCW_CMD_READ_VQ_CONF 0x32 + #define CCW_CMD_SET_VIRTIO_REV 0x83 +\end{lstlisting} + +The virtio-ccw device acts like a normal channel device, as specified +in [S390 PoP] and [S390 Common I/O]. In particular: + +- A device must post a unit check with command reject for any command + it does not support. + +- If a driver did not suppress length checks for a channel command, + the device must present a subchannel status as detailed in the + architecture when the actual length did not match the expected length. + +- If a driver did suppress length checks for a channel command, the + device must present a check condition if the transmitted data does + not contain enough data to process the command. If the driver submitted + a buffer that was too long, the device should accept the command. + The driver should attempt to provide the correct length even if it + suppresses length checks. + +\subsection{Device Initialization}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization} + +virtio-ccw uses several channel commands to set up a device. + +\subsubsection{Setting the Virtio Revision}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting the Virtio Revision} + +CCW_CMD_SET_VIRTIO_REV is issued by the driver to set the revision of +the virtio-ccw transport it intends to drive the device with. It uses the +following communication structure: + +\begin{lstlisting} + struct virtio_rev_info { + __u16 revision; + __u16 length; + __u8 data[]; + }; +\end{lstlisting} + +revision contains the desired revision id, length the length of the +data portion and data revision-dependent additional desired options. + +The following values are supported: + +\begin{verbatim} ++----------+--------+-----------+--------------------------------+ +| revision | length | data | remarks | +|----------|--------|-----------|--------------------------------| +| 0 | 0 | <empty> | legacy interface; transitional | +| | | | devices only | +|----------|--------|-----------|--------------------------------| +| 1 | 0 | <empty> | Virtio 1.0 | +|----------|--------|-----------|--------------------------------| +| 2-n | | | reserved for later revisions | ++----------+--------+-----------+--------------------------------+ +\end{verbatim} + +Note that a change in the virtio standard does not neccessarily +correspond to a change in the virtio-ccw revision. + +A device must post a unit check with command reject for any revision +it does not support. For any invalid combination of revision, length +and data, it must post a unit check with command reject as well. A +non-transitional device must reject revision id 0. + +A driver should start with trying to set the highest revision it +supports and continue with lower revisions if it gets a command reject. + +A driver must not issue any other virtio-ccw specific channel commands +prior to setting the revision. + +A device must answer with command reject to any virtio-ccw specific +channel command that is not contained in the revision selected by the +driver. + +After a revision has been successfully selected by the driver, it +must not attempt to select a different revision. A device must answer +to any such attempt with a command reject. + +A device must treat the revision as unset from the time the associated +subchannel has been enabled until a revision has been successfully set +by the driver. This implies that revisions are not persistent across +disabling and enabling of the associated subchannel. + +\paragraph{Legacy Interfaces: A Note on Setting the Virtio Revision}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting the Virtio Revision / Legacy Interfaces: A Note on Setting the Virtio Revision} + +A legacy device will not support the CCW_CMD_SET_VIRTIO_REV and answer +with a command reject. A non-transitional driver must stop trying to +operate this device in that case. A transitional driver must operate +the device as if it had been able to set revision 0. + +A legacy driver will not issue the CCW_CMD_SET_VIRTIO_REV prior to +issueing other virtio-ccw specific channel commands. A non-transitional +device therefore must answer any such attempts with a command reject. +A transitional device must assume in this case that the driver is a +legacy driver and continue as if the driver selected revision 0. This +implies that the device must reject any command not valid for revision +0, including a subsequent CCW_CMD_SET_VIRTIO_REV. + +\subsubsection{Configuring a Virtqueue}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Configuring a Virtqueue} + +CCW_CMD_READ_VQ_CONF is issued by the guest to obtain information +about a queue. It uses the following structure for communicating: + +\begin{lstlisting} + struct vq_config_block { + be16 index; + be16 max_num; + } __attribute__ ((packed)); +\end{lstlisting} + +The requested number of buffers for queue index is returned in +max_num. + +Afterwards, CCW_CMD_SET_VQ is issued by the guest to inform the +host about the location used for its queue. The transmitted +structure is + +\begin{lstlisting} + struct vq_info_block { + be64 desc; + be32 res0; + be16 index; + be16 num; + be64 avail; + be64 used; + } __attribute__ ((packed)); +\end{lstlisting} + +desc, avail and used contain the guest addresses for the descriptor table, +available ring and used ring for queue index, respectively. The actual +virtqueue size (number of allocated buffers) is transmitted in num. +res0 is reserved and must be ignored by the device. + +\paragraph{Legacy Interface: A Note on Configuring a Virtqueue}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Configuring a Virtqueue / Legacy Interface: A Note on Configuring a Virtqueue} + +For a legacy driver or for a driver that selected revision 0, +CCW_CMD_SET_VQ uses the following communication block: + +\begin{lstlisting} + struct vq_info_block_legacy { + be64 queue; + be32 align; + be16 index; + be16 num; + } __attribute__ ((packed)); +\end{lstlisting} + +queue contains the guest address for queue index, num the number of buffers +and align the alignment. + +\subsubsection{Virtqueue Layout}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Virtqueue Layout} + +The virtqueue is physically contiguous, with padded added to make the +used ring meet the align value: + +\begin{verbatim} ++-------------------+-----------------------------------+-----------+ +| Descriptor Table | Available Ring (padding) | Used Ring | ++-------------------+-----------------------------------+-----------+ +\end{verbatim} + +The calculation for total size is as follows: + +\begin{lstlisting} + #define ALIGN(x) (((x) + align) & ~align) + static inline unsigned vring_size(unsigned int num) + { + return ALIGN(sizeof(struct vring_desc)*num + + sizeof(u16)*(3 + num)) + + ALIGN(sizeof(u16)*3 + sizeof(struct vring_used_elem)*num); + } +\end{lstlisting} + +\subsubsection{Communicating Status Information}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Communicating Status Information} + +The guest can change the status of a device via the +CCW_CMD_WRITE_STATUS command, which transmits an 8 bit status +value. + +\subsubsection{Handling Device Features}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Handling Device Features} + +Feature bits are arranged in an array of 32 bit values, making +for a total of 8192 feature bits. Feature bits are in +little-endian byte order. + +The CCW commands dealing with features use the following +communication block: + +\begin{lstlisting} + struct virtio_feature_desc { + be32 features; + u8 index; + } __attribute__ ((packed)); +\end{lstlisting} + +features are the 32 bits of features currently accessed, while +index describes which of the feature bit values is to be +accessed. + +The guest may obtain the host's device feature set via the +CCW_CMD_READ_FEAT command. The host stores the features at index +to features. + +For communicating its device features to the host, the guest may +use the CCW_CMD_WRITE_FEAT command, denoting a features/index +combination. + +\subsubsection{Device Configuration}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Device Configuration} + +The device's configuration space is located in host memory. It is +the same size as the standard PCI configuration space. + +To obtain information from the configuration space, the guest may +use CCW_CMD_READ_CONF, specifying the guest memory for the host +to write to. + +For changing configuration information, the guest may use +CCW_CMD_WRITE_CONF, specifying the guest memory for the host to +read from. + +In both cases, the complete configuration space is transmitted. This +allows the guest to compare the new configuration space with the old +version, and keep a generation count internally whenever it changes. + +\subsubsection{Setting Up Indicators}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting Up Indicators} + +In order to set up the indicator bits for host->guest notification, +the driver uses different channel commands depending on whether it +wishes to use traditional I/O interrupts tied to a subchannel or +adapter I/O interrupts for virtqueue notifications. For any given +device, the two mechanisms are mutually exclusive. + +For the configuration change indicators, only a mechanism using +traditional I/O interrupts is provided, regardless of whether +traditional or adapter I/O interrupts are used for virtqueue +notifications. + +\paragraph{Setting Up Classic Queue Indicators}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting Up Indicators / Setting Up Classic Queue Indicators} + +Indicators for notification via classic I/O interrupts are contained +in a 64 bit value per virtio-ccw proxy device. + +To communicate the location of the indicator bits for host->guest +notification, the guest uses the CCW_CMD_SET_IND command, +pointing to a location containing the guest address of the +indicators in a 64 bit value. + +If the driver has already set up two-staged queue indicators via the +CCW_CMD_SET_IND_ADAPTER command, the device MUST post a unit check +with command reject to any subsequent CCW_CMD_SET_IND command. + +\paragraph{Setting Up Configuration Change Indicators}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting Up Indicators / Setting Up Configuration Change Indicators} + +Indicators for configuration change host->guest notification are +contained in a 64 bit value per virtio-ccw proxy device. + +To communicate the location of the indicator bits used in the +configuration change host->guest notification, the driver issues the +CCW_CMD_SET_CONF_IND command, pointing to a location containing the +guest address of the indicators in a 64 bit value. + +\paragraph{Setting Up Two-Stage Queue Indicators}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting Up Indicators / Setting Up Two-Stage Queue Indicators} + +Indicators for notification via adapter I/O interrupts consist of +two stages: +- a summary indicator byte covering the virtqueues for one or more + virtio-ccw proxy devices +- a set of contigous indicator bits for the virtqueues for a + virtio-ccw proxy device + +To communicate the location of the summary and queue indicator bits, +the driver uses the CCW_CMD_SET_IND_ADAPTER command with the following +payload: + +\begin{lstlisting} + struct virtio_thinint_area { + be64 summary_indicator; + be64 indicator; + be64 bit_nr; + u8 isc; + } __attribute__ ((packed)); +\end{lstlisting} + +summary_indicator contains the guest address of the 8 bit summary +indicator. +indicator contains the guest address of an area wherin the indicators +for the devices are contained, starting at bit_nr, one bit per +virtqueue of the device. Bit numbers start at the left. +isc contains the I/O interruption subclass to be used for the adapter +I/O interrupt. It may be different from the isc used by the proxy +virtio-ccw device's subchannel. + +If the driver has already set up classic queue indicators via the +CCW_CMD_SET_IND command, the device MUST post a unit check with +command reject to any subsequent CCW_CMD_SET_IND_ADAPTER command. + +\paragraph{Legacy Interfaces: A Note on Setting Up Indicators}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting Up Indicators / Legacy Interfaces: A Note on Setting Up Indicators} + +Legacy devices will only support classic queue indicators; they will +reject CCW_CMD_SET_IND_ADAPTER as they don't know that command. + +\subsection{Device Operation}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Operation} + +\subsubsection{Host->Guest Notification}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Operation / Host->Guest Notification} + +There are two modes of operation regarding host->guest notifcation, +classic I/O interrupts and adapter I/O interrupts. The mode to be +used is determined by the driver by using CCW_CMD_SET_IND respectively +CCW_CMD_SET_IND_ADAPTER to set up queue indicators. + +For configuration changes, the driver will always use classic I/O +interrupts. + +\paragraph{Notification via Classic I/O Interrupts}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Operation / Host->Guest Notification / Notification via Classic I/O Interrupts} + +If the driver used the CCW_CMD_SET_IND command to set up queue +indicators, the device will use classic I/O interrupts for +host->guest notification about virtqueue activity. + +For notifying the guest of virtqueue buffers, the host sets the +corresponding bit in the guest-provided indicators. If an +interrupt is not already pending for the subchannel, the host +generates an unsolicited I/O interrupt. + +If the host wants to notify the guest about configuration +changes, it sets bit 0 in the configuration indicators and +generates an unsolicited I/O interrupt, if needed. This also +applies if adapter I/O interrupts are used for queue notifications. + +\paragraph{Notification via Adapter I/O Interrupts}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Operation / Host->Guest Notification / Notification via Adapter I/O Interrupts} + +If the driver used the CCW_CMD_SET_IND_ADAPTER command to set up +queue indicators, the device will use adapter I/O interrupts for +host->guest notification about virtqueue activity. + +For notifying the guest of virtqueue buffers, the host sets the +bit in the guest-provided indicator area at the corresponding offset. +The guest-provided summary indicator is also set. An adapter I/O +interrupt for the corresponding interruption subclass is generated. +The device SHOULD only generate an adapter I/O interrupt if the +summary indicator had not been set prior to notification. The driver +MUST clear the summary indicator after receiving an adapter I/O +interrupt before it processes the queue indicators. + +\paragraph{Legacy Interfaces: A Note on Host->Guest Notification}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Operation / Host->Guest Notification / Legacy Interfaces: A Note on Host->Guest Notification} + +As legacy devices and drivers support only classic queue indicators, +host->guest notification will always be done via classic I/O interrupts. + +\subsubsection{Guest->Host Notification}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Operation / Guest->Host Notification} + +For notifying the host of virtqueue buffers, the guest +unfortunately can't use a channel command (the asynchronous +characteristics of channel I/O interact badly with the host block +I/O backend). Instead, it uses a diagnose 0x500 call with subcode +3 specifying the queue, as follows: + +\begin{verbatim} ++------+-------------------+--------------+ +| GPR | Input Value | Output Value | ++------+-------------------+--------------+ ++------+-------------------+--------------+ +| 1 | 0x3 | | ++------+-------------------+--------------+ +| 2 | Subchannel ID | Host Cookie | ++------+-------------------+--------------+ +| 3 | Virtqueue number | | ++------+-------------------+--------------+ +| 4 | Host Cookie | | ++------+-------------------+--------------+ +\end{verbatim} + +Host cookie is an optional per-virtqueue 64 bit value that can be +used by the hypervisor to speed up the notification execution. +For each notification, the output value is returned in GPR2 and +should be passed in GPR4 for the next notification: + + info->cookie = do_notify(schid, + virtqueue_get_queue_index(vq), + info->cookie); + +\subsubsection{Early printk for Virtio Consoles}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Operation / Early printk for Virtio Consoles} + +For the early printk mechanism, diagnose 0x500 with subcode 0 is +used. + +\subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio over channel I/O / Device Operation / Resetting Devices} + +In order to reset a device, a guest may send the +CCW_CMD_VDEV_RESET command. + + +\chapter{Device Types}\label{sec:Device Types} + +On top of the queues, config space and feature negotiation facilities +built into virtio, several specific devices are defined. + +The following device IDs are used to identify different types of virtio +devices. Some device IDs are reserved for devices which are not currently +defined in this standard. + +Discovering what devices are available and their type is bus-dependent. + +\begin{verbatim} ++------------+--------------------+ +| Device ID | Virtio Device | ++------------+--------------------+ ++------------+--------------------+ +| 0 | reserved (invalid) | ++------------+--------------------+ +| 1 | network card | ++------------+--------------------+ +| 2 | block device | ++------------+--------------------+ +| 3 | console | ++------------+--------------------+ +| 4 | entropy source | ++------------+--------------------+ +| 5 | memory ballooning | ++------------+--------------------+ +| 6 | ioMemory | ++------------+--------------------+ +| 7 | rpmsg | ++------------+--------------------+ +| 8 | SCSI host | ++------------+--------------------+ +| 9 | 9P transport | ++------------+--------------------+ +| 10 | mac80211 wlan | ++------------+--------------------+ +| 11 | rproc serial | ++------------+--------------------+ +| 12 | virtio CAIF | ++------------+--------------------+ +\end{verbatim} + +\section{Network Device}\label{sec:Device Types / Network Device} + +The virtio network device is a virtual ethernet card, and is the +most complex of the devices supported so far by virtio. It has +enhanced rapidly and demonstrates clearly how support for new +features should be added to an existing device. Empty buffers are +placed in one virtqueue for receiving packets, and outgoing +packets are enqueued into another for transmission in that order. +A third command queue is used to control advanced filtering +features. + +\subsection{Device ID}\label{sec:Device Types / Network Device / Device ID} + + 1 + +\subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues} + + 0:receiveq. 1:transmitq. 2:controlq + + Virtqueue 2 only exists if VIRTIO_NET_F_CTRL_VQ set. + +\subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits} + + VIRTIO_NET_F_CSUM (0) Device handles packets with partial checksum + + VIRTIO_NET_F_GUEST_CSUM (1) Guest handles packets with partial checksum + + VIRTIO_NET_F_CTRL_GUEST_OFFLOADS (2) Control channel offloads + reconfiguration support. + + VIRTIO_NET_F_MAC (5) Device has given MAC address. + + VIRTIO_NET_F_GUEST_TSO4 (7) Guest can receive TSOv4. + + VIRTIO_NET_F_GUEST_TSO6 (8) Guest can receive TSOv6. + + VIRTIO_NET_F_GUEST_ECN (9) Guest can receive TSO with ECN. + + VIRTIO_NET_F_GUEST_UFO (10) Guest can receive UFO. + + VIRTIO_NET_F_HOST_TSO4 (11) Device can receive TSOv4. + + VIRTIO_NET_F_HOST_TSO6 (12) Device can receive TSOv6. + + VIRTIO_NET_F_HOST_ECN (13) Device can receive TSO with ECN. + + VIRTIO_NET_F_HOST_UFO (14) Device can receive UFO. + + VIRTIO_NET_F_MRG_RXBUF (15) Guest can merge receive buffers. + + VIRTIO_NET_F_STATUS (16) Configuration status field is + available. + + VIRTIO_NET_F_CTRL_VQ (17) Control channel is available. + + VIRTIO_NET_F_CTRL_RX (18) Control channel RX mode support. + + VIRTIO_NET_F_CTRL_VLAN (19) Control channel VLAN filtering. + + VIRTIO_NET_F_GUEST_ANNOUNCE(21) Guest can send gratuitous + packets. + +\subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits} +VIRTIO_NET_F_GSO (6) Device handles packets with any GSO type. + +This was supposed to indicate segmentation offload support, but +upon further investigation it became clear that multiple bits +were required. + +\subsection{Device configuration layout}\label{sec:Device Types / Network Device / Device configuration layout} + +Two configuration fields are currently defined. The mac address field +always exists (though is only valid if VIRTIO_NET_F_MAC is set), and +the status field only exists if VIRTIO_NET_F_STATUS is set. Two +read-only bits are currently defined for the status field: +VIRTIO_NET_S_LINK_UP and VIRTIO_NET_S_ANNOUNCE. + +\begin{lstlisting} + #define VIRTIO_NET_S_LINK_UP 1 + #define VIRTIO_NET_S_ANNOUNCE 2 + + struct virtio_net_config { + u8 mac[6]; + le16 status; + }; +\end{lstlisting} + +\subsubsection{Legacy Interface: Device configuration layout}\label{sec:Device Types / Network Device / Device configuration layout / Legacy Interface: Device configuration layout} +For legacy devices, the status field in struct virtio_net_config is the +native endian of the guest rather than (necessarily) little-endian. + + +\subsection{Device Initialization}\label{sec:Device Types / Network Device / Device Initialization} + +1. The initialization routine should identify the receive and + transmission virtqueues. + +2. If the VIRTIO_NET_F_MAC feature bit is set, the configuration + space “mac” entry indicates the “physical” address of the the + network card, otherwise a private MAC address should be + assigned. All guests are expected to negotiate this feature if + it is set. + +3. If the VIRTIO_NET_F_CTRL_VQ feature bit is negotiated, + identify the control virtqueue. + +4. If the VIRTIO_NET_F_STATUS feature bit is negotiated, the link + status can be read from the bottom bit of the “status” config + field. Otherwise, the link should be assumed active. + +5. The receive virtqueue should be filled with receive buffers. + This is described in detail below in “Setting Up Receive + Buffers”. + +6. A driver can indicate that it will generate checksumless + packets by negotating the VIRTIO_NET_F_CSUM feature. This “ + checksum offload” is a common feature on modern network cards. + +7. If that feature is negotiated +\footnote{ie. VIRTIO_NET_F_HOST_TSO* and VIRTIO_NET_F_HOST_UFO are +dependent on VIRTIO_NET_F_CSUM; a dvice which offers the offload +features must offer the checksum feature, and a driver which +accepts the offload features must accept the checksum feature. +Similar logic applies to the VIRTIO_NET_F_GUEST_TSO4 features +depending on VIRTIO_NET_F_GUEST_CSUM. +}, a driver can use TCP or UDP + segmentation offload by negotiating the VIRTIO_NET_F_HOST_TSO4 (IPv4 + TCP), VIRTIO_NET_F_HOST_TSO6 (IPv6 TCP) and VIRTIO_NET_F_HOST_UFO + (UDP fragmentation) features. It should not send TCP packets + requiring segmentation offload which have the Explicit Congestion + Notification bit set, unless the VIRTIO_NET_F_HOST_ECN feature is + negotiated. +\footnote{This is a common restriction in real, older network cards. +} + +8. The converse features are also available: a driver can save + the virtual device some work by negotiating these features. +\footnote{For example, a network packet transported between two guests on +the same system may not require checksumming at all, nor segmentation, +if both guests are amenable. +} + The VIRTIO_NET_F_GUEST_CSUM feature indicates that partially + checksummed packets can be received, and if it can do that then + the VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6, + VIRTIO_NET_F_GUEST_UFO and VIRTIO_NET_F_GUEST_ECN are the input + equivalents of the features described above. + See \ref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}~\nameref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers} and \ref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}~\nameref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers} below. + +\subsection{Device Operation}\label{sec:Device Types / Network Device / Device Operation} + +Packets are transmitted by placing them in the transmitq, and +buffers for incoming packets are placed in the receiveq. In each +case, the packet itself is preceeded by a header: + +\begin{lstlisting} + struct virtio_net_hdr { + #define VIRTIO_NET_HDR_F_NEEDS_CSUM 1 + u8 flags; + #define VIRTIO_NET_HDR_GSO_NONE 0 + #define VIRTIO_NET_HDR_GSO_TCPV4 1 + #define VIRTIO_NET_HDR_GSO_UDP 3 + #define VIRTIO_NET_HDR_GSO_TCPV6 4 + #define VIRTIO_NET_HDR_GSO_ECN 0x80 + u8 gso_type; + le16 hdr_len; + le16 gso_size; + le16 csum_start; + le16 csum_offset; + /* Only if VIRTIO_NET_F_MRG_RXBUF: */ + le16 num_buffers; + }; +\end{lstlisting} + +The controlq is used to control device features such as +filtering. + +\subsubsection{Legacy Interface: Device Operation}\label{sec:Device Types / Network Device / Device Operation / Legacy Interface: Device Operation} +For legacy devices, the fields in struct virtio_net_hdr are the +native endian of the guest rather than (necessarily) little-endian. + +\subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / Device Operation / Packet Transmission} + +Transmitting a single packet is simple, but varies depending on +the different features the driver negotiated. + +1. If the driver negotiated VIRTIO_NET_F_CSUM, and the packet has + not been fully checksummed, then the virtio_net_hdr's fields + are set as follows. Otherwise, the packet must be fully + checksummed, and flags is zero. + + • flags has the VIRTIO_NET_HDR_F_NEEDS_CSUM set, + + • csum_start is set to the offset within the packet to begin checksumming, + and + + • csum_offset indicates how many bytes after the csum_start the + new (16 bit ones' complement) checksum should be placed. +\footnote{For example, consider a partially checksummed TCP (IPv4) packet. +It will have a 14 byte ethernet header and 20 byte IP header +followed by the TCP header (with the TCP checksum field 16 bytes +into that header). csum_start will be 14+20 = 34 (the TCP +checksum includes the header), and csum_offset will be 16. The +value in the TCP checksum field should be initialized to the sum +of the TCP pseudo header, so that replacing it by the ones' +complement checksum of the TCP header and body will give the +correct result. +} + +2. If the driver negotiated + VIRTIO_NET_F_HOST_TSO4, TSO6 or UFO, and the packet requires + TCP segmentation or UDP fragmentation, then the “gso_type” + field is set to VIRTIO_NET_HDR_GSO_TCPV4, TCPV6 or UDP. + (Otherwise, it is set to VIRTIO_NET_HDR_GSO_NONE). In this + case, packets larger than 1514 bytes can be transmitted: the + metadata indicates how to replicate the packet header to cut it + into smaller packets. The other gso fields are set: + + • hdr_len is a hint to the device as to how much of the header + needs to be kept to copy into each packet, usually set to the + length of the headers, including the transport header. +\footnote{Due to various bugs in implementations, this field is not useful +as a guarantee of the transport header size. +} + + • gso_size is the maximum size of each packet beyond that + header (ie. MSS). + + • If the driver negotiated the VIRTIO_NET_F_HOST_ECN feature, + the VIRTIO_NET_HDR_GSO_ECN bit may be set in “gso_type” as + well, indicating that the TCP packet has the ECN bit set. +\footnote{This case is not handled by some older hardware, so is called out +specifically in the protocol. +} + +3. If the driver negotiated the VIRTIO_NET_F_MRG_RXBUF feature, + the num_buffers field is set to zero. + +4. The header and packet are added as one output buffer to the + transmitq, and the device is notified of the new entry + (see \ref{sec:Device Types / Network Device / Device Initialization}~\nameref{sec:Device Types / Network Device / Device Initialization}). +\footnote{Note that the header will be two bytes longer for the +VIRTIO_NET_F_MRG_RXBUF case. +} + +\paragraph{Packet Transmission Interrupt}\label{sec:Device Types / Network Device / Device Operation / Packet Transmission / Packet Transmission Interrupt} + +Often a driver will suppress transmission interrupts using the +VRING_AVAIL_F_NO_INTERRUPT flag + (see \ref{sec:Device Types / Block Device}~\nameref{sec:Device Types / Block Device}) +and check for used packets in the transmit path of following +packets. + +The normal behavior in this interrupt handler is to retrieve and +new descriptors from the used ring and free the corresponding +headers and packets. + +\subsubsection{Setting Up Receive Buffers}\label{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers} + +It is generally a good idea to keep the receive virtqueue as +fully populated as possible: if it runs out, network performance +will suffer. + +If the VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6 or +VIRTIO_NET_F_GUEST_UFO features are used, the Guest will need to +accept packets of up to 65550 bytes long (the maximum size of a +TCP or UDP packet, plus the 14 byte ethernet header), otherwise +1514. bytes. So unless VIRTIO_NET_F_MRG_RXBUF is negotiated, every +buffer in the receive queue needs to be at least this length +\footnote{Obviously each one can be split across multiple descriptor +elements. +} + +If VIRTIO_NET_F_MRG_RXBUF is negotiated, each buffer must be at +least the size of the struct virtio_net_hdr. + +\paragraph{Packet Receive Interrupt}\label{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers / Packet Receive Interrupt} + +When a packet is copied into a buffer in the receiveq, the +optimal path is to disable further interrupts for the receiveq +(see 2.2.2.2. Receiving Used Buffers From The Device) and process +packets until no more are found, then re-enable them. + +Processing packet involves: + +1. If the driver negotiated the VIRTIO_NET_F_MRG_RXBUF feature, + then the “num_buffers” field indicates how many descriptors + this packet is spread over (including this one). This allows + receipt of large packets without having to allocate large + buffers. In this case, there will be at least “num_buffers” in + the used ring, and they should be chained together to form a + single packet. The other buffers will not begin with a struct + virtio_net_hdr. + +2. If the VIRTIO_NET_F_MRG_RXBUF feature was not negotiated, or + the “num_buffers” field is one, then the entire packet will be + contained within this buffer, immediately following the struct + virtio_net_hdr. + +3. If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the + VIRTIO_NET_HDR_F_NEEDS_CSUM bit in the “flags” field may be + set: if so, the checksum on the packet is incomplete and the “ + csum_start” and “csum_offset” fields indicate how to calculate + it (see Packet Transmission point 1). + +4. If the VIRTIO_NET_F_GUEST_TSO4, TSO6 or UFO options were + negotiated, then the “gso_type” may be something other than + VIRTIO_NET_HDR_GSO_NONE, and the “gso_size” field indicates the + desired MSS (see Packet Transmission point 2). + +\subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue} + +The driver uses the control virtqueue (if VIRTIO_NET_F_VTRL_VQ is +negotiated) to send commands to manipulate various features of +the device which would not easily map into the configuration +space. + +All commands are of the following form: + +\begin{lstlisting} + struct virtio_net_ctrl { + u8 class; + u8 command; + u8 command-specific-data[]; + u8 ack; + }; + + /* ack values */ + #define VIRTIO_NET_OK 0 + #define VIRTIO_NET_ERR 1 +\end{lstlisting} + +The class, command and command-specific-data are set by the +driver, and the device sets the ack byte. There is little it can +do except issue a diagnostic if the ack byte is not +VIRTIO_NET_OK. + +\paragraph{Packet Receive Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Packet Receive Filtering} + +If the VIRTIO_NET_F_CTRL_RX feature is negotiated, the driver can +send control commands for promiscuous mode, multicast receiving, +and filtering of MAC addresses. + +Note that in general, these commands are best-effort: unwanted +packets may still arrive. + +Setting Promiscuous Mode + +\begin{lstlisting} + #define VIRTIO_NET_CTRL_RX 0 + #define VIRTIO_NET_CTRL_RX_PROMISC 0 + #define VIRTIO_NET_CTRL_RX_ALLMULTI 1 +\end{lstlisting} + +The class VIRTIO_NET_CTRL_RX has two commands: +VIRTIO_NET_CTRL_RX_PROMISC turns promiscuous mode on and off, and +VIRTIO_NET_CTRL_RX_ALLMULTI turns all-multicast receive on and +off. The command-specific-data is one byte containing 0 (off) or +1 (on). + +\paragraph{Setting MAC Address Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Setting MAC Address Filtering} + +\begin{lstlisting} + struct virtio_net_ctrl_mac { + le32 entries; + u8 macs[entries][ETH_ALEN]; + }; + + #define VIRTIO_NET_CTRL_MAC 1 + #define VIRTIO_NET_CTRL_MAC_TABLE_SET 0 +\end{lstlisting} + +The device can filter incoming packets by any number of destination +MAC addresses. +\footnote{Since there are no guarentees, it can use a hash filter or +silently switch to allmulti or promiscuous mode if it is given too +many addresses. +} This table is set using the class +VIRTIO_NET_CTRL_MAC and the command VIRTIO_NET_CTRL_MAC_TABLE_SET. The +command-specific-data is two variable length tables of 6-byte MAC +addresses. The first table contains unicast addresses, and the second +contains multicast addresses. + +\subparagraph{Legacy Interface: Setting MAC Address Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Setting MAC Address Filtering / Legacy Interface: Setting MAC Address Filtering} +For legacy devices, the entries field in struct virtio_net_ctrl_mac is the +native endian of the guest rather than (necessarily) little-endian. + +\paragraph{VLAN Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / VLAN Filtering} + +If the driver negotiates the VIRTION_NET_F_CTRL_VLAN feature, it +can control a VLAN filter table in the device. + +\begin{lstlisting} + #define VIRTIO_NET_CTRL_VLAN 2 + #define VIRTIO_NET_CTRL_VLAN_ADD 0 + #define VIRTIO_NET_CTRL_VLAN_DEL 1 +\end{lstlisting} + +Both the VIRTIO_NET_CTRL_VLAN_ADD and VIRTIO_NET_CTRL_VLAN_DEL +command take a little-endian 16-bit VLAN id as the command-specific-data. + +\subparagraph{Legacy Interface: VLAN Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / VLAN Filtering / Legacy Interface: VLAN Filtering} +For legacy devices, the VLAN id is in the +native endian of the guest rather than (necessarily) little-endian. + +\paragraph{Gratuitous Packet Sending}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending} + +If the driver negotiates the VIRTIO_NET_F_GUEST_ANNOUNCE (depends +on VIRTIO_NET_F_CTRL_VQ), it can ask the guest to send gratuitous +packets; this is usually done after the guest has been physically +migrated, and needs to announce its presence on the new network +links. (As hypervisor does not have the knowledge of guest +network configuration (eg. tagged vlan) it is simplest to prod +the guest in this way). + +\begin{lstlisting} + #define VIRTIO_NET_CTRL_ANNOUNCE 3 + #define VIRTIO_NET_CTRL_ANNOUNCE_ACK 0 +\end{lstlisting} + +The Guest needs to check VIRTIO_NET_S_ANNOUNCE bit in status +field when it notices the changes of device configuration. The +command VIRTIO_NET_CTRL_ANNOUNCE_ACK is used to indicate that +driver has recevied the notification and device would clear the +VIRTIO_NET_S_ANNOUNCE bit in the status filed after it received +this command. + +Processing this notification involves: + +1. Sending the gratuitous packets or marking there are pending + gratuitous packets to be sent and letting deferred routine to + send them. + +2. Sending VIRTIO_NET_CTRL_ANNOUNCE_ACK command through control + vq. + +\paragraph{Offloads State Configuration}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration} + +If the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is negotiated, the driver can +send control commands for dynamic offloads state configuration. + +\subparagraph{Setting Offloads State}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State} + +\begin{lstlisting} + le64 offloads; + + #define VIRTIO_NET_F_GUEST_CSUM 1 + #define VIRTIO_NET_F_GUEST_TSO4 7 + #define VIRTIO_NET_F_GUEST_TSO6 8 + #define VIRTIO_NET_F_GUEST_ECN 9 + #define VIRTIO_NET_F_GUEST_UFO 10 + + #define VIRTIO_NET_CTRL_GUEST_OFFLOADS 5 + #define VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET 0 +\end{lstlisting} + +The class VIRTIO_NET_CTRL_GUEST_OFFLOADS has one command: +VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET applies the new offloads configuration. + +le64 value passed as command data is a bitmask, bits set define +offloads to be enabled, bits cleared - offloads to be disabled. + +There is a corresponding device feature for each offload. Upon feature +negotiation corresponding offload gets enabled to preserve backward +compartibility. + +Corresponding feature must be negotiated at startup in order to allow dynamic +change of specific offload state. + + +\subparagraph{Legacy Interface: Setting Offloads State}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State / Legacy Interface: Setting Offloads State} +For legacy devices, the offloads field is the +native endian of the guest rather than (necessarily) little-endian. + + +\section{Block Device}\label{sec:Device Types / Block Device} + +The virtio block device is a simple virtual block device (ie. +disk). Read and write requests (and other exotic requests) are +placed in the queue, and serviced (probably out of order) by the +device except where noted. + +\subsection{Device ID}\label{sec:Device Types / Block Device / Device ID} + 2 + +\subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} + 0:requestq + +\subsection{Feature bits}\label{sec:Device Types / Block Device / Feature bits} + + VIRTIO_BLK_F_SIZE_MAX (1) Maximum size of any single segment is + in “size_max”. + + VIRTIO_BLK_F_SEG_MAX (2) Maximum number of segments in a + request is in “seg_max”. + + VIRTIO_BLK_F_GEOMETRY (4) Disk-style geometry specified in “ + geometry”. + + VIRTIO_BLK_F_RO (5) Device is read-only. + + VIRTIO_BLK_F_BLK_SIZE (6) Block size of disk is in “blk_size”. + + VIRTIO_BLK_F_TOPOLOGY (10) Device exports information on optimal I/O + alignment. + +\subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Block Device / Feature bits / Legacy Interface: Feature bits} + VIRTIO_BLK_F_BARRIER (0) Host supports request barriers. + + VIRTIO_BLK_F_SCSI (7) Device supports scsi packet commands. + + VIRTIO_BLK_F_FLUSH (9) Cache flush command support. + + VIRTIO_BLK_F_CONFIG_WCE (11) Device can toggle its cache between writeback + and writethrough modes. + +VIRTIO_BLK_F_FLUSH was also called VIRTIO_BLK_F_WCE: Legacy drivers +should only negotiate this feature if they are capable of sending +VIRTIO_BLK_T_FLUSH commands. + +\subsubsection{Device configuration layout}\label{sec:Device Types / Block Device / Feature bits / Device configuration layout} + +The capacity of the device (expressed in 512-byte sectors) is always +present. The availability of the others all depend on various feature +bits as indicated above. + +\begin{lstlisting} + struct virtio_blk_config { + le64 capacity; + le32 size_max; + le32 seg_max; + struct virtio_blk_geometry { + le16 cylinders; + u8 heads; + u8 sectors; + } geometry; + le32 blk_size; + struct virtio_blk_topology { + // # of logical blocks per physical block (log2) + u8 physical_block_exp; + // offset of first aligned logical block + u8 alignment_offset; + // suggested minimum I/O size in blocks + le16 min_io_size; + // optimal (suggested maximum) I/O size in blocks + le32 opt_io_size; + } topology; + u8 reserved; + }; +\end{lstlisting} + + +\paragraph{Legacy Interface: Device configuration layout}\label{sec:Device Types / Block Device / Feature bits / Device configuration layout / Legacy Interface: Device configuration layout} +For legacy devices, the fields in struct virtio_blk_config are the +native endian of the guest rather than (necessarily) little-endian. + + +\subsection{Device Initialization}\label{sec:Device Types / Block Device / Device Initialization} + +1. The device size should be read from the “capacity” + configuration field. No requests should be submitted which goes + beyond this limit. + +2. If the VIRTIO_BLK_F_BLK_SIZE feature is negotiated, the + blk_size field can be read to determine the optimal sector size + for the driver to use. This does not affect the units used in + the protocol (always 512 bytes), but awareness of the correct + value can affect performance. + +3. If the VIRTIO_BLK_F_RO feature is set by the device, any write + requests will fail. + +4. If the VIRTIO_BLK_F_TOPOLOGY feature is negotiated, the fields in the + topology struct can be read to determine the physical block size and optimal + I/O lengths for the driver to use. This also does not affect the units + in the protocol, only performance. + +\subsubsection{Legacy Interface: Device Initialization}\label{sec:Device Types / Block Device / Device Initialization / Legacy Interface: Device Initialization} + +The reserved field used to be called writeback. If the +VIRTIO_BLK_F_CONFIG_WCE feature is offered, the cache mode should be +read from the writeback field of the configuration if available; the +driver can also write to the field in order to toggle the cache +between writethrough (0) and writeback (1) mode. If the feature is +not available, the driver can instead look at the result of +negotiating VIRTIO_BLK_F_FLUSH: the cache will be in writeback mode +after reset if and only if VIRTIO_BLK_F_FLUSH is negotiated. + +Some older legacy devices did not operate in writethrough mode even +after a guest announced lack of support for VIRTIO_BLK_F_FLUSH. + +\subsection{Device Operation}\label{sec:Device Types / Block Device / Device Operation} + +The driver queues requests to the virtqueue, and they are used by +the device (not necessarily in order). Each request is of form: + +\begin{lstlisting} + struct virtio_blk_req { + le32 type; + le32 reserved; + le64 sector; + char data[][512]; + u8 status; + }; +\end{lstlisting} + +The type of the request is either a read (VIRTIO_BLK_T_IN), a write +(VIRTIO_BLK_T_OUT), or a flush (VIRTIO_BLK_T_FLUSH or +VIRTIO_BLK_T_FLUSH_OUT +\footnote{The FLUSH and FLUSH_OUT types are equivalent, the device does not +distinguish between them +}). + +\begin{lstlisting} + #define VIRTIO_BLK_T_IN 0 + #define VIRTIO_BLK_T_OUT 1 + #define VIRTIO_BLK_T_FLUSH 4 + #define VIRTIO_BLK_T_FLUSH_OUT 5 +\end{lstlisting} + +The sector number indicates the offset (multiplied by 512) where +the read or write is to occur. This field is unused and set to 0 +for scsi packet commands and for flush commands. + +The final status byte is written by the device: either +VIRTIO_BLK_S_OK for success, VIRTIO_BLK_S_IOERR for host or guest +error or VIRTIO_BLK_S_UNSUPP for a request unsupported by host: + +\begin{lstlisting} + #define VIRTIO_BLK_S_OK 0 + #define VIRTIO_BLK_S_IOERR 1 + #define VIRTIO_BLK_S_UNSUPP 2 +\end{lstlisting} + +Any writes completed before the submission of the flush command should +be committed to non-volatile storage by the device. + +\subsubsection{Legacy Interface: Device Operation}\label{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation} +For legacy devices, the fields in struct virtio_blk_req are the +native endian of the guest rather than (necessarily) little-endian. + +The 'reserved' field was previously called ioprio. The ioprio field +is a hint about the relative priorities of requests to the device: +higher numbers indicate more important requests. + +\begin{lstlisting} + #define VIRTIO_BLK_T_BARRIER 0x80000000 +\end{lstlisting} + +If the device has VIRTIO_BLK_F_BARRIER +feature the high bit (VIRTIO_BLK_T_BARRIER) indicates that this +request acts as a barrier and that all preceeding requests must be +complete before this one, and all following requests must not be +started until this is complete. Note that a barrier does not flush +caches in the underlying backend device in host, and thus does not +serve as data consistency guarantee. Driver must use FLUSH request to +flush the host cache. + +If the device has VIRTIO_BLK_F_SCSI feature, it can also support +scsi packet command requests, each of these requests is of form: + +\begin{lstlisting} + /* All fields are in guest's native endian. */ + struct virtio_scsi_pc_req { + u32 type; + u32 ioprio; + u64 sector; + char cmd[]; + char data[][512]; + #define SCSI_SENSE_BUFFERSIZE 96 + u8 sense[SCSI_SENSE_BUFFERSIZE]; + u32 errors; + u32 data_len; + u32 sense_len; + u32 residual; + u8 status; + }; +\end{lstlisting} + +A request type can also be a scsi packet command (VIRTIO_BLK_T_SCSI_CMD or +VIRTIO_BLK_T_SCSI_CMD_OUT). The two types are equivalent, the device +does not distinguish between them: + +\begin{lstlisting} + #define VIRTIO_BLK_T_SCSI_CMD 2 + #define VIRTIO_BLK_T_SCSI_CMD_OUT 3 +\end{lstlisting} + +The cmd field is only present for scsi packet command requests, +and indicates the command to perform. This field must reside in a +single, separate read-only buffer; command length can be derived +from the length of this buffer. + +Note that these first three (four for scsi packet commands) +fields are always read-only: the data field is either read-only +or write-only, depending on the request. The size of the read or +write can be derived from the total size of the request buffers. + +The sense field is only present for scsi packet command requests, +and indicates the buffer for scsi sense data. + +The data_len field is only present for scsi packet command +requests, this field is deprecated, and should be ignored by the +driver. Historically, devices copied data length there. + +The sense_len field is only present for scsi packet command +requests and indicates the number of bytes actually written to +the sense buffer. + +The residual field is only present for scsi packet command +requests and indicates the residual size, calculated as data +length - number of bytes actually transferred. + +Historically, devices assumed that the fields type, ioprio and +sector reside in a single, separate read-only buffer; the fields +errors, data_len, sense_len and residual reside in a single, +separate write-only buffer; the sense field in a separate +write-only buffer of size 96 bytes, by itself; the fields errors, +data_len, sense_len and residual in a single write-only buffer; +and the status field is a separate read-only buffer of size 1 +byte, by itself. + + +\section{Console Device}\label{sec:Device Types / Console Device} + +The virtio console device is a simple device for data input and +output. A device may have one or more ports. Each port has a pair +of input and output virtqueues. Moreover, a device has a pair of +control IO virtqueues. The control virtqueues are used to +communicate information between the device and the driver about +ports being opened and closed on either side of the connection, +indication from the host about whether a particular port is a +console port, adding new ports, port hot-plug/unplug, etc., and +indication from the guest about whether a port or a device was +successfully added, port open/close, etc.. For data IO, one or +more empty buffers are placed in the receive queue for incoming +data and outgoing characters are placed in the transmit queue. + +\subsection{Device ID}\label{sec:Device Types / Console Device / Device ID} + + 3 + +\subsection{Virtqueues}\label{sec:Device Types / Console Device / Virtqueues} + + 0:receiveq(port0). 1:transmitq(port0), 2:control receiveq, 3:control transmitq, 4:receiveq(port1), 5:transmitq(port1), + ... + + Ports 2 onwards only exist if VIRTIO_CONSOLE_F_MULTIPORT is set. + +\subsection{Feature bits}\label{sec:Device Types / Console Device / Feature bits} + + VIRTIO_CONSOLE_F_SIZE (0) Configuration cols and rows fields + are valid. + + VIRTIO_CONSOLE_F_MULTIPORT(1) Device has support for multiple + ports; configuration fields nr_ports and max_nr_ports are + valid and control virtqueues will be used. + +\subsection{Device configuration layout}\label{sec:Device Types / Console Device / Device configuration layout} + + The size of the console is supplied + in the configuration space if the VIRTIO_CONSOLE_F_SIZE feature + is set. Furthermore, if the VIRTIO_CONSOLE_F_MULTIPORT feature + is set, the maximum number of ports supported by the device can + be fetched. + +\begin{lstlisting} + struct virtio_console_config { + le16 cols; + le16 rows; + le32 max_nr_ports; + }; +\end{lstlisting} + +\subsubsection{Legacy Interface: Device configuration layout}\label{sec:Device Types / Console Device / Device configuration layout / Legacy Interface: Device configuration layout} +For legacy devices, the fields in struct virtio_console_config are the +native endian of the guest rather than (necessarily) little-endian. + +\subsection{Device Initialization}\label{sec:Device Types / Console Device / Device Initialization} + +1. If the VIRTIO_CONSOLE_F_SIZE feature is negotiated, the driver + can read the console dimensions from the configuration fields. + +2. If the VIRTIO_CONSOLE_F_MULTIPORT feature is negotiated, the + driver can spawn multiple ports, not all of which may be + attached to a console. Some could be generic ports. In this + case, the control virtqueues are enabled and according to the + max_nr_ports configuration-space value, the appropriate number + of virtqueues are created. A control message indicating the + driver is ready is sent to the host. The host can then send + control messages for adding new ports to the device. After + creating and initializing each port, a + VIRTIO_CONSOLE_PORT_READY control message is sent to the host + for that port so the host can let us know of any additional + configuration options set for that port. + +3. The receiveq for each port is populated with one or more + receive buffers. + +\subsection{Device Operation}\label{sec:Device Types / Console Device / Device Operation} + +1. For output, a buffer containing the characters is placed in + the port's transmitq. +\footnote{Because this is high importance and low bandwidth, the current +Linux implementation polls for the buffer to be used, rather than +waiting for an interrupt, simplifying the implementation +significantly. However, for generic serial ports with the +O_NONBLOCK flag set, the polling limitation is relaxed and the +consumed buffers are freed upon the next write or poll call or +when a port is closed or hot-unplugged. +} + +2. When a buffer is used in the receiveq (signalled by an + interrupt), the contents is the input to the port associated + with the virtqueue for which the notification was received. + +3. If the driver negotiated the VIRTIO_CONSOLE_F_SIZE feature, a + configuration change interrupt may occur. The updated size can + be read from the configuration fields. + +4. If the driver negotiated the VIRTIO_CONSOLE_F_MULTIPORT + feature, active ports are announced by the host using the + VIRTIO_CONSOLE_PORT_ADD control message. The same message is + used for port hot-plug as well. + +5. If the host specified a port `name', a sysfs attribute is + created with the name filled in, so that udev rules can be + written that can create a symlink from the port's name to the + char device for port discovery by applications in the guest. + +6. Changes to ports' state are effected by control messages. + Appropriate action is taken on the port indicated in the + control message. The layout of the structure of the control + buffer and the events associated are: + +\begin{lstlisting} + struct virtio_console_control { + le32 id; /* Port number */ + le16 event; /* The kind of control event */ + le16 value; /* Extra information for the event */ + }; + + /* Some events for the internal messages (control packets) */ + #define VIRTIO_CONSOLE_DEVICE_READY 0 + #define VIRTIO_CONSOLE_PORT_ADD 1 + #define VIRTIO_CONSOLE_PORT_REMOVE 2 + #define VIRTIO_CONSOLE_PORT_READY 3 + #define VIRTIO_CONSOLE_CONSOLE_PORT 4 + #define VIRTIO_CONSOLE_RESIZE 5 + #define VIRTIO_CONSOLE_PORT_OPEN 6 + #define VIRTIO_CONSOLE_PORT_NAME 7 +\end{lstlisting} + +\subsubsection{Legacy Interface: Device Operation}\label{sec:Device Types / Console Device / Device Operation / Legacy Interface: Device Operation} +For legacy devices, the fields in struct virtio_console_control are the +native endian of the guest rather than (necessarily) little-endian. + + +\section{Entropy Device}\label{sec:Device Types / Entropy Device} + +The virtio entropy device supplies high-quality randomness for +guest use. + +\subsection{Device ID}\label{sec:Device Types / Entropy Device / Device ID} + 4 + +\subsection{Virtqueues}\label{sec:Device Types / Entropy Device / Virtqueues} + 0:requestq. + +\subsection{Feature bits}\label{sec:Device Types / Entropy Device / Feature bits} + None currently defined + +\subsection{Device configuration layout}\label{sec:Device Types / Entropy Device / Device configuration layout} + None currently defined. + +\subsection{Device Initialization}\label{sec:Device Types / Entropy Device / Device Initialization} + +1. The virtqueue is initialized + +\subsection{Device Operation}\label{sec:Device Types / Entropy Device / Device Operation} + +When the driver requires random bytes, it places the descriptor +of one or more buffers in the queue. It will be completely filled +by random data by the device. + +\section{Memory Balloon Device}\label{sec:Device Types / Memory Balloon Device} + +The virtio memory balloon device is a primitive device for +managing guest memory: the device asks for a certain amount of +memory, and the guest supplies it (or withdraws it, if the device +has more than it asks for). This allows the guest to adapt to +changes in allowance of underlying physical memory. If the +feature is negotiated, the device can also be used to communicate +guest memory statistics to the host. + +\subsection{Device ID}\label{sec:Device Types / Memory Balloon Device / Device ID} + 5 + +\subsection{Virtqueues}\label{sec:Device Types / Memory Balloon Device / Virtqueues} + 0:inflateq. 1:deflateq. 2:statsq. + + Virtqueue 2 only exists if VIRTIO_BALLON_F_STATS_VQ set. + +\subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Feature bits} + VIRTIO_BALLOON_F_MUST_TELL_HOST (0) Host must be told before + pages from the balloon are used. + + VIRTIO_BALLOON_F_STATS_VQ (1) A virtqueue for reporting guest + memory statistics is present. + +\subsection{Device configuration layout}\label{sec:Device Types / Memory Balloon Device / Device configuration layout} + Both fields of this configuration + are always available. + +\begin{lstlisting} + struct virtio_balloon_config { + le32 num_pages; + le32 actual; + }; +\end{lstlisting} + +\subsubsection{Legacy Interface: Device configuration layout}\label{sec:Device Types / Memory Balloon Device / Device configuration layout / Legacy Interface: Device configuration layout} +Note that these fields are always little endian, despite convention +that legacy device fields are guest endian. + +\subsection{Device Initialization}\label{sec:Device Types / Memory Balloon Device / Device Initialization} + +1. The inflate and deflate virtqueues are identified. + +2. If the VIRTIO_BALLOON_F_STATS_VQ feature bit is negotiated: + + (a) Identify the stats virtqueue. + + (b) Add one empty buffer to the stats virtqueue and notify the + host. + +Device operation begins immediately. + +\subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / Device Operation} + +Memory Ballooning The device is driven by the receipt of a +configuration change interrupt. + +1. The “num_pages” configuration field is examined. If this is + greater than the “actual” number of pages, memory must be given + to the balloon. If it is less than the “actual” number of + pages, memory may be taken back from the balloon for general + use. + +2. To supply memory to the balloon (aka. inflate): + + (a) The driver constructs an array of addresses of unused memory + pages. These addresses are divided by 4096 +\footnote{This is historical, and independent of the guest page size +} and the descriptor + describing the resulting 32-bit array is added to the inflateq. + +3. To remove memory from the balloon (aka. deflate): + + (a) The driver constructs an array of addresses of memory pages + it has previously given to the balloon, as described above. + This descriptor is added to the deflateq. + + (b) If the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is negotiated, the + guest may not use these requested pages until that descriptor + in the deflateq has been used by the device. + + (c) Otherwise, the guest may begin to re-use pages previously + given to the balloon before the device has acknowledged their + withdrawl. +\footnote{In this case, deflation advice is merely a courtesy +} + +4. In either case, once the device has completed the inflation or + deflation, the “actual” field of the configuration should be + updated to reflect the new number of pages in the balloon. +\footnote{As updates to configuration space are not atomic, this field +isn't particularly reliable, but can be used to diagnose buggy guests. +} + +\subsubsection{Memory Statistics}\label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics} + +The stats virtqueue is atypical because communication is driven +by the device (not the driver). The channel becomes active at +driver initialization time when the driver adds an empty buffer +and notifies the device. A request for memory statistics proceeds +as follows: + +1. The device pushes the buffer onto the used ring and sends an + interrupt. + +2. The driver pops the used buffer and discards it. + +3. The driver collects memory statistics and writes them into a + new buffer. + +4. The driver adds the buffer to the virtqueue and notifies the + device. + +5. The device pops the buffer (retaining it to initiate a + subsequent request) and consumes the statistics. + + Memory Statistics Format Each statistic consists of a 16 bit + tag and a 64 bit value. All statistics are optional and the + driver may choose which ones to supply. To guarantee backwards + compatibility, unsupported statistics should be omitted. + +\begin{lstlisting} + struct virtio_balloon_stat { + #define VIRTIO_BALLOON_S_SWAP_IN 0 + #define VIRTIO_BALLOON_S_SWAP_OUT 1 + #define VIRTIO_BALLOON_S_MAJFLT 2 + #define VIRTIO_BALLOON_S_MINFLT 3 + #define VIRTIO_BALLOON_S_MEMFREE 4 + #define VIRTIO_BALLOON_S_MEMTOT 5 + le16 tag; + le64 val; + } __attribute__((packed)); +\end{lstlisting} + +\paragraph{Legacy Interface: Memory Statistics}\label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics / Legacy Interface: Memory Statistics} +For legacy devices, the fields in struct virtio_balloon_stat are the +native endian of the guest rather than (necessarily) little-endian. + + +\subsubsection{Memory Statistics Tags}\label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics Tags} + + VIRTIO_BALLOON_S_SWAP_IN The amount of memory that has been + swapped in (in bytes). + + VIRTIO_BALLOON_S_SWAP_OUT The amount of memory that has been + swapped out to disk (in bytes). + + VIRTIO_BALLOON_S_MAJFLT The number of major page faults that + have occurred. + + VIRTIO_BALLOON_S_MINFLT The number of minor page faults that + have occurred. + + VIRTIO_BALLOON_S_MEMFREE The amount of memory not being used + for any purpose (in bytes). + + VIRTIO_BALLOON_S_MEMTOT The total amount of memory available + (in bytes). + + +\section{SCSI Host Device}\label{sec:Device Types / SCSI Host Device} + +The virtio SCSI host device groups together one or more virtual +logical units (such as disks), and allows communicating to them +using the SCSI protocol. An instance of the device represents a +SCSI host to which many targets and LUNs are attached. + +The virtio SCSI device services two kinds of requests: + +• command requests for a logical unit; + +• task management functions related to a logical unit, target or + command. + +The device is also able to send out notifications about added and +removed logical units. Together, these capabilities provide a +SCSI transport protocol that uses virtqueues as the transfer +medium. In the transport protocol, the virtio driver acts as the +initiator, while the virtio SCSI host provides one or more +targets that receive and process the requests. + +\subsection{Device ID}\label{sec:Device Types / SCSI Host Device / Device ID} + 8 + +\subsection{Virtqueues}\label{sec:Device Types / SCSI Host Device / Virtqueues} + 0:controlq; 1:eventq; 2..n:request queues. + +\subsection{Feature bits}\label{sec:Device Types / SCSI Host Device / Feature bits} + + VIRTIO_SCSI_F_INOUT (0) A single request can include both + read-only and write-only data buffers. + + VIRTIO_SCSI_F_HOTPLUG (1) The host should enable + hot-plug/hot-unplug of new LUNs and targets on the SCSI bus. + + VIRTIO_SCSI_F_CHANGE (2) The host will report changes to LUN + parameters via a VIRTIO_SCSI_T_PARAM_CHANGE event. + +\subsection{Device configuration layout}\label{sec:Device Types / SCSI Host Device / Device configuration layout} + + All fields of this configuration are always available. sense_size + and cdb_size are writable by the guest. + +\begin{lstlisting} + struct virtio_scsi_config { + le32 num_queues; + le32 seg_max; + le32 max_sectors; + le32 cmd_per_lun; + le32 event_info_size; + le32 sense_size; + le32 cdb_size; + le16 max_channel; + le16 max_target; + le32 max_lun; + }; +\end{lstlisting} + + num_queues is the total number of request virtqueues exposed by + the device. The driver is free to use only one request queue, + or it can use more to achieve better performance. + + seg_max is the maximum number of segments that can be in a + command. A bidirectional command can include seg_max input + segments and seg_max output segments. + + max_sectors is a hint to the guest about the maximum transfer + size it should use. + + cmd_per_lun is a hint to the guest about the maximum number of + linked commands it should send to one LUN. The actual value + to be used is the minimum of cmd_per_lun and the virtqueue + size. + + event_info_size is the maximum size that the device will fill + for buffers that the driver places in the eventq. The driver + should always put buffers at least of this size. It is + written by the device depending on the set of negotated + features. + + sense_size is the maximum size of the sense data that the + device will write. The default value is written by the device + and will always be 96, but the driver can modify it. It is + restored to the default when the device is reset. + + cdb_size is the maximum size of the CDB that the driver will + write. The default value is written by the device and will + always be 32, but the driver can likewise modify it. It is + restored to the default when the device is reset. + + max_channel, max_target and max_lun can be used by the driver + as hints to constrain scanning the logical units on the + host.h + +\subsubsection{Legacy Interface: Device configuration layout}\label{sec:Device Types / SCSI Host Device / Device configuration layout / Legacy Interface: Device configuration layout} +For legacy devices, the fields in struct virtio_scsi_config are the +native endian of the guest rather than (necessarily) little-endian. + +\subsection{Device Initialization}\label{sec:Device Types / SCSI Host Device / Device Initialization} + +The initialization routine should first of all discover the +device's virtqueues. + +If the driver uses the eventq, it should then place at least a +buffer in the eventq. + +The driver can immediately issue requests (for example, INQUIRY +or REPORT LUNS) or task management functions (for example, I_T +RESET). + +\subsection{Device Operation}\label{sec:Device Types / SCSI Host Device / Device Operation} + +Device operation consists of operating request queues, the control +queue and the event queue. + +\subsubsection{Device Operation: Request Queues}\label{sec:Device Types / SCSI Host Device / Device Operation / Device Operation: Request Queues} + +The driver queues requests to an arbitrary request queue, and +they are used by the device on that same queue. It is the +responsibility of the driver to ensure strict request ordering +for commands placed on different queues, because they will be +consumed with no order constraints. + +Requests have the following format: + +\begin{lstlisting} + struct virtio_scsi_req_cmd { + // Read-only + u8 lun[8]; + le64 id; + u8 task_attr; + u8 prio; + u8 crn; + char cdb[cdb_size]; + char dataout[]; + // Write-only part + le32 sense_len; + le32 residual; + le16 status_qualifier; + u8 status; + u8 response; + u8 sense[sense_size]; + char datain[]; + }; + + + /* command-specific response values */ + #define VIRTIO_SCSI_S_OK 0 + #define VIRTIO_SCSI_S_OVERRUN 1 + #define VIRTIO_SCSI_S_ABORTED 2 + #define VIRTIO_SCSI_S_BAD_TARGET 3 + #define VIRTIO_SCSI_S_RESET 4 + #define VIRTIO_SCSI_S_BUSY 5 + #define VIRTIO_SCSI_S_TRANSPORT_FAILURE 6 + #define VIRTIO_SCSI_S_TARGET_FAILURE 7 + #define VIRTIO_SCSI_S_NEXUS_FAILURE 8 + #define VIRTIO_SCSI_S_FAILURE 9 + + /* task_attr */ + #define VIRTIO_SCSI_S_SIMPLE 0 + #define VIRTIO_SCSI_S_ORDERED 1 + #define VIRTIO_SCSI_S_HEAD 2 + #define VIRTIO_SCSI_S_ACA 3 +\end{lstlisting} + +The lun field addresses a target and logical unit in the +virtio-scsi device's SCSI domain. The only supported format for +the LUN field is: first byte set to 1, second byte set to target, +third and fourth byte representing a single level LUN structure, +followed by four zero bytes. With this representation, a +virtio-scsi device can serve up to 256 targets and 16384 LUNs per +target. + +The id field is the command identifier (“tag”). + +task_attr, prio and crn should be left to zero. task_attr defines +the task attribute as in the table above, but all task attributes +may be mapped to SIMPLE by the device; crn may also be provided +by clients, but is generally expected to be 0. The maximum CRN +value defined by the protocol is 255, since CRN is stored in an +8-bit integer. + +All of these fields are defined in SAM. They are always +read-only, as are the cdb and dataout field. The cdb_size is +taken from the configuration space. + +sense and subsequent fields are always write-only. The sense_len +field indicates the number of bytes actually written to the sense +buffer. The residual field indicates the residual size, +calculated as “data_length - number_of_transferred_bytes”, for +read or write operations. For bidirectional commands, the +number_of_transferred_bytes includes both read and written bytes. +A residual field that is less than the size of datain means that +the dataout field was processed entirely. A residual field that +exceeds the size of datain means that the dataout field was +processed partially and the datain field was not processed at +all. + +The status byte is written by the device to be the status code as +defined in SAM. + +The response byte is written by the device to be one of the +following: + + VIRTIO_SCSI_S_OK when the request was completed and the status + byte is filled with a SCSI status code (not necessarily + "GOOD"). + + VIRTIO_SCSI_S_OVERRUN if the content of the CDB requires + transferring more data than is available in the data buffers. + + VIRTIO_SCSI_S_ABORTED if the request was cancelled due to an + ABORT TASK or ABORT TASK SET task management function. + + VIRTIO_SCSI_S_BAD_TARGET if the request was never processed + because the target indicated by the lun field does not exist. + + VIRTIO_SCSI_S_RESET if the request was cancelled due to a bus + or device reset (including a task management function). + + VIRTIO_SCSI_S_TRANSPORT_FAILURE if the request failed due to a + problem in the connection between the host and the target + (severed link). + + VIRTIO_SCSI_S_TARGET_FAILURE if the target is suffering a + failure and the guest should not retry on other paths. + + VIRTIO_SCSI_S_NEXUS_FAILURE if the nexus is suffering a failure + but retrying on other paths might yield a different result. + + VIRTIO_SCSI_S_BUSY if the request failed but retrying on the + same path should work. + + VIRTIO_SCSI_S_FAILURE for other host or guest error. In + particular, if neither dataout nor datain is empty, and the + VIRTIO_SCSI_F_INOUT feature has not been negotiated, the + request will be immediately returned with a response equal to + VIRTIO_SCSI_S_FAILURE. + +\paragraph{Legacy Interface: Device Operation: Request Queues}\label{sec:Device Types / SCSI Host Device / Device Operation / Device Operation: Request Queues / Legacy Interface: Device Operation: Request Queues} +For legacy devices, the fields in struct virtio_scsi_req_cmd are the +native endian of the guest rather than (necessarily) little-endian. + +\subsubsection{Device Operation: controlq}\label{sec:Device Types / SCSI Host Device / Device Operation / Device Operation: controlq} + +The controlq is used for other SCSI transport operations. +Requests have the following format: + +\begin{lstlisting} + struct virtio_scsi_ctrl { + le32 type; + ... + u8 response; + }; + + /* response values valid for all commands */ + #define VIRTIO_SCSI_S_OK 0 + #define VIRTIO_SCSI_S_BAD_TARGET 3 + #define VIRTIO_SCSI_S_BUSY 5 + #define VIRTIO_SCSI_S_TRANSPORT_FAILURE 6 + #define VIRTIO_SCSI_S_TARGET_FAILURE 7 + #define VIRTIO_SCSI_S_NEXUS_FAILURE 8 + #define VIRTIO_SCSI_S_FAILURE 9 + #define VIRTIO_SCSI_S_INCORRECT_LUN 12 +\end{lstlisting} + +The type identifies the remaining fields. + +The following commands are defined: + + Task management function +\begin{lstlisting} + #define VIRTIO_SCSI_T_TMF 0 + + #define VIRTIO_SCSI_T_TMF_ABORT_TASK 0 + #define VIRTIO_SCSI_T_TMF_ABORT_TASK_SET 1 + #define VIRTIO_SCSI_T_TMF_CLEAR_ACA 2 + #define VIRTIO_SCSI_T_TMF_CLEAR_TASK_SET 3 + #define VIRTIO_SCSI_T_TMF_I_T_NEXUS_RESET 4 + #define VIRTIO_SCSI_T_TMF_LOGICAL_UNIT_RESET 5 + #define VIRTIO_SCSI_T_TMF_QUERY_TASK 6 + #define VIRTIO_SCSI_T_TMF_QUERY_TASK_SET 7 + + struct virtio_scsi_ctrl_tmf + { + // Read-only part + le32 type; + le32 subtype; + u8 lun[8]; + le64 id; + // Write-only part + u8 response; + } + + /* command-specific response values */ + #define VIRTIO_SCSI_S_FUNCTION_COMPLETE 0 + #define VIRTIO_SCSI_S_FUNCTION_SUCCEEDED 10 + #define VIRTIO_SCSI_S_FUNCTION_REJECTED 11 +\end{lstlisting} + + The type is VIRTIO_SCSI_T_TMF; the subtype field defines. All + fields except response are filled by the driver. The subtype + field must always be specified and identifies the requested + task management function. + + Other fields may be irrelevant for the requested TMF; if so, + they are ignored but they should still be present. The lun + field is in the same format specified for request queues; the + single level LUN is ignored when the task management function + addresses a whole I_T nexus. When relevant, the value of the id + field is matched against the id values passed on the requestq. + + The outcome of the task management function is written by the + device in the response field. The command-specific response + values map 1-to-1 with those defined in SAM. + + Asynchronous notification query + +\begin{lstlisting} + #define VIRTIO_SCSI_T_AN_QUERY 1 + + struct virtio_scsi_ctrl_an { + // Read-only part + le32 type; + u8 lun[8]; + le32 event_requested; + // Write-only part + le32 event_actual; + u8 response; + } + + #define VIRTIO_SCSI_EVT_ASYNC_OPERATIONAL_CHANGE 2 + #define VIRTIO_SCSI_EVT_ASYNC_POWER_MGMT 4 + #define VIRTIO_SCSI_EVT_ASYNC_EXTERNAL_REQUEST 8 + #define VIRTIO_SCSI_EVT_ASYNC_MEDIA_CHANGE 16 + #define VIRTIO_SCSI_EVT_ASYNC_MULTI_HOST 32 + #define VIRTIO_SCSI_EVT_ASYNC_DEVICE_BUSY 64 +\end{lstlisting} + + By sending this command, the driver asks the device which + events the given LUN can report, as described in paragraphs 6.6 + and A.6 of the SCSI MMC specification. The driver writes the + events it is interested in into the event_requested; the device + responds by writing the events that it supports into + event_actual. + + The type is VIRTIO_SCSI_T_AN_QUERY. The lun and event_requested + fields are written by the driver. The event_actual and response + fields are written by the device. + + No command-specific values are defined for the response byte. + + Asynchronous notification subscription +\begin{lstlisting} + #define VIRTIO_SCSI_T_AN_SUBSCRIBE 2 + + struct virtio_scsi_ctrl_an { + // Read-only part + le32 type; + u8 lun[8]; + le32 event_requested; + // Write-only part + le32 event_actual; + u8 response; + } +\end{lstlisting} + + By sending this command, the driver asks the specified LUN to + report events for its physical interface, again as described in + the SCSI MMC specification. The driver writes the events it is + interested in into the event_requested; the device responds by + writing the events that it supports into event_actual. + + Event types are the same as for the asynchronous notification + query message. + + The type is VIRTIO_SCSI_T_AN_SUBSCRIBE. The lun and + event_requested fields are written by the driver. The + event_actual and response fields are written by the device. + + No command-specific values are defined for the response byte. + +\paragraph{Legacy Interface: Device Operation: controlq}\label{sec:Device Types / SCSI Host Device / Device Operation / Device Operation: controlq / Legacy Interface: Device Operation: controlq} + +For legacy devices, the fields in struct virtio_scsi_ctrl, struct +virtio_scsi_ctrl_tmf, struct virtio_scsi_ctrl_an and struct +virtio_scsi_ctrl_an are the native endian of the guest rather than +(necessarily) little-endian. + + +\subsubsection{Device Operation: eventq}\label{sec:Device Types / SCSI Host Device / Device Operation / Device Operation: eventq} + +The eventq is used by the device to report information on logical +units that are attached to it. The driver should always leave a +few buffers ready in the eventq. In general, the device will not +queue events to cope with an empty eventq, and will end up +dropping events if it finds no buffer ready. However, when +reporting events for many LUNs (e.g. when a whole target +disappears), the device can throttle events to avoid dropping +them. For this reason, placing 10-15 buffers on the event queue +should be enough. + +Buffers are placed in the eventq and filled by the device when +interesting events occur. The buffers should be strictly +write-only (device-filled) and the size of the buffers should be +at least the value given in the device's configuration +information. + +Buffers returned by the device on the eventq will be referred to +as "events" in the rest of this section. Events have the +following format: + +\begin{lstlisting} + #define VIRTIO_SCSI_T_EVENTS_MISSED 0x80000000 + + struct virtio_scsi_event { + // Write-only part + le32 event; + u8 lun[8]; + le32 reason; + } +\end{lstlisting} + +If bit 31 is set in the event field, the device failed to report +an event due to missing buffers. In this case, the driver should +poll the logical units for unit attention conditions, and/or do +whatever form of bus scan is appropriate for the guest operating +system. + +The meaning of the reason field depends on the +contents of the event field. The following events are defined: + + No event +\begin{lstlisting} + #define VIRTIO_SCSI_T_NO_EVENT 0 +\end{lstlisting} + + This event is fired in the following cases: + + • When the device detects in the eventq a buffer that is + shorter than what is indicated in the configuration field, it + might use it immediately and put this dummy value in the + event field. A well-written driver will never observe this + situation. + + • When events are dropped, the device may signal this event as + soon as the drivers makes a buffer available, in order to + request action from the driver. In this case, of course, this + event will be reported with the VIRTIO_SCSI_T_EVENTS_MISSED + flag. + + Transport reset +\begin{lstlisting} + #define VIRTIO_SCSI_T_TRANSPORT_RESET 1 + + #define VIRTIO_SCSI_EVT_RESET_HARD 0 + #define VIRTIO_SCSI_EVT_RESET_RESCAN 1 + #define VIRTIO_SCSI_EVT_RESET_REMOVED 2 +\end{lstlisting} + + By sending this event, the device signals that a logical unit + on a target has been reset, including the case of a new device + appearing or disappearing on the bus.The device fills in all + fields. The event field is set to + VIRTIO_SCSI_T_TRANSPORT_RESET. The lun field addresses a + logical unit in the SCSI host. + + The reason value is one of the three \#define values appearing + above: + + • VIRTIO_SCSI_EVT_RESET_REMOVED (“LUN/target removed”) is used + if the target or logical unit is no longer able to receive + commands. + + • VIRTIO_SCSI_EVT_RESET_HARD (“LUN hard reset”) is used if the + logical unit has been reset, but is still present. + + • VIRTIO_SCSI_EVT_RESET_RESCAN (“rescan LUN/target”) is used if + a target or logical unit has just appeared on the device. + + The “removed” and “rescan” events, when sent for LUN 0, may + apply to the entire target. After receiving them the driver + should ask the initiator to rescan the target, in order to + detect the case when an entire target has appeared or + disappeared. These two events will never be reported unless the + VIRTIO_SCSI_F_HOTPLUG feature was negotiated between the host + and the guest. + + Events will also be reported via sense codes (this obviously + does not apply to newly appeared buses or targets, since the + application has never discovered them): + + • “LUN/target removed” maps to sense key ILLEGAL REQUEST, asc + 0x25, ascq 0x00 (LOGICAL UNIT NOT SUPPORTED) + + • “LUN hard reset” maps to sense key UNIT ATTENTION, asc 0x29 + (POWER ON, RESET OR BUS DEVICE RESET OCCURRED) + + • “rescan LUN/target” maps to sense key UNIT ATTENTION, asc + 0x3f, ascq 0x0e (REPORTED LUNS DATA HAS CHANGED) + + The preferred way to detect transport reset is always to use + events, because sense codes are only seen by the driver when it + sends a SCSI command to the logical unit or target. However, in + case events are dropped, the initiator will still be able to + synchronize with the actual state of the controller if the + driver asks the initiator to rescan of the SCSI bus. During the + rescan, the initiator will be able to observe the above sense + codes, and it will process them as if it the driver had + received the equivalent event. + + Asynchronous notification +\begin{lstlisting} + #define VIRTIO_SCSI_T_ASYNC_NOTIFY 2 +\end{lstlisting} + + By sending this event, the device signals that an asynchronous + event was fired from a physical interface. + + All fields are written by the device. The event field is set to + VIRTIO_SCSI_T_ASYNC_NOTIFY. The lun field addresses a logical + unit in the SCSI host. The reason field is a subset of the + events that the driver has subscribed to via the "Asynchronous + notification subscription" command. + + When dropped events are reported, the driver should poll for + asynchronous events manually using SCSI commands. + + LUN parameter change +\begin{lstlisting} + #define VIRTIO_SCSI_T_PARAM_CHANGE 3 +\end{lstlisting} + + By sending this event, the device signals that the configuration parameters + (for example the capacity) of a logical unit have changed. + The event field is set to VIRTIO_SCSI_T_PARAM_CHANGE. + The lun field addresses a logical unit in the SCSI host. + + The same event is also reported as a unit attention condition. + The reason field contains the additional sense code and additional sense code qualifier, + respectively in bits 0..7 and 8..15. + For example, a change in capacity will be reported as asc 0x2a, ascq 0x09 + (CAPACITY DATA HAS CHANGED). + + For MMC devices (inquiry type 5) there would be some overlap between this + event and the asynchronous notification event. + For simplicity, as of this version of the specification the host must + never report this event for MMC devices. + +\paragraph{Legacy Interface: Device Operation: eventq}\label{sec:Device Types / SCSI Host Device / Device Operation / Device Operation: eventq / Legacy Interface: Device Operation: eventq} +For legacy devices, the fields in struct virtio_scsi_event are the +native endian of the guest rather than (necessarily) little-endian. + +\chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} + +Currently there are four device-independent feature bits defined: + + VIRTIO_F_RING_INDIRECT_DESC (28) Negotiating this feature indicates + that the driver can use descriptors with the VRING_DESC_F_INDIRECT + flag set, as described in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Descriptor Table / Indirect Descriptors}. + + VIRTIO_F_RING_EVENT_IDX(29) This feature enables the used_event + and the avail_event fields. If set, it indicates that the + device should ignore the flags field in the available ring + structure. Instead, the used_event field in this structure is + used by guest to suppress device interrupts. Further, the + driver should ignore the flags field in the used ring + structure. Instead, the avail_event field in this structure is + used by the device to suppress notifications. If unset, the + driver should ignore the used_event field; the device should + ignore the avail_event field; the flags field is used + + VIRTIO_F_VERSION_1(32) This feature must be offered by any device + compliant with this specification, and acknowledged by all device + drivers. + +In addition, bit 30 is used by qemu's implementation to check for experimental +early versions of virtio which did not perform correct feature negotiation, +and should not be used. + +\section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits} + +Legacy or transitional devices may offer the following: + +VIRTIO_F_NOTIFY_ON_EMPTY (24) Negotiating this feature + indicates that the driver wants an interrupt if the device runs + out of available descriptors on a virtqueue, even though + interrupts are suppressed using the VRING_AVAIL_F_NO_INTERRUPT + flag or the used_event field. An example of this is the + networking driver: it doesn't need to know every time a packet + is transmitted, but it does need to free the transmitted + packets a finite time after they are transmitted. It can avoid + using a timer if the device interrupts it when all the packets + are transmitted. + +VIRTIO_F_ANY_LAYOUT (27) This feature indicates that the device + accepts arbitrary descriptor layouts, as described in Section + \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}~\nameref{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}. + +\chapter{virtio_ring.h}\label{sec:virtio-ring.h} + +\begin{lstlisting} +#ifndef VIRTIO_RING_H +#define VIRTIO_RING_H +/* An interface for efficient virtio implementation. + * + * This header is BSD licensed so anyone can use the definitions + * to implement compatible drivers/servers. + * + * Copyright 2007, 2009, IBM Corporation + * Copyright 2011, Red Hat, Inc + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of IBM nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL IBM OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ +#include <stdint.h> + +/* This marks a buffer as continuing via the next field. */ +#define VRING_DESC_F_NEXT 1 +/* This marks a buffer as write-only (otherwise read-only). */ +#define VRING_DESC_F_WRITE 2 +/* This means the buffer contains a list of buffer descriptors. */ +#define VRING_DESC_F_INDIRECT 4 + +/* The Host uses this in used->flags to advise the Guest: don't kick me + * when you add a buffer. It's unreliable, so it's simply an + * optimization. Guest will still kick if it's out of buffers. */ +#define VRING_USED_F_NO_NOTIFY 1 +/* The Guest uses this in avail->flags to advise the Host: don't + * interrupt me when you consume a buffer. It's unreliable, so it's + * simply an optimization. */ +#define VRING_AVAIL_F_NO_INTERRUPT 1 + +/* Support for indirect descriptors */ +#define VIRTIO_RING_F_INDIRECT_DESC 28 + +/* Support for avail_idx and used_idx fields */ +#define VIRTIO_RING_F_EVENT_IDX 29 + +/* Arbitrary descriptor layouts. */ +#define VIRTIO_F_ANY_LAYOUT 27 + +/* Virtio ring descriptors: 16 bytes. + * These can chain together via "next". */ +struct vring_desc { + /* Address (guest-physical). */ + le64 addr; + /* Length. */ + le32 len; + /* The flags as indicated above. */ + le16 flags; + /* We chain unused descriptors via this, too */ + le16 next; +}; + +struct vring_avail { + le16 flags; + le16 idx; + le16 ring[]; + /* Only if VIRTIO_RING_F_EVENT_IDX: le16 used_event; */ +}; + +/* le32 is used here for ids for padding reasons. */ +struct vring_used_elem { + /* Index of start of used descriptor chain. */ + le32 id; + /* Total length of the descriptor chain which was written to. */ + le32 len; +}; + +struct vring_used { + le16 flags; + le16 idx; + struct vring_used_elem ring[]; + /* Only if VIRTIO_RING_F_EVENT_IDX: le16 avail_event; */ +}; + +struct vring { + unsigned int num; + + struct vring_desc *desc; + struct vring_avail *avail; + struct vring_used *used; +}; + +/* The standard layout for the ring is a continuous chunk of memory which + * looks like this. We assume num is a power of 2. + * + * struct vring { + * // The actual descriptors (16 bytes each) + * struct vring_desc desc[num]; + * + * // A ring of available descriptor heads with free-running index. + * le16 avail_flags; + * le16 avail_idx; + * le16 available[num]; + * le16 used_event_idx; // Only if VIRTIO_RING_F_EVENT_IDX + * + * // Padding to the next align boundary. + * char pad[]; + * + * // A ring of used descriptor heads with free-running index. + * le16 used_flags; + * le16 used_idx; + * struct vring_used_elem used[num]; + * le16 avail_event_idx; // Only if VIRTIO_RING_F_EVENT_IDX + * }; + * Note: for virtio PCI, align is 4096. + */ +static inline void vring_init(struct vring *vr, unsigned int num, void *p, + unsigned long align) +{ + vr->num = num; + vr->desc = p; + vr->avail = p + num*sizeof(struct vring_desc); + vr->used = (void *)(((unsigned long)&vr->avail->ring[num] + sizeof(le16) + + align-1) + & ~(align - 1)); +} + +static inline unsigned vring_size(unsigned int num, unsigned long align) +{ + return ((sizeof(struct vring_desc)*num + sizeof(le16)*(3+num) + + align - 1) & ~(align - 1)) + + sizeof(le16)*3 + sizeof(struct vring_used_elem)*num; +} + +static inline int vring_need_event(uint16_t event_idx, uint16_t new_idx, uint16_t old_idx) +{ + return (uint16_t)(new_idx - event_idx - 1) < (uint16_t)(new_idx - old_idx); +} + +/* Get location of event indices (only with VIRTIO_RING_F_EVENT_IDX) */ +static inline le16 *vring_used_event(struct vring *vr) +{ + /* For backwards compat, used event index is at *end* of avail ring. */ + return &vr->avail->ring[vr->num]; +} + +static inline le16 *vring_avail_event(struct vring *vr) +{ + /* For backwards compat, avail event index is at *end* of used ring. */ + return (le16 *)&vr->used->ring[vr->num]; +} +#endif /* VIRTIO_RING_H */ +\end{lstlisting} + + + +\chapter{Creating New Device Types}\label{sec:Creating New Device Types} + +Various considerations are necessary when creating a new device +type. + +\section{How Many Virtqueues?}\label{sec:Creating New Device Types / How Many Virtqueues?} + +It is possible that a very simple device will operate entirely +through its configuration space, but most will need at least one +virtqueue in which it will place requests. A device with both +input and output (eg. console and network devices described here) +need two queues: one which the driver fills with buffers to +receive input, and one which the driver places buffers to +transmit output. + +\section{What Configuration Space Layout?}\label{sec:Creating New Device Types / What Configuration Space Layout?} + +Configuration space should only be used for initialization-time +parameters. It is a limited resource with no synchronization between +writable fields, so for most uses it is better to use a virtqueue to update +configuration information (the network device does this for filtering, +otherwise the table in the config space could potentially be very +large). + +Devices must not assume that configuration fields over 32 bits wide +are atomically writable. + +\section{What Device Number?}\label{sec:Creating New Device Types / What Device Number?} + +Currently device numbers are assigned quite freely: a simple +request mail to the author of this document or the Linux +virtualization mailing list +\footnote{https://lists.linux-foundation.org/mailman/listinfo/virtualization +} will be sufficient to secure a unique one. + +Meanwhile for experimental drivers, use 65535 and work backwards. + +\section{How many MSI-X vectors? (for PCI)}\label{sec:Creating New Device Types / How many MSI-X vectors? (for PCI)} + +Using the optional MSI-X capability devices can speed up +interrupt processing by removing the need to read ISR Status +register by guest driver (which might be an expensive operation), +reducing interrupt sharing between devices and queues within the +device, and handling interrupts from multiple CPUs. However, some +systems impose a limit (which might be as low as 256) on the +total number of MSI-X vectors that can be allocated to all +devices. Devices and/or device drivers should take this into +account, limiting the number of vectors used unless the device is +expected to cause a high volume of interrupts. Devices can +control the number of vectors used by limiting the MSI-X Table +Size or not presenting MSI-X capability in PCI configuration +space. Drivers can control this by mapping events to as small +number of vectors as possible, or disabling MSI-X capability +altogether. + +\section{Device Improvements}\label{sec:Creating New Device Types / Device Improvements} + +Any change to configuration space, or new virtqueues, or +behavioural changes, should be indicated by negotiation of a new +feature bit. This establishes clarity +\footnote{Even if it does mean documenting design or implementation +mistakes! +} and avoids future expansion problems. + +Clusters of functionality which are always implemented together +can use a single bit, but if one feature makes sense without the +others they should not be gratuitously grouped together to +conserve feature bits. + + |