balloon: transitional device support

Virtio 1.0 cs02 doesn't include a modern balloon device.  At some
point we'll likely define an incompatible interface with a
different ID and different semantics.  But for now, it's not a
big effort to support a transitional balloon device: this has the
advantage of supporting existing drivers, transparently, as well
as transports that don't allow mixing virtio 0 and virtio 1
devices. And balloon is an easy device to test, so it's also
useful for people to test virtio core handling of transitional
devices.

Three issues with legacy hypervisors have been identified:
1. Actual value is actually used, and is necessary for management
   to work. Luckily 4 byte config space writes are now atomic.
   When using old guests, hypervisors can detect access to the last byte.
   When using old hypervisors, drivers can use atomic 4-byte accesses.
2. Hypervisors actually didn't ignore the stats from the first
   buffer supplied. This means the values there would be
   incorrect until hypervisor resends the request.
   Add a note suggesting hypervisors ignore the 1st buffer.
3. QEMU simply over-writes stats from each buffer it gets.
   Thus if driver supplies a different subset of stats
   on each request, stale values will be there.
   Require drivers to supply the same subset on each
   request. This also gives us a simple way to figure out
   which stats are supported.

VIRTIO-143

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

git-svn-id: https://tools.oasis-open.org/version-control/svn/virtio/branches/v1.0@526 0c8fb4dd-22a2-4bb5-bc14-6c75a5f43652

2 files changed, 168 insertions, 29 deletions

diff --git a/conformance.tex b/conformance.tex
index 80b333f..279b5cb 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -15,13 +15,13 @@ Conformance targets:
   \begin{itemize}
     \item Clause \ref{sec:Conformance / Driver Conformance},
     \item One of clauses \ref{sec:Conformance / Driver Conformance / PCI Driver Conformance}, \ref{sec:Conformance / Driver Conformance / MMIO Driver Conformance} or \ref{sec:Conformance / Driver Conformance / Channel I/O Driver Conformance}.
-    \item One of clauses \ref{sec:Conformance / Driver Conformance / Network Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Block Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Console Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Entropy Driver Conformance} or \ref{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance}.
+    \item One of clauses \ref{sec:Conformance / Driver Conformance / Network Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Block Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Console Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Entropy Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Traditional Memory Balloon Driver Conformance} or \ref{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance}.
   \end{itemize}
 \item[Device] A device MUST conform to three conformance clauses:
   \begin{itemize}
     \item Clause \ref{sec:Conformance / Device Conformance},
     \item One of clauses \ref{sec:Conformance / Device Conformance / PCI Device Conformance}, \ref{sec:Conformance / Device Conformance / MMIO Device Conformance} or \ref{sec:Conformance / Device Conformance / Channel I/O Device Conformance}.
-    \item One of clauses \ref{sec:Conformance / Device Conformance / Network Device Conformance}, \ref{sec:Conformance / Device Conformance / Block Device Conformance}, \ref{sec:Conformance / Device Conformance / Console Device Conformance}, \ref{sec:Conformance / Device Conformance / Entropy Device Conformance} or \ref{sec:Conformance / Device Conformance / SCSI Host Device Conformance}.
+    \item One of clauses \ref{sec:Conformance / Device Conformance / Network Device Conformance}, \ref{sec:Conformance / Device Conformance / Block Device Conformance}, \ref{sec:Conformance / Device Conformance / Console Device Conformance}, \ref{sec:Conformance / Device Conformance / Entropy Device Conformance}, \ref{sec:Conformance / Device Conformance / Traditional Memory Balloon Device Conformance} or \ref{sec:Conformance / Device Conformance / SCSI Host Device Conformance}.
   \end{itemize}
 \end{description}
 
@@ -123,6 +123,16 @@ An entropy driver MUST conform to the following normative statements:
 \item \ref{drivernormative:Device Types / Entropy Device / Device Operation}
 \end{itemize}
 
+\subsection{Traditional Memory Balloon Driver Conformance}\label{sec:Conformance / Driver Conformance / Traditional Memory Balloon Driver Conformance}
+
+A traditional memory balloon driver MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{drivernormative:Device Types / Memory Balloon Device / Feature bits}
+\item \ref{drivernormative:Device Types / Memory Balloon Device / Device Operation}
+\item \ref{drivernormative:Device Types / Memory Balloon Device / Device Operation / Memory Statistics}
+\end{itemize}
+
 \subsection{SCSI Host Driver Conformance}\label{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance}
 
 An SCSI host driver MUST conform to the following normative statements:
@@ -230,6 +240,16 @@ An entropy device MUST conform to the following normative statements:
 \item \ref{devicenormative:Device Types / Entropy Device / Device Operation}
 \end{itemize}
 
+\subsection{Traditional Memory Balloon Device Conformance}\label{sec:Conformance / Device Conformance / Traditional Memory Balloon Device Conformance}
+
+A traditional memory balloon device MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{devicenormative:Device Types / Memory Balloon Device / Feature bits}
+\item \ref{devicenormative:Device Types / Memory Balloon Device / Device Operation}
+\item \ref{devicenormative:Device Types / Memory Balloon Device / Device Operation / Memory Statistics}
+\end{itemize}
+
 \subsection{SCSI Host Device Conformance}\label{sec:Conformance / Device Conformance / SCSI Host Device Conformance}
 
 An SCSI host device MUST conform to the following normative statements:
@@ -292,7 +312,7 @@ Feature Bits / Legacy Interface: A Note on Feature Bits}
 \item Section \ref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation}
 \item Section \ref{sec:Device Types / Console Device / Device configuration layout / Legacy Interface: Device configuration layout}
 \item Section \ref{sec:Device Types / Console Device / Device Operation / Legacy Interface: Device Operation}
-\item Section \ref{drivernormative:Device Types / Memory Balloon Device / Device Operation}
+\item Section \ref{sec:Device Types / Memory Balloon Device / Feature bits / Legacy Interface: Feature bits}
 \item Section \ref{sec:Device Types / Memory Balloon Device / Device Operation / Legacy Interface: Device Operation}
 \item Section \ref{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics / Legacy Interface: Memory Statistics}
 \item Section \ref{sec:Device Types / SCSI Host Device / Device configuration layout / Legacy Interface: Device configuration layout}
diff --git a/content.tex b/content.tex
index 11015a5..b1cd7bf 100644
--- a/content.tex
+++ b/content.tex
@@ -1046,7 +1046,7 @@ Transitional PCI Device ID  &  Virtio Device    \\
 \hline
 0x1001     &   block device     \\
 \hline
-0x1002     & memory ballooning (legacy)  \\
+0x1002     & memory ballooning (traditional)  \\
 \hline
 0x1003     &      console       \\
 \hline
@@ -2966,7 +2966,7 @@ Device ID  &  Virtio Device    \\
 \hline
 4          &  entropy source    \\
 \hline
-5          & memory ballooning (legacy)  \\
+5          & memory ballooning (traditional)  \\
 \hline
 6          &     ioMemory       \\
 \hline
@@ -4357,14 +4357,13 @@ how many random bytes were received.
 The device MUST place one or more random bytes into the buffer, but it
 MAY use less than the entire buffer length.
 
-\section{Legacy Interface: Memory Balloon Device}\label{sec:Device Types / Memory Balloon Device}
+\section{Traditional Memory Balloon Device}\label{sec:Device Types / Memory Balloon Device}
 
-This device is deprecated, and thus only exists as a legacy device
-illustrated here for reference.  The device number 13 is reserved for
-a new memory balloon interface which is expected in a future version
-of the standard.
+This is the traditional balloon device.  The device number 13 is
+reserved for a new memory balloon interface, with different
+semantics, which is expected in a future version of the standard.
 
-The virtio memory balloon device is a primitive device for
+The traditional virtio memory balloon device is a primitive device for
 managing guest memory: the device asks for a certain amount of
 memory, and the driver supplies it (or withdraws it, if the device
 has more than it asks for). This allows the guest to adapt to
@@ -4393,6 +4392,25 @@ guest memory statistics to the host.
     memory statistics is present.
 \end{description}
 
+\drivernormative{\subsubsection}{Feature bits}{Device Types / Memory Balloon Device / Feature bits}
+The driver SHOULD accept the VIRTIO_BALLOON_F_MUST_TELL_HOST
+feature if offered by the device.
+
+\devicenormative{\subsubsection}{Feature bits}{Device Types / Memory Balloon Device / Feature bits}
+If the device offers the VIRTIO_BALLOON_F_MUST_TELL_HOST feature
+bit, and if the driver did not accept this feature bit, the
+device MAY signal failure by failing to set FEATURES_OK
+\field{device status} bit when the driver writes it.
+\subparagraph{Legacy Interface: Feature bits}\label{sec:Device
+Types / Memory Balloon Device / Feature bits / Legacy Interface:
+Feature bits}
+As the legacy interface does not have a way to gracefully report feature
+negotiation failure, when using the legacy interface,
+transitional devices MUST support guests which do not negotiate
+VIRTIO_BALLOON_F_MUST_TELL_HOST feature, and SHOULD
+allow guest to use memory before notifying host if
+VIRTIO_BALLOON_F_MUST_TELL_HOST is not negotiated.
+
 \subsection{Device configuration layout}\label{sec:Device Types / Memory Balloon Device / Device configuration layout}
   Both fields of this configuration
   are always available.
@@ -4404,25 +4422,31 @@ struct virtio_balloon_config {
 };
 \end{lstlisting}
 
-Note that these fields are always little endian, despite convention
-that legacy device fields are guest endian.
+\subparagraph{Legacy Interface: Device configuration layout}\label{sec:Device Types / Memory Balloon Device / Device
+configuration layout / Legacy Interface: Device configuration layout}
+When using the legacy interface, transitional devices and drivers
+MUST format the fields in struct virtio_balloon_config
+according to the little-endian format.
+\begin{note}
+This is unlike the usual convention that legacy device fields are guest endian.
+\end{note}
 
 \subsection{Device Initialization}\label{sec:Device Types / Memory Balloon Device / Device Initialization}
 
+The device initialization process is outlined below:
+
 \begin{enumerate}
 \item The inflate and deflate virtqueues are identified.
 
 \item If the VIRTIO_BALLOON_F_STATS_VQ feature bit is negotiated:
   \begin{enumerate}
   \item Identify the stats virtqueue.
-
-  \item Add one empty buffer to the stats virtqueue and notify the
-    device.
+  \item Add one empty buffer to the stats virtqueue.
+  \item DRIVER_OK is set: device operation begins.
+  \item Notify the device about the stats virtqueue buffer.
   \end{enumerate}
 \end{enumerate}
 
-Device operation begins immediately.
-
 \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / Device Operation}
 
 The device is driven by the receipt of a
@@ -4451,16 +4475,16 @@ configuration change interrupt.
   \item If the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is negotiated, the
     guest informs the device of pages before it uses them.
 
-  \item Otherwise, the guest MAY begin to re-use pages previously
+  \item Otherwise, the guest is allowed to re-use pages previously
     given to the balloon before the device has acknowledged their
     withdrawal\footnote{In this case, deflation advice is merely a courtesy.
 }.
   \end{enumerate}
 
-\item In either case, once the device has completed the inflation or
-  deflation, the driver updates \field{actual} to reflect the new number of pages in the balloon\footnote{As updates to device-specific configuration space are not atomic, this field
-isn't particularly reliable, but can be used to diagnose buggy guests.
-}.
+\item In either case, the device acknowledges inflate and deflate
+requests by using the descriptor.
+\item Once the device has acknowledged the inflation or
+  deflation, the driver updates \field{actual} to reflect the new number of pages in the balloon.
 \end{enumerate}
 
 \drivernormative{\subsubsection}{Device Operation}{Device Types / Memory Balloon Device / Device Operation}
@@ -4474,12 +4498,38 @@ The driver MUST use the deflateq to inform the device of pages that it
 wants to use from the balloon.
 
 If the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is negotiated, the
-driver MUST wait until the device has used the deflateq descriptor
-before using the pages.
+driver MUST NOT use pages from the balloon until
+the device has acknowledged the deflate request.
+
+Otherwise, if the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is not
+negotiated, the driver MAY begin to re-use pages previously
+given to the balloon before the device has acknowledged the
+deflate request.
+
+In any case, the driver MUST NOT use pages from the balloon
+after adding the pages to the balloon, but before the device has
+acknowledged the inflate request.
+
+The driver MUST NOT request deflation of pages in
+the balloon before the device has acknowledged the inflate
+request.
 
 The driver MUST update \field{actual} after changing the number
 of pages in the balloon.
 
+\devicenormative{\subsubsection}{Device Operation}{Device Types / Memory Balloon Device / Device Operation}
+
+The device MAY modify the contents of a page in the balloon
+after detecting its physical number in an inflate request
+and before acknowledging the inflate request by using the inflateq
+descriptor.
+
+If the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is negotiated, the
+device MAY modify the contents of a page in the balloon
+after detecting its physical number in an inflate request
+and before detecting its physical number in a deflate request
+and acknowledging the deflate request.
+
 \paragraph{Legacy Interface: Device Operation}\label{sec:Device
 Types / Memory Balloon Device / Device Operation / Legacy
 Interface: Device Operation}
@@ -4488,7 +4538,25 @@ When using the legacy interface, the driver SHOULD ignore the \field{len} value
 Historically, some devices put the total descriptor length there,
 even though no data was actually written.
 \end{note}
-
+When using the legacy interface, the driver MUST write out all
+4 bytes each time it updates the \field{actual} value in the
+configuration space, using a single atomic operation.
+
+When using the legacy interface, the device SHOULD NOT use the
+\field{actual} value written by the driver in the configuration
+space, until the last, most-significant byte of the value has been
+written.
+\begin{note}
+Historically, devices used the \field{actual} value, even though
+when using Virtio Over PCI Bus the device-specific configuration
+space was not guaranteed to be atomic. Using intermediate
+values during update by driver is best avoided, except for
+debugging.
+
+Historically, drivers using Virtio Over PCI Bus wrote the
+\field{actual} value by using multiple single-byte writes in
+order, from the least-significant to the most-significant value.
+\end{note}
 \subsubsection{Memory Statistics}\label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics}
 
 The stats virtqueue is atypical because communication is driven
@@ -4513,10 +4581,11 @@ as follows:
   subsequent request) and consumes the statistics.
 \end{enumerate}
 
+  Within the buffer, statistics are an array of 6-byte entries.
   Each statistic consists of a 16 bit
   tag and a 64 bit value. All statistics are optional and the
   driver chooses which ones to supply. To guarantee backwards
-  compatibility, the driver SHOULD omit unsupported statistics.
+  compatibility, devices omit unsupported statistics.
 
 \begin{lstlisting}
 struct virtio_balloon_stat {
@@ -4526,17 +4595,67 @@ struct virtio_balloon_stat {
 #define VIRTIO_BALLOON_S_MINFLT   3
 #define VIRTIO_BALLOON_S_MEMFREE  4
 #define VIRTIO_BALLOON_S_MEMTOT   5
-        u16 tag;
-        u64 val;
+        le16 tag;
+        le64 val;
 } __attribute__((packed));
 \end{lstlisting}
 
+\drivernormative{\paragraph}{Memory Statistics}{Device Types / Memory Balloon Device / Device Operation / Memory Statistics}
+Normative statements in this section apply if and only if the
+VIRTIO_BALLOON_F_STATS_VQ feature has been negotiated.
+
+The driver MUST make at most one buffer available to the device
+in the statsq, at all times.
+
+After initializing the device, the driver MUST make an output
+buffer available in the statsq.
+
+Upon detecting that device has used a buffer in the statsq, the
+driver MUST make an output buffer available in the statsq.
+
+Before making an output buffer available in the statsq, the
+driver MUST initialize it, including one struct
+virtio_balloon_stat entry for each statistic that it supports.
+
+Driver MUST use an output buffer size which is a multiple of 6
+bytes for all buffers submitted to the statsq.
+
+Driver MAY supply struct virtio_balloon_stat entries in the
+output buffer submitted to the statsq in any order, without
+regard to \field{tag} values.
+
+Driver MAY supply a subset of all statistics in the output buffer
+submitted to the statsq.
+
+Driver MUST supply the same subset of statistics in all buffers
+submitted to the statsq.
+
+\devicenormative{\paragraph}{Memory Statistics}{Device Types / Memory Balloon Device / Device Operation / Memory Statistics}
+Normative statements in this section apply if and only if  the
+VIRTIO_BALLOON_F_STATS_VQ feature has been negotiated.
+
+Within an output buffer submitted to the statsq,
+the device MUST ignore entries with \field{tag} values that it does not recognize.
+
+Within an output buffer submitted to the statsq,
+the device MUST accept struct virtio_balloon_stat entries in any
+order without regard to \field{tag} values.
+
 \paragraph{Legacy Interface: Memory Statistics}\label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics / Legacy Interface: Memory Statistics}
+
 When using the legacy interface, transitional devices and drivers
 MUST format the fields in struct virtio_balloon_stat
 according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
+When using the legacy interface,
+the device SHOULD ignore all values in the first buffer in the
+statsq supplied by the driver after device initialization.
+\begin{note}
+Historically, drivers supplied an uninitialized buffer in the
+first buffer.
+\end{note}
+
 \subsubsection{Memory Statistics Tags}\label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics Tags}
 
 \begin{description}