[virtio-comment] Re: [PATCH V2 6/6] virtio-pci: implement dirty page tracking

["Michael S. Tsirkin" <mst@redhat.com>]

On Fri, Nov 03, 2023 at 10:32:59PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 11/3/2023 6:50 PM, Michael S. Tsirkin wrote:
> 
>     On Fri, Nov 03, 2023 at 06:34:37PM +0800, Zhu Lingshan wrote:
> 
>         +\item[\field{bitmap_addr}]
>         +       The driver use this to set the address of the bitmap which records the dirty pages
>         +       caused by the device.
>         +       Each bit in the bitmap represents one memory page, bit 0 in the bitmap
>         +       reprsents page 0 at address 0, bit 1 represents page 1, and so on in a linear manner.
>         +       When \field{enable} is set to 1 and the device writes to a memory page,
>         +       the device MUST set the corresponding bit to 1 which indicating the page is dirty.
>         +\item[\field{bitmap_length}]
>         +       The driver use this to set the length in bytes of the bitmap.
>         +\end{description}
>         +
>         +\devicenormative{\subsubsection}{Memory Dirty Pages Tracker Capability}{Virtio Transport Options / Virtio Over PCI Bus / Memory Dirty Pages Tracker Capability}
>         +
>         +The device MUST NOT set any bits beyond bitmap_length when reporting dirty pages.
>         +
>         +To prevent a read-modify-write procedure, if a memory page is dirty,
>         +optionally the device is permitted to set the entire byte, which encompasses the relevant bit, to 1.
>         +
>         +The device MAY increase \field{gra_power} to reduce \field{bitmap_length}.
>         +
>         +The device must ignore any writes to \field{pasid} if PASID Extended Capability is absent or
>         +the PASID functionality is disabled in PASID Extended Capability
> 
> 
>     I have to say this is going to work very badly when the number of dirty
>     pages is small: you will end up scanning and re-scanning all of bitmap.
> 
> The driver needs to scan anyway,

Not with e.g. Parav's proposal - device reports individual pages
changed. This is analogous to PML.

> Intel production work with similar bitmap
> based dirty page tracking solution for years.

and then VMs became bigger and PML was introduced.

> Otherwise the device should report PFN which is not very practical.

Why not?

>     And the resolution is apparently 8 pages? You have just multiplied
>     the migration bandwidth by a factor of 8.
> 
> No, as described in the comments, the tacking granularity is controlled by \
> field{gra_power}, one bit represents a page with page_size = 2^(12 +
> gra_power). This can also be used to reduce the size of the bitmap.

.. at the cost of increasing migration bandwidth.

> "To prevent a read-modify-write procedure, if a memory page is dirty,
> optionally the device is permitted to set the entire byte, which encompasses the relevant bit, to 1."
> 
> This is optional and DMA is very likely to write a neighbor page, and the device transmit a whole byte anyway
> when a bit is dirty.
> 
> How about we use platform dirty page tracking facility then implement this in virtio, as Jason suggested?
> 

Without something like PML it likely won't scale either.

-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/