Regarding to the recent Intel IAA/DSA/QAT support on migration

Regarding to the recent Intel IAA/DSA/QAT support on migration

[Peter Xu]

Copy qemu-devel.

On Wed, Feb 07, 2024 at 04:07:40PM +0800, Peter Xu wrote:
> Hi,
> 
> I'm sending this email just to leave a generic comment to the recent
> migration efforts to enable these new Intel technologies.
> 
> The relevant patchsets (latest version so far) we're discussing are:
> 
>   [PATCH v3 0/4] Live Migration Acceleration with IAA Compression
>   https://lore.kernel.org/r/20240103112851.908082-1-yuan1.liu@intel.com
>   
>   [PATCH v3 00/20] Use Intel DSA accelerator to offload zero page checking in multifd live migration.
>   https://lore.kernel.org/r/20240104004452.324068-1-hao.xiang@bytedance.com
>   
>   [PATCH 0/5] *** Implement using Intel QAT to offload ZLIB
>   https://lore.kernel.org/r/20231231205804.2366509-1-bryan.zhang@bytedance.com
> 
> I want to comment in a generic way since this should apply to all these
> series:
> 
>   - A heads-up that multifd code is rapidly changing recently, I apologize
>     that you'll need a rebase.  It's just that it's probably much better to
>     do this before anything lands there.
> 
>     IIUC the good thing is we found that send_prepare() doesn't need to be
>     changed that much, however there's still some change; please refer to
>     the new code (I'll prepare a pull tomorrow to include most of the
>     changes, and we should have a major thread race fixed too with Fabiano
>     & Avihai's help). I hope this will also provide some kind of isolation
>     to e.g. other works that may touch other areas.  E.g., I hope fixed-ram
>     won't need to conflict much with any of the above series now.
> 
>   - When posting the new patchset (if there is a plan..), please make sure
>     we have:
> 
>     - Proper unit tests for the new code (probably mostly software
>       fallbacks to be tested on the new libraries being introduced; just to
>       make sure the new library code paths can get some torture please).
> 
>     - Proper documentation for the new code.  Please feel free to start
>       creating your own .rst file under docs/devel/migration/, we can try
>       to merge them later.  It should help avoid conflictions.  Please also
>       link the new file into index.rst there.
> 
>       IMHO the document can contain many things, the important ones could
>       start from: who should enable such feature; what one can get from
>       having it enabled; what is the HW requirement to enable it; how
>       should one tune the new parameters, and so on... some links to the
>       technology behinds it would be nice too to be referenced.
> 
>     - Try to add new code (especially HW/library based) into new file.
>       I see that QPL & QAT already proposed its own files (multifd-pql.c,
>       multifd-qatzip.c) which is great.
> 
>       Xiang, please also consider doing so for the DSA based zero page
>       detection.  It can be called multifd-zero-page.c, for example, and
>       you can create it when working on the
>       offload-zero-page-detect-to-multifd patchset already.
> 
>     - Please provide someone who can potentially maintain this code if ever
>       possible.  Pushing these code upstream is great, but maintaining will
>       also take effort.  It might be impractical this keeps growing for
>       migration maintainers (currently Fabiano and myself), so we may like
>       to have people covering these areas, especially when the new codes
>       are not directly relevant to migration framework.
> 
>       I'd suggest for each of the project we can add an entry in
>       MAINTAINERS below "Migration" section, adding relevant files (and
>       these files should exist in both the new section and "Migration").  I
>       am not sure whether Bytedance would be able to cover this, or we
>       should try to find someone from Intel?  If you're willing to add
>       yourself to maintain such codes, please attach the maintainers file
>       change together with the series.  It will be very much appreciated.
> 
> Thanks,
> 
> -- 
> Peter Xu

-- 
Peter Xu

RE: Regarding to the recent Intel IAA/DSA/QAT support on migration

[Liu, Yuan1]

Thank you very much for your reminder and the rapid updates to the
multifd function. I will incorporate your suggestions into the next
version (IAA Accelerated Live Migration solution). 

Regarding the QAT and DSA optimization, my colleagues and I have 
already started reviewing and testing them, and it seems like a 
promising optimization direction. I am more than willing to contribute 
further efforts to the long-term maintenance of Intel accelerators in 
live migration.

> -----Original Message-----
> From: Peter Xu <peterx@redhat.com>
> Sent: Wednesday, February 7, 2024 4:10 PM
> To: Bryan Zhang <bryan.zhang@bytedance.com>; Hao Xiang
> <hao.xiang@bytedance.com>; Liu, Yuan1 <yuan1.liu@intel.com>
> Cc: Fabiano Rosas <farosas@suse.de>; QEMU Devel Mailing List <qemu-
> devel@nongnu.org>
> Subject: Regarding to the recent Intel IAA/DSA/QAT support on migration
> 
> Copy qemu-devel.
> 
> On Wed, Feb 07, 2024 at 04:07:40PM +0800, Peter Xu wrote:
> > Hi,
> >
> > I'm sending this email just to leave a generic comment to the recent
> > migration efforts to enable these new Intel technologies.
> >
> > The relevant patchsets (latest version so far) we're discussing are:
> >
> >   [PATCH v3 0/4] Live Migration Acceleration with IAA Compression
> >
> > https://lore.kernel.org/r/20240103112851.908082-1-yuan1.liu@intel.com
> >
> >   [PATCH v3 00/20] Use Intel DSA accelerator to offload zero page
> checking in multifd live migration.
> >
> > https://lore.kernel.org/r/20240104004452.324068-1-hao.xiang@bytedance.
> > com
> >
> >   [PATCH 0/5] *** Implement using Intel QAT to offload ZLIB
> >
> > https://lore.kernel.org/r/20231231205804.2366509-1-bryan.zhang@bytedan
> > ce.com
> >
> > I want to comment in a generic way since this should apply to all
> > these
> > series:
> >
> >   - A heads-up that multifd code is rapidly changing recently, I
> apologize
> >     that you'll need a rebase.  It's just that it's probably much better
> to
> >     do this before anything lands there.
> >
> >     IIUC the good thing is we found that send_prepare() doesn't need to
> be
> >     changed that much, however there's still some change; please refer
> to
> >     the new code (I'll prepare a pull tomorrow to include most of the
> >     changes, and we should have a major thread race fixed too with
> Fabiano
> >     & Avihai's help). I hope this will also provide some kind of
> isolation
> >     to e.g. other works that may touch other areas.  E.g., I hope fixed-
> ram
> >     won't need to conflict much with any of the above series now.
> >
> >   - When posting the new patchset (if there is a plan..), please make
> sure
> >     we have:
> >
> >     - Proper unit tests for the new code (probably mostly software
> >       fallbacks to be tested on the new libraries being introduced; just
> to
> >       make sure the new library code paths can get some torture please).
> >
> >     - Proper documentation for the new code.  Please feel free to start
> >       creating your own .rst file under docs/devel/migration/, we can
> try
> >       to merge them later.  It should help avoid conflictions.  Please
> also
> >       link the new file into index.rst there.
> >
> >       IMHO the document can contain many things, the important ones
> could
> >       start from: who should enable such feature; what one can get from
> >       having it enabled; what is the HW requirement to enable it; how
> >       should one tune the new parameters, and so on... some links to the
> >       technology behinds it would be nice too to be referenced.
> >
> >     - Try to add new code (especially HW/library based) into new file.
> >       I see that QPL & QAT already proposed its own files (multifd-
> pql.c,
> >       multifd-qatzip.c) which is great.
> >
> >       Xiang, please also consider doing so for the DSA based zero page
> >       detection.  It can be called multifd-zero-page.c, for example, and
> >       you can create it when working on the
> >       offload-zero-page-detect-to-multifd patchset already.
> >
> >     - Please provide someone who can potentially maintain this code if
> ever
> >       possible.  Pushing these code upstream is great, but maintaining
> will
> >       also take effort.  It might be impractical this keeps growing for
> >       migration maintainers (currently Fabiano and myself), so we may
> like
> >       to have people covering these areas, especially when the new codes
> >       are not directly relevant to migration framework.
> >
> >       I'd suggest for each of the project we can add an entry in
> >       MAINTAINERS below "Migration" section, adding relevant files (and
> >       these files should exist in both the new section and "Migration").
> I
> >       am not sure whether Bytedance would be able to cover this, or we
> >       should try to find someone from Intel?  If you're willing to add
> >       yourself to maintain such codes, please attach the maintainers
> file
> >       change together with the series.  It will be very much
> appreciated.
> >
> > Thanks,
> >
> > --
> > Peter Xu
> 
> --
> Peter Xu

Re: [External] RE: Regarding to the recent Intel IAA/DSA/QAT support on migration

[Hao Xiang]

On Wed, Feb 7, 2024 at 12:38 AM Liu, Yuan1 <yuan1.liu@intel.com> wrote:
>
> Thank you very much for your reminder and the rapid updates to the
> multifd function. I will incorporate your suggestions into the next
> version (IAA Accelerated Live Migration solution).
>
> Regarding the QAT and DSA optimization, my colleagues and I have
> already started reviewing and testing them, and it seems like a
> promising optimization direction. I am more than willing to contribute
> further efforts to the long-term maintenance of Intel accelerators in
> live migration.
>
> > -----Original Message-----
> > From: Peter Xu <peterx@redhat.com>
> > Sent: Wednesday, February 7, 2024 4:10 PM
> > To: Bryan Zhang <bryan.zhang@bytedance.com>; Hao Xiang
> > <hao.xiang@bytedance.com>; Liu, Yuan1 <yuan1.liu@intel.com>
> > Cc: Fabiano Rosas <farosas@suse.de>; QEMU Devel Mailing List <qemu-
> > devel@nongnu.org>
> > Subject: Regarding to the recent Intel IAA/DSA/QAT support on migration
> >
> > Copy qemu-devel.
> >
> > On Wed, Feb 07, 2024 at 04:07:40PM +0800, Peter Xu wrote:
> > > Hi,
> > >
> > > I'm sending this email just to leave a generic comment to the recent
> > > migration efforts to enable these new Intel technologies.
> > >
> > > The relevant patchsets (latest version so far) we're discussing are:
> > >
> > >   [PATCH v3 0/4] Live Migration Acceleration with IAA Compression
> > >
> > > https://lore.kernel.org/r/20240103112851.908082-1-yuan1.liu@intel.com
> > >
> > >   [PATCH v3 00/20] Use Intel DSA accelerator to offload zero page
> > checking in multifd live migration.
> > >
> > > https://lore.kernel.org/r/20240104004452.324068-1-hao.xiang@bytedance.
> > > com
> > >
> > >   [PATCH 0/5] *** Implement using Intel QAT to offload ZLIB
> > >
> > > https://lore.kernel.org/r/20231231205804.2366509-1-bryan.zhang@bytedan
> > > ce.com
> > >
> > > I want to comment in a generic way since this should apply to all
> > > these
> > > series:
> > >
> > >   - A heads-up that multifd code is rapidly changing recently, I
> > apologize
> > >     that you'll need a rebase.  It's just that it's probably much better
> > to
> > >     do this before anything lands there.
> > >
> > >     IIUC the good thing is we found that send_prepare() doesn't need to
> > be
> > >     changed that much, however there's still some change; please refer
> > to
> > >     the new code (I'll prepare a pull tomorrow to include most of the
> > >     changes, and we should have a major thread race fixed too with
> > Fabiano
> > >     & Avihai's help). I hope this will also provide some kind of
> > isolation
> > >     to e.g. other works that may touch other areas.  E.g., I hope fixed-
> > ram
> > >     won't need to conflict much with any of the above series now.

Thanks for the update. The rebase shouldn't be that bad so no worries.

> > >
> > >   - When posting the new patchset (if there is a plan..), please make
> > sure
> > >     we have:
> > >
> > >     - Proper unit tests for the new code (probably mostly software
> > >       fallbacks to be tested on the new libraries being introduced; just
> > to
> > >       make sure the new library code paths can get some torture please).
> > >
> > >     - Proper documentation for the new code.  Please feel free to start
> > >       creating your own .rst file under docs/devel/migration/, we can
> > try
> > >       to merge them later.  It should help avoid conflictions.  Please
> > also
> > >       link the new file into index.rst there.
> > >
> > >       IMHO the document can contain many things, the important ones
> > could
> > >       start from: who should enable such feature; what one can get from
> > >       having it enabled; what is the HW requirement to enable it; how
> > >       should one tune the new parameters, and so on... some links to the
> > >       technology behinds it would be nice too to be referenced.
> > >
> > >     - Try to add new code (especially HW/library based) into new file.
> > >       I see that QPL & QAT already proposed its own files (multifd-
> > pql.c,
> > >       multifd-qatzip.c) which is great.
> > >
> > >       Xiang, please also consider doing so for the DSA based zero page
> > >       detection.  It can be called multifd-zero-page.c, for example, and
> > >       you can create it when working on the
> > >       offload-zero-page-detect-to-multifd patchset already.

Sounds good.

> > >
> > >     - Please provide someone who can potentially maintain this code if
> > ever
> > >       possible.  Pushing these code upstream is great, but maintaining
> > will
> > >       also take effort.  It might be impractical this keeps growing for
> > >       migration maintainers (currently Fabiano and myself), so we may
> > like
> > >       to have people covering these areas, especially when the new codes
> > >       are not directly relevant to migration framework.
> > >
> > >       I'd suggest for each of the project we can add an entry in
> > >       MAINTAINERS below "Migration" section, adding relevant files (and
> > >       these files should exist in both the new section and "Migration").
> > I
> > >       am not sure whether Bytedance would be able to cover this, or we
> > >       should try to find someone from Intel?  If you're willing to add
> > >       yourself to maintain such codes, please attach the maintainers
> > file
> > >       change together with the series.  It will be very much
> > appreciated.

I will add myself to maintain this. In addition, I will talk to Intel
about adding someone from Intel (hopefully Yuan or someone from his
team?) as well to maintain the DSA work.

> > >
> > > Thanks,
> > >
> > > --
> > > Peter Xu
> >
> > --
> > Peter Xu
>