Re: post-copy is broken?

Re: post-copy is broken?

[Andrea Arcangeli]

Adding linux-mm too,

On Thu, Apr 14, 2016 at 01:34:41PM +0100, Dr. David Alan Gilbert wrote:
> * Andrea Arcangeli (aarcange@redhat.com) wrote:
> 
> > The next suspect is the massive THP refcounting change that went
> > upstream recently:
> 
> > As further debug hint, can you try to disable THP and see if that
> > makes the problem go away?
> 
> Yep, this seems to be the problem (cc'ing in Kirill).
> 
> 122afea9626ab3f717b250a8dd3d5ebf57cdb56c - works (just before Kirill disables THP)
> 61f5d698cc97600e813ca5cf8e449b1ea1c11492 - breaks (when THP is reenabled)
> 
> It's pretty reliable; as you say disabling THP makes it work again
> and putting it back to THP/madvise mode makes it break.  And you need
> to test on a machine with some free ram to make sure THP has a chance
> to have happened.
> 
> I'm not sure of all of the rework that happened in that series,
> but my reading of it is that splitting of THP pages gets deferred;
> so I wonder if when I do the madvise to turn THP off, if it's actually
> still got THP pages and thus we end up with a whole THP mapped
> when I'm expecting to be userfaulting those pages.

Good thing at least I didn't make UFFDIO_COPY THP aware yet so there's
less variables (as no user was interested to handle userfaults at THP
granularity yet, and from userland such an improvement would be
completely invisible in terms of API, so if an user starts doing that
we can just optimize the kernel for it, criu restore could do that as
the faults will come from disk-I/O, when network is involved THP
userfaults wouldn't have a great tradeoff with regard to the increased
fault latency).

I suspect there is an handle_userfault missing somewhere in connection
with trans_huge_pmd splits (not anymore THP splits) that you're doing
with MADV_DONTNEED to zap those pages in the destination that got
redirtied in source during the last precopy stage. Or more simply
MADV_DONTNEED isn't zapping all the right ptes after the trans huge
pmd got splitted.

The fact the page isn't splitted shouldn't matter too much, all we care
about is the pte triggers handle_userfault after MADV_DONTNEED.

The userfaultfd testcase in the kernel isn't exercising this case
unfortunately, that should probably be improved too, so there is a
simpler way to reproduce than running precopy before postcopy in qemu.

Thanks,
Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Kirill A. Shutemov]

On Thu, Apr 14, 2016 at 12:22:30PM -0400, Andrea Arcangeli wrote:
> Adding linux-mm too,
> 
> On Thu, Apr 14, 2016 at 01:34:41PM +0100, Dr. David Alan Gilbert wrote:
> > * Andrea Arcangeli (aarcange@redhat.com) wrote:
> > 
> > > The next suspect is the massive THP refcounting change that went
> > > upstream recently:
> > 
> > > As further debug hint, can you try to disable THP and see if that
> > > makes the problem go away?
> > 
> > Yep, this seems to be the problem (cc'ing in Kirill).
> > 
> > 122afea9626ab3f717b250a8dd3d5ebf57cdb56c - works (just before Kirill disables THP)
> > 61f5d698cc97600e813ca5cf8e449b1ea1c11492 - breaks (when THP is reenabled)
> > 
> > It's pretty reliable; as you say disabling THP makes it work again
> > and putting it back to THP/madvise mode makes it break.  And you need
> > to test on a machine with some free ram to make sure THP has a chance
> > to have happened.
> > 
> > I'm not sure of all of the rework that happened in that series,
> > but my reading of it is that splitting of THP pages gets deferred;
> > so I wonder if when I do the madvise to turn THP off, if it's actually
> > still got THP pages and thus we end up with a whole THP mapped
> > when I'm expecting to be userfaulting those pages.
> 
> Good thing at least I didn't make UFFDIO_COPY THP aware yet so there's
> less variables (as no user was interested to handle userfaults at THP
> granularity yet, and from userland such an improvement would be
> completely invisible in terms of API, so if an user starts doing that
> we can just optimize the kernel for it, criu restore could do that as
> the faults will come from disk-I/O, when network is involved THP
> userfaults wouldn't have a great tradeoff with regard to the increased
> fault latency).
> 
> I suspect there is an handle_userfault missing somewhere in connection
> with trans_huge_pmd splits (not anymore THP splits) that you're doing
> with MADV_DONTNEED to zap those pages in the destination that got
> redirtied in source during the last precopy stage. Or more simply
> MADV_DONTNEED isn't zapping all the right ptes after the trans huge
> pmd got splitted.
> 
> The fact the page isn't splitted shouldn't matter too much, all we care
> about is the pte triggers handle_userfault after MADV_DONTNEED.
> 
> The userfaultfd testcase in the kernel isn't exercising this case
> unfortunately, that should probably be improved too, so there is a
> simpler way to reproduce than running precopy before postcopy in qemu.

I've tested current Linus' tree and v4.5 using qemu postcopy test case for
both x86-64 and i386 and it never failed for me:

/x86_64/postcopy: first_byte = 7e last_byte = 7d hit_edge = 1 OK
OK
/i386/postcopy: first_byte = f6 last_byte = f5 hit_edge = 1 OK
OK

I've run it directly, setting relevant QTEST_QEMU_BINARY.

-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Dr. David Alan Gilbert]

* Kirill A. Shutemov (kirill@shutemov.name) wrote:
> On Thu, Apr 14, 2016 at 12:22:30PM -0400, Andrea Arcangeli wrote:
> > Adding linux-mm too,
> > 
> > On Thu, Apr 14, 2016 at 01:34:41PM +0100, Dr. David Alan Gilbert wrote:
> > > * Andrea Arcangeli (aarcange@redhat.com) wrote:
> > > 
> > > > The next suspect is the massive THP refcounting change that went
> > > > upstream recently:
> > > 
> > > > As further debug hint, can you try to disable THP and see if that
> > > > makes the problem go away?
> > > 
> > > Yep, this seems to be the problem (cc'ing in Kirill).
> > > 
> > > 122afea9626ab3f717b250a8dd3d5ebf57cdb56c - works (just before Kirill disables THP)
> > > 61f5d698cc97600e813ca5cf8e449b1ea1c11492 - breaks (when THP is reenabled)
> > > 
> > > It's pretty reliable; as you say disabling THP makes it work again
> > > and putting it back to THP/madvise mode makes it break.  And you need
> > > to test on a machine with some free ram to make sure THP has a chance
> > > to have happened.
> > > 
> > > I'm not sure of all of the rework that happened in that series,
> > > but my reading of it is that splitting of THP pages gets deferred;
> > > so I wonder if when I do the madvise to turn THP off, if it's actually
> > > still got THP pages and thus we end up with a whole THP mapped
> > > when I'm expecting to be userfaulting those pages.
> > 
> > Good thing at least I didn't make UFFDIO_COPY THP aware yet so there's
> > less variables (as no user was interested to handle userfaults at THP
> > granularity yet, and from userland such an improvement would be
> > completely invisible in terms of API, so if an user starts doing that
> > we can just optimize the kernel for it, criu restore could do that as
> > the faults will come from disk-I/O, when network is involved THP
> > userfaults wouldn't have a great tradeoff with regard to the increased
> > fault latency).
> > 
> > I suspect there is an handle_userfault missing somewhere in connection
> > with trans_huge_pmd splits (not anymore THP splits) that you're doing
> > with MADV_DONTNEED to zap those pages in the destination that got
> > redirtied in source during the last precopy stage. Or more simply
> > MADV_DONTNEED isn't zapping all the right ptes after the trans huge
> > pmd got splitted.
> > 
> > The fact the page isn't splitted shouldn't matter too much, all we care
> > about is the pte triggers handle_userfault after MADV_DONTNEED.
> > 
> > The userfaultfd testcase in the kernel isn't exercising this case
> > unfortunately, that should probably be improved too, so there is a
> > simpler way to reproduce than running precopy before postcopy in qemu.
> 
> I've tested current Linus' tree and v4.5 using qemu postcopy test case for
> both x86-64 and i386 and it never failed for me:
> 
> /x86_64/postcopy: first_byte = 7e last_byte = 7d hit_edge = 1 OK
> OK
> /i386/postcopy: first_byte = f6 last_byte = f5 hit_edge = 1 OK
> OK
> 
> I've run it directly, setting relevant QTEST_QEMU_BINARY.

Interesting; it's failing reliably for me - but only with a reasonably
freshly booted machine (so that the pages get THPd).

Dave

> 
> -- 
>  Kirill A. Shutemov
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Kirill A. Shutemov]

On Fri, Apr 15, 2016 at 02:42:33PM +0100, Dr. David Alan Gilbert wrote:
> * Kirill A. Shutemov (kirill@shutemov.name) wrote:
> > On Thu, Apr 14, 2016 at 12:22:30PM -0400, Andrea Arcangeli wrote:
> > > Adding linux-mm too,
> > > 
> > > On Thu, Apr 14, 2016 at 01:34:41PM +0100, Dr. David Alan Gilbert wrote:
> > > > * Andrea Arcangeli (aarcange@redhat.com) wrote:
> > > > 
> > > > > The next suspect is the massive THP refcounting change that went
> > > > > upstream recently:
> > > > 
> > > > > As further debug hint, can you try to disable THP and see if that
> > > > > makes the problem go away?
> > > > 
> > > > Yep, this seems to be the problem (cc'ing in Kirill).
> > > > 
> > > > 122afea9626ab3f717b250a8dd3d5ebf57cdb56c - works (just before Kirill disables THP)
> > > > 61f5d698cc97600e813ca5cf8e449b1ea1c11492 - breaks (when THP is reenabled)
> > > > 
> > > > It's pretty reliable; as you say disabling THP makes it work again
> > > > and putting it back to THP/madvise mode makes it break.  And you need
> > > > to test on a machine with some free ram to make sure THP has a chance
> > > > to have happened.
> > > > 
> > > > I'm not sure of all of the rework that happened in that series,
> > > > but my reading of it is that splitting of THP pages gets deferred;
> > > > so I wonder if when I do the madvise to turn THP off, if it's actually
> > > > still got THP pages and thus we end up with a whole THP mapped
> > > > when I'm expecting to be userfaulting those pages.
> > > 
> > > Good thing at least I didn't make UFFDIO_COPY THP aware yet so there's
> > > less variables (as no user was interested to handle userfaults at THP
> > > granularity yet, and from userland such an improvement would be
> > > completely invisible in terms of API, so if an user starts doing that
> > > we can just optimize the kernel for it, criu restore could do that as
> > > the faults will come from disk-I/O, when network is involved THP
> > > userfaults wouldn't have a great tradeoff with regard to the increased
> > > fault latency).
> > > 
> > > I suspect there is an handle_userfault missing somewhere in connection
> > > with trans_huge_pmd splits (not anymore THP splits) that you're doing
> > > with MADV_DONTNEED to zap those pages in the destination that got
> > > redirtied in source during the last precopy stage. Or more simply
> > > MADV_DONTNEED isn't zapping all the right ptes after the trans huge
> > > pmd got splitted.
> > > 
> > > The fact the page isn't splitted shouldn't matter too much, all we care
> > > about is the pte triggers handle_userfault after MADV_DONTNEED.
> > > 
> > > The userfaultfd testcase in the kernel isn't exercising this case
> > > unfortunately, that should probably be improved too, so there is a
> > > simpler way to reproduce than running precopy before postcopy in qemu.
> > 
> > I've tested current Linus' tree and v4.5 using qemu postcopy test case for
> > both x86-64 and i386 and it never failed for me:
> > 
> > /x86_64/postcopy: first_byte = 7e last_byte = 7d hit_edge = 1 OK
> > OK
> > /i386/postcopy: first_byte = f6 last_byte = f5 hit_edge = 1 OK
> > OK
> > 
> > I've run it directly, setting relevant QTEST_QEMU_BINARY.
> 
> Interesting; it's failing reliably for me - but only with a reasonably
> freshly booted machine (so that the pages get THPd).

The same here. Freshly booted machine with 64GiB ram. I've checked
/proc/vmstat: huge pages were allocated

-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Dr. David Alan Gilbert]

* Kirill A. Shutemov (kirill@shutemov.name) wrote:
> On Fri, Apr 15, 2016 at 02:42:33PM +0100, Dr. David Alan Gilbert wrote:
> > * Kirill A. Shutemov (kirill@shutemov.name) wrote:
> > > On Thu, Apr 14, 2016 at 12:22:30PM -0400, Andrea Arcangeli wrote:
> > > > Adding linux-mm too,
> > > > 
> > > > On Thu, Apr 14, 2016 at 01:34:41PM +0100, Dr. David Alan Gilbert wrote:
> > > > > * Andrea Arcangeli (aarcange@redhat.com) wrote:
> > > > > 
> > > > > > The next suspect is the massive THP refcounting change that went
> > > > > > upstream recently:
> > > > > 
> > > > > > As further debug hint, can you try to disable THP and see if that
> > > > > > makes the problem go away?
> > > > > 
> > > > > Yep, this seems to be the problem (cc'ing in Kirill).
> > > > > 
> > > > > 122afea9626ab3f717b250a8dd3d5ebf57cdb56c - works (just before Kirill disables THP)
> > > > > 61f5d698cc97600e813ca5cf8e449b1ea1c11492 - breaks (when THP is reenabled)
> > > > > 
> > > > > It's pretty reliable; as you say disabling THP makes it work again
> > > > > and putting it back to THP/madvise mode makes it break.  And you need
> > > > > to test on a machine with some free ram to make sure THP has a chance
> > > > > to have happened.
> > > > > 
> > > > > I'm not sure of all of the rework that happened in that series,
> > > > > but my reading of it is that splitting of THP pages gets deferred;
> > > > > so I wonder if when I do the madvise to turn THP off, if it's actually
> > > > > still got THP pages and thus we end up with a whole THP mapped
> > > > > when I'm expecting to be userfaulting those pages.
> > > > 
> > > > Good thing at least I didn't make UFFDIO_COPY THP aware yet so there's
> > > > less variables (as no user was interested to handle userfaults at THP
> > > > granularity yet, and from userland such an improvement would be
> > > > completely invisible in terms of API, so if an user starts doing that
> > > > we can just optimize the kernel for it, criu restore could do that as
> > > > the faults will come from disk-I/O, when network is involved THP
> > > > userfaults wouldn't have a great tradeoff with regard to the increased
> > > > fault latency).
> > > > 
> > > > I suspect there is an handle_userfault missing somewhere in connection
> > > > with trans_huge_pmd splits (not anymore THP splits) that you're doing
> > > > with MADV_DONTNEED to zap those pages in the destination that got
> > > > redirtied in source during the last precopy stage. Or more simply
> > > > MADV_DONTNEED isn't zapping all the right ptes after the trans huge
> > > > pmd got splitted.
> > > > 
> > > > The fact the page isn't splitted shouldn't matter too much, all we care
> > > > about is the pte triggers handle_userfault after MADV_DONTNEED.
> > > > 
> > > > The userfaultfd testcase in the kernel isn't exercising this case
> > > > unfortunately, that should probably be improved too, so there is a
> > > > simpler way to reproduce than running precopy before postcopy in qemu.
> > > 
> > > I've tested current Linus' tree and v4.5 using qemu postcopy test case for
> > > both x86-64 and i386 and it never failed for me:
> > > 
> > > /x86_64/postcopy: first_byte = 7e last_byte = 7d hit_edge = 1 OK
> > > OK
> > > /i386/postcopy: first_byte = f6 last_byte = f5 hit_edge = 1 OK
> > > OK
> > > 
> > > I've run it directly, setting relevant QTEST_QEMU_BINARY.
> > 
> > Interesting; it's failing reliably for me - but only with a reasonably
> > freshly booted machine (so that the pages get THPd).
> 
> The same here. Freshly booted machine with 64GiB ram. I've checked
> /proc/vmstat: huge pages were allocated

Thanks for testing.

Damn; this is confusing now.  I've got a RHEL7 box with 4.6.0-rc3 on where it
works, and a fedora24 VM where it fails (the f24 VM is where I did the bisect
so it works fine with the older kernel on the f24 userspace in that VM).

So lets see:
   works: Kirill's (64GB machine)
          Dave's RHEL7 host (24GB RAM, dual xeon, RHEL7 userspace and kernel config)
   fails: Dave's f24 VM (4GB RAM, 4 vcpus VM on my laptop24 userspace and kernel config)

So it's any of userspace, kernel config, machine hardware or hmm.

My f24 box has transparent_hugepage_madvise, where my rhel7 has transparent_hugepage_always
(but still works if I flip it to madvise at run time).  I'll try and get the configs
closer together.

Liang Li: Can you run my test on your setup which fails the migrate and tell
me what your userspace is?

(If you've not built my test yet, you might find you need to add a :
   tests/postcopy-test$(EXESUF): tests/postcopy-test.o

  to the tests/Makefile)


Dave
> 
> -- 
>  Kirill A. Shutemov
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: post-copy is broken?

[Li, Liang Z]

> > > > I've run it directly, setting relevant QTEST_QEMU_BINARY.
> > >
> > > Interesting; it's failing reliably for me - but only with a
> > > reasonably freshly booted machine (so that the pages get THPd).
> >
> > The same here. Freshly booted machine with 64GiB ram. I've checked
> > /proc/vmstat: huge pages were allocated
> 
> Thanks for testing.
> 
> Damn; this is confusing now.  I've got a RHEL7 box with 4.6.0-rc3 on where it
> works, and a fedora24 VM where it fails (the f24 VM is where I did the bisect
> so it works fine with the older kernel on the f24 userspace in that VM).
> 
> So lets see:
>    works: Kirill's (64GB machine)
>           Dave's RHEL7 host (24GB RAM, dual xeon, RHEL7 userspace and kernel
> config)
>    fails: Dave's f24 VM (4GB RAM, 4 vcpus VM on my laptop24 userspace and
> kernel config)
> 
> So it's any of userspace, kernel config, machine hardware or hmm.
> 
> My f24 box has transparent_hugepage_madvise, where my rhel7 has
> transparent_hugepage_always (but still works if I flip it to madvise at run
> time).  I'll try and get the configs closer together.
> 
> Liang Li: Can you run my test on your setup which fails the migrate and tell
> me what your userspace is?
> 
> (If you've not built my test yet, you might find you need to add a :
>    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> 
>   to the tests/Makefile)
> 

Hi Dave,

  How to build and run you test? I didn't do that before.

Thanks!
Liang

> 
> Dave
> >
> > --
> >  Kirill A. Shutemov
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Dr. David Alan Gilbert]

* Li, Liang Z (liang.z.li@intel.com) wrote:
> > > > > I've run it directly, setting relevant QTEST_QEMU_BINARY.
> > > >
> > > > Interesting; it's failing reliably for me - but only with a
> > > > reasonably freshly booted machine (so that the pages get THPd).
> > >
> > > The same here. Freshly booted machine with 64GiB ram. I've checked
> > > /proc/vmstat: huge pages were allocated
> > 
> > Thanks for testing.
> > 
> > Damn; this is confusing now.  I've got a RHEL7 box with 4.6.0-rc3 on where it
> > works, and a fedora24 VM where it fails (the f24 VM is where I did the bisect
> > so it works fine with the older kernel on the f24 userspace in that VM).
> > 
> > So lets see:
> >    works: Kirill's (64GB machine)
> >           Dave's RHEL7 host (24GB RAM, dual xeon, RHEL7 userspace and kernel
> > config)
> >    fails: Dave's f24 VM (4GB RAM, 4 vcpus VM on my laptop24 userspace and
> > kernel config)
> > 
> > So it's any of userspace, kernel config, machine hardware or hmm.
> > 
> > My f24 box has transparent_hugepage_madvise, where my rhel7 has
> > transparent_hugepage_always (but still works if I flip it to madvise at run
> > time).  I'll try and get the configs closer together.
> > 
> > Liang Li: Can you run my test on your setup which fails the migrate and tell
> > me what your userspace is?
> > 
> > (If you've not built my test yet, you might find you need to add a :
> >    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> > 
> >   to the tests/Makefile)
> > 
> 
> Hi Dave,
> 
>   How to build and run you test? I didn't do that before.

Apply the code in:
http://lists.gnu.org/archive/html/qemu-devel/2016-04/msg02138.html

fix the:
+            if ( ((b + 1) % 255) == last_byte && !hit_edge) {
to:
+            if ( ((b + 1) % 256) == last_byte && !hit_edge) {

to tests/Makefile
   tests/postcopy-test$(EXESUF): tests/postcopy-test.o

and do a:
    make check

in qemu.
Then you can rerun the test with:
    QTEST_QEMU_BINARY=path/to/qemu-system-x86_64 ./tests/postcopy-test

if it works, reboot and check it still works from a fresh boot.

Can you describe the system which your full test failed on? What distro on
the host? What type of host was it tested on?

Dave

> 
> Thanks!
> Liang
> 
> > 
> > Dave
> > >
> > > --
> > >  Kirill A. Shutemov
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: post-copy is broken?

[Li, Liang Z]

> * Li, Liang Z (liang.z.li@intel.com) wrote:
> > > > > > I've run it directly, setting relevant QTEST_QEMU_BINARY.
> > > > >
> > > > > Interesting; it's failing reliably for me - but only with a
> > > > > reasonably freshly booted machine (so that the pages get THPd).
> > > >
> > > > The same here. Freshly booted machine with 64GiB ram. I've checked
> > > > /proc/vmstat: huge pages were allocated
> > >
> > > Thanks for testing.
> > >
> > > Damn; this is confusing now.  I've got a RHEL7 box with 4.6.0-rc3 on
> > > where it works, and a fedora24 VM where it fails (the f24 VM is
> > > where I did the bisect so it works fine with the older kernel on the f24
> userspace in that VM).
> > >
> > > So lets see:
> > >    works: Kirill's (64GB machine)
> > >           Dave's RHEL7 host (24GB RAM, dual xeon, RHEL7 userspace
> > > and kernel
> > > config)
> > >    fails: Dave's f24 VM (4GB RAM, 4 vcpus VM on my laptop24
> > > userspace and kernel config)
> > >
> > > So it's any of userspace, kernel config, machine hardware or hmm.
> > >
> > > My f24 box has transparent_hugepage_madvise, where my rhel7 has
> > > transparent_hugepage_always (but still works if I flip it to madvise
> > > at run time).  I'll try and get the configs closer together.
> > >
> > > Liang Li: Can you run my test on your setup which fails the migrate
> > > and tell me what your userspace is?
> > >
> > > (If you've not built my test yet, you might find you need to add a :
> > >    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> > >
> > >   to the tests/Makefile)
> > >
> >
> > Hi Dave,
> >
> >   How to build and run you test? I didn't do that before.
> 
> Apply the code in:
> http://lists.gnu.org/archive/html/qemu-devel/2016-04/msg02138.html
> 
> fix the:
> +            if ( ((b + 1) % 255) == last_byte && !hit_edge) {
> to:
> +            if ( ((b + 1) % 256) == last_byte && !hit_edge) {
> 
> to tests/Makefile
>    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> 
> and do a:
>     make check
> 
> in qemu.
> Then you can rerun the test with:
>     QTEST_QEMU_BINARY=path/to/qemu-system-x86_64 ./tests/postcopy-
> test
> 
> if it works, reboot and check it still works from a fresh boot.
> 
> Can you describe the system which your full test failed on? What distro on
> the host? What type of host was it tested on?
> 
> Dave
> 


Thanks, Dave

The host is CenOS7, its original kernel is 3.10.0-327.el7.x86_64 (CentOS 7.1?),
The hardware platform is HSW-EP with 64GB RAM.


> >
> > Thanks!
> > Liang
> >
> > >
> > > Dave
> > > >
> > > > --
> > > >  Kirill A. Shutemov
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Dr. David Alan Gilbert]

* Li, Liang Z (liang.z.li@intel.com) wrote:
> > * Li, Liang Z (liang.z.li@intel.com) wrote:
> > > > > > > I've run it directly, setting relevant QTEST_QEMU_BINARY.
> > > > > >
> > > > > > Interesting; it's failing reliably for me - but only with a
> > > > > > reasonably freshly booted machine (so that the pages get THPd).
> > > > >
> > > > > The same here. Freshly booted machine with 64GiB ram. I've checked
> > > > > /proc/vmstat: huge pages were allocated
> > > >
> > > > Thanks for testing.
> > > >
> > > > Damn; this is confusing now.  I've got a RHEL7 box with 4.6.0-rc3 on
> > > > where it works, and a fedora24 VM where it fails (the f24 VM is
> > > > where I did the bisect so it works fine with the older kernel on the f24
> > userspace in that VM).
> > > >
> > > > So lets see:
> > > >    works: Kirill's (64GB machine)
> > > >           Dave's RHEL7 host (24GB RAM, dual xeon, RHEL7 userspace
> > > > and kernel
> > > > config)
> > > >    fails: Dave's f24 VM (4GB RAM, 4 vcpus VM on my laptop24
> > > > userspace and kernel config)
> > > >
> > > > So it's any of userspace, kernel config, machine hardware or hmm.
> > > >
> > > > My f24 box has transparent_hugepage_madvise, where my rhel7 has
> > > > transparent_hugepage_always (but still works if I flip it to madvise
> > > > at run time).  I'll try and get the configs closer together.
> > > >
> > > > Liang Li: Can you run my test on your setup which fails the migrate
> > > > and tell me what your userspace is?
> > > >
> > > > (If you've not built my test yet, you might find you need to add a :
> > > >    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> > > >
> > > >   to the tests/Makefile)
> > > >
> > >
> > > Hi Dave,
> > >
> > >   How to build and run you test? I didn't do that before.
> > 
> > Apply the code in:
> > http://lists.gnu.org/archive/html/qemu-devel/2016-04/msg02138.html
> > 
> > fix the:
> > +            if ( ((b + 1) % 255) == last_byte && !hit_edge) {
> > to:
> > +            if ( ((b + 1) % 256) == last_byte && !hit_edge) {
> > 
> > to tests/Makefile
> >    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> > 
> > and do a:
> >     make check
> > 
> > in qemu.
> > Then you can rerun the test with:
> >     QTEST_QEMU_BINARY=path/to/qemu-system-x86_64 ./tests/postcopy-
> > test
> > 
> > if it works, reboot and check it still works from a fresh boot.
> > 
> > Can you describe the system which your full test failed on? What distro on
> > the host? What type of host was it tested on?
> > 
> > Dave
> > 
> 
> 
> Thanks, Dave
> 
> The host is CenOS7, its original kernel is 3.10.0-327.el7.x86_64 (CentOS 7.1?),
> The hardware platform is HSW-EP with 64GB RAM.

OK, so your test fails on real hardware; my guess is that my test will work
on there.
Can you try your test with THP disabled on the host:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Dave

> 
> 
> > >
> > > Thanks!
> > > Liang
> > >
> > > >
> > > > Dave
> > > > >
> > > > > --
> > > > >  Kirill A. Shutemov
> > > > --
> > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: post-copy is broken?

[Li, Liang Z]

> > > > > > > Interesting; it's failing reliably for me - but only with a
> > > > > > > reasonably freshly booted machine (so that the pages get THPd).
> > > > > >
> > > > > > The same here. Freshly booted machine with 64GiB ram. I've
> > > > > > checked
> > > > > > /proc/vmstat: huge pages were allocated
> > > > >
> > > > > Thanks for testing.
> > > > >
> > > > > Damn; this is confusing now.  I've got a RHEL7 box with
> > > > > 4.6.0-rc3 on where it works, and a fedora24 VM where it fails
> > > > > (the f24 VM is where I did the bisect so it works fine with the
> > > > > older kernel on the f24
> > > userspace in that VM).
> > > > >
> > > > > So lets see:
> > > > >    works: Kirill's (64GB machine)
> > > > >           Dave's RHEL7 host (24GB RAM, dual xeon, RHEL7
> > > > > userspace and kernel
> > > > > config)
> > > > >    fails: Dave's f24 VM (4GB RAM, 4 vcpus VM on my laptop24
> > > > > userspace and kernel config)
> > > > >
> > > > > So it's any of userspace, kernel config, machine hardware or hmm.
> > > > >
> > > > > My f24 box has transparent_hugepage_madvise, where my rhel7 has
> > > > > transparent_hugepage_always (but still works if I flip it to
> > > > > madvise at run time).  I'll try and get the configs closer together.
> > > > >
> > > > > Liang Li: Can you run my test on your setup which fails the
> > > > > migrate and tell me what your userspace is?
> > > > >
> > > > > (If you've not built my test yet, you might find you need to add a :
> > > > >    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> > > > >
> > > > >   to the tests/Makefile)
> > > > >
> > > >
> > > > Hi Dave,
> > > >
> > > >   How to build and run you test? I didn't do that before.
> > >
> > > Apply the code in:
> > > http://lists.gnu.org/archive/html/qemu-devel/2016-04/msg02138.html
> > >
> > > fix the:
> > > +            if ( ((b + 1) % 255) == last_byte && !hit_edge) {
> > > to:
> > > +            if ( ((b + 1) % 256) == last_byte && !hit_edge) {
> > >
> > > to tests/Makefile
> > >    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> > >
> > > and do a:
> > >     make check
> > >
> > > in qemu.
> > > Then you can rerun the test with:
> > >     QTEST_QEMU_BINARY=path/to/qemu-system-
> x86_64 ./tests/postcopy-
> > > test
> > >
> > > if it works, reboot and check it still works from a fresh boot.
> > >
> > > Can you describe the system which your full test failed on? What
> > > distro on the host? What type of host was it tested on?
> > >
> > > Dave
> > >
> >
> >
> > Thanks, Dave
> >
> > The host is CenOS7, its original kernel is 3.10.0-327.el7.x86_64
> > (CentOS 7.1?), The hardware platform is HSW-EP with 64GB RAM.
> 
> OK, so your test fails on real hardware; my guess is that my test will work on
> there.
> Can you try your test with THP disabled on the host:
> 
> echo never > /sys/kernel/mm/transparent_hugepage/enabled
> 

If the THP is disabled, no fails.
And your test was always passed, even when  real post-copy was failed. 

In my env, the output of 
'cat /sys/kernel/mm/transparent_hugepage/enabled'  is:

 [always] ...

Liang

> Dave
> 
> >
> >
> > > >
> > > > Thanks!
> > > > Liang
> > > >
> > > > >
> > > > > Dave
> > > > > >
> > > > > > --
> > > > > >  Kirill A. Shutemov
> > > > > --
> > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Dr. David Alan Gilbert]

* Li, Liang Z (liang.z.li@intel.com) wrote:
> > > > > > > > Interesting; it's failing reliably for me - but only with a
> > > > > > > > reasonably freshly booted machine (so that the pages get THPd).
> > > > > > >
> > > > > > > The same here. Freshly booted machine with 64GiB ram. I've
> > > > > > > checked
> > > > > > > /proc/vmstat: huge pages were allocated
> > > > > >
> > > > > > Thanks for testing.
> > > > > >
> > > > > > Damn; this is confusing now.  I've got a RHEL7 box with
> > > > > > 4.6.0-rc3 on where it works, and a fedora24 VM where it fails
> > > > > > (the f24 VM is where I did the bisect so it works fine with the
> > > > > > older kernel on the f24
> > > > userspace in that VM).
> > > > > >
> > > > > > So lets see:
> > > > > >    works: Kirill's (64GB machine)
> > > > > >           Dave's RHEL7 host (24GB RAM, dual xeon, RHEL7
> > > > > > userspace and kernel
> > > > > > config)
> > > > > >    fails: Dave's f24 VM (4GB RAM, 4 vcpus VM on my laptop24
> > > > > > userspace and kernel config)
> > > > > >
> > > > > > So it's any of userspace, kernel config, machine hardware or hmm.
> > > > > >
> > > > > > My f24 box has transparent_hugepage_madvise, where my rhel7 has
> > > > > > transparent_hugepage_always (but still works if I flip it to
> > > > > > madvise at run time).  I'll try and get the configs closer together.
> > > > > >
> > > > > > Liang Li: Can you run my test on your setup which fails the
> > > > > > migrate and tell me what your userspace is?
> > > > > >
> > > > > > (If you've not built my test yet, you might find you need to add a :
> > > > > >    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> > > > > >
> > > > > >   to the tests/Makefile)
> > > > > >
> > > > >
> > > > > Hi Dave,
> > > > >
> > > > >   How to build and run you test? I didn't do that before.
> > > >
> > > > Apply the code in:
> > > > http://lists.gnu.org/archive/html/qemu-devel/2016-04/msg02138.html
> > > >
> > > > fix the:
> > > > +            if ( ((b + 1) % 255) == last_byte && !hit_edge) {
> > > > to:
> > > > +            if ( ((b + 1) % 256) == last_byte && !hit_edge) {
> > > >
> > > > to tests/Makefile
> > > >    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> > > >
> > > > and do a:
> > > >     make check
> > > >
> > > > in qemu.
> > > > Then you can rerun the test with:
> > > >     QTEST_QEMU_BINARY=path/to/qemu-system-
> > x86_64 ./tests/postcopy-
> > > > test
> > > >
> > > > if it works, reboot and check it still works from a fresh boot.
> > > >
> > > > Can you describe the system which your full test failed on? What
> > > > distro on the host? What type of host was it tested on?
> > > >
> > > > Dave
> > > >
> > >
> > >
> > > Thanks, Dave
> > >
> > > The host is CenOS7, its original kernel is 3.10.0-327.el7.x86_64
> > > (CentOS 7.1?), The hardware platform is HSW-EP with 64GB RAM.
> > 
> > OK, so your test fails on real hardware; my guess is that my test will work on
> > there.
> > Can you try your test with THP disabled on the host:
> > 
> > echo never > /sys/kernel/mm/transparent_hugepage/enabled
> > 
> 
> If the THP is disabled, no fails.
> And your test was always passed, even when  real post-copy was failed. 
> 
> In my env, the output of 
> 'cat /sys/kernel/mm/transparent_hugepage/enabled'  is:
> 
>  [always] ...

OK, I can't get my test to fail on real hardware - only in a VM; but my
suspicion is we're looking at the same bug; both of them it goes away
if we disable THP, both of them work on 4.4.x and fail on 4.5.x.
I'd love to be able to find a nice easy test to be able to give to Andrea
and Kirill

I've also just confirmed that running (in a VM) a fedora-24 4.5.0 kernel
with a fedora-23 userspace (qemu built under f23) still fails with my test.
So the problem there is definitely triggered by the newer kernel not
the newer userspace.

Dave

> 
> Liang
> 
> > Dave
> > 
> > >
> > >
> > > > >
> > > > > Thanks!
> > > > > Liang
> > > > >
> > > > > >
> > > > > > Dave
> > > > > > >
> > > > > > > --
> > > > > > >  Kirill A. Shutemov
> > > > > > --
> > > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > > > --
> > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Dr. David Alan Gilbert]

* Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> * Li, Liang Z (liang.z.li@intel.com) wrote:
> > > > > > > > > Interesting; it's failing reliably for me - but only with a
> > > > > > > > > reasonably freshly booted machine (so that the pages get THPd).
> > > > > > > >
> > > > > > > > The same here. Freshly booted machine with 64GiB ram. I've
> > > > > > > > checked
> > > > > > > > /proc/vmstat: huge pages were allocated
> > > > > > >
> > > > > > > Thanks for testing.
> > > > > > >
> > > > > > > Damn; this is confusing now.  I've got a RHEL7 box with
> > > > > > > 4.6.0-rc3 on where it works, and a fedora24 VM where it fails
> > > > > > > (the f24 VM is where I did the bisect so it works fine with the
> > > > > > > older kernel on the f24
> > > > > userspace in that VM).
> > > > > > >
> > > > > > > So lets see:
> > > > > > >    works: Kirill's (64GB machine)
> > > > > > >           Dave's RHEL7 host (24GB RAM, dual xeon, RHEL7
> > > > > > > userspace and kernel
> > > > > > > config)
> > > > > > >    fails: Dave's f24 VM (4GB RAM, 4 vcpus VM on my laptop24
> > > > > > > userspace and kernel config)
> > > > > > >
> > > > > > > So it's any of userspace, kernel config, machine hardware or hmm.
> > > > > > >
> > > > > > > My f24 box has transparent_hugepage_madvise, where my rhel7 has
> > > > > > > transparent_hugepage_always (but still works if I flip it to
> > > > > > > madvise at run time).  I'll try and get the configs closer together.
> > > > > > >
> > > > > > > Liang Li: Can you run my test on your setup which fails the
> > > > > > > migrate and tell me what your userspace is?
> > > > > > >
> > > > > > > (If you've not built my test yet, you might find you need to add a :
> > > > > > >    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> > > > > > >
> > > > > > >   to the tests/Makefile)
> > > > > > >
> > > > > >
> > > > > > Hi Dave,
> > > > > >
> > > > > >   How to build and run you test? I didn't do that before.
> > > > >
> > > > > Apply the code in:
> > > > > http://lists.gnu.org/archive/html/qemu-devel/2016-04/msg02138.html
> > > > >
> > > > > fix the:
> > > > > +            if ( ((b + 1) % 255) == last_byte && !hit_edge) {
> > > > > to:
> > > > > +            if ( ((b + 1) % 256) == last_byte && !hit_edge) {
> > > > >
> > > > > to tests/Makefile
> > > > >    tests/postcopy-test$(EXESUF): tests/postcopy-test.o
> > > > >
> > > > > and do a:
> > > > >     make check
> > > > >
> > > > > in qemu.
> > > > > Then you can rerun the test with:
> > > > >     QTEST_QEMU_BINARY=path/to/qemu-system-
> > > x86_64 ./tests/postcopy-
> > > > > test
> > > > >
> > > > > if it works, reboot and check it still works from a fresh boot.
> > > > >
> > > > > Can you describe the system which your full test failed on? What
> > > > > distro on the host? What type of host was it tested on?
> > > > >
> > > > > Dave
> > > > >
> > > >
> > > >
> > > > Thanks, Dave
> > > >
> > > > The host is CenOS7, its original kernel is 3.10.0-327.el7.x86_64
> > > > (CentOS 7.1?), The hardware platform is HSW-EP with 64GB RAM.
> > > 
> > > OK, so your test fails on real hardware; my guess is that my test will work on
> > > there.
> > > Can you try your test with THP disabled on the host:
> > > 
> > > echo never > /sys/kernel/mm/transparent_hugepage/enabled
> > > 
> > 
> > If the THP is disabled, no fails.
> > And your test was always passed, even when  real post-copy was failed. 
> > 
> > In my env, the output of 
> > 'cat /sys/kernel/mm/transparent_hugepage/enabled'  is:
> > 
> >  [always] ...
> 
> OK, I can't get my test to fail on real hardware - only in a VM; but my
> suspicion is we're looking at the same bug; both of them it goes away
> if we disable THP, both of them work on 4.4.x and fail on 4.5.x.
> I'd love to be able to find a nice easy test to be able to give to Andrea
> and Kirill
> 
> I've also just confirmed that running (in a VM) a fedora-24 4.5.0 kernel
> with a fedora-23 userspace (qemu built under f23) still fails with my test.
> So the problem there is definitely triggered by the newer kernel not
> the newer userspace.

OK, some more results - I *can* get it to fail on real hardware - it's just
really really rare, and the failure is slightly different than in the nest.

I'm using the following magic:
count=0; while true; do count=$(($count+1)); echo 3 >/proc/sys/vm/drop_caches; echo >/proc/sys/vm/compact_memory; echo "Iteration $count"; QTEST_QEMU_BINARY=./bin/qemu-system-x86_64 ./tests/postcopy-test || break; done

I've had about 4 failures out of about 5000 runs (ouch);

On the real hardware the failure addresses are always 2MB aligned, even though
other than the start address, everything in the test is 4K page based - so again
this is pointing the finger at THP:

/x86_64/postcopy: Memory content inconsistency at 4200000 first_byte = 48 last_byte = 47 current = 1 hit_edge = 1
postcopy-test: /root/git/qemu/tests/postcopy-test.c:274: check_guests_ram: Assertion `0' failed.
/x86_64/postcopy: Memory content inconsistency at 4200000 first_byte = e last_byte = d current = 9b hit_edge = 1
postcopy-test: /root/git/qemu/tests/postcopy-test.c:274: check_guests_ram: Assertion `0' failed.
/x86_64/postcopy: Memory content inconsistency at 4800000 first_byte = 19 last_byte = 18 current = 1 hit_edge = 1
postcopy-test: /root/git/qemu/tests/postcopy-test.c:274: check_guests_ram: Assertion `0' failed.
/x86_64/postcopy: Memory content inconsistency at 5e00000 first_byte = d6 last_byte = d5 current = 1 hit_edge = 1
postcopy-test: /root/git/qemu/tests/postcopy-test.c:274: check_guests_ram: Assertion `0' failed.

(My test host for the real hardware is 2x E5-2640 v3 running fedora 24)

where as in the VM I'm seeing immediate failures with addresses just on any 4k alignment.

You can run a couple in parallel; but if your load is too high the test will fail with an
assertion (postcopy-test.c:196 ...(qdict_haskey(rsp, "return")) - but that's
my test - so don't worry if you hit that; decreasing the migrate_speed_set value should
avoid that if you're hitting it repeatedly.

(Could this be something like a missing TLB flush?)

Dave


> 
> Dave
> 
> > 
> > Liang
> > 
> > > Dave
> > > 
> > > >
> > > >
> > > > > >
> > > > > > Thanks!
> > > > > > Liang
> > > > > >
> > > > > > >
> > > > > > > Dave
> > > > > > > >
> > > > > > > > --
> > > > > > > >  Kirill A. Shutemov
> > > > > > > --
> > > > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > > > > --
> > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Dr. David Alan Gilbert]

Hi,
  Just a follow up with a little more debug;

I modified the test so it doesn't quit after the first miscomparison (see
diff below), and looking on the failures on real hardware I've seen:

/x86_64/postcopy: Memory content inconsistency at 3800000 first_byte = 30 last_byte = 30 current = 10 hit_edge = 0
                  Memory content inconsistency at 38fe000 first_byte = 30 last_byte = 10 current = 30 hit_edge = 0

and then another time:
/x86_64/postcopy: Memory content inconsistency at 4c00000 first_byte = 9a last_byte = 99 current = 1 hit_edge = 1
                  Memory content inconsistency at 4cec000 first_byte = 9a last_byte = 1 current = 99 hit_edge = 1

so in both cases what we're seeing there is starting on a 2M page boundary, a page
that is read on the destination as zero instead of getting the migrated value -
but somewhere later in the page it starts behaving. (in the first example the counter
had reached 0x30 - except for those pages which hadn't been transferred where
the counter is much lower at 0x10).

Testing it in my VM, I added some debug for where I'd been doing an madvise DONTNEED
previously:

ram_discard_range: pc.ram:0xf51000 for 42094592
ram_discard_range: pc.ram:0x5259000 for 18509824
Memory content inconsistency at f51000 first_byte = 6d last_byte = 6d current = 9e hit_edge = 0
Memory content inconsistency at 1000000 first_byte = 6d last_byte = 9e current = 6d hit_edge = 0

   So that's saying that from f51000..1000000 it was wrong - so not just one page, but upto the THP edge.
(It then got back to the right value - 6d - on the page edge).  Note how the start corresponds
to the address I'd previously done a discard on, but not the whole discard range - just
upto the THP page boundary.  Nothing in my userspace code knows about THP
(other than turning it off).

Dave



@@ -251,6 +251,7 @@ static void check_guests_ram(void)
     uint8_t first_byte;
     uint8_t last_byte;
     bool hit_edge = false;
+    bool bad = false;
 
     qtest_memread(global_qtest, start_address, &first_byte, 1);
     last_byte = first_byte;
@@ -271,11 +272,12 @@ static void check_guests_ram(void)
                                 " first_byte = %x last_byte = %x current = %x"
                                 " hit_edge = %x\n",
                                 address, first_byte, last_byte, b, hit_edge);
-                assert(0);
+                bad = true;
             }
         }
         last_byte = b;
     }
+    assert(!bad);
     fprintf(stderr, "first_byte = %x last_byte = %x hit_edge = %x OK\n",
                     first_byte, last_byte, hit_edge);
 }

--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Dr. David Alan Gilbert]

Hi Andrea,

I'm wondering if this bug is the opposite way around from what I originally
thought it was - I don't think the problem is 0 pages on the destination; I think
it's more subtle.

I added some debug to print the source VMs memory and also
the byte in the destination's 1st page (this is in the nest):

nhp_range: block: pc.ram @ 0x7fc59a800000
Destination 1st byte: e8,df <stop> df

   OK, so that tells us that the destination is running OK, and that it
stops running when we tell it to.

Memory content inconsistency at f79000 first_byte = df last_byte = de current = 9 hit_edge = 1 src_byte = 9

'src_byte' is saying that the source VM had the byte 9 in that page (we've still got the source VMs memory - it's
paused at this point in the test)
  so adding the start of pc.ram we get that being a host address of 0x7FC59B779000 and in the logs I see:
postcopy_place_page: 0x55ba64503f7d->0x7fc59b779000 copy=4096 1stbyte=9/9

  OK, so that shows that when the destination received the page it was also '9' and after the uffdio_copy
it read as 9 - so the page made it into RAM; it wasn't 0.

But that also means, that page hasn't changed *after* migration; why not?

We can see that the other pages are changing (that Destination 1st byte
line shows the 1st byte of the test memory changed) - so the incrementer
loop has apparently incremented every byte of the test memory multiple
times - except these pages are still stuck at the '9' it got when we
placed the page into it atomically.

I've been unable to trigger this bug in a standalone test case that ran
without kvm.

Is it possible that the guest KVM CPU isn't noticing some change to
the mapping?

Dave


--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Andrea Arcangeli]

Hello Liang,

On Mon, Apr 18, 2016 at 10:33:14AM +0000, Li, Liang Z wrote:
> If the THP is disabled, no fails.
> And your test was always passed, even when  real post-copy was failed. 
> 
> In my env, the output of 
> 'cat /sys/kernel/mm/transparent_hugepage/enabled'  is:
> 
>  [always] ...
> 

Can you test the fix?
https://marc.info/?l=linux-mm&m=146175869123580&w=2

This was not a breakage in userfaultfd nor in postcopy. userfaultfd
had no bugs and is fully rock solid and with zero chances of
generating undetected memory corruption like it was happening in v4.5.

As I suspected, the same problem would have happened with any THP
pmd_trans_huge split (swapping/inflating-balloon etc..). Postcopy just
makes it easier to reproduce the problem because it does a scattered
MADV_DONTNEED on the destination qemu guest memory for the pages
redirtied during the last precopy pass that run, or not transferred
(to allow THP faults in destination qemu during precopy), just before
starting the guest in the destination node.

Other reports of KVM memory corruption happening on v4.5 with THP
enabled will also be taken care of by the above fix.

I hope I managed to fix this in time for v4.6 final (current is
v4.6-rc5-69), so the only kernel where KVM must not be used with THP
enabled will be v4.5.

On a side note, this MADV_DONTEED trigger reminded me as soon as the
madvisev syscall is merged, loadvm_postcopy_ram_handle_discard should
start using it to reduce the enter/exit kernel to just 1 (or a few
madvisev in case we want to give a limit to the temporary buffer to
avoid the risk of allocating too much temporary RAM for very large
guests) to do the MADV_DONTNEED scattered zapping. Same thing in
virtio_balloon_handle_output.

Thanks,
Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: post-copy is broken?

[Li, Liang Z]

> -----Original Message-----
> From: Andrea Arcangeli [mailto:aarcange@redhat.com]
> Sent: Wednesday, April 27, 2016 10:48 PM
> To: Li, Liang Z
> Cc: Dr. David Alan Gilbert; Kirill A. Shutemov; kirill.shutemov@linux.intel.com;
> Amit Shah; qemu-devel@nongnu.org; quintela@redhat.com; linux-
> mm@kvack.org
> Subject: Re: post-copy is broken?
> 
> Hello Liang,
> 
> On Mon, Apr 18, 2016 at 10:33:14AM +0000, Li, Liang Z wrote:
> > If the THP is disabled, no fails.
> > And your test was always passed, even when  real post-copy was failed.
> >
> > In my env, the output of
> > 'cat /sys/kernel/mm/transparent_hugepage/enabled'  is:
> >
> >  [always] ...
> >
> 
> Can you test the fix?
> https://marc.info/?l=linux-mm&m=146175869123580&w=2
> 
> This was not a breakage in userfaultfd nor in postcopy. userfaultfd had no
> bugs and is fully rock solid and with zero chances of generating undetected
> memory corruption like it was happening in v4.5.
> 
> As I suspected, the same problem would have happened with any THP
> pmd_trans_huge split (swapping/inflating-balloon etc..). Postcopy just
> makes it easier to reproduce the problem because it does a scattered
> MADV_DONTNEED on the destination qemu guest memory for the pages
> redirtied during the last precopy pass that run, or not transferred (to allow
> THP faults in destination qemu during precopy), just before starting the
> guest in the destination node.
> 
> Other reports of KVM memory corruption happening on v4.5 with THP
> enabled will also be taken care of by the above fix.
> 
> I hope I managed to fix this in time for v4.6 final (current is v4.6-rc5-69), so
> the only kernel where KVM must not be used with THP enabled will be v4.5.
> 
> On a side note, this MADV_DONTEED trigger reminded me as soon as the
> madvisev syscall is merged, loadvm_postcopy_ram_handle_discard should
> start using it to reduce the enter/exit kernel to just 1 (or a few madvisev in
> case we want to give a limit to the temporary buffer to avoid the risk of
> allocating too much temporary RAM for very large
> guests) to do the MADV_DONTNEED scattered zapping. Same thing in
> virtio_balloon_handle_output.
> 

I have test the patch, guest doesn't crash anymore, I think the issue is fixed. Thanks!

Liang
> Thanks,
> Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Dr. David Alan Gilbert]

* Li, Liang Z (liang.z.li@intel.com) wrote:

> 
> I have test the patch, guest doesn't crash anymore, I think the issue is fixed. Thanks!

Great!  Thanks for reporting it.

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Andrea Arcangeli]

On Fri, Apr 15, 2016 at 06:23:30PM +0300, Kirill A. Shutemov wrote:
> The same here. Freshly booted machine with 64GiB ram. I've checked
> /proc/vmstat: huge pages were allocated

I tried the test in a loop and I can't reproduce it here.

Tested with gcc 4.9.3 and glibc 2.21 and glibc 2.22 so far,
qemu&kernel/KVM latest upstream (4.6-rc3..).

You can run this in between each invocation to guarantee all memory is
backed by THP (no need of reboot):

# echo 3 >/proc/sys/vm/drop_caches
# echo >/proc/sys/vm/compact_memory

4.5 kernel built with gcc 5.3.1 run on a older userland worked fine
too.

Next thing to test would be if there's something wrong with qemu built
with gcc 5.3.1 if run on top of a 4.4 kernel?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: post-copy is broken?

[Dr. David Alan Gilbert]

* Andrea Arcangeli (aarcange@redhat.com) wrote:
> On Fri, Apr 15, 2016 at 06:23:30PM +0300, Kirill A. Shutemov wrote:
> > The same here. Freshly booted machine with 64GiB ram. I've checked
> > /proc/vmstat: huge pages were allocated
> 
> I tried the test in a loop and I can't reproduce it here.
> 
> Tested with gcc 4.9.3 and glibc 2.21 and glibc 2.22 so far,
> qemu&kernel/KVM latest upstream (4.6-rc3..).
> 
> You can run this in between each invocation to guarantee all memory is
> backed by THP (no need of reboot):
> 
> # echo 3 >/proc/sys/vm/drop_caches
> # echo >/proc/sys/vm/compact_memory
> 
> 4.5 kernel built with gcc 5.3.1 run on a older userland worked fine
> too.
> 
> Next thing to test would be if there's something wrong with qemu built
> with gcc 5.3.1 if run on top of a 4.4 kernel?

It's also working for me on f24 (4.5.0-320 packaged kernel) on a real machine;
so we currently have two sets that break:

   a) Liang Li's setup (that breaks with the migrate of a real VM but I don't
                        think we have any details of the setup);
                        works with 4.4.x breaks with 4.5.x

   b) f24 nested with my test, with THP enabled after Kirill's changes.

Dave

--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: post-copy is broken?

[Li, Liang Z]

> * Andrea Arcangeli (aarcange@redhat.com) wrote:
> > On Fri, Apr 15, 2016 at 06:23:30PM +0300, Kirill A. Shutemov wrote:
> > > The same here. Freshly booted machine with 64GiB ram. I've checked
> > > /proc/vmstat: huge pages were allocated
> >
> > I tried the test in a loop and I can't reproduce it here.
> >
> > Tested with gcc 4.9.3 and glibc 2.21 and glibc 2.22 so far,
> > qemu&kernel/KVM latest upstream (4.6-rc3..).
> >
> > You can run this in between each invocation to guarantee all memory is
> > backed by THP (no need of reboot):
> >
> > # echo 3 >/proc/sys/vm/drop_caches
> > # echo >/proc/sys/vm/compact_memory
> >
> > 4.5 kernel built with gcc 5.3.1 run on a older userland worked fine
> > too.
> >
> > Next thing to test would be if there's something wrong with qemu built
> > with gcc 5.3.1 if run on top of a 4.4 kernel?
> 
> It's also working for me on f24 (4.5.0-320 packaged kernel) on a real machine;
> so we currently have two sets that break:
> 
>    a) Liang Li's setup (that breaks with the migrate of a real VM but I don't
>                         think we have any details of the setup);
>                         works with 4.4.x breaks with 4.5.x
> 

The host OS is Centos 7, I just replace the kernel with 4.4/4.5, I can provide the .config for the kernel 
if you think it's helpful.
The guest OS is CentOS6.6, 2.6.32-504.el6.x86_64

Liang
>    b) f24 nested with my test, with THP enabled after Kirill's changes.
> 
> Dave
> 
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>