[Qemu-devel] Overlapping buffers in I/O requests

[Qemu-devel] Overlapping buffers in I/O requests

[Stefan Hajnoczi]

There is a block I/O corner case that I don't fully understand.  I'd
appreciate thoughts on the expected behavior.

At one point during a Windows Server 2008 install to an IDE disk the
guest sends a read request with overlapping sglist buffers.  It looks
like this:
[0] addr=A len=4k
[1] addr=B len=4k
[2] addr=C len=4k
[3] addr=B len=4k

Buffers 1 and 3 are the same guest memory, their addresses match.

If I understand correctly, IDE will perform each operation in turn and
DMA the result back to the buffers in order.  Therefore disk contents
at +12k should be written to address B.

Unfortunately QEMU does not guarantee this today.  Sometimes the disk
contents at +4k (buffer 1) are read and other times the disk contents
at +12k (buffer 3) are read.

QEMU can be taken out of the picture and replaced by a simple test
program that calls preadv(2) directly with the same overlapping buffer
pattern.  There doesn't appear to be a guarantee that the disk
contents at +12k (buffer 3) will be read instead of +4k (buffer 1).

When the page cache is active preadv(2) produces consistent results.

When the page cache is bypassed (O_DIRECT) preadv(2) produces
consistent results against a physical disk:
               a-22904 [001]  3042.186790: block_bio_queue: 8,0 R 2048 + 32 [a]
               a-22904 [001]  3042.186807: block_getrq: 8,0 R 2048 + 32 [a]
               a-22904 [001]  3042.186812: block_plug: [a]
               a-22904 [001]  3042.186816: block_rq_insert: 8,0 R 0 ()
2048 + 32 [a]
               a-22904 [001]  3042.186822: block_unplug_io: [a] 1
               a-22904 [001]  3042.186829: block_rq_issue: 8,0 R 0 ()
2048 + 32 [a]
 pam-foreground--22912 [001]  3042.187066: block_rq_complete: 8,0 R ()
2048 + 32 [0]

Notice that a single 32 sector read is issued on /dev/sda (8,0).  This
makes sense under the assumption that the disk honors DMA buffer
ordering within a request.

However, when the page cache is bypassed preadv(2) produces
inconsistent results against a file on ext3 -> LVM -> dm-crypt ->
/dev/sda.
               a-22834 [001]  3038.425802: block_bio_queue: 254,3 R
32616672 + 8 [a]
               a-22834 [001]  3038.425812: block_remap: 254,0 R
58544736 + 8 <- (254,3) 32616672
               a-22834 [001]  3038.425813: block_bio_queue: 254,0 R
58544736 + 8 [a]
      kcryptd_io-379   [001]  3038.425832: block_remap: 8,0 R 59044807
+ 8 <- (8,2) 58546792
      kcryptd_io-379   [001]  3038.425833: block_bio_queue: 8,0 R
59044807 + 8 [kcryptd_io]
      kcryptd_io-379   [001]  3038.425841: block_getrq: 8,0 R 59044807
+ 8 [kcryptd_io]
      kcryptd_io-379   [001]  3038.425845: block_plug: [kcryptd_io]
      kcryptd_io-379   [001]  3038.425848: block_rq_insert: 8,0 R 0 ()
59044807 + 8 [kcryptd_io]
      kcryptd_io-379   [001]  3038.425859: block_rq_issue: 8,0 R 0 ()
59044807 + 8 [kcryptd_io]
               a-22834 [001]  3038.425894: block_bio_queue: 254,3 R
32616792 + 16 [a]
               a-22834 [001]  3038.425898: block_remap: 254,0 R
58544856 + 16 <- (254,3) 32616792
               a-22834 [001]  3038.425899: block_bio_queue: 254,0 R
58544856 + 16 [a]
      kcryptd_io-379   [001]  3038.425908: block_remap: 8,0 R 59044927
+ 16 <- (8,2) 58546912
      kcryptd_io-379   [001]  3038.425909: block_bio_queue: 8,0 R
59044927 + 16 [kcryptd_io]
      kcryptd_io-379   [001]  3038.425911: block_getrq: 8,0 R 59044927
+ 16 [kcryptd_io]
      kcryptd_io-379   [001]  3038.425913: block_plug: [kcryptd_io]
      kcryptd_io-379   [001]  3038.425914: block_rq_insert: 8,0 R 0 ()
59044927 + 16 [kcryptd_io]
               a-22834 [001]  3038.425920: block_bio_queue: 254,3 R
32616992 + 8 [a]
               a-22834 [001]  3038.425922: block_remap: 254,0 R
58545056 + 8 <- (254,3) 32616992
               a-22834 [001]  3038.425923: block_bio_queue: 254,0 R
58545056 + 8 [a]
               a-22834 [001]  3038.425929: block_unplug_io: [a] 0
               a-22834 [001]  3038.425930: block_unplug_io: [a] 0
               a-22834 [001]  3038.425931: block_unplug_io: [a] 2
               a-22834 [001]  3038.425934: block_rq_issue: 8,0 R 0 ()
59044927 + 16 [a]
      kcryptd_io-379   [001]  3038.425948: block_remap: 8,0 R 59045127
+ 8 <- (8,2) 58547112
      kcryptd_io-379   [001]  3038.425949: block_bio_queue: 8,0 R
59045127 + 8 [kcryptd_io]
      kcryptd_io-379   [001]  3038.425951: block_getrq: 8,0 R 59045127
+ 8 [kcryptd_io]
      kcryptd_io-379   [001]  3038.425953: block_plug: [kcryptd_io]
      kcryptd_io-379   [001]  3038.425954: block_rq_insert: 8,0 R 0 ()
59045127 + 8 [kcryptd_io]
          <idle>-0     [001]  3038.427414: block_unplug_timer: [swapper] 3
       kblockd/1-21    [001]  3038.427437: block_unplug_io: [kblockd/1] 3
       kblockd/1-21    [001]  3038.427440: block_rq_issue: 8,0 R 0 ()
59045127 + 8 [kblockd/1]
          <idle>-0     [000]  3038.436786: block_rq_complete: 8,0 R ()
59044807 + 8 [0]
         kcryptd-380   [001]  3038.436960: block_bio_complete: 254,0 R
58544736 + 8 [0]
         kcryptd-380   [001]  3038.436963: block_bio_complete: 254,3 R
32616672 + 8 [0]
          <idle>-0     [001]  3038.437070: block_rq_complete: 8,0 R ()
59044927 + 16 [0]
         kcryptd-380   [000]  3038.437343: block_bio_complete: 254,0 R
58544856 + 16 [611733513]
         kcryptd-380   [000]  3038.437346: block_bio_complete: 254,3 R
32616792 + 16 [-815025730]
          <idle>-0     [000]  3038.437428: block_rq_complete: 8,0 R ()
59045127 + 8 [0]
         kcryptd-380   [000]  3038.437569: block_bio_complete: 254,0 R
58545056 + 8 [-2107963545]
         kcryptd-380   [000]  3038.437571: block_bio_complete: 254,3 R
32616992 + 8 [176593183]

The 32 sectors are broken up into 8, 8, and 16 sector requests.  I
believe the filesystem is doing this before LVM is reached.  This
makes sense since a file may not be contiguous on disk and several
extents need to be read.

These 3 independent requests can complete in any order.  The order
will affect what contents are visible at address B when the read
completes.

So now my question:

Is QEMU risking data corruption when buffers overlap?  If IDE
guarantees that buffers are read in order then we are doing it wrong
(at least when O_DIRECT is used).

Perhaps there is no ordering guarantee in IDE, Windows is doing
something crazy, and QEMU is within its writes to use preadv(2) like
this.

Stefan