[Qemu-devel] qemu/virtio issue due to non-atomic data access

[Qemu-devel] qemu/virtio issue due to non-atomic data access

[Paul Guo]

Hello,

I'm developing the qemu io support for kvm on arch/tile. During virtio-net testing I always saw the following similar message:

"Guest moved used index from 46573 to 46592"

The guest os then exits immediately. The qemu version is 0.13.0.

Here is the code that reports the error message:

static int virtqueue_num_heads(VirtQueue *vq, unsigned int idx)
{
    uint16_t num_heads = vring_avail_idx(vq) - idx;

    /* Check it isn't doing very strange things with descriptor numbers. */
    if (num_heads > vq->vring.num) {
        fprintf(stderr, "Guest moved used index from %u to %u",
                idx, vring_avail_idx(vq));
        exit(1);
    }

    return num_heads;
}

I looked into this issue a bit, it seems that this is due to the non-atomic data access of some virtio variables in qemu. In the above case, vq->vring.avail.idx is modified by kernel and is read in qemu via lduw_le_p() (for our default hw configuration case). lduw_le_p() loads the 16bit values byte by byte. If the kernel is updating the value from 0xB5FF to 0xB600 (i.e. 46592), qemu probably reads 0xB6FF and then virtqueue_num_heads() enters the error handling branch.

static inline int lduw_le_p(const void *ptr)
{
#ifdef _ARCH_PPC
    int val;
    __asm__ __volatile__ ("lhbrx %0,0,%1" : "=r" (val) : "r" (ptr));
    return val;
#else
    const uint8_t *p = ptr;
    return p[0] | (p[1] << 8);
#endif
}

Latest qemu changes to use memcpy() in lduw_le_p(), but if the alignment of the destination pointer in memcpy() is not implied, the compiler will probably still have to load byte by byte, thus vring_avail_idx() still has this issue.

A proper fix for this issue seems to be: Judge whether the address is aligned, do direct loading for the aligned case in ldq_le_p(), etc?

Thanks,
Paul

Re: [Qemu-devel] qemu/virtio issue due to non-atomic data access

[Richard Henderson]

On 2013-05-02 10:12, Paul Guo wrote:
> A proper fix for this issue seems to be: Judge whether the address is aligned, do direct loading for the aligned case in ldq_le_p(), etc?

No, I would think the proper fix would be to change the bits of virtio that are 
known to access aligned memory to not use the pointer wrapper functions, but to 
use a normal C memory access followed by leN_to_cpu et al.


r~

Re: [Qemu-devel] qemu/virtio issue due to non-atomic data access

[Paul Guo]

Yes, sounds to be a better idea. I'm wondering whether the final real virtual address for loading in lduw_phys(), etc is aligned also in most cases? given the load address (the input argument with type hwaddr) is required to be aligned (according to the comment). If so we should make the lduw_phys(), etc be the aligned cases by default and add the unaligned versions instead.

Thanks,
Paul

> On 2013-05-02 10:12, Paul Guo wrote:
>> A proper fix for this issue seems to be: Judge whether the address is aligned, do direct loading for the aligned case in ldq_le_p(), etc?
> 
> No, I would think the proper fix would be to change the bits of virtio that are known to access aligned memory to not use the pointer wrapper functions, but to use a normal C memory access followed by leN_to_cpu et al.
> 
> 
> r~