All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Cc: axboe@kernel.dk, martin.petersen@oracle.com,
	gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org,
	kasan-dev@googlegroups.com,
	virtualization@lists.linux-foundation.org,
	James.Bottomley@hansenpartnership.com,
	torvalds@linux-foundation.org, edumazet@google.com,
	linux@roeck-us.net, netdev@vger.kernel.org, c@redhat.com,
	kuba@kernel.org, pabeni@redhat.com, andres@anarazel.de,
	davem@davemloft.net, Dmitry Vyukov <dvyukov@google.com>
Subject: Re: upstream kernel crashes
Date: Wed, 17 Aug 2022 06:53:53 -0400	[thread overview]
Message-ID: <20220817065207-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <1660718191.3631961-1-xuanzhuo@linux.alibaba.com>

On Wed, Aug 17, 2022 at 02:36:31PM +0800, Xuan Zhuo wrote:
> On Wed, 17 Aug 2022 08:13:59 +0200, Dmitry Vyukov <dvyukov@google.com> wrote:
> > On Mon, 15 Aug 2022 17:32:06 -0400, Michael wrote:
> > > So if you pass the size parameter for a legacy device it will
> > > try to make the ring smaller and that is not legal with
> > > legacy at all. But the driver treats legacy and modern
> > > the same, it allocates a smaller queue anyway.
> > >
> > > Lo and behold, I pass disable-modern=on to qemu and it happily
> > > corrupts memory exactly the same as GCP does.
> >
> > Ouch!
> >
> > I understand that the host does the actual corruption,
> > but could you think of any additional debug checking in the guest
> > that would caught this in future? Potentially only when KASAN
> > is enabled which can verify validity of memory ranges.
> > Some kind of additional layer of sanity checking.
> >
> > This caused a bit of a havoc for syzbot with almost 100 unique
> > crash signatures, so would be useful to catch such issues more
> > reliably in future.
> 
> We can add a check to vring size before calling vp_legacy_set_queue_address().
> Checking the memory range directly is a bit cumbersome.
> 
> Thanks.

With a comment along the lines of

/* Legacy virtio pci has no way to communicate a change in vq size to
 * the hypervisor. If ring sizes don't match hypervisor will happily
 * corrupt memory.
 */


> diff --git a/drivers/virtio/virtio_pci_legacy.c b/drivers/virtio/virtio_pci_legacy.c
> index 2257f1b3d8ae..0673831f45b6 100644
> --- a/drivers/virtio/virtio_pci_legacy.c
> +++ b/drivers/virtio/virtio_pci_legacy.c
> @@ -146,6 +146,8 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev,
>                 goto out_del_vq;
>         }
> 
> +       BUG_ON(num != virtqueue_get_vring_size(vq));
> +
>         /* activate the queue */
>         vp_legacy_set_queue_address(&vp_dev->ldev, index, q_pfn);
> 
> 
> >
> > Thanks

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Cc: Dmitry Vyukov <dvyukov@google.com>,
	James.Bottomley@hansenpartnership.com, andres@anarazel.de,
	axboe@kernel.dk, c@redhat.com, davem@davemloft.net,
	edumazet@google.com, gregkh@linuxfoundation.org,
	jasowang@redhat.com, kuba@kernel.org,
	linux-kernel@vger.kernel.org, linux@roeck-us.net,
	martin.petersen@oracle.com, netdev@vger.kernel.org,
	pabeni@redhat.com, torvalds@linux-foundation.org,
	virtualization@lists.linux-foundation.org,
	kasan-dev@googlegroups.com
Subject: Re: upstream kernel crashes
Date: Wed, 17 Aug 2022 06:53:53 -0400	[thread overview]
Message-ID: <20220817065207-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <1660718191.3631961-1-xuanzhuo@linux.alibaba.com>

On Wed, Aug 17, 2022 at 02:36:31PM +0800, Xuan Zhuo wrote:
> On Wed, 17 Aug 2022 08:13:59 +0200, Dmitry Vyukov <dvyukov@google.com> wrote:
> > On Mon, 15 Aug 2022 17:32:06 -0400, Michael wrote:
> > > So if you pass the size parameter for a legacy device it will
> > > try to make the ring smaller and that is not legal with
> > > legacy at all. But the driver treats legacy and modern
> > > the same, it allocates a smaller queue anyway.
> > >
> > > Lo and behold, I pass disable-modern=on to qemu and it happily
> > > corrupts memory exactly the same as GCP does.
> >
> > Ouch!
> >
> > I understand that the host does the actual corruption,
> > but could you think of any additional debug checking in the guest
> > that would caught this in future? Potentially only when KASAN
> > is enabled which can verify validity of memory ranges.
> > Some kind of additional layer of sanity checking.
> >
> > This caused a bit of a havoc for syzbot with almost 100 unique
> > crash signatures, so would be useful to catch such issues more
> > reliably in future.
> 
> We can add a check to vring size before calling vp_legacy_set_queue_address().
> Checking the memory range directly is a bit cumbersome.
> 
> Thanks.

With a comment along the lines of

/* Legacy virtio pci has no way to communicate a change in vq size to
 * the hypervisor. If ring sizes don't match hypervisor will happily
 * corrupt memory.
 */


> diff --git a/drivers/virtio/virtio_pci_legacy.c b/drivers/virtio/virtio_pci_legacy.c
> index 2257f1b3d8ae..0673831f45b6 100644
> --- a/drivers/virtio/virtio_pci_legacy.c
> +++ b/drivers/virtio/virtio_pci_legacy.c
> @@ -146,6 +146,8 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev,
>                 goto out_del_vq;
>         }
> 
> +       BUG_ON(num != virtqueue_get_vring_size(vq));
> +
>         /* activate the queue */
>         vp_legacy_set_queue_address(&vp_dev->ldev, index, q_pfn);
> 
> 
> >
> > Thanks


  reply	other threads:[~2022-08-17 10:54 UTC|newest]

Thread overview: 62+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-08-14 21:26 upstream kernel crashes Guenter Roeck
2022-08-14 21:40 ` Linus Torvalds
2022-08-14 22:37   ` Andres Freund
2022-08-14 22:47     ` Linus Torvalds
2022-08-15  1:04       ` Jens Axboe
2022-08-15  1:36         ` Andres Freund
2022-08-15  3:18           ` Linus Torvalds
2022-08-15  7:11             ` Andres Freund
2022-08-15  7:29               ` Michael S. Tsirkin
2022-08-15  7:46                 ` Andres Freund
2022-08-15  7:53                   ` Michael S. Tsirkin
2022-08-15  8:02                   ` Michael S. Tsirkin
2022-08-15  8:02                     ` Michael S. Tsirkin
2022-08-15  7:51               ` Michael S. Tsirkin
2022-08-15  8:15                 ` Andres Freund
2022-08-15  8:28                   ` Michael S. Tsirkin
2022-08-15  8:34                     ` Andres Freund
2022-08-15 15:40                       ` Michael S. Tsirkin
2022-08-15 15:40                         ` Michael S. Tsirkin
2022-08-15 16:45                         ` Andres Freund
2022-08-15 16:45                           ` Andres Freund
2022-08-15 16:50                           ` Michael S. Tsirkin
2022-08-15 16:50                             ` Michael S. Tsirkin
2022-08-15 17:46                             ` Andres Freund
2022-08-15 17:46                               ` Andres Freund
2022-08-15 20:21                               ` Michael S. Tsirkin
2022-08-15 20:21                                 ` Michael S. Tsirkin
2022-08-15 20:53                                 ` Andres Freund
2022-08-15 20:53                                   ` Andres Freund
2022-08-15 21:04                                   ` Andres Freund
2022-08-15 21:04                                     ` Andres Freund
2022-08-15 21:10                                     ` Andres Freund
2022-08-15 21:10                                       ` Andres Freund
2022-08-15 21:32                                   ` Michael S. Tsirkin
2022-08-15 21:32                                     ` Michael S. Tsirkin
2022-08-16  2:45                                     ` Xuan Zhuo
2022-08-16  2:45                                       ` Xuan Zhuo
2022-08-17  6:13                                     ` Dmitry Vyukov via Virtualization
2022-08-17  6:13                                       ` Dmitry Vyukov
2022-08-17  6:36                                       ` Xuan Zhuo
2022-08-17  6:36                                         ` Xuan Zhuo
2022-08-17 10:53                                         ` Michael S. Tsirkin [this message]
2022-08-17 10:53                                           ` Michael S. Tsirkin
2022-08-17 15:58                                         ` Linus Torvalds
2022-08-17 15:58                                           ` Linus Torvalds
2022-08-18  1:55                                           ` Xuan Zhuo
2022-08-18  1:55                                             ` Xuan Zhuo
2022-08-15 20:45                             ` Guenter Roeck
2022-08-15 20:45                               ` Guenter Roeck
2022-08-15  6:36           ` Michael S. Tsirkin
2022-08-15  7:17             ` Andres Freund
2022-08-15  7:43               ` Michael S. Tsirkin
2022-08-15  1:17       ` Guenter Roeck
2022-08-15  1:29         ` Jens Axboe
2022-08-15  9:43 ` Michael S. Tsirkin
2022-08-15 15:49   ` Guenter Roeck
2022-08-15 16:01     ` Michael S. Tsirkin
2022-08-15 18:22       ` Guenter Roeck
2022-08-15 18:37         ` Linus Torvalds
2022-08-15 20:38           ` Guenter Roeck
2022-08-17 17:12 ` Linus Torvalds
2022-08-18  1:08   ` Andres Freund

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220817065207-mutt-send-email-mst@kernel.org \
    --to=mst@redhat.com \
    --cc=James.Bottomley@hansenpartnership.com \
    --cc=andres@anarazel.de \
    --cc=axboe@kernel.dk \
    --cc=c@redhat.com \
    --cc=davem@davemloft.net \
    --cc=dvyukov@google.com \
    --cc=edumazet@google.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=kasan-dev@googlegroups.com \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux@roeck-us.net \
    --cc=martin.petersen@oracle.com \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=torvalds@linux-foundation.org \
    --cc=virtualization@lists.linux-foundation.org \
    --cc=xuanzhuo@linux.alibaba.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.