[PATCH RFC] vhost: fix barrier pairing

[PATCH RFC] vhost: fix barrier pairing

[Michael S. Tsirkin]

According to memory-barriers.txt, an smp memory barrier
should always be paired with another smp memory barrier,
and I quote "a lack of appropriate pairing is almost certainly an
error".

In case of vhost, failure to flush out used index
update before looking at the interrupt disable flag
could result in missed interrupts, resulting in
networking hang under stress.

This might happen when flags read bypasses used index write.
So we see interrupts disabled and do not interrupt, at the
same time guest writes flags value to enable interrupt,
reads an old used index value, thinks that
used ring is empty and waits for interrupt.

Note: the barrier we pair with here is in
drivers/virtio/virtio_ring.c, function
vring_enable_cb.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---

Dave, I think this is needed in 2.6.34, I'll send a pull
request after doing some more testing.

Rusty, Juan, could you take a look as well please?
Thanks!

 drivers/vhost/vhost.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index e69d238..14fa2f5 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1035,7 +1035,10 @@ int vhost_add_used(struct vhost_virtqueue *vq, unsigned int head, int len)
 /* This actually signals the guest, using eventfd. */
 void vhost_signal(struct vhost_dev *dev, struct vhost_virtqueue *vq)
 {
-	__u16 flags = 0;
+	__u16 flags;
+	/* Flush out used index updates. */
+	smp_mb();
+
 	if (get_user(flags, &vq->avail->flags)) {
 		vq_err(vq, "Failed to get flags");
 		return;
-- 
1.7.1.12.g42b7f

[PATCH RFC] vhost: fix barrier pairing

[Michael S. Tsirkin]

According to memory-barriers.txt, an smp memory barrier
should always be paired with another smp memory barrier,
and I quote "a lack of appropriate pairing is almost certainly an
error".

In case of vhost, failure to flush out used index
update before looking at the interrupt disable flag
could result in missed interrupts, resulting in
networking hang under stress.

This might happen when flags read bypasses used index write.
So we see interrupts disabled and do not interrupt, at the
same time guest writes flags value to enable interrupt,
reads an old used index value, thinks that
used ring is empty and waits for interrupt.

Note: the barrier we pair with here is in
drivers/virtio/virtio_ring.c, function
vring_enable_cb.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---

Dave, I think this is needed in 2.6.34, I'll send a pull
request after doing some more testing.

Rusty, Juan, could you take a look as well please?
Thanks!

 drivers/vhost/vhost.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index e69d238..14fa2f5 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1035,7 +1035,10 @@ int vhost_add_used(struct vhost_virtqueue *vq, unsigned int head, int len)
 /* This actually signals the guest, using eventfd. */
 void vhost_signal(struct vhost_dev *dev, struct vhost_virtqueue *vq)
 {
-	__u16 flags = 0;
+	__u16 flags;
+	/* Flush out used index updates. */
+	smp_mb();
+
 	if (get_user(flags, &vq->avail->flags)) {
 		vq_err(vq, "Failed to get flags");
 		return;
-- 
1.7.1.12.g42b7f

Re: [PATCH RFC] vhost: fix barrier pairing

[Juan Quintela]

"Michael S. Tsirkin" <mst@redhat.com> wrote:
> According to memory-barriers.txt, an smp memory barrier
> should always be paired with another smp memory barrier,
> and I quote "a lack of appropriate pairing is almost certainly an
> error".
>
> In case of vhost, failure to flush out used index
> update before looking at the interrupt disable flag
> could result in missed interrupts, resulting in
> networking hang under stress.
>
> This might happen when flags read bypasses used index write.
> So we see interrupts disabled and do not interrupt, at the
> same time guest writes flags value to enable interrupt,
> reads an old used index value, thinks that
> used ring is empty and waits for interrupt.
>
> Note: the barrier we pair with here is in
> drivers/virtio/virtio_ring.c, function
> vring_enable_cb.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>
> Dave, I think this is needed in 2.6.34, I'll send a pull
> request after doing some more testing.
>
> Rusty, Juan, could you take a look as well please?
> Thanks!

I would have prefered to put it:

void vhost_add_used_and_signal(struct vhost_dev *dev,
			       struct vhost_virtqueue *vq,
			       unsigned int head, int len)
{
	vhost_add_used(vq, head, len);
>>>>    smp_mb();
	vhost_signal(dev, vq);
}

Because it looks strange to have a barrier as the 1st instruction of a
function.  And this way it is clearer (at least to me) what we are
protecting.

But on the other hand, we would have to put a comment explainingthat all
users of vhost_signal() have to put that smp_mb() so .....

Perhaps just improving the commet stating that the corresponding barrier
is there?

> Note: the barrier we pair with here is in
> drivers/virtio/virtio_ring.c, function
> vring_enable_cb.

Good catch.

Later, Juan.

[PATCH RFC] vhost: fix barrier pairing

[Michael S. Tsirkin]

According to memory-barriers.txt, an smp memory barrier
should always be paired with another smp memory barrier,
and I quote "a lack of appropriate pairing is almost certainly an
error".

In case of vhost, failure to flush out used index
update before looking at the interrupt disable flag
could result in missed interrupts, resulting in
networking hang under stress.

This might happen when flags read bypasses used index write.
So we see interrupts disabled and do not interrupt, at the
same time guest writes flags value to enable interrupt,
reads an old used index value, thinks that
used ring is empty and waits for interrupt.

Note: the barrier we pair with here is in
drivers/virtio/virtio_ring.c, function
vring_enable_cb.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---

Dave, I think this is needed in 2.6.34, I'll send a pull
request after doing some more testing.

Rusty, Juan, could you take a look as well please?
Thanks!

 drivers/vhost/vhost.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index e69d238..14fa2f5 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1035,7 +1035,10 @@ int vhost_add_used(struct vhost_virtqueue *vq, unsigned int head, int len)
 /* This actually signals the guest, using eventfd. */
 void vhost_signal(struct vhost_dev *dev, struct vhost_virtqueue *vq)
 {
-	__u16 flags = 0;
+	__u16 flags;
+	/* Flush out used index updates. */
+	smp_mb();
+
 	if (get_user(flags, &vq->avail->flags)) {
 		vq_err(vq, "Failed to get flags");
 		return;
-- 
1.7.1.12.g42b7f

[PATCH RFC] vhost: fix barrier pairing

[Michael S. Tsirkin]

According to memory-barriers.txt, an smp memory barrier
should always be paired with another smp memory barrier,
and I quote "a lack of appropriate pairing is almost certainly an
error".

In case of vhost, failure to flush out used index
update before looking at the interrupt disable flag
could result in missed interrupts, resulting in
networking hang under stress.

This might happen when flags read bypasses used index write.
So we see interrupts disabled and do not interrupt, at the
same time guest writes flags value to enable interrupt,
reads an old used index value, thinks that
used ring is empty and waits for interrupt.

Note: the barrier we pair with here is in
drivers/virtio/virtio_ring.c, function
vring_enable_cb.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---

Dave, I think this is needed in 2.6.34, I'll send a pull
request after doing some more testing.

Rusty, Juan, could you take a look as well please?
Thanks!

 drivers/vhost/vhost.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index e69d238..14fa2f5 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1035,7 +1035,10 @@ int vhost_add_used(struct vhost_virtqueue *vq, unsigned int head, int len)
 /* This actually signals the guest, using eventfd. */
 void vhost_signal(struct vhost_dev *dev, struct vhost_virtqueue *vq)
 {
-	__u16 flags = 0;
+	__u16 flags;
+	/* Flush out used index updates. */
+	smp_mb();
+
 	if (get_user(flags, &vq->avail->flags)) {
 		vq_err(vq, "Failed to get flags");
 		return;
-- 
1.7.1.12.g42b7f