* Re: [PATCH net 2/3] net: sctp: fix panic on duplicate ASCONF chunks
From: Daniel Borkmann @ 2014-10-12 23:25 UTC (permalink / raw)
To: Neil Horman; +Cc: davem, linux-sctp, netdev, Vlad Yasevich
In-Reply-To: <20141012014233.GA16360@localhost.localdomain>
On 10/12/2014 03:42 AM, Neil Horman wrote:
> On Sat, Oct 11, 2014 at 12:02:31AM +0200, Daniel Borkmann wrote:
>> On 10/10/2014 05:39 PM, Neil Horman wrote:
>> ...
>>> Is it worth adding a WARN_ON, to indicate that two ASCONF chunks have been
>>> received with duplicate serials?
>>
>> Don't think so, as this would be triggerable from outside.
>>
> WARN_ON_ONCE then, per serial number?
Sorry, but no. If someone seriously runs that in production and it
triggers a WARN() from outside, admins will start sending us bug
reports that apparently something with the kernel code is wrong.
WARN() should only be used if we have some *internal* unexpected bug,
but can still fail gracefully. This would neither be an actual code bug
nor would it be an internally triggered one, plus we add unnecessary
complexity to the code. Similarly, for those reasons we don't WARN()
and throw a stack trace when we receive, say, an skb of invalid length
elsewhere.
I'd also like to avoid any additional pr_debug().
I don't think people enable them in production, and if they really do,
it's too late anyway as we already have received this chunk. If anything,
I'd rather like to see debugging code further removed as we have already
different facilities in the kernel for runtime debugging that are much
more powerful.
^ permalink raw reply
* RE: [PATCH] net: fec: Fix sparse warnings with different lock contexts for basic block
From: fugang.duan @ 2014-10-13 1:35 UTC (permalink / raw)
To: Fabio Estevam
Cc: David S. Miller, netdev@vger.kernel.org, sparse@chrisli.org,
Frank.Li@freescale.com
In-Reply-To: <CAOMZO5AhnRFUiCNBQ8FwdvjqVM+1fHfa3DUoHbqKTDcdg7AfBA@mail.gmail.com>
From: Fabio Estevam <festevam@gmail.com> Sent: Sunday, October 12, 2014 5:23 AM
>To: Duan Fugang-B38611
>Cc: David S. Miller; netdev@vger.kernel.org; sparse@chrisli.org; Li Frank-
>B20596
>Subject: Re: [PATCH] net: fec: Fix sparse warnings with different lock
>contexts for basic block
>
>Hi Fugang,
>
>On Sat, Oct 11, 2014 at 12:00 AM, Fugang Duan <b38611@freescale.com> wrote:
>> reproduce:
>> make ARCH=arm C=1 2>fec.txt drivers/net/ethernet/freescale/fec_main.o
>> cat fec.txt
>>
>> sparse warnings:
>> drivers/net/ethernet/freescale/fec_main.c:2916:12: warning: context
>> imbalance in 'fec_set_features' - different lock contexts for basic
>> block
>>
>> Christopher Li suggest to change as below:
>> if (need_lock) {
>> lock();
>> do_something_real();
>> unlock();
>> } else {
>> do_something_real();
>> }
>>
>> Reported-by: Fabio Estevam <festevam@gmail.com>
>> Signed-off-by: Fugang Duan <B38611@freescale.com>
>> Signed-off-by: Christopher Li <sparse@chrisli.org>
>
>It looks like Suggested-by is a more appropriate tag for Christopher.
Agree, thanks. I will change it for V2.
Regards,
Andy
^ permalink raw reply
* [PATCH v2 1/1] net: fec: Fix sparse warnings with different lock contexts for basic block
From: Fugang Duan @ 2014-10-13 2:53 UTC (permalink / raw)
To: davem; +Cc: netdev, festevam, sparse, B20596, b38611
reproduce:
make ARCH=arm C=1 2>fec.txt drivers/net/ethernet/freescale/fec_main.o
cat fec.txt
sparse warnings:
drivers/net/ethernet/freescale/fec_main.c:2916:12: warning: context imbalance
in 'fec_set_features' - different lock contexts for basic block
Christopher Li suggest to change as below:
if (need_lock) {
lock();
do_something_real();
unlock();
} else {
do_something_real();
}
Reported-by: Fabio Estevam <festevam@gmail.com>
Suggested-by: Christopher Li <sparse@chrisli.org>
Signed-off-by: Fugang Duan <B38611@freescale.com>
---
drivers/net/ethernet/freescale/fec_main.c | 24 ++++++++++++++----------
1 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c
index 87975b5..7a8209e 100644
--- a/drivers/net/ethernet/freescale/fec_main.c
+++ b/drivers/net/ethernet/freescale/fec_main.c
@@ -2912,20 +2912,12 @@ static void fec_poll_controller(struct net_device *dev)
#endif
#define FEATURES_NEED_QUIESCE NETIF_F_RXCSUM
-
-static int fec_set_features(struct net_device *netdev,
+static inline void fec_enet_set_netdev_features(struct net_device *netdev,
netdev_features_t features)
{
struct fec_enet_private *fep = netdev_priv(netdev);
netdev_features_t changed = features ^ netdev->features;
- /* Quiesce the device if necessary */
- if (netif_running(netdev) && changed & FEATURES_NEED_QUIESCE) {
- napi_disable(&fep->napi);
- netif_tx_lock_bh(netdev);
- fec_stop(netdev);
- }
-
netdev->features = features;
/* Receive checksum has been changed */
@@ -2935,13 +2927,25 @@ static int fec_set_features(struct net_device *netdev,
else
fep->csum_flags &= ~FLAG_RX_CSUM_ENABLED;
}
+}
+
+static int fec_set_features(struct net_device *netdev,
+ netdev_features_t features)
+{
+ struct fec_enet_private *fep = netdev_priv(netdev);
+ netdev_features_t changed = features ^ netdev->features;
- /* Resume the device after updates */
if (netif_running(netdev) && changed & FEATURES_NEED_QUIESCE) {
+ napi_disable(&fep->napi);
+ netif_tx_lock_bh(netdev);
+ fec_stop(netdev);
+ fec_enet_set_netdev_features(netdev, features);
fec_restart(netdev);
netif_tx_wake_all_queues(netdev);
netif_tx_unlock_bh(netdev);
napi_enable(&fep->napi);
+ } else {
+ fec_enet_set_netdev_features(netdev, features);
}
return 0;
--
1.7.8
^ permalink raw reply related
* Re: [PATCH net-next RFC 3/3] virtio-net: conditionally enable tx interrupt
From: Jason Wang @ 2014-10-13 6:02 UTC (permalink / raw)
To: Eric Dumazet; +Cc: kvm, mst, netdev, linux-kernel, virtualization, linux-api
In-Reply-To: <1413038899.9362.43.camel@edumazet-glaptop2.roam.corp.google.com>
On 10/11/2014 10:48 PM, Eric Dumazet wrote:
> On Sat, 2014-10-11 at 15:16 +0800, Jason Wang wrote:
>> We free transmitted packets in ndo_start_xmit() in the past to get better
>> performance in the past. One side effect is that skb_orphan() needs to be
>> called in ndo_start_xmit() which makes sk_wmem_alloc not accurate in
>> fact. For TCP protocol, this means several optimization could not work well
>> such as TCP small queue and auto corking. This can lead extra low
>> throughput of small packets stream.
>>
>> Thanks to the urgent descriptor support. This patch tries to solve this
>> issue by enable the tx interrupt selectively for stream packets. This means
>> we don't need to orphan TCP stream packets in ndo_start_xmit() but enable
>> tx interrupt for those packets. After we get tx interrupt, a tx napi was
>> scheduled to free those packets.
>>
>> With this method, sk_wmem_alloc of TCP socket were more accurate than in
>> the past which let TCP can batch more through TSQ and auto corking.
>>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>> ---
>> drivers/net/virtio_net.c | 164 ++++++++++++++++++++++++++++++++++++-----------
>> 1 file changed, 128 insertions(+), 36 deletions(-)
>>
>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
>> index 5810841..b450fc4 100644
>> --- a/drivers/net/virtio_net.c
>> +++ b/drivers/net/virtio_net.c
>> @@ -72,6 +72,8 @@ struct send_queue {
>>
>> /* Name of the send queue: output.$index */
>> char name[40];
>> +
>> + struct napi_struct napi;
>> };
>>
>> /* Internal representation of a receive virtqueue */
>> @@ -217,15 +219,40 @@ static struct page *get_a_page(struct receive_queue *rq, gfp_t gfp_mask)
>> return p;
>> }
>>
>> +static int free_old_xmit_skbs(struct send_queue *sq, int budget)
>> +{
>> + struct sk_buff *skb;
>> + unsigned int len;
>> + struct virtnet_info *vi = sq->vq->vdev->priv;
>> + struct virtnet_stats *stats = this_cpu_ptr(vi->stats);
>> + int sent = 0;
>> +
>> + while (sent < budget &&
>> + (skb = virtqueue_get_buf(sq->vq, &len)) != NULL) {
>> + pr_debug("Sent skb %p\n", skb);
>> +
>> + u64_stats_update_begin(&stats->tx_syncp);
>> + stats->tx_bytes += skb->len;
>> + stats->tx_packets++;
>> + u64_stats_update_end(&stats->tx_syncp);
>> +
>> + dev_kfree_skb_any(skb);
>> + sent++;
>> + }
>> +
> You could accumulate skb->len in a totlen var, and perform a single
>
> u64_stats_update_begin(&stats->tx_syncp);
> stats->tx_bytes += totlen;
> stats->tx_packets += sent;
> u64_stats_update_end(&stats->tx_syncp);
>
> after the loop.
>
Yes, will do this in a separated patch.
>> + return sent;
>> +}
>> +
> ...
>
>> +
>> +static bool virtnet_skb_needs_intr(struct sk_buff *skb)
>> +{
>> + union {
>> + unsigned char *network;
>> + struct iphdr *ipv4;
>> + struct ipv6hdr *ipv6;
>> + } hdr;
>> + struct tcphdr *th = tcp_hdr(skb);
>> + u16 payload_len;
>> +
>> + hdr.network = skb_network_header(skb);
>> +
>> + /* Only IPv4/IPv6 with TCP is supported */
> Oh well, yet another packet flow dissector :)
>
> If most packets were caught by your implementation, you could use it
> for fast patj and fallback to skb_flow_dissect() for encapsulated
> traffic.
>
> struct flow_keys keys;
>
> if (!skb_flow_dissect(skb, &keys))
> return false;
>
> if (keys.ip_proto != IPPROTO_TCP)
> return false;
>
> then check __skb_get_poff() how to get th, and check if there is some
> payload...
>
>
Yes, but we don't know if most of packets were TCP or encapsulated TCP,
it depends on userspace application. If not, looks like
skb_flow_dissect() can bring some overhead, or it could be ignored?
skb_flow_dissect
^ permalink raw reply
* Re: [PATCH net-next RFC 1/3] virtio: support for urgent descriptors
From: Jason Wang @ 2014-10-13 6:22 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: kvm, netdev, linux-kernel, virtualization, linux-api
In-Reply-To: <20141012092744.GA9567@redhat.com>
On 10/12/2014 05:27 PM, Michael S. Tsirkin wrote:
> On Sat, Oct 11, 2014 at 03:16:44PM +0800, Jason Wang wrote:
>> Below should be useful for some experiments Jason is doing.
>> I thought I'd send it out for early review/feedback.
>>
>> event idx feature allows us to defer interrupts until
>> a specific # of descriptors were used.
>> Sometimes it might be useful to get an interrupt after
>> a specific descriptor, regardless.
>> This adds a descriptor flag for this, and an API
>> to create an urgent output descriptor.
>> This is still an RFC:
>> we'll need a feature bit for drivers to detect this,
>> but we've run out of feature bits for virtio 0.X.
>> For experimentation purposes, drivers can assume
>> this is set, or add a driver-specific feature bit.
>>
>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
> I see that as compared to my original patch, you have
> added a new flag: VRING_AVAIL_F_NO_URGENT_INTERRUPT
> I don't think it's necessary, see below.
>
> As such, I think this patch should be split:
> - original patch adding support for urgent descriptors
> - a patch adding virtqueue_enable/disable_cb_urgent(_prepare)?
Not sure this is a good idea, since the api of first patch is in-completed.
>
>> ---
>> drivers/virtio/virtio_ring.c | 75 +++++++++++++++++++++++++++++++++++++---
>> include/linux/virtio.h | 14 ++++++++
>> include/uapi/linux/virtio_ring.h | 5 ++-
>> 3 files changed, 89 insertions(+), 5 deletions(-)
>>
[...]
>>
>> +unsigned virtqueue_enable_cb_prepare_urgent(struct virtqueue *_vq)
>> +{
>> + struct vring_virtqueue *vq = to_vvq(_vq);
>> + u16 last_used_idx;
>> +
>> + START_USE(vq);
>> + vq->vring.avail->flags &= ~VRING_AVAIL_F_NO_URGENT_INTERRUPT;
>> + last_used_idx = vq->last_used_idx;
>> + END_USE(vq);
>> + return last_used_idx;
>> +}
>> +EXPORT_SYMBOL_GPL(virtqueue_enable_cb_prepare_urgent);
>> +
> You can implement virtqueue_enable_cb_prepare_urgent
> simply by clearing ~VRING_AVAIL_F_NO_INTERRUPT.
>
> The effect is same: host sends interrupts only if there
> is an urgent descriptor.
Seems not, consider the case when event index was disabled. This will
turn on all interrupts.
^ permalink raw reply
* Re:
From: geohughes @ 2014-10-13 6:18 UTC (permalink / raw)
I am Mr Tan Wong and i have a Business Proposal for you.If Interested do
contact me at my email for further details tan.wong4040@yahoo.com.hk
^ permalink raw reply
* [PATCH] netfilter: release skbuf when nlmsg put fail
From: Houcheng Lin @ 2014-10-13 6:50 UTC (permalink / raw)
To: pablo, kaber, kadlec, davem, netfilter-devel
Cc: coreteam, netdev, Linux Kernel Mailing List
When system is under heavy loading, the __nfulnl_send() may may failed
to put nlmsg into skbuf of nfulnl_instance. If not clear the skbuff on failed,
the __nfulnl_send() will still try to put next nlmsg onto this half-full skbuf
and cause the user program can never receive packet.
This patch fix this issue by releasing skbuf immediately after nlmst put
failed.
Signed-off-by: Houcheng Lin <houcheng@gmail.com>
---
net/netfilter/nfnetlink_log.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/net/netfilter/nfnetlink_log.c b/net/netfilter/nfnetlink_log.c
index a11c5ff..0cb9ede 100644
--- a/net/netfilter/nfnetlink_log.c
+++ b/net/netfilter/nfnetlink_log.c
@@ -358,10 +358,9 @@ __nfulnl_send(struct nfulnl_instance *inst)
}
status = nfnetlink_unicast(inst->skb, inst->net, inst->peer_portid,
MSG_DONTWAIT);
-
+out:
inst->qlen = 0;
inst->skb = NULL;
-out:
return status;
}
--
1.7.9.5
^ permalink raw reply related
* Re: [PATCH v3 10/25] virtio: add API to enable VQs early
From: Rusty Russell @ 2014-10-13 6:52 UTC (permalink / raw)
To: Michael S. Tsirkin, linux-kernel
Cc: linux-s390, linux-scsi, kvm, Christian Borntraeger, netdev,
virtualization, Paolo Bonzini, Amit Shah, v9fs-developer,
David S. Miller
In-Reply-To: <1413114332-626-11-git-send-email-mst@redhat.com>
"Michael S. Tsirkin" <mst@redhat.com> writes:
> virtio spec 0.9.X requires DRIVER_OK to be set before
> VQs are used, but some drivers use VQs before probe
> function returns.
> Since DRIVER_OK is set after probe, this violates the spec.
>
> Even though under virtio 1.0 transitional devices support this
> behaviour, we want to make it possible for those early callers to become
> spec compliant and eventually support non-transitional devices.
>
> Add API for drivers to call before using VQs.
>
> Sets DRIVER_OK internally.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
OK, this all looks good, but I think we should do two things:
1) rename virtio_enable_vqs_early() to virtio_device_ready().
2) Add a BUG_ON in the virtio_ring code to make sure device is ready
before any ops are called.
Cheers,
Rusty.
^ permalink raw reply
* Re: [PATCH net-next RFC 1/3] virtio: support for urgent descriptors
From: Michael S. Tsirkin @ 2014-10-13 7:16 UTC (permalink / raw)
To: Jason Wang; +Cc: kvm, netdev, linux-kernel, virtualization, linux-api
In-Reply-To: <543B6F94.2090107@redhat.com>
On Mon, Oct 13, 2014 at 02:22:12PM +0800, Jason Wang wrote:
> On 10/12/2014 05:27 PM, Michael S. Tsirkin wrote:
> > On Sat, Oct 11, 2014 at 03:16:44PM +0800, Jason Wang wrote:
> >> Below should be useful for some experiments Jason is doing.
> >> I thought I'd send it out for early review/feedback.
> >>
> >> event idx feature allows us to defer interrupts until
> >> a specific # of descriptors were used.
> >> Sometimes it might be useful to get an interrupt after
> >> a specific descriptor, regardless.
> >> This adds a descriptor flag for this, and an API
> >> to create an urgent output descriptor.
> >> This is still an RFC:
> >> we'll need a feature bit for drivers to detect this,
> >> but we've run out of feature bits for virtio 0.X.
> >> For experimentation purposes, drivers can assume
> >> this is set, or add a driver-specific feature bit.
> >>
> >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >> Signed-off-by: Jason Wang <jasowang@redhat.com>
> > I see that as compared to my original patch, you have
> > added a new flag: VRING_AVAIL_F_NO_URGENT_INTERRUPT
> > I don't think it's necessary, see below.
> >
> > As such, I think this patch should be split:
> > - original patch adding support for urgent descriptors
> > - a patch adding virtqueue_enable/disable_cb_urgent(_prepare)?
>
> Not sure this is a good idea, since the api of first patch is in-completed.
> >
> >> ---
> >> drivers/virtio/virtio_ring.c | 75 +++++++++++++++++++++++++++++++++++++---
> >> include/linux/virtio.h | 14 ++++++++
> >> include/uapi/linux/virtio_ring.h | 5 ++-
> >> 3 files changed, 89 insertions(+), 5 deletions(-)
> >>
> [...]
> >>
> >> +unsigned virtqueue_enable_cb_prepare_urgent(struct virtqueue *_vq)
> >> +{
> >> + struct vring_virtqueue *vq = to_vvq(_vq);
> >> + u16 last_used_idx;
> >> +
> >> + START_USE(vq);
> >> + vq->vring.avail->flags &= ~VRING_AVAIL_F_NO_URGENT_INTERRUPT;
> >> + last_used_idx = vq->last_used_idx;
> >> + END_USE(vq);
> >> + return last_used_idx;
> >> +}
> >> +EXPORT_SYMBOL_GPL(virtqueue_enable_cb_prepare_urgent);
> >> +
> > You can implement virtqueue_enable_cb_prepare_urgent
> > simply by clearing ~VRING_AVAIL_F_NO_INTERRUPT.
> >
> > The effect is same: host sends interrupts only if there
> > is an urgent descriptor.
>
> Seems not, consider the case when event index was disabled. This will
> turn on all interrupts.
This means that a legacy device without event index support
will get more interrupts.
Sounds reasonable.
--
MST
^ permalink raw reply
* Re: [PATCH v3 10/25] virtio: add API to enable VQs early
From: Michael S. Tsirkin @ 2014-10-13 7:35 UTC (permalink / raw)
To: Rusty Russell
Cc: linux-s390, kvm, linux-scsi, Christian Borntraeger, netdev,
linux-kernel, virtualization, Paolo Bonzini, Amit Shah,
v9fs-developer, David S. Miller
In-Reply-To: <8738asea14.fsf@rustcorp.com.au>
On Mon, Oct 13, 2014 at 05:22:39PM +1030, Rusty Russell wrote:
> "Michael S. Tsirkin" <mst@redhat.com> writes:
> > virtio spec 0.9.X requires DRIVER_OK to be set before
> > VQs are used, but some drivers use VQs before probe
> > function returns.
> > Since DRIVER_OK is set after probe, this violates the spec.
> >
> > Even though under virtio 1.0 transitional devices support this
> > behaviour, we want to make it possible for those early callers to become
> > spec compliant and eventually support non-transitional devices.
> >
> > Add API for drivers to call before using VQs.
> >
> > Sets DRIVER_OK internally.
> >
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
>
> OK, this all looks good, but I think we should do two things:
>
> 1) rename virtio_enable_vqs_early() to virtio_device_ready().
I considered this, I was concerned that the usage isn't clear from name:
- this has to be called before VQs are used
- this is optional, happens automatically after probe
device_ready seems to imply driver is fully initialized,
if it was it could just call probe.
But it's easy enough to make the change, so I'll go ahead and
post V4. It will be up to you which one to apply then.
> 2) Add a BUG_ON in the virtio_ring code to make sure device is ready
> before any ops are called.
>
> Cheers,
> Rusty.
On kick? This seems too expensive, especially considering that
we don't track the status.
Anyway, can be a patch on top?
--
MST
^ permalink raw reply
* [PATCH v4 00/25] virtio: fix spec compliance issues
From: Michael S. Tsirkin @ 2014-10-13 7:48 UTC (permalink / raw)
To: linux-kernel
Cc: linux-s390, kvm, linux-scsi, Christian Borntraeger, netdev,
virtualization, Paolo Bonzini, Amit Shah, v9fs-developer,
David S. Miller
Changes from v4:
rename virtio_enable_vqs_early() to virtio_device_ready()
Note: Rusty requested we add a BUG_ON in the virtio_ring code.
This can be done by a separate patch on top.
Good for bisectability in case BUG_ON starts triggering :)
Rusty, please review this, and consider for this merge window.
This fixes the following virtio spec compliance issues:
1. on restore, drivers use device before setting ACKNOWLEDGE and DRIVER bits
2. on probe, drivers aren't prepared to handle config interrupts
arriving before probe returns
3. on probe, drivers use device before DRIVER_OK it set
Note that 1 is a clear violation of virtio spec 0.9 and 1.0,
so I am proposing the fix for stable. OTOH 2 is against 1.0 rules but
is a known documented bug in many drivers, so let's fix it going
forward, but it does not seem to be worth it to backport
the changes.
An error handling bugfix for virtio-net and a fix for a
theoretical race condition in virtio scsi is also included.
2 is merely a theoretical race condition, but it seems important
to address to make sure that changes to address 3 do not introduce
instability.
I also included patch to drop scan callback use from virtio
scsi: using this callback creates asymmetry between probe
and resume, and we actually had to add code comments so
people can figure out the flow. Code flow becomes clearer
with the new API.
After this change, the only user of the scan callback is virtio rng.
It's possible to modify it trivially and then drop scan callback
from core, but I'm rather inclined to look at ways to
make some rng core changes so that we don't need to
have so many variables tracking device state.
So this is deferred for now.
Michael S. Tsirkin (24):
virtio_pci: fix virtio spec compliance on restore
virtio: unify config_changed handling
virtio-pci: move freeze/restore to virtio core
virtio: defer config changed notifications
virtio_blk: drop config_enable
virtio-blk: drop config_mutex
virtio_net: drop config_enable
virtio-net: drop config_mutex
virtio_net: minor cleanup
virtio: add API to enable VQs early
virtio_net: enable VQs early
virtio_blk: enable VQs early
virtio_console: enable VQs early
9p/trans_virtio: enable VQs early
virtio_net: fix use after free on allocation failure
virtio_scsi: move kick event out from virtscsi_init
virtio_blk: enable VQs early on restore
virtio_scsi: enable VQs early on restore
virtio_console: enable VQs early on restore
virtio_net: enable VQs early on restore
virtio_scsi: fix race on device removal
virtio_balloon: enable VQs early on restore
virtio_scsi: drop scan callback
virtio-rng: refactor probe error handling
Paolo Bonzini (1):
virito_scsi: use freezable WQ for events
include/linux/virtio.h | 14 +++++
include/linux/virtio_config.h | 17 ++++++
drivers/block/virtio_blk.c | 40 ++++----------
drivers/char/hw_random/virtio-rng.c | 15 +++---
drivers/char/virtio_console.c | 4 ++
drivers/misc/mic/card/mic_virtio.c | 6 +--
drivers/net/virtio_net.c | 44 +++++-----------
drivers/s390/kvm/kvm_virtio.c | 9 +---
drivers/s390/kvm/virtio_ccw.c | 6 +--
drivers/scsi/virtio_scsi.c | 42 +++++++++------
drivers/virtio/virtio.c | 102 ++++++++++++++++++++++++++++++++++++
drivers/virtio/virtio_balloon.c | 2 +
drivers/virtio/virtio_mmio.c | 7 +--
drivers/virtio/virtio_pci.c | 33 ++----------
net/9p/trans_virtio.c | 2 +
15 files changed, 206 insertions(+), 137 deletions(-)
--
MST
^ permalink raw reply
* [PATCH v4 01/25] virtio_pci: fix virtio spec compliance on restore
From: Michael S. Tsirkin @ 2014-10-13 7:48 UTC (permalink / raw)
To: linux-kernel
Cc: linux-s390, kvm, linux-scsi, Christian Borntraeger, netdev,
stable, virtualization, Paolo Bonzini, Amit Shah, v9fs-developer,
David S. Miller
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
On restore, virtio pci does the following:
+ set features
+ init vqs etc - device can be used at this point!
+ set ACKNOWLEDGE,DRIVER and DRIVER_OK status bits
This is in violation of the virtio spec, which
requires the following order:
- ACKNOWLEDGE
- DRIVER
- init vqs
- DRIVER_OK
This behaviour will break with hypervisors that assume spec compliant
behaviour. It seems like a good idea to have this patch applied to
stable branches to reduce the support butden for the hypervisors.
Cc: stable@vger.kernel.org
Cc: Amit Shah <amit.shah@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
drivers/virtio/virtio_pci.c | 33 ++++++++++++++++++++++++++++++---
1 file changed, 30 insertions(+), 3 deletions(-)
diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
index 3d1463c..add40d0 100644
--- a/drivers/virtio/virtio_pci.c
+++ b/drivers/virtio/virtio_pci.c
@@ -789,6 +789,7 @@ static int virtio_pci_restore(struct device *dev)
struct pci_dev *pci_dev = to_pci_dev(dev);
struct virtio_pci_device *vp_dev = pci_get_drvdata(pci_dev);
struct virtio_driver *drv;
+ unsigned status = 0;
int ret;
drv = container_of(vp_dev->vdev.dev.driver,
@@ -799,14 +800,40 @@ static int virtio_pci_restore(struct device *dev)
return ret;
pci_set_master(pci_dev);
+ /* We always start by resetting the device, in case a previous
+ * driver messed it up. */
+ vp_reset(&vp_dev->vdev);
+
+ /* Acknowledge that we've seen the device. */
+ status |= VIRTIO_CONFIG_S_ACKNOWLEDGE;
+ vp_set_status(&vp_dev->vdev, status);
+
+ /* Maybe driver failed before freeze.
+ * Restore the failed status, for debugging. */
+ status |= vp_dev->saved_status & VIRTIO_CONFIG_S_FAILED;
+ vp_set_status(&vp_dev->vdev, status);
+
+ if (!drv)
+ return 0;
+
+ /* We have a driver! */
+ status |= VIRTIO_CONFIG_S_DRIVER;
+ vp_set_status(&vp_dev->vdev, status);
+
vp_finalize_features(&vp_dev->vdev);
- if (drv && drv->restore)
+ if (drv->restore) {
ret = drv->restore(&vp_dev->vdev);
+ if (ret) {
+ status |= VIRTIO_CONFIG_S_FAILED;
+ vp_set_status(&vp_dev->vdev, status);
+ return ret;
+ }
+ }
/* Finally, tell the device we're all set */
- if (!ret)
- vp_set_status(&vp_dev->vdev, vp_dev->saved_status);
+ status |= VIRTIO_CONFIG_S_DRIVER_OK;
+ vp_set_status(&vp_dev->vdev, status);
return ret;
}
--
MST
^ permalink raw reply related
* [PATCH v4 02/25] virtio: unify config_changed handling
From: Michael S. Tsirkin @ 2014-10-13 7:48 UTC (permalink / raw)
To: linux-kernel
Cc: kvm, Wolfram Sang, Heiko Carstens, Sudeep Dutt, virtualization,
linux-s390, linux-scsi, Christian Borntraeger, v9fs-developer,
Siva Yerramreddy, Martin Schwidefsky, Paolo Bonzini, netdev,
Ashutosh Dixit, Greg Kroah-Hartman, Amit Shah, linux390,
Andrew Morton, David S. Miller
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
Replace duplicated code in all transports with a single wrapper in
virtio.c.
The only functional change is in virtio_mmio.c: if a buggy device sends
us an interrupt before driver is set, we previously returned IRQ_NONE,
now we return IRQ_HANDLED.
As this must not happen in practice, this does not look like a big deal.
See also commit 3fff0179e33cd7d0a688dab65700c46ad089e934
virtio-pci: do not oops on config change if driver not loaded.
for the original motivation behind the driver check.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
include/linux/virtio.h | 2 ++
drivers/misc/mic/card/mic_virtio.c | 6 +-----
drivers/s390/kvm/kvm_virtio.c | 9 +--------
drivers/s390/kvm/virtio_ccw.c | 6 +-----
drivers/virtio/virtio.c | 9 +++++++++
drivers/virtio/virtio_mmio.c | 7 ++-----
drivers/virtio/virtio_pci.c | 6 +-----
7 files changed, 17 insertions(+), 28 deletions(-)
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index b46671e..3c19bd3 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -108,6 +108,8 @@ void unregister_virtio_device(struct virtio_device *dev);
void virtio_break_device(struct virtio_device *dev);
+void virtio_config_changed(struct virtio_device *dev);
+
/**
* virtio_driver - operations for a virtio I/O driver
* @driver: underlying device driver (populate name and owner).
diff --git a/drivers/misc/mic/card/mic_virtio.c b/drivers/misc/mic/card/mic_virtio.c
index f14b600..e647947 100644
--- a/drivers/misc/mic/card/mic_virtio.c
+++ b/drivers/misc/mic/card/mic_virtio.c
@@ -462,16 +462,12 @@ static void mic_handle_config_change(struct mic_device_desc __iomem *d,
struct mic_device_ctrl __iomem *dc
= (void __iomem *)d + mic_aligned_desc_size(d);
struct mic_vdev *mvdev = (struct mic_vdev *)ioread64(&dc->vdev);
- struct virtio_driver *drv;
if (ioread8(&dc->config_change) != MIC_VIRTIO_PARAM_CONFIG_CHANGED)
return;
dev_dbg(mdrv->dev, "%s %d\n", __func__, __LINE__);
- drv = container_of(mvdev->vdev.dev.driver,
- struct virtio_driver, driver);
- if (drv->config_changed)
- drv->config_changed(&mvdev->vdev);
+ virtio_config_changed(&mvdev->vdev);
iowrite8(1, &dc->guest_ack);
}
diff --git a/drivers/s390/kvm/kvm_virtio.c b/drivers/s390/kvm/kvm_virtio.c
index a134965..6431290 100644
--- a/drivers/s390/kvm/kvm_virtio.c
+++ b/drivers/s390/kvm/kvm_virtio.c
@@ -406,15 +406,8 @@ static void kvm_extint_handler(struct ext_code ext_code,
switch (param) {
case VIRTIO_PARAM_CONFIG_CHANGED:
- {
- struct virtio_driver *drv;
- drv = container_of(vq->vdev->dev.driver,
- struct virtio_driver, driver);
- if (drv->config_changed)
- drv->config_changed(vq->vdev);
-
+ virtio_config_changed(vq->vdev);
break;
- }
case VIRTIO_PARAM_DEV_ADD:
schedule_work(&hotplug_work);
break;
diff --git a/drivers/s390/kvm/virtio_ccw.c b/drivers/s390/kvm/virtio_ccw.c
index d2c0b44..6cbe6ef 100644
--- a/drivers/s390/kvm/virtio_ccw.c
+++ b/drivers/s390/kvm/virtio_ccw.c
@@ -940,11 +940,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
vring_interrupt(0, vq);
}
if (test_bit(0, &vcdev->indicators2)) {
- drv = container_of(vcdev->vdev.dev.driver,
- struct virtio_driver, driver);
-
- if (drv && drv->config_changed)
- drv->config_changed(&vcdev->vdev);
+ virtio_config_changed(&vcdev->vdev);
clear_bit(0, &vcdev->indicators2);
}
}
diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index fed0ce1..3980687 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -239,6 +239,15 @@ void unregister_virtio_device(struct virtio_device *dev)
}
EXPORT_SYMBOL_GPL(unregister_virtio_device);
+void virtio_config_changed(struct virtio_device *dev)
+{
+ struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
+
+ if (drv && drv->config_changed)
+ drv->config_changed(dev);
+}
+EXPORT_SYMBOL_GPL(virtio_config_changed);
+
static int virtio_init(void)
{
if (bus_register(&virtio_bus) != 0)
diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
index c600ccf..ef9a165 100644
--- a/drivers/virtio/virtio_mmio.c
+++ b/drivers/virtio/virtio_mmio.c
@@ -234,8 +234,6 @@ static irqreturn_t vm_interrupt(int irq, void *opaque)
{
struct virtio_mmio_device *vm_dev = opaque;
struct virtio_mmio_vq_info *info;
- struct virtio_driver *vdrv = container_of(vm_dev->vdev.dev.driver,
- struct virtio_driver, driver);
unsigned long status;
unsigned long flags;
irqreturn_t ret = IRQ_NONE;
@@ -244,9 +242,8 @@ static irqreturn_t vm_interrupt(int irq, void *opaque)
status = readl(vm_dev->base + VIRTIO_MMIO_INTERRUPT_STATUS);
writel(status, vm_dev->base + VIRTIO_MMIO_INTERRUPT_ACK);
- if (unlikely(status & VIRTIO_MMIO_INT_CONFIG)
- && vdrv && vdrv->config_changed) {
- vdrv->config_changed(&vm_dev->vdev);
+ if (unlikely(status & VIRTIO_MMIO_INT_CONFIG)) {
+ virtio_config_changed(&vm_dev->vdev);
ret = IRQ_HANDLED;
}
diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
index add40d0..f39f4e7 100644
--- a/drivers/virtio/virtio_pci.c
+++ b/drivers/virtio/virtio_pci.c
@@ -211,12 +211,8 @@ static bool vp_notify(struct virtqueue *vq)
static irqreturn_t vp_config_changed(int irq, void *opaque)
{
struct virtio_pci_device *vp_dev = opaque;
- struct virtio_driver *drv;
- drv = container_of(vp_dev->vdev.dev.driver,
- struct virtio_driver, driver);
- if (drv && drv->config_changed)
- drv->config_changed(&vp_dev->vdev);
+ virtio_config_changed(&vp_dev->vdev);
return IRQ_HANDLED;
}
--
MST
^ permalink raw reply related
* [PATCH v4 03/25] virtio-pci: move freeze/restore to virtio core
From: Michael S. Tsirkin @ 2014-10-13 7:48 UTC (permalink / raw)
To: linux-kernel
Cc: linux-s390, kvm, linux-scsi, Christian Borntraeger, netdev,
virtualization, Paolo Bonzini, Amit Shah, v9fs-developer,
David S. Miller
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
This is in preparation to extending config changed event handling
in core.
Wrapping these in an API also seems to make for a cleaner code.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
include/linux/virtio.h | 6 +++++
drivers/virtio/virtio.c | 54 +++++++++++++++++++++++++++++++++++++++++++++
drivers/virtio/virtio_pci.c | 54 ++-------------------------------------------
3 files changed, 62 insertions(+), 52 deletions(-)
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index 3c19bd3..8df7ba8 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -78,6 +78,7 @@ bool virtqueue_is_broken(struct virtqueue *vq);
/**
* virtio_device - representation of a device using virtio
* @index: unique position on the virtio bus
+ * @failed: saved value for CONFIG_S_FAILED bit (for restore)
* @dev: underlying device.
* @id: the device type identification (used to match it with a driver).
* @config: the configuration ops for this device.
@@ -88,6 +89,7 @@ bool virtqueue_is_broken(struct virtqueue *vq);
*/
struct virtio_device {
int index;
+ bool failed;
struct device dev;
struct virtio_device_id id;
const struct virtio_config_ops *config;
@@ -109,6 +111,10 @@ void unregister_virtio_device(struct virtio_device *dev);
void virtio_break_device(struct virtio_device *dev);
void virtio_config_changed(struct virtio_device *dev);
+#ifdef CONFIG_PM_SLEEP
+int virtio_device_freeze(struct virtio_device *dev);
+int virtio_device_restore(struct virtio_device *dev);
+#endif
/**
* virtio_driver - operations for a virtio I/O driver
diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 3980687..8216b73 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -248,6 +248,60 @@ void virtio_config_changed(struct virtio_device *dev)
}
EXPORT_SYMBOL_GPL(virtio_config_changed);
+#ifdef CONFIG_PM_SLEEP
+int virtio_device_freeze(struct virtio_device *dev)
+{
+ struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
+
+ dev->failed = dev->config->get_status(dev) & VIRTIO_CONFIG_S_FAILED;
+
+ if (drv && drv->freeze)
+ return drv->freeze(dev);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(virtio_device_freeze);
+
+int virtio_device_restore(struct virtio_device *dev)
+{
+ struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
+
+ /* We always start by resetting the device, in case a previous
+ * driver messed it up. */
+ dev->config->reset(dev);
+
+ /* Acknowledge that we've seen the device. */
+ add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE);
+
+ /* Maybe driver failed before freeze.
+ * Restore the failed status, for debugging. */
+ if (dev->failed)
+ add_status(dev, VIRTIO_CONFIG_S_FAILED);
+
+ if (!drv)
+ return 0;
+
+ /* We have a driver! */
+ add_status(dev, VIRTIO_CONFIG_S_DRIVER);
+
+ dev->config->finalize_features(dev);
+
+ if (drv->restore) {
+ int ret = drv->restore(dev);
+ if (ret) {
+ add_status(dev, VIRTIO_CONFIG_S_FAILED);
+ return ret;
+ }
+ }
+
+ /* Finally, tell the device we're all set */
+ add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(virtio_device_restore);
+#endif
+
static int virtio_init(void)
{
if (bus_register(&virtio_bus) != 0)
diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
index f39f4e7..d34ebfa 100644
--- a/drivers/virtio/virtio_pci.c
+++ b/drivers/virtio/virtio_pci.c
@@ -57,9 +57,6 @@ struct virtio_pci_device
/* Vectors allocated, excluding per-vq vectors if any */
unsigned msix_used_vectors;
- /* Status saved during hibernate/restore */
- u8 saved_status;
-
/* Whether we have vector per vq */
bool per_vq_vectors;
};
@@ -764,16 +761,9 @@ static int virtio_pci_freeze(struct device *dev)
{
struct pci_dev *pci_dev = to_pci_dev(dev);
struct virtio_pci_device *vp_dev = pci_get_drvdata(pci_dev);
- struct virtio_driver *drv;
int ret;
- drv = container_of(vp_dev->vdev.dev.driver,
- struct virtio_driver, driver);
-
- ret = 0;
- vp_dev->saved_status = vp_get_status(&vp_dev->vdev);
- if (drv && drv->freeze)
- ret = drv->freeze(&vp_dev->vdev);
+ ret = virtio_device_freeze(&vp_dev->vdev);
if (!ret)
pci_disable_device(pci_dev);
@@ -784,54 +774,14 @@ static int virtio_pci_restore(struct device *dev)
{
struct pci_dev *pci_dev = to_pci_dev(dev);
struct virtio_pci_device *vp_dev = pci_get_drvdata(pci_dev);
- struct virtio_driver *drv;
- unsigned status = 0;
int ret;
- drv = container_of(vp_dev->vdev.dev.driver,
- struct virtio_driver, driver);
-
ret = pci_enable_device(pci_dev);
if (ret)
return ret;
pci_set_master(pci_dev);
- /* We always start by resetting the device, in case a previous
- * driver messed it up. */
- vp_reset(&vp_dev->vdev);
-
- /* Acknowledge that we've seen the device. */
- status |= VIRTIO_CONFIG_S_ACKNOWLEDGE;
- vp_set_status(&vp_dev->vdev, status);
-
- /* Maybe driver failed before freeze.
- * Restore the failed status, for debugging. */
- status |= vp_dev->saved_status & VIRTIO_CONFIG_S_FAILED;
- vp_set_status(&vp_dev->vdev, status);
-
- if (!drv)
- return 0;
-
- /* We have a driver! */
- status |= VIRTIO_CONFIG_S_DRIVER;
- vp_set_status(&vp_dev->vdev, status);
-
- vp_finalize_features(&vp_dev->vdev);
-
- if (drv->restore) {
- ret = drv->restore(&vp_dev->vdev);
- if (ret) {
- status |= VIRTIO_CONFIG_S_FAILED;
- vp_set_status(&vp_dev->vdev, status);
- return ret;
- }
- }
-
- /* Finally, tell the device we're all set */
- status |= VIRTIO_CONFIG_S_DRIVER_OK;
- vp_set_status(&vp_dev->vdev, status);
-
- return ret;
+ return virtio_device_restore(&vp_dev->vdev);
}
static const struct dev_pm_ops virtio_pci_pm_ops = {
--
MST
^ permalink raw reply related
* [PATCH v4 04/25] virtio: defer config changed notifications
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: linux-s390, kvm, linux-scsi, Christian Borntraeger, netdev,
virtualization, Paolo Bonzini, Amit Shah, v9fs-developer,
David S. Miller
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
Defer config changed notifications that arrive during
probe/scan/freeze/restore.
This will allow drivers to set DRIVER_OK earlier, without worrying about
racing with config change interrupts.
This change will also benefit old hypervisors (before 2009)
that send interrupts without checking DRIVER_OK: previously,
the callback could race with driver-specific initialization.
This will also help simplify drivers.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
include/linux/virtio.h | 6 ++++++
drivers/virtio/virtio.c | 57 +++++++++++++++++++++++++++++++++++++++++--------
2 files changed, 54 insertions(+), 9 deletions(-)
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index 8df7ba8..5636b11 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -79,6 +79,9 @@ bool virtqueue_is_broken(struct virtqueue *vq);
* virtio_device - representation of a device using virtio
* @index: unique position on the virtio bus
* @failed: saved value for CONFIG_S_FAILED bit (for restore)
+ * @config_enabled: configuration change reporting enabled
+ * @config_changed: configuration change reported while disabled
+ * @config_lock: protects configuration change reporting
* @dev: underlying device.
* @id: the device type identification (used to match it with a driver).
* @config: the configuration ops for this device.
@@ -90,6 +93,9 @@ bool virtqueue_is_broken(struct virtqueue *vq);
struct virtio_device {
int index;
bool failed;
+ bool config_enabled;
+ bool config_changed;
+ spinlock_t config_lock;
struct device dev;
struct virtio_device_id id;
const struct virtio_config_ops *config;
diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 8216b73..2536701 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -117,6 +117,42 @@ void virtio_check_driver_offered_feature(const struct virtio_device *vdev,
}
EXPORT_SYMBOL_GPL(virtio_check_driver_offered_feature);
+static void __virtio_config_changed(struct virtio_device *dev)
+{
+ struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
+
+ if (!dev->config_enabled)
+ dev->config_changed = true;
+ else if (drv && drv->config_changed)
+ drv->config_changed(dev);
+}
+
+void virtio_config_changed(struct virtio_device *dev)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&dev->config_lock, flags);
+ __virtio_config_changed(dev);
+ spin_unlock_irqrestore(&dev->config_lock, flags);
+}
+EXPORT_SYMBOL_GPL(virtio_config_changed);
+
+static void virtio_config_disable(struct virtio_device *dev)
+{
+ spin_lock_irq(&dev->config_lock);
+ dev->config_enabled = false;
+ spin_unlock_irq(&dev->config_lock);
+}
+
+static void virtio_config_enable(struct virtio_device *dev)
+{
+ spin_lock_irq(&dev->config_lock);
+ dev->config_enabled = true;
+ __virtio_config_changed(dev);
+ dev->config_changed = false;
+ spin_unlock_irq(&dev->config_lock);
+}
+
static int virtio_dev_probe(struct device *_d)
{
int err, i;
@@ -153,6 +189,8 @@ static int virtio_dev_probe(struct device *_d)
add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK);
if (drv->scan)
drv->scan(dev);
+
+ virtio_config_enable(dev);
}
return err;
@@ -163,6 +201,8 @@ static int virtio_dev_remove(struct device *_d)
struct virtio_device *dev = dev_to_virtio(_d);
struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
+ virtio_config_disable(dev);
+
drv->remove(dev);
/* Driver should have reset device. */
@@ -211,6 +251,10 @@ int register_virtio_device(struct virtio_device *dev)
dev->index = err;
dev_set_name(&dev->dev, "virtio%u", dev->index);
+ spin_lock_init(&dev->config_lock);
+ dev->config_enabled = false;
+ dev->config_changed = false;
+
/* We always start by resetting the device, in case a previous
* driver messed it up. This also tests that code path a little. */
dev->config->reset(dev);
@@ -239,20 +283,13 @@ void unregister_virtio_device(struct virtio_device *dev)
}
EXPORT_SYMBOL_GPL(unregister_virtio_device);
-void virtio_config_changed(struct virtio_device *dev)
-{
- struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
-
- if (drv && drv->config_changed)
- drv->config_changed(dev);
-}
-EXPORT_SYMBOL_GPL(virtio_config_changed);
-
#ifdef CONFIG_PM_SLEEP
int virtio_device_freeze(struct virtio_device *dev)
{
struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
+ virtio_config_disable(dev);
+
dev->failed = dev->config->get_status(dev) & VIRTIO_CONFIG_S_FAILED;
if (drv && drv->freeze)
@@ -297,6 +334,8 @@ int virtio_device_restore(struct virtio_device *dev)
/* Finally, tell the device we're all set */
add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK);
+ virtio_config_enable(dev);
+
return 0;
}
EXPORT_SYMBOL_GPL(virtio_device_restore);
--
MST
^ permalink raw reply related
* [PATCH v4 05/25] virtio_blk: drop config_enable
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: linux-s390, kvm, linux-scsi, Christian Borntraeger, netdev,
virtualization, Paolo Bonzini, Amit Shah, v9fs-developer,
David S. Miller
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
Now that virtio core ensures config changes don't
arrive during probing, drop config_enable flag
in virtio blk.
On removal, flush is now sufficient to guarantee that
no change work is queued.
This help simplify the driver, and will allow
setting DRIVER_OK earlier without losing config
change notifications.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
drivers/block/virtio_blk.c | 23 ++++-------------------
1 file changed, 4 insertions(+), 19 deletions(-)
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 0a58140..91272f1a 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -44,9 +44,6 @@ struct virtio_blk
/* Lock for config space updates */
struct mutex config_lock;
- /* enable config space updates */
- bool config_enable;
-
/* What host tells us, plus 2 for header & tailer. */
unsigned int sg_elems;
@@ -348,8 +345,6 @@ static void virtblk_config_changed_work(struct work_struct *work)
u64 capacity, size;
mutex_lock(&vblk->config_lock);
- if (!vblk->config_enable)
- goto done;
/* Host must always specify the capacity. */
virtio_cread(vdev, struct virtio_blk_config, capacity, &capacity);
@@ -374,7 +369,7 @@ static void virtblk_config_changed_work(struct work_struct *work)
set_capacity(vblk->disk, capacity);
revalidate_disk(vblk->disk);
kobject_uevent_env(&disk_to_dev(vblk->disk)->kobj, KOBJ_CHANGE, envp);
-done:
+
mutex_unlock(&vblk->config_lock);
}
@@ -609,7 +604,6 @@ static int virtblk_probe(struct virtio_device *vdev)
mutex_init(&vblk->config_lock);
INIT_WORK(&vblk->config_work, virtblk_config_changed_work);
- vblk->config_enable = true;
err = init_vq(vblk);
if (err)
@@ -771,10 +765,8 @@ static void virtblk_remove(struct virtio_device *vdev)
int index = vblk->index;
int refc;
- /* Prevent config work handler from accessing the device. */
- mutex_lock(&vblk->config_lock);
- vblk->config_enable = false;
- mutex_unlock(&vblk->config_lock);
+ /* Make sure no work handler is accessing the device. */
+ flush_work(&vblk->config_work);
del_gendisk(vblk->disk);
blk_cleanup_queue(vblk->disk->queue);
@@ -784,8 +776,6 @@ static void virtblk_remove(struct virtio_device *vdev)
/* Stop all the virtqueues. */
vdev->config->reset(vdev);
- flush_work(&vblk->config_work);
-
refc = atomic_read(&disk_to_dev(vblk->disk)->kobj.kref.refcount);
put_disk(vblk->disk);
vdev->config->del_vqs(vdev);
@@ -805,11 +795,7 @@ static int virtblk_freeze(struct virtio_device *vdev)
/* Ensure we don't receive any more interrupts */
vdev->config->reset(vdev);
- /* Prevent config work handler from accessing the device. */
- mutex_lock(&vblk->config_lock);
- vblk->config_enable = false;
- mutex_unlock(&vblk->config_lock);
-
+ /* Make sure no work handler is accessing the device. */
flush_work(&vblk->config_work);
blk_mq_stop_hw_queues(vblk->disk->queue);
@@ -823,7 +809,6 @@ static int virtblk_restore(struct virtio_device *vdev)
struct virtio_blk *vblk = vdev->priv;
int ret;
- vblk->config_enable = true;
ret = init_vq(vdev->priv);
if (!ret)
blk_mq_start_stopped_hw_queues(vblk->disk->queue, true);
--
MST
^ permalink raw reply related
* [PATCH v4 06/25] virtio-blk: drop config_mutex
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: linux-s390, kvm, linux-scsi, Christian Borntraeger, netdev,
virtualization, Paolo Bonzini, Amit Shah, v9fs-developer,
David S. Miller
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
config_mutex served two purposes: prevent multiple concurrent config
change handlers, and synchronize access to config_enable flag.
Since commit dbf2576e37da0fcc7aacbfbb9fd5d3de7888a3c1
workqueue: make all workqueues non-reentrant
all workqueues are non-reentrant, and config_enable
is now gone.
Get rid of the unnecessary lock.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
drivers/block/virtio_blk.c | 8 --------
1 file changed, 8 deletions(-)
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 91272f1a..89ba8d6 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -41,9 +41,6 @@ struct virtio_blk
/* Process context for config space updates */
struct work_struct config_work;
- /* Lock for config space updates */
- struct mutex config_lock;
-
/* What host tells us, plus 2 for header & tailer. */
unsigned int sg_elems;
@@ -344,8 +341,6 @@ static void virtblk_config_changed_work(struct work_struct *work)
char *envp[] = { "RESIZE=1", NULL };
u64 capacity, size;
- mutex_lock(&vblk->config_lock);
-
/* Host must always specify the capacity. */
virtio_cread(vdev, struct virtio_blk_config, capacity, &capacity);
@@ -369,8 +364,6 @@ static void virtblk_config_changed_work(struct work_struct *work)
set_capacity(vblk->disk, capacity);
revalidate_disk(vblk->disk);
kobject_uevent_env(&disk_to_dev(vblk->disk)->kobj, KOBJ_CHANGE, envp);
-
- mutex_unlock(&vblk->config_lock);
}
static void virtblk_config_changed(struct virtio_device *vdev)
@@ -601,7 +594,6 @@ static int virtblk_probe(struct virtio_device *vdev)
vblk->vdev = vdev;
vblk->sg_elems = sg_elems;
- mutex_init(&vblk->config_lock);
INIT_WORK(&vblk->config_work, virtblk_config_changed_work);
--
MST
^ permalink raw reply related
* [PATCH v4 07/25] virtio_net: drop config_enable
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: linux-s390, kvm, linux-scsi, Christian Borntraeger, netdev,
virtualization, Paolo Bonzini, Amit Shah, v9fs-developer,
David S. Miller
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
Now that virtio core ensures config changes don't arrive during probing,
drop config_enable flag in virtio net.
On removal, flush is now sufficient to guarantee that no change work is
queued.
This help simplify the driver, and will allow setting DRIVER_OK earlier
without losing config change notifications.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
drivers/net/virtio_net.c | 27 ++++-----------------------
1 file changed, 4 insertions(+), 23 deletions(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 59caa06..743fb04 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -123,9 +123,6 @@ struct virtnet_info {
/* Host can handle any s/g split between our header and packet data */
bool any_header_sg;
- /* enable config space updates */
- bool config_enable;
-
/* Active statistics */
struct virtnet_stats __percpu *stats;
@@ -1408,9 +1405,6 @@ static void virtnet_config_changed_work(struct work_struct *work)
u16 v;
mutex_lock(&vi->config_lock);
- if (!vi->config_enable)
- goto done;
-
if (virtio_cread_feature(vi->vdev, VIRTIO_NET_F_STATUS,
struct virtio_net_config, status, &v) < 0)
goto done;
@@ -1758,7 +1752,6 @@ static int virtnet_probe(struct virtio_device *vdev)
}
mutex_init(&vi->config_lock);
- vi->config_enable = true;
INIT_WORK(&vi->config_work, virtnet_config_changed_work);
/* If we can receive ANY GSO packets, we must allocate large ones. */
@@ -1875,17 +1868,13 @@ static void virtnet_remove(struct virtio_device *vdev)
unregister_hotcpu_notifier(&vi->nb);
- /* Prevent config work handler from accessing the device. */
- mutex_lock(&vi->config_lock);
- vi->config_enable = false;
- mutex_unlock(&vi->config_lock);
+ /* Make sure no work handler is accessing the device. */
+ flush_work(&vi->config_work);
unregister_netdev(vi->dev);
remove_vq_common(vi);
- flush_work(&vi->config_work);
-
free_percpu(vi->stats);
free_netdev(vi->dev);
}
@@ -1898,10 +1887,8 @@ static int virtnet_freeze(struct virtio_device *vdev)
unregister_hotcpu_notifier(&vi->nb);
- /* Prevent config work handler from accessing the device */
- mutex_lock(&vi->config_lock);
- vi->config_enable = false;
- mutex_unlock(&vi->config_lock);
+ /* Make sure no work handler is accessing the device */
+ flush_work(&vi->config_work);
netif_device_detach(vi->dev);
cancel_delayed_work_sync(&vi->refill);
@@ -1916,8 +1903,6 @@ static int virtnet_freeze(struct virtio_device *vdev)
remove_vq_common(vi);
- flush_work(&vi->config_work);
-
return 0;
}
@@ -1941,10 +1926,6 @@ static int virtnet_restore(struct virtio_device *vdev)
netif_device_attach(vi->dev);
- mutex_lock(&vi->config_lock);
- vi->config_enable = true;
- mutex_unlock(&vi->config_lock);
-
rtnl_lock();
virtnet_set_queues(vi, vi->curr_queue_pairs);
rtnl_unlock();
--
MST
^ permalink raw reply related
* [PATCH v4 08/25] virtio-net: drop config_mutex
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: Rusty Russell, virtualization, linux-scsi, linux-s390,
v9fs-developer, netdev, kvm, Amit Shah, Cornelia Huck,
Christian Borntraeger, David S. Miller, Paolo Bonzini
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
config_mutex served two purposes: prevent multiple concurrent config
change handlers, and synchronize access to config_enable flag.
Since commit dbf2576e37da0fcc7aacbfbb9fd5d3de7888a3c1
workqueue: make all workqueues non-reentrant
all workqueues are non-reentrant, and config_enable
is now gone.
Get rid of the unnecessary lock.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
drivers/net/virtio_net.c | 7 +------
1 file changed, 1 insertion(+), 6 deletions(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 743fb04..23e4a69 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -132,9 +132,6 @@ struct virtnet_info {
/* Work struct for config space updates */
struct work_struct config_work;
- /* Lock for config space updates */
- struct mutex config_lock;
-
/* Does the affinity hint is set for virtqueues? */
bool affinity_hint_set;
@@ -1404,7 +1401,6 @@ static void virtnet_config_changed_work(struct work_struct *work)
container_of(work, struct virtnet_info, config_work);
u16 v;
- mutex_lock(&vi->config_lock);
if (virtio_cread_feature(vi->vdev, VIRTIO_NET_F_STATUS,
struct virtio_net_config, status, &v) < 0)
goto done;
@@ -1430,7 +1426,7 @@ static void virtnet_config_changed_work(struct work_struct *work)
netif_tx_stop_all_queues(vi->dev);
}
done:
- mutex_unlock(&vi->config_lock);
+ return;
}
static void virtnet_config_changed(struct virtio_device *vdev)
@@ -1751,7 +1747,6 @@ static int virtnet_probe(struct virtio_device *vdev)
u64_stats_init(&virtnet_stats->rx_syncp);
}
- mutex_init(&vi->config_lock);
INIT_WORK(&vi->config_work, virtnet_config_changed_work);
/* If we can receive ANY GSO packets, we must allocate large ones. */
--
MST
^ permalink raw reply related
* [PATCH v4 09/25] virtio_net: minor cleanup
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: linux-s390, kvm, linux-scsi, Christian Borntraeger, netdev,
virtualization, Paolo Bonzini, Amit Shah, v9fs-developer,
David S. Miller
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
goto done;
done:
return;
is ugly, it was put there to make diff review easier.
replace by open-coded return.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Acked-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
drivers/net/virtio_net.c | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 23e4a69..ef04d23 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1403,7 +1403,7 @@ static void virtnet_config_changed_work(struct work_struct *work)
if (virtio_cread_feature(vi->vdev, VIRTIO_NET_F_STATUS,
struct virtio_net_config, status, &v) < 0)
- goto done;
+ return;
if (v & VIRTIO_NET_S_ANNOUNCE) {
netdev_notify_peers(vi->dev);
@@ -1414,7 +1414,7 @@ static void virtnet_config_changed_work(struct work_struct *work)
v &= VIRTIO_NET_S_LINK_UP;
if (vi->status == v)
- goto done;
+ return;
vi->status = v;
@@ -1425,8 +1425,6 @@ static void virtnet_config_changed_work(struct work_struct *work)
netif_carrier_off(vi->dev);
netif_tx_stop_all_queues(vi->dev);
}
-done:
- return;
}
static void virtnet_config_changed(struct virtio_device *vdev)
--
MST
^ permalink raw reply related
* [PATCH v4 10/25] virtio: add API to enable VQs early
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: Rusty Russell, virtualization, linux-scsi, linux-s390,
v9fs-developer, netdev, kvm, Amit Shah, Cornelia Huck,
Christian Borntraeger, David S. Miller, Paolo Bonzini
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
virtio spec 0.9.X requires DRIVER_OK to be set before
VQs are used, but some drivers use VQs before probe
function returns.
Since DRIVER_OK is set after probe, this violates the spec.
Even though under virtio 1.0 transitional devices support this
behaviour, we want to make it possible for those early callers to become
spec compliant and eventually support non-transitional devices.
Add API for drivers to call before using VQs.
Sets DRIVER_OK internally.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
include/linux/virtio_config.h | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index e8f8f71..e36403b 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -109,6 +109,23 @@ struct virtqueue *virtio_find_single_vq(struct virtio_device *vdev,
return vq;
}
+/**
+ * virtio_device_ready - enable vq use in probe function
+ * @vdev: the device
+ *
+ * Driver must call this to use vqs in the probe function.
+ *
+ * Note: vqs are enabled automatically after probe returns.
+ */
+static inline
+void virtio_device_ready(struct virtio_device *dev)
+{
+ unsigned status = dev->config->get_status(dev);
+
+ BUG_ON(status & VIRTIO_CONFIG_S_DRIVER_OK);
+ dev->config->set_status(dev, status | VIRTIO_CONFIG_S_DRIVER_OK);
+}
+
static inline
const char *virtio_bus_name(struct virtio_device *vdev)
{
--
MST
^ permalink raw reply related
* [PATCH v4 11/25] virtio_net: enable VQs early
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: linux-s390, kvm, linux-scsi, Christian Borntraeger, netdev,
virtualization, Paolo Bonzini, Amit Shah, v9fs-developer,
David S. Miller
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
virtio spec requires drivers to set DRIVER_OK before using VQs.
This is set automatically after probe returns, virtio net violated this
rule by using receive VQs within probe.
To fix, call virtio_device_ready before using VQs.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
drivers/net/virtio_net.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index ef04d23..430f3ae 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1792,6 +1792,8 @@ static int virtnet_probe(struct virtio_device *vdev)
goto free_vqs;
}
+ virtio_device_ready(vdev);
+
/* Last of all, set up some receive buffers. */
for (i = 0; i < vi->curr_queue_pairs; i++) {
try_fill_recv(&vi->rq[i], GFP_KERNEL);
--
MST
^ permalink raw reply related
* [PATCH v4 12/25] virtio_blk: enable VQs early
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: Rusty Russell, virtualization, linux-scsi, linux-s390,
v9fs-developer, netdev, kvm, Amit Shah, Cornelia Huck,
Christian Borntraeger, David S. Miller, Paolo Bonzini
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
virtio spec requires drivers to set DRIVER_OK before using VQs.
This is set automatically after probe returns, virtio block violated this
rule by calling add_disk, which causes the VQ to be used directly within
probe.
To fix, call virtio_device_ready before using VQs.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
---
drivers/block/virtio_blk.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 89ba8d6..46b04bf 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -719,6 +719,8 @@ static int virtblk_probe(struct virtio_device *vdev)
if (!err && opt_io_size)
blk_queue_io_opt(q, blk_size * opt_io_size);
+ virtio_device_ready(vdev);
+
add_disk(vblk->disk);
err = device_create_file(disk_to_dev(vblk->disk), &dev_attr_serial);
if (err)
--
MST
^ permalink raw reply related
* [PATCH v4 13/25] virtio_console: enable VQs early
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: Rusty Russell, virtualization, linux-scsi, linux-s390,
v9fs-developer, netdev, kvm, Amit Shah, Cornelia Huck,
Christian Borntraeger, David S. Miller, Paolo Bonzini,
Arnd Bergmann, Greg Kroah-Hartman
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
virtio spec requires drivers to set DRIVER_OK before using VQs.
This is set automatically after probe returns, virtio console violated this
rule by adding inbufs, which causes the VQ to be used directly within
probe.
To fix, call virtio_device_ready before using VQs.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
drivers/char/virtio_console.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index b585b47..6ebe8f6 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -1449,6 +1449,8 @@ static int add_port(struct ports_device *portdev, u32 id)
spin_lock_init(&port->outvq_lock);
init_waitqueue_head(&port->waitqueue);
+ virtio_device_ready(portdev->vdev);
+
/* Fill the in_vq with buffers so the host can send us data. */
nr_added_bufs = fill_queue(port->in_vq, &port->inbuf_lock);
if (!nr_added_bufs) {
--
MST
^ permalink raw reply related
* [PATCH v4 14/25] 9p/trans_virtio: enable VQs early
From: Michael S. Tsirkin @ 2014-10-13 7:50 UTC (permalink / raw)
To: linux-kernel
Cc: linux-s390, Ron Minnich, kvm, linux-scsi, Christian Borntraeger,
Eric Van Hensbergen, netdev, virtualization, Paolo Bonzini,
Amit Shah, v9fs-developer, David S. Miller
In-Reply-To: <1413114332-626-1-git-send-email-mst-v4@redhat.com>
virtio spec requires drivers to set DRIVER_OK before using VQs.
This is set automatically after probe returns, but virtio 9p device
adds self to channel list within probe, at which point VQ can be
used in violation of the spec.
To fix, call virtio_device_ready before using VQs.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
net/9p/trans_virtio.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/net/9p/trans_virtio.c b/net/9p/trans_virtio.c
index 6940d8f..766ba48 100644
--- a/net/9p/trans_virtio.c
+++ b/net/9p/trans_virtio.c
@@ -575,6 +575,8 @@ static int p9_virtio_probe(struct virtio_device *vdev)
/* Ceiling limit to avoid denial of service attacks */
chan->p9_max_pages = nr_free_buffer_pages()/4;
+ virtio_device_ready(vdev);
+
mutex_lock(&virtio_9p_lock);
list_add_tail(&chan->chan_list, &virtio_chan_list);
mutex_unlock(&virtio_9p_lock);
--
MST
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox