linuxppc-dev.lists.ozlabs.org archive mirror
 help / color / mirror / Atom feed
* [PATCH kernel v3] vfio/spapr: Add cond_resched() for huge updates
@ 2017-09-28  9:16 Alexey Kardashevskiy
  2017-09-29  0:39 ` David Gibson
  2017-09-29 22:17 ` Alex Williamson
  0 siblings, 2 replies; 3+ messages in thread
From: Alexey Kardashevskiy @ 2017-09-28  9:16 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: Alexey Kardashevskiy, David Gibson, kvm-ppc, kvm, Alex Williamson,
	Nicholas Piggin

Clearing very big IOMMU tables can trigger soft lockups. This adds
cond_resched() to allow the scheduler to do context switching when
it decides to.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---

The testcase is POWER9 box with 264GB guest, 4 VFIO devices from
independent IOMMU groups, 64K IOMMU pages. This configuration produces
4325376 TCE entries, each entry update incurs 4 OPAL calls to update
an individual PE TCE cache; this produced lockups for more than 20s.
Reducing table size to 4194304 (i.e. 256GB guest) or removing one
of 4 VFIO devices makes the problem go away.

---
Changes:
v3:
* cond_resched() checks for should_resched() so we just call resched()
and let the cpu scheduler decide whether to switch or not

v2:
* replaced with time based solution
---
 drivers/vfio/vfio_iommu_spapr_tce.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 63112c36ab2d..759a5bdd40e1 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -507,6 +507,8 @@ static int tce_iommu_clear(struct tce_container *container,
 	enum dma_data_direction direction;
 
 	for ( ; pages; --pages, ++entry) {
+		cond_resched();
+
 		direction = DMA_NONE;
 		oldhpa = 0;
 		ret = iommu_tce_xchg(tbl, entry, &oldhpa, &direction);
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 3+ messages in thread

* Re: [PATCH kernel v3] vfio/spapr: Add cond_resched() for huge updates
  2017-09-28  9:16 [PATCH kernel v3] vfio/spapr: Add cond_resched() for huge updates Alexey Kardashevskiy
@ 2017-09-29  0:39 ` David Gibson
  2017-09-29 22:17 ` Alex Williamson
  1 sibling, 0 replies; 3+ messages in thread
From: David Gibson @ 2017-09-29  0:39 UTC (permalink / raw)
  To: Alexey Kardashevskiy
  Cc: linuxppc-dev, kvm-ppc, kvm, Alex Williamson, Nicholas Piggin

[-- Attachment #1: Type: text/plain, Size: 1793 bytes --]

On Thu, Sep 28, 2017 at 07:16:12PM +1000, Alexey Kardashevskiy wrote:
> Clearing very big IOMMU tables can trigger soft lockups. This adds
> cond_resched() to allow the scheduler to do context switching when
> it decides to.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

> ---
> 
> The testcase is POWER9 box with 264GB guest, 4 VFIO devices from
> independent IOMMU groups, 64K IOMMU pages. This configuration produces
> 4325376 TCE entries, each entry update incurs 4 OPAL calls to update
> an individual PE TCE cache; this produced lockups for more than 20s.
> Reducing table size to 4194304 (i.e. 256GB guest) or removing one
> of 4 VFIO devices makes the problem go away.
> 
> ---
> Changes:
> v3:
> * cond_resched() checks for should_resched() so we just call resched()
> and let the cpu scheduler decide whether to switch or not
> 
> v2:
> * replaced with time based solution
> ---
>  drivers/vfio/vfio_iommu_spapr_tce.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> index 63112c36ab2d..759a5bdd40e1 100644
> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> @@ -507,6 +507,8 @@ static int tce_iommu_clear(struct tce_container *container,
>  	enum dma_data_direction direction;
>  
>  	for ( ; pages; --pages, ++entry) {
> +		cond_resched();
> +
>  		direction = DMA_NONE;
>  		oldhpa = 0;
>  		ret = iommu_tce_xchg(tbl, entry, &oldhpa, &direction);

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH kernel v3] vfio/spapr: Add cond_resched() for huge updates
  2017-09-28  9:16 [PATCH kernel v3] vfio/spapr: Add cond_resched() for huge updates Alexey Kardashevskiy
  2017-09-29  0:39 ` David Gibson
@ 2017-09-29 22:17 ` Alex Williamson
  1 sibling, 0 replies; 3+ messages in thread
From: Alex Williamson @ 2017-09-29 22:17 UTC (permalink / raw)
  To: Alexey Kardashevskiy
  Cc: linuxppc-dev, David Gibson, kvm-ppc, kvm, Nicholas Piggin

On Thu, 28 Sep 2017 19:16:12 +1000
Alexey Kardashevskiy <aik@ozlabs.ru> wrote:

> Clearing very big IOMMU tables can trigger soft lockups. This adds
> cond_resched() to allow the scheduler to do context switching when
> it decides to.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
> 
> The testcase is POWER9 box with 264GB guest, 4 VFIO devices from
> independent IOMMU groups, 64K IOMMU pages. This configuration produces
> 4325376 TCE entries, each entry update incurs 4 OPAL calls to update
> an individual PE TCE cache; this produced lockups for more than 20s.
> Reducing table size to 4194304 (i.e. 256GB guest) or removing one
> of 4 VFIO devices makes the problem go away.
> 
> ---
> Changes:
> v3:
> * cond_resched() checks for should_resched() so we just call resched()
> and let the cpu scheduler decide whether to switch or not
> 
> v2:
> * replaced with time based solution
> ---
>  drivers/vfio/vfio_iommu_spapr_tce.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> index 63112c36ab2d..759a5bdd40e1 100644
> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> @@ -507,6 +507,8 @@ static int tce_iommu_clear(struct tce_container *container,
>  	enum dma_data_direction direction;
>  
>  	for ( ; pages; --pages, ++entry) {
> +		cond_resched();
> +
>  		direction = DMA_NONE;
>  		oldhpa = 0;
>  		ret = iommu_tce_xchg(tbl, entry, &oldhpa, &direction);

This looks fine to me, I've applied it to my local next branch for
v4.15.  I'll push that branch next week, once I can rebase to
4.14-rc3.  Thanks,

Alex

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2017-09-29 22:17 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-28  9:16 [PATCH kernel v3] vfio/spapr: Add cond_resched() for huge updates Alexey Kardashevskiy
2017-09-29  0:39 ` David Gibson
2017-09-29 22:17 ` Alex Williamson

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).