* Re: [PATCH v4 2/7] KVM: arm64: clean up redundant 'kvm_run' parameters
From: Marc Zyngier @ 2020-05-05 8:39 UTC (permalink / raw)
To: Tianjia Zhang
Cc: wanpengli, kvm, david, heiko.carstens, peterx, linux-mips, hpa,
kvmarm, linux-s390, frankja, chenhuacai, joro, x86, borntraeger,
mingo, julien.thierry.kdev, thuth, gor, suzuki.poulose, kvm-ppc,
bp, tglx, linux-arm-kernel, jmattson, tsbogend, cohuck,
christoffer.dall, sean.j.christopherson, linux-kernel,
james.morse, pbonzini, vkuznets, linuxppc-dev
In-Reply-To: <20200427043514.16144-3-tianjia.zhang@linux.alibaba.com>
Hi Tianjia,
On 2020-04-27 05:35, Tianjia Zhang wrote:
> In the current kvm version, 'kvm_run' has been included in the
> 'kvm_vcpu'
> structure. For historical reasons, many kvm-related function parameters
> retain the 'kvm_run' and 'kvm_vcpu' parameters at the same time. This
> patch does a unified cleanup of these remaining redundant parameters.
>
> Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
On the face of it, this looks OK, but I haven't tried to run the
resulting kernel. I'm not opposed to taking this patch *if* there
is an agreement across architectures to take the series (I value
consistency over the janitorial exercise).
Another thing is that this is going to conflict with the set of
patches that move the KVM/arm code back where it belongs
(arch/arm64/kvm),
so I'd probably cherry-pick that one directly.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply
* Re: [PATCH v7 25/28] powerpc: Test prefixed instructions in feature fixups
From: Jordan Niethe @ 2020-05-05 7:34 UTC (permalink / raw)
To: Alistair Popple
Cc: Nicholas Piggin, Balamuruhan S, naveen.n.rao, linuxppc-dev,
Daniel Axtens
In-Reply-To: <5803808.3aLiq0rt1P@townsend>
On Tue, May 5, 2020 at 5:15 PM Alistair Popple <alistair@popple.id.au> wrote:
>
> Hmm, I was hoping to add a tested by but I'm seeing the following failure in
> Mambo:
>
> [ 1.475459] feature-fixups: test failed at line 730
>
> Based on the name of the test it looks like you probably made a copy/paste
> error in ftr_fixup_prefix2_expected. I suspect you probably meant to use the alt
> fixup:
>
> globl(ftr_fixup_prefix2_expected)
> or 1,1,1
> .long 0x7000000
> .long 0x0000001
> or 2,2,2
Thanks, I changed from using 0x7000000 to 1 << 26 but missed here.
Changing that fixes this.
>
> Also for some reason these tests (and one of the code-patching tests) aren't
> passing on big endian.
Okay, will fix that.
>
> - Alistair
>
> On Friday, 1 May 2020 1:42:17 PM AEST Jordan Niethe wrote:
> > Expand the feature-fixups self-tests to includes tests for prefixed
> > instructions.
> >
> > Signed-off-by: Jordan Niethe <jniethe5@gmail.com>
> > ---
> > v6: New to series
> > ---
> > arch/powerpc/lib/feature-fixups-test.S | 68 +++++++++++++++++++++++
> > arch/powerpc/lib/feature-fixups.c | 74 ++++++++++++++++++++++++++
> > 2 files changed, 142 insertions(+)
> >
> > diff --git a/arch/powerpc/lib/feature-fixups-test.S
> > b/arch/powerpc/lib/feature-fixups-test.S index b12168c2447a..6e2da9123a9b
> > 100644
> > --- a/arch/powerpc/lib/feature-fixups-test.S
> > +++ b/arch/powerpc/lib/feature-fixups-test.S
> > @@ -791,3 +791,71 @@ globl(lwsync_fixup_test_expected_SYNC)
> > 1: or 1,1,1
> > sync
> >
> > +globl(ftr_fixup_prefix1)
> > + or 1,1,1
> > + .long 1 << 26
> > + .long 0x0000000
> > + or 2,2,2
> > +globl(end_ftr_fixup_prefix1)
> > +
> > +globl(ftr_fixup_prefix1_orig)
> > + or 1,1,1
> > + .long 1 << 26
> > + .long 0x0000000
> > + or 2,2,2
> > +
> > +globl(ftr_fixup_prefix1_expected)
> > + or 1,1,1
> > + nop
> > + nop
> > + or 2,2,2
> > +
> > +globl(ftr_fixup_prefix2)
> > + or 1,1,1
> > + .long 1 << 26
> > + .long 0x0000000
> > + or 2,2,2
> > +globl(end_ftr_fixup_prefix2)
> > +
> > +globl(ftr_fixup_prefix2_orig)
> > + or 1,1,1
> > + .long 1 << 26
> > + .long 0x0000000
> > + or 2,2,2
> > +
> > +globl(ftr_fixup_prefix2_alt)
> > + .long 0x7000000
> > + .long 0x0000001
> > +
> > +globl(ftr_fixup_prefix2_expected)
> > + or 1,1,1
> > + .long 1 << 26
> > + .long 0x0000001
> > + or 2,2,2
> > +
> > +globl(ftr_fixup_prefix3)
> > + or 1,1,1
> > + .long 1 << 26
> > + .long 0x0000000
> > + or 2,2,2
> > + or 3,3,3
> > +globl(end_ftr_fixup_prefix3)
> > +
> > +globl(ftr_fixup_prefix3_orig)
> > + or 1,1,1
> > + .long 1 << 26
> > + .long 0x0000000
> > + or 2,2,2
> > + or 3,3,3
> > +
> > +globl(ftr_fixup_prefix3_alt)
> > + .long 1 << 26
> > + .long 0x0000001
> > + nop
> > +
> > +globl(ftr_fixup_prefix3_expected)
> > + or 1,1,1
> > + .long 1 << 26
> > + .long 0x0000001
> > + nop
> > + or 3,3,3
> > diff --git a/arch/powerpc/lib/feature-fixups.c
> > b/arch/powerpc/lib/feature-fixups.c index 243011f85287..6fc499b1d63e 100644
> > --- a/arch/powerpc/lib/feature-fixups.c
> > +++ b/arch/powerpc/lib/feature-fixups.c
> > @@ -687,6 +687,75 @@ static void test_lwsync_macros(void)
> > }
> > }
> >
> > +#ifdef __powerpc64__
> > +static void __init test_prefix_patching(void)
> > +{
> > + extern unsigned int ftr_fixup_prefix1[];
> > + extern unsigned int end_ftr_fixup_prefix1[];
> > + extern unsigned int ftr_fixup_prefix1_orig[];
> > + extern unsigned int ftr_fixup_prefix1_expected[];
> > + int size = sizeof(unsigned int) * (end_ftr_fixup_prefix1 -
> > ftr_fixup_prefix1); +
> > + fixup.value = fixup.mask = 8;
> > + fixup.start_off = calc_offset(&fixup, ftr_fixup_prefix1 + 1);
> > + fixup.end_off = calc_offset(&fixup, ftr_fixup_prefix1 + 3);
> > + fixup.alt_start_off = fixup.alt_end_off = 0;
> > +
> > + /* Sanity check */
> > + check(memcmp(ftr_fixup_prefix1, ftr_fixup_prefix1_orig, size) == 0);
> > +
> > + patch_feature_section(0, &fixup);
> > + check(memcmp(ftr_fixup_prefix1, ftr_fixup_prefix1_expected, size) == 0);
> > + check(memcmp(ftr_fixup_prefix1, ftr_fixup_prefix1_orig, size) != 0);
> > +}
> > +
> > +static void __init test_prefix_alt_patching(void)
> > +{
> > + extern unsigned int ftr_fixup_prefix2[];
> > + extern unsigned int end_ftr_fixup_prefix2[];
> > + extern unsigned int ftr_fixup_prefix2_orig[];
> > + extern unsigned int ftr_fixup_prefix2_expected[];
> > + extern unsigned int ftr_fixup_prefix2_alt[];
> > + int size = sizeof(unsigned int) * (end_ftr_fixup_prefix2 -
> > ftr_fixup_prefix2); +
> > + fixup.value = fixup.mask = 8;
> > + fixup.start_off = calc_offset(&fixup, ftr_fixup_prefix2 + 1);
> > + fixup.end_off = calc_offset(&fixup, ftr_fixup_prefix2 + 3);
> > + fixup.alt_start_off = calc_offset(&fixup, ftr_fixup_prefix2_alt);
> > + fixup.alt_end_off = calc_offset(&fixup, ftr_fixup_prefix2_alt + 2);
> > + /* Sanity check */
> > + check(memcmp(ftr_fixup_prefix2, ftr_fixup_prefix2_orig, size) == 0);
> > +
> > + patch_feature_section(0, &fixup);
> > + check(memcmp(ftr_fixup_prefix2, ftr_fixup_prefix2_expected, size) == 0);
> > + patch_feature_section(0, &fixup);
> > + check(memcmp(ftr_fixup_prefix2, ftr_fixup_prefix2_orig, size) != 0);
> > +}
> > +
> > +static void __init test_prefix_word_alt_patching(void)
> > +{
> > + extern unsigned int ftr_fixup_prefix3[];
> > + extern unsigned int end_ftr_fixup_prefix3[];
> > + extern unsigned int ftr_fixup_prefix3_orig[];
> > + extern unsigned int ftr_fixup_prefix3_expected[];
> > + extern unsigned int ftr_fixup_prefix3_alt[];
> > + int size = sizeof(unsigned int) * (end_ftr_fixup_prefix3 -
> > ftr_fixup_prefix3); +
> > + fixup.value = fixup.mask = 8;
> > + fixup.start_off = calc_offset(&fixup, ftr_fixup_prefix3 + 1);
> > + fixup.end_off = calc_offset(&fixup, ftr_fixup_prefix3 + 4);
> > + fixup.alt_start_off = calc_offset(&fixup, ftr_fixup_prefix3_alt);
> > + fixup.alt_end_off = calc_offset(&fixup, ftr_fixup_prefix3_alt + 3);
> > + /* Sanity check */
> > + check(memcmp(ftr_fixup_prefix3, ftr_fixup_prefix3_orig, size) == 0);
> > +
> > + patch_feature_section(0, &fixup);
> > + check(memcmp(ftr_fixup_prefix3, ftr_fixup_prefix3_expected, size) == 0);
> > + patch_feature_section(0, &fixup);
> > + check(memcmp(ftr_fixup_prefix3, ftr_fixup_prefix3_orig, size) != 0);
> > +}
> > +#endif /* __powerpc64__ */
> > +
> > static int __init test_feature_fixups(void)
> > {
> > printk(KERN_DEBUG "Running feature fixup self-tests ...\n");
> > @@ -701,6 +770,11 @@ static int __init test_feature_fixups(void)
> > test_cpu_macros();
> > test_fw_macros();
> > test_lwsync_macros();
> > +#ifdef __powerpc64__
> > + test_prefix_patching();
> > + test_prefix_alt_patching();
> > + test_prefix_word_alt_patching();
> > +#endif
> >
> > return 0;
> > }
>
>
>
>
^ permalink raw reply
* Re: [PATCH v7 04/28] powerpc/xmon: Use bitwise calculations in_breakpoint_table()
From: Jordan Niethe @ 2020-05-05 7:31 UTC (permalink / raw)
To: Michael Ellerman
Cc: Alistair Popple, Nicholas Piggin, Balamuruhan S, naveen.n.rao,
linuxppc-dev, Daniel Axtens
In-Reply-To: <871rnyeu4t.fsf@mpe.ellerman.id.au>
On Tue, May 5, 2020 at 5:08 PM Michael Ellerman <mpe@ellerman.id.au> wrote:
>
> Jordan Niethe <jniethe5@gmail.com> writes:
> > A modulo operation is used for calculating the current offset from a
> > breakpoint within the breakpoint table. As instruction lengths are
> > always a power of 2, this can be replaced with a bitwise 'and'. The
> > current check for word alignment can be replaced with checking that the
> > lower 2 bits are not set.
> >
> > Suggested-by: Christophe Leroy <christophe.leroy@c-s.fr>
> > Signed-off-by: Jordan Niethe <jniethe5@gmail.com>
> > ---
> > v6: New to series
> > ---
> > arch/powerpc/xmon/xmon.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
> > index bbfea22f4a96..e122f0c8a044 100644
> > --- a/arch/powerpc/xmon/xmon.c
> > +++ b/arch/powerpc/xmon/xmon.c
> > @@ -857,8 +857,8 @@ static struct bpt *in_breakpoint_table(unsigned long nip, unsigned long *offp)
> > off = nip - (unsigned long) bpt_table;
> > if (off >= sizeof(bpt_table))
> > return NULL;
> > - *offp = off % BPT_SIZE;
> > - if (*offp != 0 && *offp != 4)
> > + *offp = off & (BPT_SIZE - 1);
> > + if (off & 3)
> > return NULL;
>
> It would be even better if you didn't hard code the 3 wouldn't it?
>
The three is just checking word alignment, which I think was the
intention of the previous
- if (*offp != 0 && *offp != 4)
But using BPT_SIZE is is a different calculation.
BPT_SIZE == 2 * sizeof(unsigned int) == 8
Which would mean the trap of the breakpoint pair of instructions would
return NULL.
> eg:
>
> + *offp = off & (BPT_SIZE - 1);
> + if (off & (BPT_SIZE - 1))
> return NULL;
>
> cheers
^ permalink raw reply
* Re: [PATCH 1/2] powerpc/spufs: fix copy_to_user while atomic
From: Christoph Hellwig @ 2020-05-05 7:30 UTC (permalink / raw)
To: Michael Ellerman
Cc: linuxppc-dev, Christoph Hellwig, Arnd Bergmann, Jeremy Kerr
In-Reply-To: <87y2q6dezt.fsf@mpe.ellerman.id.au>
On Tue, May 05, 2020 at 05:20:54PM +1000, Michael Ellerman wrote:
> Christoph Hellwig <hch@lst.de> writes:
> > powerpc mantainers,
>
> There's only one of me.
>
> > are you going to pick this up for the next -rc1? I'm waiting for it to
> > hit upstream before resending the coredump series.
>
> I thought you were going to take it in your series.
Ok, I'll pick it up.
^ permalink raw reply
* Re: [PATCH 1/2] powerpc/spufs: fix copy_to_user while atomic
From: Michael Ellerman @ 2020-05-05 7:20 UTC (permalink / raw)
To: Christoph Hellwig, Jeremy Kerr
Cc: linuxppc-dev, Christoph Hellwig, Arnd Bergmann
In-Reply-To: <20200504130443.GA18125@lst.de>
Christoph Hellwig <hch@lst.de> writes:
> powerpc mantainers,
There's only one of me.
> are you going to pick this up for the next -rc1? I'm waiting for it to
> hit upstream before resending the coredump series.
I thought you were going to take it in your series.
Otherwise you'll be waiting 4 or more weeks before this hits rc1.
I can put it in a topic branch if you're worried about merge conflicts.
There's also the fcheck() RCU fix I need to repost as a proper patch, it
seems to work.
cheers
> On Wed, Apr 29, 2020 at 03:03:02PM +0800, Jeremy Kerr wrote:
>> Currently, we may perform a copy_to_user (through
>> simple_read_from_buffer()) while holding a context's register_lock,
>> while accessing the context save area.
>>
>> This change uses a temporary buffer for the context save area data,
>> which we then pass to simple_read_from_buffer.
>>
>> Includes changes from Christoph Hellwig <hch@lst.de>.
>>
>> Fixes: bf1ab978be23 ("[POWERPC] coredump: Add SPU elf notes to coredump.")
>> Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
>> Reviewed-by: Arnd Bergmann <arnd@arndb.de>
>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>> ---
>> arch/powerpc/platforms/cell/spufs/file.c | 113 +++++++++++++++--------
>> 1 file changed, 75 insertions(+), 38 deletions(-)
>>
>> diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
>> index c0f950a3f4e1..b4e1ef650b40 100644
>> --- a/arch/powerpc/platforms/cell/spufs/file.c
>> +++ b/arch/powerpc/platforms/cell/spufs/file.c
>> @@ -1978,8 +1978,9 @@ static ssize_t __spufs_mbox_info_read(struct spu_context *ctx,
>> static ssize_t spufs_mbox_info_read(struct file *file, char __user *buf,
>> size_t len, loff_t *pos)
>> {
>> - int ret;
>> struct spu_context *ctx = file->private_data;
>> + u32 stat, data;
>> + int ret;
>>
>> if (!access_ok(buf, len))
>> return -EFAULT;
>> @@ -1988,11 +1989,16 @@ static ssize_t spufs_mbox_info_read(struct file *file, char __user *buf,
>> if (ret)
>> return ret;
>> spin_lock(&ctx->csa.register_lock);
>> - ret = __spufs_mbox_info_read(ctx, buf, len, pos);
>> + stat = ctx->csa.prob.mb_stat_R;
>> + data = ctx->csa.prob.pu_mb_R;
>> spin_unlock(&ctx->csa.register_lock);
>> spu_release_saved(ctx);
>>
>> - return ret;
>> + /* EOF if there's no entry in the mbox */
>> + if (!(stat & 0x0000ff))
>> + return 0;
>> +
>> + return simple_read_from_buffer(buf, len, pos, &data, sizeof(data));
>> }
>>
>> static const struct file_operations spufs_mbox_info_fops = {
>> @@ -2019,6 +2025,7 @@ static ssize_t spufs_ibox_info_read(struct file *file, char __user *buf,
>> size_t len, loff_t *pos)
>> {
>> struct spu_context *ctx = file->private_data;
>> + u32 stat, data;
>> int ret;
>>
>> if (!access_ok(buf, len))
>> @@ -2028,11 +2035,16 @@ static ssize_t spufs_ibox_info_read(struct file *file, char __user *buf,
>> if (ret)
>> return ret;
>> spin_lock(&ctx->csa.register_lock);
>> - ret = __spufs_ibox_info_read(ctx, buf, len, pos);
>> + stat = ctx->csa.prob.mb_stat_R;
>> + data = ctx->csa.priv2.puint_mb_R;
>> spin_unlock(&ctx->csa.register_lock);
>> spu_release_saved(ctx);
>>
>> - return ret;
>> + /* EOF if there's no entry in the ibox */
>> + if (!(stat & 0xff0000))
>> + return 0;
>> +
>> + return simple_read_from_buffer(buf, len, pos, &data, sizeof(data));
>> }
>>
>> static const struct file_operations spufs_ibox_info_fops = {
>> @@ -2041,6 +2053,11 @@ static const struct file_operations spufs_ibox_info_fops = {
>> .llseek = generic_file_llseek,
>> };
>>
>> +static size_t spufs_wbox_info_cnt(struct spu_context *ctx)
>> +{
>> + return (4 - ((ctx->csa.prob.mb_stat_R & 0x00ff00) >> 8)) * sizeof(u32);
>> +}
>> +
>> static ssize_t __spufs_wbox_info_read(struct spu_context *ctx,
>> char __user *buf, size_t len, loff_t *pos)
>> {
>> @@ -2049,7 +2066,7 @@ static ssize_t __spufs_wbox_info_read(struct spu_context *ctx,
>> u32 wbox_stat;
>>
>> wbox_stat = ctx->csa.prob.mb_stat_R;
>> - cnt = 4 - ((wbox_stat & 0x00ff00) >> 8);
>> + cnt = spufs_wbox_info_cnt(ctx);
>> for (i = 0; i < cnt; i++) {
>> data[i] = ctx->csa.spu_mailbox_data[i];
>> }
>> @@ -2062,7 +2079,8 @@ static ssize_t spufs_wbox_info_read(struct file *file, char __user *buf,
>> size_t len, loff_t *pos)
>> {
>> struct spu_context *ctx = file->private_data;
>> - int ret;
>> + u32 data[ARRAY_SIZE(ctx->csa.spu_mailbox_data)];
>> + int ret, count;
>>
>> if (!access_ok(buf, len))
>> return -EFAULT;
>> @@ -2071,11 +2089,13 @@ static ssize_t spufs_wbox_info_read(struct file *file, char __user *buf,
>> if (ret)
>> return ret;
>> spin_lock(&ctx->csa.register_lock);
>> - ret = __spufs_wbox_info_read(ctx, buf, len, pos);
>> + count = spufs_wbox_info_cnt(ctx);
>> + memcpy(&data, &ctx->csa.spu_mailbox_data, sizeof(data));
>> spin_unlock(&ctx->csa.register_lock);
>> spu_release_saved(ctx);
>>
>> - return ret;
>> + return simple_read_from_buffer(buf, len, pos, &data,
>> + count * sizeof(u32));
>> }
>>
>> static const struct file_operations spufs_wbox_info_fops = {
>> @@ -2084,27 +2104,33 @@ static const struct file_operations spufs_wbox_info_fops = {
>> .llseek = generic_file_llseek,
>> };
>>
>> -static ssize_t __spufs_dma_info_read(struct spu_context *ctx,
>> - char __user *buf, size_t len, loff_t *pos)
>> +static void ___spufs_dma_info_read(struct spu_context *ctx,
>> + struct spu_dma_info *info)
>> {
>> - struct spu_dma_info info;
>> - struct mfc_cq_sr *qp, *spuqp;
>> int i;
>>
>> - info.dma_info_type = ctx->csa.priv2.spu_tag_status_query_RW;
>> - info.dma_info_mask = ctx->csa.lscsa->tag_mask.slot[0];
>> - info.dma_info_status = ctx->csa.spu_chnldata_RW[24];
>> - info.dma_info_stall_and_notify = ctx->csa.spu_chnldata_RW[25];
>> - info.dma_info_atomic_command_status = ctx->csa.spu_chnldata_RW[27];
>> + info->dma_info_type = ctx->csa.priv2.spu_tag_status_query_RW;
>> + info->dma_info_mask = ctx->csa.lscsa->tag_mask.slot[0];
>> + info->dma_info_status = ctx->csa.spu_chnldata_RW[24];
>> + info->dma_info_stall_and_notify = ctx->csa.spu_chnldata_RW[25];
>> + info->dma_info_atomic_command_status = ctx->csa.spu_chnldata_RW[27];
>> for (i = 0; i < 16; i++) {
>> - qp = &info.dma_info_command_data[i];
>> - spuqp = &ctx->csa.priv2.spuq[i];
>> + struct mfc_cq_sr *qp = &info->dma_info_command_data[i];
>> + struct mfc_cq_sr *spuqp = &ctx->csa.priv2.spuq[i];
>>
>> qp->mfc_cq_data0_RW = spuqp->mfc_cq_data0_RW;
>> qp->mfc_cq_data1_RW = spuqp->mfc_cq_data1_RW;
>> qp->mfc_cq_data2_RW = spuqp->mfc_cq_data2_RW;
>> qp->mfc_cq_data3_RW = spuqp->mfc_cq_data3_RW;
>> }
>> +}
>> +
>> +static ssize_t __spufs_dma_info_read(struct spu_context *ctx,
>> + char __user *buf, size_t len, loff_t *pos)
>> +{
>> + struct spu_dma_info info;
>> +
>> + ___spufs_dma_info_read(ctx, &info);
>>
>> return simple_read_from_buffer(buf, len, pos, &info,
>> sizeof info);
>> @@ -2114,6 +2140,7 @@ static ssize_t spufs_dma_info_read(struct file *file, char __user *buf,
>> size_t len, loff_t *pos)
>> {
>> struct spu_context *ctx = file->private_data;
>> + struct spu_dma_info info;
>> int ret;
>>
>> if (!access_ok(buf, len))
>> @@ -2123,11 +2150,12 @@ static ssize_t spufs_dma_info_read(struct file *file, char __user *buf,
>> if (ret)
>> return ret;
>> spin_lock(&ctx->csa.register_lock);
>> - ret = __spufs_dma_info_read(ctx, buf, len, pos);
>> + ___spufs_dma_info_read(ctx, &info);
>> spin_unlock(&ctx->csa.register_lock);
>> spu_release_saved(ctx);
>>
>> - return ret;
>> + return simple_read_from_buffer(buf, len, pos, &info,
>> + sizeof(info));
>> }
>>
>> static const struct file_operations spufs_dma_info_fops = {
>> @@ -2136,13 +2164,31 @@ static const struct file_operations spufs_dma_info_fops = {
>> .llseek = no_llseek,
>> };
>>
>> +static void ___spufs_proxydma_info_read(struct spu_context *ctx,
>> + struct spu_proxydma_info *info)
>> +{
>> + int i;
>> +
>> + info->proxydma_info_type = ctx->csa.prob.dma_querytype_RW;
>> + info->proxydma_info_mask = ctx->csa.prob.dma_querymask_RW;
>> + info->proxydma_info_status = ctx->csa.prob.dma_tagstatus_R;
>> +
>> + for (i = 0; i < 8; i++) {
>> + struct mfc_cq_sr *qp = &info->proxydma_info_command_data[i];
>> + struct mfc_cq_sr *puqp = &ctx->csa.priv2.puq[i];
>> +
>> + qp->mfc_cq_data0_RW = puqp->mfc_cq_data0_RW;
>> + qp->mfc_cq_data1_RW = puqp->mfc_cq_data1_RW;
>> + qp->mfc_cq_data2_RW = puqp->mfc_cq_data2_RW;
>> + qp->mfc_cq_data3_RW = puqp->mfc_cq_data3_RW;
>> + }
>> +}
>> +
>> static ssize_t __spufs_proxydma_info_read(struct spu_context *ctx,
>> char __user *buf, size_t len, loff_t *pos)
>> {
>> struct spu_proxydma_info info;
>> - struct mfc_cq_sr *qp, *puqp;
>> int ret = sizeof info;
>> - int i;
>>
>> if (len < ret)
>> return -EINVAL;
>> @@ -2150,18 +2196,7 @@ static ssize_t __spufs_proxydma_info_read(struct spu_context *ctx,
>> if (!access_ok(buf, len))
>> return -EFAULT;
>>
>> - info.proxydma_info_type = ctx->csa.prob.dma_querytype_RW;
>> - info.proxydma_info_mask = ctx->csa.prob.dma_querymask_RW;
>> - info.proxydma_info_status = ctx->csa.prob.dma_tagstatus_R;
>> - for (i = 0; i < 8; i++) {
>> - qp = &info.proxydma_info_command_data[i];
>> - puqp = &ctx->csa.priv2.puq[i];
>> -
>> - qp->mfc_cq_data0_RW = puqp->mfc_cq_data0_RW;
>> - qp->mfc_cq_data1_RW = puqp->mfc_cq_data1_RW;
>> - qp->mfc_cq_data2_RW = puqp->mfc_cq_data2_RW;
>> - qp->mfc_cq_data3_RW = puqp->mfc_cq_data3_RW;
>> - }
>> + ___spufs_proxydma_info_read(ctx, &info);
>>
>> return simple_read_from_buffer(buf, len, pos, &info,
>> sizeof info);
>> @@ -2171,17 +2206,19 @@ static ssize_t spufs_proxydma_info_read(struct file *file, char __user *buf,
>> size_t len, loff_t *pos)
>> {
>> struct spu_context *ctx = file->private_data;
>> + struct spu_proxydma_info info;
>> int ret;
>>
>> ret = spu_acquire_saved(ctx);
>> if (ret)
>> return ret;
>> spin_lock(&ctx->csa.register_lock);
>> - ret = __spufs_proxydma_info_read(ctx, buf, len, pos);
>> + ___spufs_proxydma_info_read(ctx, &info);
>> spin_unlock(&ctx->csa.register_lock);
>> spu_release_saved(ctx);
>>
>> - return ret;
>> + return simple_read_from_buffer(buf, len, pos, &info,
>> + sizeof(info));
>> }
>>
>> static const struct file_operations spufs_proxydma_info_fops = {
>> --
>> 2.17.1
> ---end quoted text---
^ permalink raw reply
* [PATCH v4 22/22] powerpc/mm/book3s64: Fix MADV_DONTNEED and parallel page fault race
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
MADV_DONTNEED holds mmap_sem in read mode and that implies a
parallel page fault is possible and the kernel can end up with a level 1 PTE
entry (THP entry) converted to a level 0 PTE entry without flushing
the THP TLB entry.
Most architectures including POWER have issues with kernel instantiating a level
0 PTE entry while holding level 1 TLB entries.
The code sequence I am looking at is
down_read(mmap_sem) down_read(mmap_sem)
zap_pmd_range()
zap_huge_pmd()
pmd lock held
pmd_cleared
table details added to mmu_gather
pmd_unlock()
insert a level 0 PTE entry()
tlb_finish_mmu().
Fix this by forcing a tlb flush before releasing pmd lock if this is
not a fullmm invalidate. We can safely skip this invalidate for
task exit case (fullmm invalidate) because in that case we are sure
there can be no parallel fault handlers.
This do change the Qemu guest RAM del/unplug time as below
128 core, 496GB guest:
Without patch:
munmap start: timer = 196449 ms, PID=6681
munmap finish: timer = 196488 ms, PID=6681 - delta = 39ms
With patch:
munmap start: timer = 196345 ms, PID=6879
munmap finish: timer = 196714 ms, PID=6879 - delta = 369ms
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/include/asm/book3s/64/pgtable.h | 5 +++++
arch/powerpc/mm/book3s64/pgtable.c | 18 ++++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index 03521a8b0292..e1f551159f7d 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -1265,6 +1265,11 @@ static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
}
#define pmdp_collapse_flush pmdp_collapse_flush
+#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL
+pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
+ unsigned long addr,
+ pmd_t *pmdp, int full);
+
#define __HAVE_ARCH_PGTABLE_DEPOSIT
static inline void pgtable_trans_huge_deposit(struct mm_struct *mm,
pmd_t *pmdp, pgtable_t pgtable)
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index 127325ead505..54b6d6d103ea 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -112,6 +112,24 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
return __pmd(old_pmd);
}
+pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmdp, int full)
+{
+ pmd_t pmd;
+ VM_BUG_ON(addr & ~HPAGE_PMD_MASK);
+ VM_BUG_ON((pmd_present(*pmdp) && !pmd_trans_huge(*pmdp) &&
+ !pmd_devmap(*pmdp)) || !pmd_present(*pmdp));
+ pmd = pmdp_huge_get_and_clear(vma->vm_mm, addr, pmdp);
+ /*
+ * if it not a fullmm flush, then we can possibly end up converting
+ * this PMD pte entry to a regular level 0 PTE by a parallel page fault.
+ * Make sure we flush the tlb in this case.
+ */
+ if (!full)
+ flush_pmd_tlb_range(vma, addr, addr + HPAGE_PMD_SIZE);
+ return pmd;
+}
+
static pmd_t pmd_set_protbits(pmd_t pmd, pgprot_t pgprot)
{
return __pmd(pmd_val(pmd) | pgprot_val(pgprot));
--
2.26.2
^ permalink raw reply related
* [PATCH v4 21/22] mm: change pmdp_huge_get_and_clear_full take vm_area_struct as arg
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, akpm, npiggin, kirill
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
We will use this in later patch to do tlb flush when clearing pmd entries.
Cc: kirill@shutemov.name
Cc: akpm@linux-foundation.org
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/s390/include/asm/pgtable.h | 4 ++--
include/asm-generic/pgtable.h | 4 ++--
mm/huge_memory.c | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 6076c8c912d2..e2528e057980 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -1560,7 +1560,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
}
#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL
-static inline pmd_t pmdp_huge_get_and_clear_full(struct mm_struct *mm,
+static inline pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
unsigned long addr,
pmd_t *pmdp, int full)
{
@@ -1569,7 +1569,7 @@ static inline pmd_t pmdp_huge_get_and_clear_full(struct mm_struct *mm,
*pmdp = __pmd(_SEGMENT_ENTRY_EMPTY);
return pmd;
}
- return pmdp_xchg_lazy(mm, addr, pmdp, __pmd(_SEGMENT_ENTRY_EMPTY));
+ return pmdp_xchg_lazy(vma->vm_mm, addr, pmdp, __pmd(_SEGMENT_ENTRY_EMPTY));
}
#define __HAVE_ARCH_PMDP_HUGE_CLEAR_FLUSH
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index 329b8c8ca703..d10be362eafa 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -159,11 +159,11 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL
-static inline pmd_t pmdp_huge_get_and_clear_full(struct mm_struct *mm,
+static inline pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
unsigned long address, pmd_t *pmdp,
int full)
{
- return pmdp_huge_get_and_clear(mm, address, pmdp);
+ return pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
}
#endif
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 6ecd1045113b..16f2bd6f1549 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1852,8 +1852,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
* pgtable_trans_huge_withdraw after finishing pmdp related
* operations.
*/
- orig_pmd = pmdp_huge_get_and_clear_full(tlb->mm, addr, pmd,
- tlb->fullmm);
+ orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd,
+ tlb->fullmm);
tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
if (vma_is_special_huge(vma)) {
if (arch_needs_pgtable_deposit())
--
2.26.2
^ permalink raw reply related
* [PATCH v4 20/22] powerpc/mm/book3s64: Avoid sending IPI on clearing PMD
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
Now that all the lockless page table walk is careful w.r.t the PTE
address returned, we can now revert
commit: 13bd817bb884 ("powerpc/thp: Serialize pmd clear against a linux page table walk.")
We also drop the equivalent IPI from other pte updates routines. We still keep
IPI in hash pmdp collapse and that is to take care of parallel hash page table
insert. The radix pmdp collapse flush can possibly be removed once I am sure
generic code doesn't have the any expectations around parallel gup walk.
This speeds up Qemu guest RAM del/unplug time as below
128 core, 496GB guest:
Without patch:
munmap start: timer = 13162 ms, PID=7684
munmap finish: timer = 95312 ms, PID=7684 - delta = 82150 ms
With patch:
munmap start: timer = 196449 ms, PID=6681
munmap finish: timer = 196488 ms, PID=6681 - delta = 39ms
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/mm/book3s64/hash_pgtable.c | 11 -----------
arch/powerpc/mm/book3s64/pgtable.c | 8 --------
arch/powerpc/mm/book3s64/radix_pgtable.c | 19 +++++++------------
3 files changed, 7 insertions(+), 31 deletions(-)
diff --git a/arch/powerpc/mm/book3s64/hash_pgtable.c b/arch/powerpc/mm/book3s64/hash_pgtable.c
index 64733b9cb20a..64ca375278dc 100644
--- a/arch/powerpc/mm/book3s64/hash_pgtable.c
+++ b/arch/powerpc/mm/book3s64/hash_pgtable.c
@@ -363,17 +363,6 @@ pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm,
* hash fault look at them.
*/
memset(pgtable, 0, PTE_FRAG_SIZE);
- /*
- * Serialize against find_current_mm_pte variants which does lock-less
- * lookup in page tables with local interrupts disabled. For huge pages
- * it casts pmd_t to pte_t. Since format of pte_t is different from
- * pmd_t we want to prevent transit from pmd pointing to page table
- * to pmd pointing to huge page (and back) while interrupts are disabled.
- * We clear pmd to possibly replace it with page table pointer in
- * different code paths. So make sure we wait for the parallel
- * find_curren_mm_pte to finish.
- */
- serialize_against_pte_lookup(mm);
return old_pmd;
}
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index e0bb69c616e4..127325ead505 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -109,14 +109,6 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
old_pmd = pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, _PAGE_INVALID);
flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
- /*
- * This ensures that generic code that rely on IRQ disabling
- * to prevent a parallel THP split work as expected.
- *
- * Marking the entry with _PAGE_INVALID && ~_PAGE_PRESENT requires
- * a special case check in pmd_access_permitted.
- */
- serialize_against_pte_lookup(vma->vm_mm);
return __pmd(old_pmd);
}
diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
index 8f9edf07063a..dfb9fe92aea8 100644
--- a/arch/powerpc/mm/book3s64/radix_pgtable.c
+++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
@@ -962,7 +962,13 @@ pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long addre
pmd = *pmdp;
pmd_clear(pmdp);
- /*FIXME!! Verify whether we need this kick below */
+ /*
+ * pmdp collapse_flush need to ensure that there are no parallel gup
+ * walk after this call. This is needed so that we can have stable
+ * page ref count when collapsing a page. We don't allow a collapse page
+ * if we have gup taken on the page. We can ensure that by sending IPI
+ * because gup walk happens with IRQ disabled.
+ */
serialize_against_pte_lookup(vma->vm_mm);
radix__flush_tlb_collapsed_pmd(vma->vm_mm, address);
@@ -1023,17 +1029,6 @@ pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm,
old = radix__pmd_hugepage_update(mm, addr, pmdp, ~0UL, 0);
old_pmd = __pmd(old);
- /*
- * Serialize against find_current_mm_pte which does lock-less
- * lookup in page tables with local interrupts disabled. For huge pages
- * it casts pmd_t to pte_t. Since format of pte_t is different from
- * pmd_t we want to prevent transit from pmd pointing to page table
- * to pmd pointing to huge page (and back) while interrupts are disabled.
- * We clear pmd to possibly replace it with page table pointer in
- * different code paths. So make sure we wait for the parallel
- * find_current_mm_pte to finish.
- */
- serialize_against_pte_lookup(mm);
return old_pmd;
}
--
2.26.2
^ permalink raw reply related
* [PATCH v4 19/22] powerpc/kvm/book3s: Use pte_present instead of opencoding _PAGE_PRESENT check
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
This adds _PAGE_PTE check and makes sure we validate the pte value returned via
find_kvm_host_pte.
NOTE: this also considers _PAGE_INVALID to the software valid bit.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/include/asm/kvm_book3s_64.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h
index 1ca1f6495012..c58e64a0a74f 100644
--- a/arch/powerpc/include/asm/kvm_book3s_64.h
+++ b/arch/powerpc/include/asm/kvm_book3s_64.h
@@ -435,7 +435,7 @@ static inline pte_t kvmppc_read_update_linux_pte(pte_t *ptep, int writing)
continue;
}
/* If pte is not present return None */
- if (unlikely(!(pte_val(old_pte) & _PAGE_PRESENT)))
+ if (unlikely(!pte_present(old_pte)))
return __pte(0);
new_pte = pte_mkyoung(old_pte);
--
2.26.2
^ permalink raw reply related
* [PATCH v4 18/22] powerpc/kvm/book3s: Use find_kvm_host_pte in kvmppc_get_hpa
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kvm/book3s_hv_rm_mmu.c | 32 ++++++++++-------------------
1 file changed, 11 insertions(+), 21 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_hv_rm_mmu.c b/arch/powerpc/kvm/book3s_hv_rm_mmu.c
index 83e987fecf97..3b168c69d503 100644
--- a/arch/powerpc/kvm/book3s_hv_rm_mmu.c
+++ b/arch/powerpc/kvm/book3s_hv_rm_mmu.c
@@ -878,8 +878,8 @@ long kvmppc_h_clear_mod(struct kvm_vcpu *vcpu, unsigned long flags,
return ret;
}
-static int kvmppc_get_hpa(struct kvm_vcpu *vcpu, unsigned long gpa,
- int writing, unsigned long *hpa,
+static int kvmppc_get_hpa(struct kvm_vcpu *vcpu, unsigned long mmu_seq,
+ unsigned long gpa, int writing, unsigned long *hpa,
struct kvm_memory_slot **memslot_p)
{
struct kvm *kvm = vcpu->kvm;
@@ -898,7 +898,7 @@ static int kvmppc_get_hpa(struct kvm_vcpu *vcpu, unsigned long gpa,
hva = __gfn_to_hva_memslot(memslot, gfn);
/* Try to find the host pte for that virtual address */
- ptep = __find_linux_pte(vcpu->arch.pgdir, hva, NULL, &shift);
+ ptep = find_kvm_host_pte(kvm, mmu_seq, hva, &shift);
if (!ptep)
return H_TOO_HARD;
pte = kvmppc_read_update_linux_pte(ptep, writing);
@@ -933,16 +933,11 @@ static long kvmppc_do_h_page_init_zero(struct kvm_vcpu *vcpu,
mmu_seq = kvm->mmu_notifier_seq;
smp_rmb();
- ret = kvmppc_get_hpa(vcpu, dest, 1, &pa, &memslot);
- if (ret != H_SUCCESS)
- return ret;
-
- /* Check if we've been invalidated */
arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
- if (mmu_notifier_retry(kvm, mmu_seq)) {
- ret = H_TOO_HARD;
+
+ ret = kvmppc_get_hpa(vcpu, mmu_seq, dest, 1, &pa, &memslot);
+ if (ret != H_SUCCESS)
goto out_unlock;
- }
/* Zero the page */
for (i = 0; i < SZ_4K; i += L1_CACHE_BYTES, pa += L1_CACHE_BYTES)
@@ -966,19 +961,14 @@ static long kvmppc_do_h_page_init_copy(struct kvm_vcpu *vcpu,
mmu_seq = kvm->mmu_notifier_seq;
smp_rmb();
- ret = kvmppc_get_hpa(vcpu, dest, 1, &dest_pa, &dest_memslot);
- if (ret != H_SUCCESS)
- return ret;
- ret = kvmppc_get_hpa(vcpu, src, 0, &src_pa, NULL);
+ arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
+ ret = kvmppc_get_hpa(vcpu, mmu_seq, dest, 1, &dest_pa, &dest_memslot);
if (ret != H_SUCCESS)
- return ret;
+ goto out_unlock;
- /* Check if we've been invalidated */
- arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
- if (mmu_notifier_retry(kvm, mmu_seq)) {
- ret = H_TOO_HARD;
+ ret = kvmppc_get_hpa(vcpu, mmu_seq, src, 0, &src_pa, NULL);
+ if (ret != H_SUCCESS)
goto out_unlock;
- }
/* Copy the page */
memcpy((void *)dest_pa, (void *)src_pa, SZ_4K);
--
2.26.2
^ permalink raw reply related
* [PATCH v4 17/22] powerpc/kvm/book3s: use find_kvm_host_pte in kvmppc_book3s_instantiate_page
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kvm/book3s_64_mmu_radix.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c
index 70c4025406d8..271f1c3d8443 100644
--- a/arch/powerpc/kvm/book3s_64_mmu_radix.c
+++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c
@@ -813,12 +813,12 @@ int kvmppc_book3s_instantiate_page(struct kvm_vcpu *vcpu,
* Read the PTE from the process' radix tree and use that
* so we get the shift and attribute bits.
*/
- local_irq_disable();
- ptep = __find_linux_pte(vcpu->arch.pgdir, hva, NULL, &shift);
+ spin_lock(&kvm->mmu_lock);
+ ptep = find_kvm_host_pte(kvm, mmu_seq, hva, &shift);
pte = __pte(0);
if (ptep)
- pte = *ptep;
- local_irq_enable();
+ pte = READ_ONCE(*ptep);
+ spin_unlock(&kvm->mmu_lock);
/*
* If the PTE disappeared temporarily due to a THP
* collapse, just return and let the guest try again.
--
2.26.2
^ permalink raw reply related
* [PATCH v4 16/22] powerpc/kvm/book3s: Avoid using rmap to protect parallel page table update.
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin, Alexey Kardashevskiy
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
We now depend on kvm->mmu_lock
Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kvm/book3s_64_vio_hv.c | 38 +++++++----------------------
1 file changed, 9 insertions(+), 29 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio_hv.c b/arch/powerpc/kvm/book3s_64_vio_hv.c
index acc3ce570be7..167029e57c8f 100644
--- a/arch/powerpc/kvm/book3s_64_vio_hv.c
+++ b/arch/powerpc/kvm/book3s_64_vio_hv.c
@@ -74,8 +74,8 @@ struct kvmppc_spapr_tce_table *kvmppc_find_table(struct kvm *kvm,
EXPORT_SYMBOL_GPL(kvmppc_find_table);
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
-static long kvmppc_rm_tce_to_ua(struct kvm *kvm, unsigned long tce,
- unsigned long *ua, unsigned long **prmap)
+static long kvmppc_rm_tce_to_ua(struct kvm *kvm,
+ unsigned long tce, unsigned long *ua)
{
unsigned long gfn = tce >> PAGE_SHIFT;
struct kvm_memory_slot *memslot;
@@ -87,9 +87,6 @@ static long kvmppc_rm_tce_to_ua(struct kvm *kvm, unsigned long tce,
*ua = __gfn_to_hva_memslot(memslot, gfn) |
(tce & ~(PAGE_MASK | TCE_PCI_READ | TCE_PCI_WRITE));
- if (prmap)
- *prmap = &memslot->arch.rmap[gfn - memslot->base_gfn];
-
return 0;
}
@@ -116,7 +113,7 @@ static long kvmppc_rm_tce_validate(struct kvmppc_spapr_tce_table *stt,
if (iommu_tce_check_gpa(stt->page_shift, gpa))
return H_PARAMETER;
- if (kvmppc_rm_tce_to_ua(stt->kvm, tce, &ua, NULL))
+ if (kvmppc_rm_tce_to_ua(stt->kvm, tce, &ua))
return H_TOO_HARD;
list_for_each_entry_lockless(stit, &stt->iommu_tables, next) {
@@ -411,7 +408,7 @@ long kvmppc_rm_h_put_tce(struct kvm_vcpu *vcpu, unsigned long liobn,
return ret;
dir = iommu_tce_direction(tce);
- if ((dir != DMA_NONE) && kvmppc_rm_tce_to_ua(vcpu->kvm, tce, &ua, NULL))
+ if ((dir != DMA_NONE) && kvmppc_rm_tce_to_ua(vcpu->kvm, tce, &ua))
return H_PARAMETER;
entry = ioba >> stt->page_shift;
@@ -488,7 +485,6 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,
struct kvmppc_spapr_tce_table *stt;
long i, ret = H_SUCCESS;
unsigned long tces, entry, ua = 0;
- unsigned long *rmap = NULL;
unsigned long mmu_seq;
bool prereg = false;
struct kvmppc_spapr_tce_iommu_table *stit;
@@ -530,7 +526,7 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,
*/
struct mm_iommu_table_group_mem_t *mem;
- if (kvmppc_rm_tce_to_ua(vcpu->kvm, tce_list, &ua, NULL))
+ if (kvmppc_rm_tce_to_ua(vcpu->kvm, tce_list, &ua))
return H_TOO_HARD;
mem = mm_iommu_lookup_rm(vcpu->kvm->mm, ua, IOMMU_PAGE_SIZE_4K);
@@ -546,23 +542,9 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,
* We do not require memory to be preregistered in this case
* so lock rmap and do __find_linux_pte_or_hugepte().
*/
- if (kvmppc_rm_tce_to_ua(vcpu->kvm, tce_list, &ua, &rmap))
- return H_TOO_HARD;
-
- rmap = (void *) vmalloc_to_phys(rmap);
- if (WARN_ON_ONCE_RM(!rmap))
+ if (kvmppc_rm_tce_to_ua(vcpu->kvm, tce_list, &ua))
return H_TOO_HARD;
- /*
- * Synchronize with the MMU notifier callbacks in
- * book3s_64_mmu_hv.c (kvm_unmap_hva_range_hv etc.).
- * While we have the rmap lock, code running on other CPUs
- * cannot finish unmapping the host real page that backs
- * this guest real page, so we are OK to access the host
- * real page.
- */
- lock_rmap(rmap);
-
arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
if (kvmppc_rm_ua_to_hpa(vcpu, mmu_seq, ua, &tces)) {
ret = H_TOO_HARD;
@@ -582,7 +564,7 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,
unsigned long tce = be64_to_cpu(((u64 *)tces)[i]);
ua = 0;
- if (kvmppc_rm_tce_to_ua(vcpu->kvm, tce, &ua, NULL)) {
+ if (kvmppc_rm_tce_to_ua(vcpu->kvm, tce, &ua)) {
ret = H_PARAMETER;
goto invalidate_exit;
}
@@ -607,10 +589,8 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,
iommu_tce_kill_rm(stit->tbl, entry, npages);
unlock_exit:
- if (rmap)
- unlock_rmap(rmap);
-
- arch_spin_unlock(&kvm->mmu_lock.rlock.raw_lock);
+ if (!prereg)
+ arch_spin_unlock(&kvm->mmu_lock.rlock.raw_lock);
return ret;
}
--
2.26.2
^ permalink raw reply related
* [PATCH v4 15/22] powerpc/kvm/book3s: use find_kvm_host_pte in pute_tce functions
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin, Alexey Kardashevskiy
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
Current code just hold rmap lock to ensure parallel page table update is
prevented. That is not sufficient. The kernel should also check whether
a mmu_notifer callback was running in parallel.
Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kvm/book3s_64_vio_hv.c | 30 +++++++++++++++++++++++------
1 file changed, 24 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_vio_hv.c b/arch/powerpc/kvm/book3s_64_vio_hv.c
index 6fcaf1fa8e02..acc3ce570be7 100644
--- a/arch/powerpc/kvm/book3s_64_vio_hv.c
+++ b/arch/powerpc/kvm/book3s_64_vio_hv.c
@@ -437,8 +437,8 @@ long kvmppc_rm_h_put_tce(struct kvm_vcpu *vcpu, unsigned long liobn,
return H_SUCCESS;
}
-static long kvmppc_rm_ua_to_hpa(struct kvm_vcpu *vcpu,
- unsigned long ua, unsigned long *phpa)
+static long kvmppc_rm_ua_to_hpa(struct kvm_vcpu *vcpu, unsigned long mmu_seq,
+ unsigned long ua, unsigned long *phpa)
{
pte_t *ptep, pte;
unsigned shift = 0;
@@ -452,10 +452,17 @@ static long kvmppc_rm_ua_to_hpa(struct kvm_vcpu *vcpu,
* to exit which will agains result in the below page table walk
* to finish.
*/
- ptep = __find_linux_pte(vcpu->arch.pgdir, ua, NULL, &shift);
- if (!ptep || !pte_present(*ptep))
+ /* an rmap lock won't make it safe. because that just ensure hash
+ * page table entries are removed with rmap lock held. After that
+ * mmu notifier returns and we go ahead and removing ptes from Qemu page table.
+ */
+ ptep = find_kvm_host_pte(vcpu->kvm, mmu_seq, ua, &shift);
+ if (!ptep)
+ return -ENXIO;
+
+ pte = READ_ONCE(*ptep);
+ if (!pte_present(pte))
return -ENXIO;
- pte = *ptep;
if (!shift)
shift = PAGE_SHIFT;
@@ -477,10 +484,12 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,
unsigned long liobn, unsigned long ioba,
unsigned long tce_list, unsigned long npages)
{
+ struct kvm *kvm = vcpu->kvm;
struct kvmppc_spapr_tce_table *stt;
long i, ret = H_SUCCESS;
unsigned long tces, entry, ua = 0;
unsigned long *rmap = NULL;
+ unsigned long mmu_seq;
bool prereg = false;
struct kvmppc_spapr_tce_iommu_table *stit;
@@ -488,6 +497,12 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,
if (kvm_is_radix(vcpu->kvm))
return H_TOO_HARD;
+ /*
+ * used to check for invalidations in progress
+ */
+ mmu_seq = kvm->mmu_notifier_seq;
+ smp_rmb();
+
stt = kvmppc_find_table(vcpu->kvm, liobn);
if (!stt)
return H_TOO_HARD;
@@ -547,7 +562,9 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,
* real page.
*/
lock_rmap(rmap);
- if (kvmppc_rm_ua_to_hpa(vcpu, ua, &tces)) {
+
+ arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
+ if (kvmppc_rm_ua_to_hpa(vcpu, mmu_seq, ua, &tces)) {
ret = H_TOO_HARD;
goto unlock_exit;
}
@@ -593,6 +610,7 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,
if (rmap)
unlock_rmap(rmap);
+ arch_spin_unlock(&kvm->mmu_lock.rlock.raw_lock);
return ret;
}
--
2.26.2
^ permalink raw reply related
* [PATCH v4 14/22] powerpc/kvm/book3s: Use find_kvm_host_pte in h_enter
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
Since kvmppc_do_h_enter can get called in realmode use low level
arch_spin_lock which is safe to be called in realmode.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kvm/book3s_64_mmu_hv.c | 5 ++---
arch/powerpc/kvm/book3s_hv_rm_mmu.c | 22 ++++++----------------
2 files changed, 8 insertions(+), 19 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_mmu_hv.c b/arch/powerpc/kvm/book3s_64_mmu_hv.c
index 38e934dc1714..8d9725445e4f 100644
--- a/arch/powerpc/kvm/book3s_64_mmu_hv.c
+++ b/arch/powerpc/kvm/book3s_64_mmu_hv.c
@@ -281,11 +281,10 @@ static long kvmppc_virtmode_do_h_enter(struct kvm *kvm, unsigned long flags,
{
long ret;
- /* Protect linux PTE lookup from page table destruction */
- rcu_read_lock_sched(); /* this disables preemption too */
+ preempt_disable();
ret = kvmppc_do_h_enter(kvm, flags, pte_index, pteh, ptel,
kvm->mm->pgd, false, pte_idx_ret);
- rcu_read_unlock_sched();
+ preempt_enable();
if (ret == H_TOO_HARD) {
/* this can't happen */
pr_err("KVM: Oops, kvmppc_h_enter returned too hard!\n");
diff --git a/arch/powerpc/kvm/book3s_hv_rm_mmu.c b/arch/powerpc/kvm/book3s_hv_rm_mmu.c
index 03f8347de48b..83e987fecf97 100644
--- a/arch/powerpc/kvm/book3s_hv_rm_mmu.c
+++ b/arch/powerpc/kvm/book3s_hv_rm_mmu.c
@@ -210,7 +210,7 @@ long kvmppc_do_h_enter(struct kvm *kvm, unsigned long flags,
pte_t *ptep;
unsigned int writing;
unsigned long mmu_seq;
- unsigned long rcbits, irq_flags = 0;
+ unsigned long rcbits;
if (kvm_is_radix(kvm))
return H_FUNCTION;
@@ -248,17 +248,9 @@ long kvmppc_do_h_enter(struct kvm *kvm, unsigned long flags,
/* Translate to host virtual address */
hva = __gfn_to_hva_memslot(memslot, gfn);
- /*
- * If we had a page table table change after lookup, we would
- * retry via mmu_notifier_retry.
- */
- if (!realmode)
- local_irq_save(irq_flags);
- /*
- * If called in real mode we have MSR_EE = 0. Otherwise
- * we disable irq above.
- */
- ptep = __find_linux_pte(pgdir, hva, NULL, &hpage_shift);
+
+ arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
+ ptep = find_kvm_host_pte(kvm, mmu_seq, hva, &hpage_shift);
if (ptep) {
pte_t pte;
unsigned int host_pte_size;
@@ -272,8 +264,7 @@ long kvmppc_do_h_enter(struct kvm *kvm, unsigned long flags,
* to <= host page size, if host is using hugepage
*/
if (host_pte_size < psize) {
- if (!realmode)
- local_irq_restore(flags);
+ arch_spin_unlock(&kvm->mmu_lock.rlock.raw_lock);
return H_PARAMETER;
}
pte = kvmppc_read_update_linux_pte(ptep, writing);
@@ -287,8 +278,7 @@ long kvmppc_do_h_enter(struct kvm *kvm, unsigned long flags,
pa |= gpa & ~PAGE_MASK;
}
}
- if (!realmode)
- local_irq_restore(irq_flags);
+ arch_spin_unlock(&kvm->mmu_lock.rlock.raw_lock);
ptel &= HPTE_R_KEY | HPTE_R_PP0 | (psize-1);
ptel |= pa;
--
2.26.2
^ permalink raw reply related
* [PATCH v4 13/22] powerpc/kvm/book3s: Use find_kvm_host_pte in page fault handler
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kvm/book3s_64_mmu_hv.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_mmu_hv.c b/arch/powerpc/kvm/book3s_64_mmu_hv.c
index 2b35f9bcf892..38e934dc1714 100644
--- a/arch/powerpc/kvm/book3s_64_mmu_hv.c
+++ b/arch/powerpc/kvm/book3s_64_mmu_hv.c
@@ -602,12 +602,12 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,
* Read the PTE from the process' radix tree and use that
* so we get the shift and attribute bits.
*/
- local_irq_disable();
- ptep = __find_linux_pte(vcpu->arch.pgdir, hva, NULL, &shift);
+ spin_lock(&kvm->mmu_lock);
+ ptep = find_kvm_host_pte(kvm, mmu_seq, hva, &shift)
pte = __pte(0);
if (ptep)
- pte = *ptep;
- local_irq_enable();
+ pte = READ_ONCE(*ptep);
+ spin_unlock(&kvm->mmu_lock);
/*
* If the PTE disappeared temporarily due to a THP
* collapse, just return and let the guest try again.
--
2.26.2
^ permalink raw reply related
* [PATCH v4 12/22] powerpc/kvm/book3s: Add helper for host page table walk
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/include/asm/kvm_book3s_64.h | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h
index 2860521992b6..1ca1f6495012 100644
--- a/arch/powerpc/include/asm/kvm_book3s_64.h
+++ b/arch/powerpc/include/asm/kvm_book3s_64.h
@@ -647,6 +647,22 @@ static inline pte_t *find_kvm_secondary_pte(struct kvm *kvm, unsigned long ea,
return pte;
}
+static inline pte_t *find_kvm_host_pte(struct kvm *kvm, unsigned long mmu_seq,
+ unsigned long ea, unsigned *hshift)
+{
+ pte_t *pte;
+
+ VM_WARN(!spin_is_locked(&kvm->mmu_lock),
+ "%s called with kvm mmu_lock not held \n", __func__);
+
+ if (mmu_notifier_retry(kvm, mmu_seq))
+ return NULL;
+
+ pte = __find_linux_pte(kvm->mm->pgd, ea, NULL, hshift);
+
+ return pte;
+}
+
extern pte_t *find_kvm_nested_guest_pte(struct kvm *kvm, unsigned long lpid,
unsigned long ea, unsigned *hshift);
--
2.26.2
^ permalink raw reply related
* [PATCH v4 11/22] powerpc/kvm/book3s: Use kvm helpers to walk shadow or secondary table
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
update kvmppc_hv_handle_set_rc to use find_kvm_nested_guest_pte and
find_kvm_secondary_pte
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/include/asm/kvm_book3s.h | 2 +-
arch/powerpc/include/asm/kvm_book3s_64.h | 3 +++
arch/powerpc/kvm/book3s_64_mmu_radix.c | 18 +++++++++---------
arch/powerpc/kvm/book3s_hv_nested.c | 13 ++++++-------
4 files changed, 19 insertions(+), 17 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_book3s.h b/arch/powerpc/include/asm/kvm_book3s.h
index 506e4df2d730..37c8b50cb505 100644
--- a/arch/powerpc/include/asm/kvm_book3s.h
+++ b/arch/powerpc/include/asm/kvm_book3s.h
@@ -198,7 +198,7 @@ extern void kvmppc_unmap_pte(struct kvm *kvm, pte_t *pte, unsigned long gpa,
unsigned int shift,
const struct kvm_memory_slot *memslot,
unsigned int lpid);
-extern bool kvmppc_hv_handle_set_rc(struct kvm *kvm, pgd_t *pgtable,
+extern bool kvmppc_hv_handle_set_rc(struct kvm *kvm, bool nested,
bool writing, unsigned long gpa,
unsigned int lpid);
extern int kvmppc_book3s_instantiate_page(struct kvm_vcpu *vcpu,
diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h
index 2c2635967d6e..2860521992b6 100644
--- a/arch/powerpc/include/asm/kvm_book3s_64.h
+++ b/arch/powerpc/include/asm/kvm_book3s_64.h
@@ -647,6 +647,9 @@ static inline pte_t *find_kvm_secondary_pte(struct kvm *kvm, unsigned long ea,
return pte;
}
+extern pte_t *find_kvm_nested_guest_pte(struct kvm *kvm, unsigned long lpid,
+ unsigned long ea, unsigned *hshift);
+
#endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */
#endif /* __ASM_KVM_BOOK3S_64_H__ */
diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c
index c92d413eeaaf..70c4025406d8 100644
--- a/arch/powerpc/kvm/book3s_64_mmu_radix.c
+++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c
@@ -735,7 +735,7 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,
return ret;
}
-bool kvmppc_hv_handle_set_rc(struct kvm *kvm, pgd_t *pgtable, bool writing,
+bool kvmppc_hv_handle_set_rc(struct kvm *kvm, bool nested, bool writing,
unsigned long gpa, unsigned int lpid)
{
unsigned long pgflags;
@@ -750,12 +750,12 @@ bool kvmppc_hv_handle_set_rc(struct kvm *kvm, pgd_t *pgtable, bool writing,
pgflags = _PAGE_ACCESSED;
if (writing)
pgflags |= _PAGE_DIRTY;
- /*
- * We are walking the secondary (partition-scoped) page table here.
- * We can do this without disabling irq because the Linux MM
- * subsystem doesn't do THP splits and collapses on this tree.
- */
- ptep = __find_linux_pte(pgtable, gpa, NULL, &shift);
+
+ if (nested)
+ ptep = find_kvm_nested_guest_pte(kvm, lpid, gpa, &shift);
+ else
+ ptep = find_kvm_secondary_pte(kvm, gpa, &shift);
+
if (ptep && pte_present(*ptep) && (!writing || pte_write(*ptep))) {
kvmppc_radix_update_pte(kvm, ptep, 0, pgflags, gpa, shift);
return true;
@@ -949,8 +949,8 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,
/* Failed to set the reference/change bits */
if (dsisr & DSISR_SET_RC) {
spin_lock(&kvm->mmu_lock);
- if (kvmppc_hv_handle_set_rc(kvm, kvm->arch.pgtable,
- writing, gpa, kvm->arch.lpid))
+ if (kvmppc_hv_handle_set_rc(kvm, false, writing,
+ gpa, kvm->arch.lpid))
dsisr &= ~DSISR_SET_RC;
spin_unlock(&kvm->mmu_lock);
diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
index b2cc3eaec618..99011f1b772a 100644
--- a/arch/powerpc/kvm/book3s_hv_nested.c
+++ b/arch/powerpc/kvm/book3s_hv_nested.c
@@ -750,8 +750,8 @@ static struct kvm_nested_guest *kvmhv_find_nested(struct kvm *kvm, int lpid)
return kvm->arch.nested_guests[lpid];
}
-static pte_t *find_kvm_nested_guest_pte(struct kvm *kvm, unsigned long lpid,
- unsigned long ea, unsigned *hshift)
+pte_t *find_kvm_nested_guest_pte(struct kvm *kvm, unsigned long lpid,
+ unsigned long ea, unsigned *hshift)
{
struct kvm_nested_guest *gp;
pte_t *pte;
@@ -767,7 +767,6 @@ static pte_t *find_kvm_nested_guest_pte(struct kvm *kvm, unsigned long lpid,
return pte;
}
-
static inline bool kvmhv_n_rmap_is_equal(u64 rmap_1, u64 rmap_2)
{
return !((rmap_1 ^ rmap_2) & (RMAP_NESTED_LPID_MASK |
@@ -1226,16 +1225,16 @@ static long kvmhv_handle_nested_set_rc(struct kvm_vcpu *vcpu,
spin_lock(&kvm->mmu_lock);
/* Set the rc bit in the pte of our (L0) pgtable for the L1 guest */
- ret = kvmppc_hv_handle_set_rc(kvm, kvm->arch.pgtable, writing,
- gpte.raddr, kvm->arch.lpid);
+ ret = kvmppc_hv_handle_set_rc(kvm, false, writing,
+ gpte.raddr, kvm->arch.lpid);
if (!ret) {
ret = -EINVAL;
goto out_unlock;
}
/* Set the rc bit in the pte of the shadow_pgtable for the nest guest */
- ret = kvmppc_hv_handle_set_rc(kvm, gp->shadow_pgtable, writing, n_gpa,
- gp->shadow_lpid);
+ ret = kvmppc_hv_handle_set_rc(kvm, true, writing,
+ n_gpa, gp->shadow_lpid);
if (!ret)
ret = -EINVAL;
else
--
2.26.2
^ permalink raw reply related
* [PATCH v4 10/22] powerpc/kvm/nested: Add helper to walk nested shadow linux page table.
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
The locking rules for walking nested shadow linux page table is different from process
scoped table. Hence add a helper for nested page table walk and also
add check whether we are holding the right locks.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kvm/book3s_hv_nested.c | 28 +++++++++++++++++++++-------
1 file changed, 21 insertions(+), 7 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
index 7f1fc5db13ea..b2cc3eaec618 100644
--- a/arch/powerpc/kvm/book3s_hv_nested.c
+++ b/arch/powerpc/kvm/book3s_hv_nested.c
@@ -750,6 +750,24 @@ static struct kvm_nested_guest *kvmhv_find_nested(struct kvm *kvm, int lpid)
return kvm->arch.nested_guests[lpid];
}
+static pte_t *find_kvm_nested_guest_pte(struct kvm *kvm, unsigned long lpid,
+ unsigned long ea, unsigned *hshift)
+{
+ struct kvm_nested_guest *gp;
+ pte_t *pte;
+
+ gp = kvmhv_find_nested(kvm, lpid);
+ if (!gp)
+ return NULL;
+
+ VM_WARN(!spin_is_locked(&kvm->mmu_lock),
+ "%s called with kvm mmu_lock not held \n", __func__);
+ pte = __find_linux_pte(gp->shadow_pgtable, ea, NULL, hshift);
+
+ return pte;
+}
+
+
static inline bool kvmhv_n_rmap_is_equal(u64 rmap_1, u64 rmap_2)
{
return !((rmap_1 ^ rmap_2) & (RMAP_NESTED_LPID_MASK |
@@ -792,19 +810,15 @@ static void kvmhv_update_nest_rmap_rc(struct kvm *kvm, u64 n_rmap,
unsigned long clr, unsigned long set,
unsigned long hpa, unsigned long mask)
{
- struct kvm_nested_guest *gp;
unsigned long gpa;
unsigned int shift, lpid;
pte_t *ptep;
gpa = n_rmap & RMAP_NESTED_GPA_MASK;
lpid = (n_rmap & RMAP_NESTED_LPID_MASK) >> RMAP_NESTED_LPID_SHIFT;
- gp = kvmhv_find_nested(kvm, lpid);
- if (!gp)
- return;
/* Find the pte */
- ptep = __find_linux_pte(gp->shadow_pgtable, gpa, NULL, &shift);
+ ptep = find_kvm_nested_guest_pte(kvm, lpid, gpa, &shift);
/*
* If the pte is present and the pfn is still the same, update the pte.
* If the pfn has changed then this is a stale rmap entry, the nested
@@ -854,7 +868,7 @@ static void kvmhv_remove_nest_rmap(struct kvm *kvm, u64 n_rmap,
return;
/* Find and invalidate the pte */
- ptep = __find_linux_pte(gp->shadow_pgtable, gpa, NULL, &shift);
+ ptep = find_kvm_nested_guest_pte(kvm, lpid, gpa, &shift);
/* Don't spuriously invalidate ptes if the pfn has changed */
if (ptep && pte_present(*ptep) && ((pte_val(*ptep) & mask) == hpa))
kvmppc_unmap_pte(kvm, ptep, gpa, shift, NULL, gp->shadow_lpid);
@@ -921,7 +935,7 @@ static bool kvmhv_invalidate_shadow_pte(struct kvm_vcpu *vcpu,
int shift;
spin_lock(&kvm->mmu_lock);
- ptep = __find_linux_pte(gp->shadow_pgtable, gpa, NULL, &shift);
+ ptep = find_kvm_nested_guest_pte(kvm, gp->l1_lpid, gpa, &shift);
if (!shift)
shift = PAGE_SHIFT;
if (ptep && pte_present(*ptep)) {
--
2.26.2
^ permalink raw reply related
* [PATCH v4 09/22] powerpc/kvm/book3s: Add helper to walk partition scoped linux page table.
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
The locking rules for walking partition scoped table is different from process
scoped table. Hence add a helper for secondary linux page table walk and also
add check whether we are holding the right locks.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/include/asm/kvm_book3s_64.h | 13 +++++++++++++
arch/powerpc/kvm/book3s_64_mmu_radix.c | 12 ++++++------
arch/powerpc/kvm/book3s_hv_nested.c | 2 +-
3 files changed, 20 insertions(+), 7 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h
index 04b2b927bb5a..2c2635967d6e 100644
--- a/arch/powerpc/include/asm/kvm_book3s_64.h
+++ b/arch/powerpc/include/asm/kvm_book3s_64.h
@@ -14,6 +14,7 @@
#include <asm/book3s/64/mmu-hash.h>
#include <asm/cpu_has_feature.h>
#include <asm/ppc-opcode.h>
+#include <asm/pte-walk.h>
#ifdef CONFIG_PPC_PSERIES
static inline bool kvmhv_on_pseries(void)
@@ -634,6 +635,18 @@ extern void kvmhv_remove_nest_rmap_range(struct kvm *kvm,
unsigned long gpa, unsigned long hpa,
unsigned long nbytes);
+static inline pte_t *find_kvm_secondary_pte(struct kvm *kvm, unsigned long ea,
+ unsigned *hshift)
+{
+ pte_t *pte;
+
+ VM_WARN(!spin_is_locked(&kvm->mmu_lock),
+ "%s called with kvm mmu_lock not held \n", __func__);
+ pte = __find_linux_pte(kvm->arch.pgtable, ea, NULL, hshift);
+
+ return pte;
+}
+
#endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */
#endif /* __ASM_KVM_BOOK3S_64_H__ */
diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c
index aa12cd4078b3..c92d413eeaaf 100644
--- a/arch/powerpc/kvm/book3s_64_mmu_radix.c
+++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c
@@ -981,11 +981,11 @@ int kvm_unmap_radix(struct kvm *kvm, struct kvm_memory_slot *memslot,
return 0;
}
- ptep = __find_linux_pte(kvm->arch.pgtable, gpa, NULL, &shift);
+ ptep = find_kvm_secondary_pte(kvm, gpa, &shift);
if (ptep && pte_present(*ptep))
kvmppc_unmap_pte(kvm, ptep, gpa, shift, memslot,
kvm->arch.lpid);
- return 0;
+ return 0;
}
/* Called with kvm->mmu_lock held */
@@ -1001,7 +1001,7 @@ int kvm_age_radix(struct kvm *kvm, struct kvm_memory_slot *memslot,
if (kvm->arch.secure_guest & KVMPPC_SECURE_INIT_DONE)
return ref;
- ptep = __find_linux_pte(kvm->arch.pgtable, gpa, NULL, &shift);
+ ptep = find_kvm_secondary_pte(kvm, gpa, &shift);
if (ptep && pte_present(*ptep) && pte_young(*ptep)) {
old = kvmppc_radix_update_pte(kvm, ptep, _PAGE_ACCESSED, 0,
gpa, shift);
@@ -1028,7 +1028,7 @@ int kvm_test_age_radix(struct kvm *kvm, struct kvm_memory_slot *memslot,
if (kvm->arch.secure_guest & KVMPPC_SECURE_INIT_DONE)
return ref;
- ptep = __find_linux_pte(kvm->arch.pgtable, gpa, NULL, &shift);
+ ptep = find_kvm_secondary_pte(kvm, gpa, &shift);
if (ptep && pte_present(*ptep) && pte_young(*ptep))
ref = 1;
return ref;
@@ -1048,7 +1048,7 @@ static int kvm_radix_test_clear_dirty(struct kvm *kvm,
if (kvm->arch.secure_guest & KVMPPC_SECURE_INIT_DONE)
return ret;
- ptep = __find_linux_pte(kvm->arch.pgtable, gpa, NULL, &shift);
+ ptep = find_kvm_secondary_pte(kvm, gpa, &shift);
if (ptep && pte_present(*ptep) && pte_dirty(*ptep)) {
ret = 1;
if (shift)
@@ -1109,7 +1109,7 @@ void kvmppc_radix_flush_memslot(struct kvm *kvm,
gpa = memslot->base_gfn << PAGE_SHIFT;
spin_lock(&kvm->mmu_lock);
for (n = memslot->npages; n; --n) {
- ptep = __find_linux_pte(kvm->arch.pgtable, gpa, NULL, &shift);
+ ptep = find_kvm_secondary_pte(kvm, gpa, &shift);
if (ptep && pte_present(*ptep))
kvmppc_unmap_pte(kvm, ptep, gpa, shift, memslot,
kvm->arch.lpid);
diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
index dc97e5be76f6..7f1fc5db13ea 100644
--- a/arch/powerpc/kvm/book3s_hv_nested.c
+++ b/arch/powerpc/kvm/book3s_hv_nested.c
@@ -1362,7 +1362,7 @@ static long int __kvmhv_nested_page_fault(struct kvm_run *run,
/* See if can find translation in our partition scoped tables for L1 */
pte = __pte(0);
spin_lock(&kvm->mmu_lock);
- pte_p = __find_linux_pte(kvm->arch.pgtable, gpa, NULL, &shift);
+ pte_p = find_kvm_secondary_pte(kvm, gpa, &shift);
if (!shift)
shift = PAGE_SHIFT;
if (pte_p)
--
2.26.2
^ permalink raw reply related
* [PATCH v4 08/22] powerpc/kvm/book3s: switch from raw_spin_*lock to arch_spin_lock.
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, Suraj Jitindar Singh, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
These functions can get called in realmode. Hence use low level
arch_spin_lock which is safe to be called in realmode.
Cc: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kvm/book3s_hv_rm_mmu.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_hv_rm_mmu.c b/arch/powerpc/kvm/book3s_hv_rm_mmu.c
index 220305454c23..03f8347de48b 100644
--- a/arch/powerpc/kvm/book3s_hv_rm_mmu.c
+++ b/arch/powerpc/kvm/book3s_hv_rm_mmu.c
@@ -948,7 +948,7 @@ static long kvmppc_do_h_page_init_zero(struct kvm_vcpu *vcpu,
return ret;
/* Check if we've been invalidated */
- raw_spin_lock(&kvm->mmu_lock.rlock);
+ arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
if (mmu_notifier_retry(kvm, mmu_seq)) {
ret = H_TOO_HARD;
goto out_unlock;
@@ -960,7 +960,7 @@ static long kvmppc_do_h_page_init_zero(struct kvm_vcpu *vcpu,
kvmppc_update_dirty_map(memslot, dest >> PAGE_SHIFT, PAGE_SIZE);
out_unlock:
- raw_spin_unlock(&kvm->mmu_lock.rlock);
+ arch_spin_unlock(&kvm->mmu_lock.rlock.raw_lock);
return ret;
}
@@ -984,7 +984,7 @@ static long kvmppc_do_h_page_init_copy(struct kvm_vcpu *vcpu,
return ret;
/* Check if we've been invalidated */
- raw_spin_lock(&kvm->mmu_lock.rlock);
+ arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
if (mmu_notifier_retry(kvm, mmu_seq)) {
ret = H_TOO_HARD;
goto out_unlock;
@@ -996,7 +996,7 @@ static long kvmppc_do_h_page_init_copy(struct kvm_vcpu *vcpu,
kvmppc_update_dirty_map(dest_memslot, dest >> PAGE_SHIFT, PAGE_SIZE);
out_unlock:
- raw_spin_unlock(&kvm->mmu_lock.rlock);
+ arch_spin_unlock(&kvm->mmu_lock.rlock.raw_lock);
return ret;
}
--
2.26.2
^ permalink raw reply related
* [PATCH v4 07/22] powerpc/perf/callchain: Use __get_user_pages_fast in read_user_stack_slow
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
read_user_stack_slow is called with interrupts soft disabled and it copies contents
from the page which we find mapped to a specific address. To convert
userspace address to pfn, the kernel now uses lockless page table walk.
The kernel needs to make sure the pfn value read remains stable and is not released
and reused for another process while the contents are read from the page. This
can only be achieved by holding a page reference.
One of the first approaches I tried was to check the pte value after the kernel
copies the contents from the page. But as shown below we can still get it wrong
CPU0 CPU1
pte = READ_ONCE(*ptep);
pte_clear(pte);
put_page(page);
page = alloc_page();
memcpy(page_address(page), "secret password", nr);
memcpy(buf, kaddr + offset, nb);
put_page(page);
handle_mm_fault()
page = alloc_page();
set_pte(pte, page);
if (pte_val(pte) != pte_val(*ptep))
Hence switch to __get_user_pages_fast.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/perf/callchain_64.c | 46 ++++++++++----------------------
1 file changed, 14 insertions(+), 32 deletions(-)
diff --git a/arch/powerpc/perf/callchain_64.c b/arch/powerpc/perf/callchain_64.c
index df1ffd8b20f2..b63086b663ef 100644
--- a/arch/powerpc/perf/callchain_64.c
+++ b/arch/powerpc/perf/callchain_64.c
@@ -26,43 +26,25 @@
*/
int read_user_stack_slow(void __user *ptr, void *buf, int nb)
{
- int ret = -EFAULT;
- pgd_t *pgdir;
- pte_t *ptep, pte;
- unsigned int shift;
+
unsigned long addr = (unsigned long) ptr;
unsigned long offset;
- unsigned long pfn, flags;
+ struct page *page;
+ int nrpages;
void *kaddr;
- pgdir = current->mm->pgd;
- if (!pgdir)
- return -EFAULT;
+ nrpages = __get_user_pages_fast(addr, 1, 1, &page);
+ if (nrpages == 1) {
+ kaddr = page_address(page);
+
+ /* align address to page boundary */
+ offset = addr & ~PAGE_MASK;
- local_irq_save(flags);
- ptep = find_current_mm_pte(pgdir, addr, NULL, &shift);
- if (!ptep)
- goto err_out;
- if (!shift)
- shift = PAGE_SHIFT;
-
- /* align address to page boundary */
- offset = addr & ((1UL << shift) - 1);
-
- pte = READ_ONCE(*ptep);
- if (!pte_present(pte) || !pte_user(pte))
- goto err_out;
- pfn = pte_pfn(pte);
- if (!page_is_ram(pfn))
- goto err_out;
-
- /* no highmem to worry about here */
- kaddr = pfn_to_kaddr(pfn);
- memcpy(buf, kaddr + offset, nb);
- ret = 0;
-err_out:
- local_irq_restore(flags);
- return ret;
+ memcpy(buf, kaddr + offset, nb);
+ put_page(page);
+ return 0;
+ }
+ return -EFAULT;
}
static int read_user_stack_64(unsigned long __user *ptr, unsigned long *ret)
--
2.26.2
^ permalink raw reply related
* [PATCH v4 06/22] powerpc/mce: Don't reload pte val in addr_to_pfn
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin, Mahesh Salgaonkar
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
A lockless page table walk should be safe against parallel THP collapse, THP
split and madvise(MADV_DONTNEED)/parallel fault. This patch makes sure kernel
won't reload the pteval when checking for different conditions. The patch also added
a check for pte_present to make sure the kernel is indeed operating
on a PTE and not a pointer to level 0 table page.
The pfn value we find here can be different from the actual pfn on which
machine check happened. This can happen if we raced with a parallel update
of the page table. In such a scenario we end up isolating a wrong pfn. But that
doesn't have any other side effect.
Cc: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kernel/mce_power.c | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/kernel/mce_power.c b/arch/powerpc/kernel/mce_power.c
index 067b094bfeff..1d18991f3854 100644
--- a/arch/powerpc/kernel/mce_power.c
+++ b/arch/powerpc/kernel/mce_power.c
@@ -27,7 +27,7 @@
*/
unsigned long addr_to_pfn(struct pt_regs *regs, unsigned long addr)
{
- pte_t *ptep;
+ pte_t *ptep, pte;
unsigned int shift;
unsigned long pfn, flags;
struct mm_struct *mm;
@@ -39,19 +39,23 @@ unsigned long addr_to_pfn(struct pt_regs *regs, unsigned long addr)
local_irq_save(flags);
ptep = __find_linux_pte(mm->pgd, addr, NULL, &shift);
+ if (!ptep) {
+ pfn = ULONG_MAX;
+ goto out;
+ }
+ pte = READ_ONCE(*ptep);
- if (!ptep || pte_special(*ptep)) {
+ if (!pte_present(pte) || pte_special(pte)) {
pfn = ULONG_MAX;
goto out;
}
if (shift <= PAGE_SHIFT)
- pfn = pte_pfn(*ptep);
+ pfn = pte_pfn(pte);
else {
unsigned long rpnmask = (1ul << shift) - PAGE_SIZE;
- pfn = pte_pfn(__pte(pte_val(*ptep) | (addr & rpnmask)));
+ pfn = pte_pfn(__pte(pte_val(pte) | (addr & rpnmask)));
}
-
out:
local_irq_restore(flags);
return pfn;
--
2.26.2
^ permalink raw reply related
* [PATCH v4 05/22] powerpc/book3s64/hash: Use the pte_t address from the caller
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
Don't fetch the pte value using lockless page table walk. Instead use the value from the
caller. hash_preload is called with ptl lock held. So it is safe to use the
pte_t address directly.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/mm/book3s64/hash_utils.c | 27 +++++----------------------
1 file changed, 5 insertions(+), 22 deletions(-)
diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c
index 525eac4ee2c2..3d727f73a8db 100644
--- a/arch/powerpc/mm/book3s64/hash_utils.c
+++ b/arch/powerpc/mm/book3s64/hash_utils.c
@@ -1546,14 +1546,11 @@ static bool should_hash_preload(struct mm_struct *mm, unsigned long ea)
}
#endif
-static void hash_preload(struct mm_struct *mm, unsigned long ea,
+static void hash_preload(struct mm_struct *mm, pte_t *ptep, unsigned long ea,
bool is_exec, unsigned long trap)
{
- int hugepage_shift;
unsigned long vsid;
pgd_t *pgdir;
- pte_t *ptep;
- unsigned long flags;
int rc, ssize, update_flags = 0;
unsigned long access = _PAGE_PRESENT | _PAGE_READ | (is_exec ? _PAGE_EXEC : 0);
@@ -1575,30 +1572,18 @@ static void hash_preload(struct mm_struct *mm, unsigned long ea,
vsid = get_user_vsid(&mm->context, ea, ssize);
if (!vsid)
return;
- /*
- * Hash doesn't like irqs. Walking linux page table with irq disabled
- * saves us from holding multiple locks.
- */
- local_irq_save(flags);
- /*
- * THP pages use update_mmu_cache_pmd. We don't do
- * hash preload there. Hence can ignore THP here
- */
- ptep = find_current_mm_pte(pgdir, ea, NULL, &hugepage_shift);
- if (!ptep)
- goto out_exit;
-
- WARN_ON(hugepage_shift);
#ifdef CONFIG_PPC_64K_PAGES
/* If either H_PAGE_4K_PFN or cache inhibited is set (and we are on
* a 64K kernel), then we don't preload, hash_page() will take
* care of it once we actually try to access the page.
* That way we don't have to duplicate all of the logic for segment
* page size demotion here
+ * Called with PTL held, hence can be sure the value won't change in
+ * between.
*/
if ((pte_val(*ptep) & H_PAGE_4K_PFN) || pte_ci(*ptep))
- goto out_exit;
+ return;
#endif /* CONFIG_PPC_64K_PAGES */
/* Is that local to this CPU ? */
@@ -1623,8 +1608,6 @@ static void hash_preload(struct mm_struct *mm, unsigned long ea,
mm_ctx_user_psize(&mm->context),
mm_ctx_user_psize(&mm->context),
pte_val(*ptep));
-out_exit:
- local_irq_restore(flags);
}
/*
@@ -1675,7 +1658,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address,
return;
}
- hash_preload(vma->vm_mm, address, is_exec, trap);
+ hash_preload(vma->vm_mm, ptep, address, is_exec, trap);
}
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
--
2.26.2
^ permalink raw reply related
* [PATCH v4 04/22] powerpc/hash64: Restrict page table lookup using init_mm with __flush_hash_table_range
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
This is only used with init_mm currently. Walking init_mm is much simpler
because we don't need to handle concurrent page table like other mm_context
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
.../include/asm/book3s/64/tlbflush-hash.h | 3 +--
arch/powerpc/kernel/pci_64.c | 2 +-
arch/powerpc/mm/book3s64/hash_tlb.c | 16 +++-------------
3 files changed, 5 insertions(+), 16 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h
index 64d02a704bcb..3b95769739c7 100644
--- a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h
+++ b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h
@@ -113,8 +113,7 @@ static inline void hash__flush_tlb_kernel_range(unsigned long start,
struct mmu_gather;
extern void hash__tlb_flush(struct mmu_gather *tlb);
/* Private function for use by PCI IO mapping code */
-extern void __flush_hash_table_range(struct mm_struct *mm, unsigned long start,
- unsigned long end);
+extern void __flush_hash_table_range(unsigned long start, unsigned long end);
extern void flush_tlb_pmd_range(struct mm_struct *mm, pmd_t *pmd,
unsigned long addr);
#endif /* _ASM_POWERPC_BOOK3S_64_TLBFLUSH_HASH_H */
diff --git a/arch/powerpc/kernel/pci_64.c b/arch/powerpc/kernel/pci_64.c
index f83d1f69b1dd..30d07fc79dd1 100644
--- a/arch/powerpc/kernel/pci_64.c
+++ b/arch/powerpc/kernel/pci_64.c
@@ -100,7 +100,7 @@ int pcibios_unmap_io_space(struct pci_bus *bus)
pci_name(bus->self));
#ifdef CONFIG_PPC_BOOK3S_64
- __flush_hash_table_range(&init_mm, res->start + _IO_BASE,
+ __flush_hash_table_range(res->start + _IO_BASE,
res->end + _IO_BASE + 1);
#endif
return 0;
diff --git a/arch/powerpc/mm/book3s64/hash_tlb.c b/arch/powerpc/mm/book3s64/hash_tlb.c
index 4a70d8dd39cd..1fa2173413b5 100644
--- a/arch/powerpc/mm/book3s64/hash_tlb.c
+++ b/arch/powerpc/mm/book3s64/hash_tlb.c
@@ -176,7 +176,6 @@ void hash__tlb_flush(struct mmu_gather *tlb)
* from the hash table (and the TLB). But keeps
* the linux PTEs intact.
*
- * @mm : mm_struct of the target address space (generally init_mm)
* @start : starting address
* @end : ending address (not included in the flush)
*
@@ -189,17 +188,14 @@ void hash__tlb_flush(struct mmu_gather *tlb)
* Because of that usage pattern, it is implemented for small size rather
* than speed.
*/
-void __flush_hash_table_range(struct mm_struct *mm, unsigned long start,
- unsigned long end)
+void __flush_hash_table_range(unsigned long start, unsigned long end)
{
- bool is_thp;
int hugepage_shift;
unsigned long flags;
start = _ALIGN_DOWN(start, PAGE_SIZE);
end = _ALIGN_UP(end, PAGE_SIZE);
- BUG_ON(!mm->pgd);
/*
* Note: Normally, we should only ever use a batch within a
@@ -212,21 +208,15 @@ void __flush_hash_table_range(struct mm_struct *mm, unsigned long start,
local_irq_save(flags);
arch_enter_lazy_mmu_mode();
for (; start < end; start += PAGE_SIZE) {
- pte_t *ptep = find_current_mm_pte(mm->pgd, start, &is_thp,
- &hugepage_shift);
+ pte_t *ptep = find_init_mm_pte(start, &hugepage_shift);
unsigned long pte;
if (ptep == NULL)
continue;
pte = pte_val(*ptep);
- if (is_thp)
- trace_hugepage_invalidate(start, pte);
if (!(pte & H_PAGE_HASHPTE))
continue;
- if (unlikely(is_thp))
- hpte_do_hugepage_flush(mm, start, (pmd_t *)ptep, pte);
- else
- hpte_need_flush(mm, start, ptep, pte, hugepage_shift);
+ hpte_need_flush(&init_mm, start, ptep, pte, hugepage_shift);
}
arch_leave_lazy_mmu_mode();
local_irq_restore(flags);
--
2.26.2
^ permalink raw reply related
* [PATCH v4 03/22] powerpc/mm/hash64: use _PAGE_PTE when checking for pte_present
From: Aneesh Kumar K.V @ 2020-05-05 7:17 UTC (permalink / raw)
To: linuxppc-dev, mpe; +Cc: Aneesh Kumar K.V, npiggin
In-Reply-To: <20200505071729.54912-1-aneesh.kumar@linux.ibm.com>
This makes the pte_present check stricter by checking for additional _PAGE_PTE
bit. A level 1 pte pointer (THP pte) can be switched to a pointer to level 0 pte
page table page by following two operations.
1) THP split.
2) madvise(MADV_DONTNEED) in parallel to page fault.
A lockless page table walk need to make sure we can handle such changes
gracefully.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/include/asm/book3s/64/pgtable.h | 15 ++++++++++-----
arch/powerpc/mm/book3s64/hash_utils.c | 11 +++++++++--
2 files changed, 19 insertions(+), 7 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index 368b136517e0..03521a8b0292 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -553,6 +553,12 @@ static inline pte_t pte_clear_savedwrite(pte_t pte)
}
#endif /* CONFIG_NUMA_BALANCING */
+static inline bool pte_hw_valid(pte_t pte)
+{
+ return (pte_raw(pte) & cpu_to_be64(_PAGE_PRESENT | _PAGE_PTE)) ==
+ cpu_to_be64(_PAGE_PRESENT | _PAGE_PTE);
+}
+
static inline int pte_present(pte_t pte)
{
/*
@@ -561,12 +567,11 @@ static inline int pte_present(pte_t pte)
* invalid during ptep_set_access_flags. Hence we look for _PAGE_INVALID
* if we find _PAGE_PRESENT cleared.
*/
- return !!(pte_raw(pte) & cpu_to_be64(_PAGE_PRESENT | _PAGE_INVALID));
-}
-static inline bool pte_hw_valid(pte_t pte)
-{
- return !!(pte_raw(pte) & cpu_to_be64(_PAGE_PRESENT));
+ if (pte_hw_valid(pte))
+ return true;
+ return (pte_raw(pte) & cpu_to_be64(_PAGE_INVALID | _PAGE_PTE)) ==
+ cpu_to_be64(_PAGE_INVALID | _PAGE_PTE);
}
#ifdef CONFIG_PPC_MEM_KEYS
diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c
index e951e87a974d..525eac4ee2c2 100644
--- a/arch/powerpc/mm/book3s64/hash_utils.c
+++ b/arch/powerpc/mm/book3s64/hash_utils.c
@@ -1350,8 +1350,15 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea,
goto bail;
}
- /* Add _PAGE_PRESENT to the required access perm */
- access |= _PAGE_PRESENT;
+ /*
+ * Add _PAGE_PRESENT to the required access perm. If there are parallel
+ * updates to the pte that can possibly clear _PAGE_PTE, catch that too.
+ *
+ * We can safely use the return pte address in rest of the function
+ * because we do set H_PAGE_BUSY which prevents further updates to pte
+ * from generic code.
+ */
+ access |= _PAGE_PRESENT | _PAGE_PTE;
/*
* Pre-check access permissions (will be re-checked atomically
--
2.26.2
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox