* Re: [PATCH 6/8] simplefs: add file creation functions
From: Greg Kroah-Hartman @ 2020-04-20 14:28 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: Song Liu, linux-usb, bpf, Rafael J. Wysocki, David Airlie,
Heiko Carstens, Alexei Starovoitov, dri-devel, J. Bruce Fields,
Joseph Qi, Hugh Dickins, Paul Mackerras, John Johansen, netdev,
ocfs2-devel, Christoph Hellwig, Andrew Donnellan, Matthew Garrett,
linux-efi, Arnd Bergmann, Daniel Borkmann, Christian Borntraeger,
linux-rdma, Mark Fasheh, Anton Vorontsov, John Fastabend,
James Morris, Ard Biesheuvel, Jason Gunthorpe, Doug Ledford,
oprofile-list, Yonghong Song, Ian Kent, Andrii Nakryiko,
Alexey Dobriyan, Serge E. Hallyn, Robert Richter,
Thomas Zimmermann, Vasily Gorbik, Tony Luck, Kees Cook,
James E.J. Bottomley, autofs, Maarten Lankhorst, Mike Marciniszyn,
Maxime Ripard, linux-fsdevel, Manoj N. Kumar, Uma Krishnan,
Jakub Kicinski, KP Singh, Trond Myklebust, Matthew R. Ochs,
David S. Miller, Felipe Balbi, linux-nfs, Iurii Zaikin,
linux-scsi, Martin K. Petersen, linux-mm, linux-s390,
Dennis Dalessandro, Miklos Szeredi, linux-security-module,
linux-kernel, Anna Schumaker, Luis Chamberlain, Chuck Lever,
Jeremy Kerr, Daniel Vetter, Colin Cross, Frederic Barrat,
Paolo Bonzini, Andrew Morton, Mike Kravetz, linuxppc-dev,
Martin KaFai Lau, Joel Becker, Alexander Viro
In-Reply-To: <f371bcc0-266a-cb0b-3bde-fed336b8c9b5@redhat.com>
On Mon, Apr 20, 2020 at 03:57:48PM +0200, Emanuele Giuseppe Esposito wrote:
>
>
> On 4/14/20 2:56 PM, Greg Kroah-Hartman wrote:
> > On Tue, Apr 14, 2020 at 02:43:00PM +0200, Emanuele Giuseppe Esposito wrote:
> > > A bunch of code is duplicated between debugfs and tracefs, unify it to the
> > > simplefs library.
> > >
> > > The code is very similar, except that dentry and inode creation are unified
> > > into a single function (unlike start_creating in debugfs and tracefs, which
> > > only takes care of dentries). This adds an output parameter to the creation
> > > functions, but pushes all error recovery into fs/simplefs.c.
> > >
> > > Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> > > ---
> > > fs/simplefs.c | 150 +++++++++++++++++++++++++++++++++++++++
> > > include/linux/simplefs.h | 19 +++++
> > > 2 files changed, 169 insertions(+)
> >
> > What's wrong with libfs, isn't that supposed to be for these types of
> > "common" filesystem interactions?
> >
> > Why create a whole "new" fs for this?
>
> I assume you meant a new file. These new functions are used only by a few
> filesystems, and I didn't want to include them in vmlinux unconditionally,
> so I introduced simplefs.c and CONFIG_SIMPLEFS instead of extending libfs.c.
> In this way only fs that need this code like debugfs and tracefs will load
> it.
Nothing "loads it", why not just make these libfs functions instead? As
the difference between the two is not obvious at all, please don't make
things confusing.
thanks,
greg k-h
^ permalink raw reply
* Re: [PATCH 0/8] Simplefs: group and simplify linux fs code
From: Emanuele Giuseppe Esposito @ 2020-04-20 14:01 UTC (permalink / raw)
To: Luis Chamberlain
Cc: Song Liu, linux-usb, bpf, Rafael J. Wysocki, David Airlie,
Heiko Carstens, Alexei Starovoitov, dri-devel, J. Bruce Fields,
Joseph Qi, Hugh Dickins, Paul Mackerras, John Johansen, netdev,
linux-s390, Christoph Hellwig, Andrew Donnellan, Matthew Garrett,
linux-efi, Arnd Bergmann, Daniel Borkmann, Christian Borntraeger,
linux-rdma, Mark Fasheh, Anton Vorontsov, John Fastabend,
James Morris, Ard Biesheuvel, Jason Gunthorpe, Doug Ledford,
Yonghong Song, Ian Kent, Andrii Nakryiko, Alexey Dobriyan,
Serge E. Hallyn, Robert Richter, Thomas Zimmermann, Vasily Gorbik,
Tony Luck, Kees Cook, James E.J. Bottomley, autofs,
Maarten Lankhorst, Mike Marciniszyn, Maxime Ripard, linux-fsdevel,
Manoj N. Kumar, Uma Krishnan, Jakub Kicinski, KP Singh,
Trond Myklebust, Matthew R. Ochs, David S. Miller, Felipe Balbi,
linux-nfs, Iurii Zaikin, linux-scsi, Martin K. Petersen, linux-mm,
Greg Kroah-Hartman, Dennis Dalessandro, Miklos Szeredi,
linux-kernel, Anna Schumaker, linux-security-module, Chuck Lever,
Jeremy Kerr, Daniel Vetter, Colin Cross, Frederic Barrat,
Paolo Bonzini, Andrew Morton, Mike Kravetz, linuxppc-dev,
Martin KaFai Lau, Joel Becker, Alexander Viro
In-Reply-To: <20200416065922.GS11244@42.do-not-panic.com>
On 4/16/20 8:59 AM, Luis Chamberlain wrote:
> On Tue, Apr 14, 2020 at 02:42:54PM +0200, Emanuele Giuseppe Esposito wrote:
>> This series of patches introduce wrappers for functions,
>> arguments simplification in functions calls and most importantly
>> groups duplicated code in a single header, simplefs, to avoid redundancy
>> in the linux fs, especially debugfs and tracefs.
>
> The general goal seems worthy, but here I don't see explained why hasn't
> this gone through libfs, and what the intention was long term. For
> instance, you added some other generalizations which you have found. It
> was not clear that this was the goal here, to expand on these paths.
>
> What if common code on fs is found which are not part of debugfs and
> tracefs, how does one decide if to move to libfs or simplefs?
The idea of simplefs (that I will also explain better in the cover
letter and commit messages) is that not only it groups common code, but
also introduces a new struct simple_fs that simplifies parameter
passing. This means all fs that use these functions and the struct
should include linux/simplefs.h, while all common functions that take a
simple_fs struct will be added in simplefs.c
Thank you for all the feedback, I will incorporate it and send a new
patch series soon.
Emanuele
^ permalink raw reply
* Re: [PATCH 1/8] apparmor: just use vfs_kern_mount to make .null
From: Emanuele Giuseppe Esposito @ 2020-04-20 14:00 UTC (permalink / raw)
To: Luis Chamberlain, Goldwyn Rodrigues
Cc: Song Liu, linux-usb, bpf, Rafael J. Wysocki, David Airlie,
Heiko Carstens, Alexei Starovoitov, dri-devel, J. Bruce Fields,
Joseph Qi, Hugh Dickins, Paul Mackerras, John Johansen, netdev,
linux-s390, Christoph Hellwig, Andrew Donnellan, Matthew Garrett,
linux-efi, Arnd Bergmann, Daniel Borkmann, Christian Borntraeger,
linux-rdma, Mark Fasheh, Anton Vorontsov, John Fastabend,
James Morris, Ard Biesheuvel, Jason Gunthorpe, Doug Ledford,
Yonghong Song, Ian Kent, Andrii Nakryiko, Alexey Dobriyan,
Serge E. Hallyn, Robert Richter, Thomas Zimmermann, Vasily Gorbik,
Tony Luck, Kees Cook, James E.J. Bottomley, autofs,
Maarten Lankhorst, Mike Marciniszyn, Maxime Ripard, linux-fsdevel,
Manoj N. Kumar, Uma Krishnan, Jakub Kicinski, KP Singh,
Trond Myklebust, Matthew R. Ochs, David S. Miller, Felipe Balbi,
linux-nfs, Iurii Zaikin, linux-scsi, Martin K. Petersen, linux-mm,
Greg Kroah-Hartman, Dennis Dalessandro, Miklos Szeredi,
linux-kernel, Anna Schumaker, linux-security-module, Chuck Lever,
Jeremy Kerr, Daniel Vetter, Colin Cross, Frederic Barrat,
Paolo Bonzini, Andrew Morton, Mike Kravetz, linuxppc-dev,
Martin KaFai Lau, Joel Becker, Alexander Viro
In-Reply-To: <20200416064405.GP11244@42.do-not-panic.com>
On 4/16/20 8:44 AM, Luis Chamberlain wrote:
> On Tue, Apr 14, 2020 at 02:42:55PM +0200, Emanuele Giuseppe Esposito wrote:
>> aa_mk_null_file is using simple_pin_fs/simple_release_fs with local
>> variables as arguments, for what would amount to a simple
>> vfs_kern_mount/mntput pair if everything was inlined. Just use
>> the normal filesystem API since the reference counting is not needed
>> here.
>
> *Why* is refcounting not needed here?
The refcount is a local variable and is always 0 on entry and exit, so
it is not necessary to have refcounting across function invocations,
such as what simple_pin_fs and simple_release_fs provide.
Thank you,
Emanuele
>
> Luis
>
>>
>> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
>> ---
>> security/apparmor/apparmorfs.c | 12 ++++++------
>> 1 file changed, 6 insertions(+), 6 deletions(-)
>>
>> diff --git a/security/apparmor/apparmorfs.c b/security/apparmor/apparmorfs.c
>> index 280741fc0f5f..828bb1eb77ea 100644
>> --- a/security/apparmor/apparmorfs.c
>> +++ b/security/apparmor/apparmorfs.c
>> @@ -2525,14 +2525,14 @@ struct path aa_null;
>>
>> static int aa_mk_null_file(struct dentry *parent)
>> {
>> - struct vfsmount *mount = NULL;
>> + struct file_system_type *type = parent->d_sb->s_type;
>> + struct vfsmount *mount;
>> struct dentry *dentry;
>> struct inode *inode;
>> - int count = 0;
>> - int error = simple_pin_fs(parent->d_sb->s_type, &mount, &count);
>>
>> - if (error)
>> - return error;
>> + mount = vfs_kern_mount(type, SB_KERNMOUNT, type->name, NULL);
>> + if (IS_ERR(mount))
>> + return PTR_ERR(mount);
>>
>> inode_lock(d_inode(parent));
>> dentry = lookup_one_len(NULL_FILE_NAME, parent, strlen(NULL_FILE_NAME));
>> @@ -2561,7 +2561,7 @@ static int aa_mk_null_file(struct dentry *parent)
>> dput(dentry);
>> out:
>> inode_unlock(d_inode(parent));
>> - simple_release_fs(&mount, &count);
>> + mntput(mount);
>> return error;
>> }
>>
>> --
>> 2.25.2
>>
>
^ permalink raw reply
* Re: [PATCH 4/8] fs: introduce simple_new_inode
From: Emanuele Giuseppe Esposito @ 2020-04-20 13:58 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Song Liu, linux-usb, bpf, Rafael J. Wysocki, David Airlie,
Heiko Carstens, Alexei Starovoitov, dri-devel, J. Bruce Fields,
Joseph Qi, Hugh Dickins, Paul Mackerras, John Johansen, netdev,
ocfs2-devel, Christoph Hellwig, Andrew Donnellan, Matthew Garrett,
linux-efi, Arnd Bergmann, Daniel Borkmann, Christian Borntraeger,
linux-rdma, Mark Fasheh, Anton Vorontsov, John Fastabend,
James Morris, Ard Biesheuvel, Jason Gunthorpe, Doug Ledford,
oprofile-list, Yonghong Song, Ian Kent, Andrii Nakryiko,
Alexey Dobriyan, Serge E. Hallyn, Robert Richter,
Thomas Zimmermann, Vasily Gorbik, Tony Luck, Kees Cook,
James E.J. Bottomley, autofs, Maarten Lankhorst, Mike Marciniszyn,
Maxime Ripard, linux-fsdevel, Manoj N. Kumar, Uma Krishnan,
Jakub Kicinski, KP Singh, Trond Myklebust, Matthew R. Ochs,
David S. Miller, Felipe Balbi, linux-nfs, Iurii Zaikin,
linux-scsi, Martin K. Petersen, linux-mm, linux-s390,
Dennis Dalessandro, Miklos Szeredi, linux-security-module,
linux-kernel, Anna Schumaker, Luis Chamberlain, Chuck Lever,
Jeremy Kerr, Daniel Vetter, Colin Cross, Frederic Barrat,
Paolo Bonzini, Andrew Morton, Mike Kravetz, linuxppc-dev,
Martin KaFai Lau, Joel Becker, Alexander Viro
In-Reply-To: <20200414130140.GD720679@kroah.com>
On 4/14/20 3:01 PM, Greg Kroah-Hartman wrote:
> On Tue, Apr 14, 2020 at 02:42:58PM +0200, Emanuele Giuseppe Esposito wrote:
>> It is a common special case for new_inode to initialize the
>> time to the current time and the inode to get_next_ino().
>> Introduce a core function that does it and use it throughout
>> Linux.
>
> Shouldn't this just be called new_inode_current_time()?
>
> How is anyone going to remember what simple_new_inode() does to the
> inode structure?
I noticed that most functions in libfs.c are called "simple_*" when they
do the right thing for the majority of simple use cases (e.g.,
simple_symlink_inode_operations or simple_dir_operations). I can
certainly rename the function.
Thank you for all the feedback, I will incorporate it and send a new
patch series soon.
Emanuele
>
>> --- a/fs/libfs.c
>> +++ b/fs/libfs.c
>> @@ -595,6 +595,18 @@ int simple_write_end(struct file *file, struct address_space *mapping,
>> }
>> EXPORT_SYMBOL(simple_write_end);
>>
>> +struct inode *simple_new_inode(struct super_block *sb)
>> +{
>> + struct inode *inode = new_inode(sb);
>> + if (inode) {
>> + inode->i_ino = get_next_ino();
>> + inode->i_atime = inode->i_mtime =
>> + inode->i_ctime = current_time(inode);
>> + }
>> + return inode;
>> +}
>> +EXPORT_SYMBOL(simple_new_inode);
^ permalink raw reply
* Re: [PATCH 6/8] simplefs: add file creation functions
From: Emanuele Giuseppe Esposito @ 2020-04-20 13:57 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Song Liu, linux-usb, bpf, Rafael J. Wysocki, David Airlie,
Heiko Carstens, Alexei Starovoitov, dri-devel, J. Bruce Fields,
Joseph Qi, Hugh Dickins, Paul Mackerras, John Johansen, netdev,
ocfs2-devel, Christoph Hellwig, Andrew Donnellan, Matthew Garrett,
linux-efi, Arnd Bergmann, Daniel Borkmann, Christian Borntraeger,
linux-rdma, Mark Fasheh, Anton Vorontsov, John Fastabend,
James Morris, Ard Biesheuvel, Jason Gunthorpe, Doug Ledford,
oprofile-list, Yonghong Song, Ian Kent, Andrii Nakryiko,
Alexey Dobriyan, Serge E. Hallyn, Robert Richter,
Thomas Zimmermann, Vasily Gorbik, Tony Luck, Kees Cook,
James E.J. Bottomley, autofs, Maarten Lankhorst, Mike Marciniszyn,
Maxime Ripard, linux-fsdevel, Manoj N. Kumar, Uma Krishnan,
Jakub Kicinski, KP Singh, Trond Myklebust, Matthew R. Ochs,
David S. Miller, Felipe Balbi, linux-nfs, Iurii Zaikin,
linux-scsi, Martin K. Petersen, linux-mm, linux-s390,
Dennis Dalessandro, Miklos Szeredi, linux-security-module,
linux-kernel, Anna Schumaker, Luis Chamberlain, Chuck Lever,
Jeremy Kerr, Daniel Vetter, Colin Cross, Frederic Barrat,
Paolo Bonzini, Andrew Morton, Mike Kravetz, linuxppc-dev,
Martin KaFai Lau, Joel Becker, Alexander Viro
In-Reply-To: <20200414125626.GC720679@kroah.com>
On 4/14/20 2:56 PM, Greg Kroah-Hartman wrote:
> On Tue, Apr 14, 2020 at 02:43:00PM +0200, Emanuele Giuseppe Esposito wrote:
>> A bunch of code is duplicated between debugfs and tracefs, unify it to the
>> simplefs library.
>>
>> The code is very similar, except that dentry and inode creation are unified
>> into a single function (unlike start_creating in debugfs and tracefs, which
>> only takes care of dentries). This adds an output parameter to the creation
>> functions, but pushes all error recovery into fs/simplefs.c.
>>
>> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
>> ---
>> fs/simplefs.c | 150 +++++++++++++++++++++++++++++++++++++++
>> include/linux/simplefs.h | 19 +++++
>> 2 files changed, 169 insertions(+)
>
> What's wrong with libfs, isn't that supposed to be for these types of
> "common" filesystem interactions?
>
> Why create a whole "new" fs for this?
I assume you meant a new file. These new functions are used only by a
few filesystems, and I didn't want to include them in vmlinux
unconditionally, so I introduced simplefs.c and CONFIG_SIMPLEFS instead
of extending libfs.c. In this way only fs that need this code like
debugfs and tracefs will load it.
Thank you,
Emanuele
^ permalink raw reply
* Re: [musl] Powerpc Linux 'scv' system call ABI proposal take 2
From: Rich Felker @ 2020-04-20 17:27 UTC (permalink / raw)
To: Nicholas Piggin
Cc: libc-dev, libc-alpha, linuxppc-dev, Adhemerval Zanella, musl
In-Reply-To: <1587356128.aslvdnmtbw.astroid@bobo.none>
On Mon, Apr 20, 2020 at 02:31:58PM +1000, Nicholas Piggin wrote:
> Excerpts from Rich Felker's message of April 20, 2020 2:09 pm:
> > On Mon, Apr 20, 2020 at 12:32:21PM +1000, Nicholas Piggin wrote:
> >> Excerpts from Rich Felker's message of April 20, 2020 11:34 am:
> >> > On Mon, Apr 20, 2020 at 11:10:25AM +1000, Nicholas Piggin wrote:
> >> >> Excerpts from Rich Felker's message of April 17, 2020 4:31 am:
> >> >> > Note that because lr is clobbered we need at least once normally
> >> >> > call-clobbered register that's not syscall clobbered to save lr in.
> >> >> > Otherwise stack frame setup is required to spill it.
> >> >>
> >> >> The kernel would like to use r9-r12 for itself. We could do with fewer
> >> >> registers, but we have some delay establishing the stack (depends on a
> >> >> load which depends on a mfspr), and entry code tends to be quite store
> >> >> heavy whereas on the caller side you have r1 set up (modulo stack
> >> >> updates), and the system call is a long delay during which time the
> >> >> store queue has significant time to drain.
> >> >>
> >> >> My feeling is it would be better for kernel to have these scratch
> >> >> registers.
> >> >
> >> > If your new kernel syscall mechanism requires the caller to make a
> >> > whole stack frame it otherwise doesn't need and spill registers to it,
> >> > it becomes a lot less attractive. Some of those 90 cycles saved are
> >> > immediately lost on the userspace side, plus you either waste icache
> >> > at the call point or require the syscall to go through a
> >> > userspace-side helper function that performs the spill and restore.
> >>
> >> You would be surprised how few cycles that takes on a high end CPU. Some
> >> might be a couple of %. I am one for counting cycles mind you, I'm not
> >> being flippant about it. If we can come up with something faster I'd be
> >> up for it.
> >
> > If the cycle count is trivial then just do it on the kernel side.
>
> The cycle count for user is, because you have r1 ready. Kernel does not
> have its stack ready, it has to mfspr rX ; ld rY,N(rX); to get stack to
> save into.
>
> Which is also wasted work for a userspace.
>
> Now that I think about it, no stack frame is even required! lr is saved
> into the caller's stack when its clobbered with an asm, just as when
> it's used for a function call.
No. If there is a non-clobbered register, lr can be moved to the
non-clobbered register rather than saved to the stack. However it
looks like (1) gcc doesn't take advantage of that possibility, but (2)
the caller already arranged for there to be space on the stack to save
lr, so the cost is only one store and one load, not any stack
adjustment or other frame setup. So it's probably not a really big
deal. However, just adding "lr" clobber to existing syscall in musl
increased the size of a simple syscall function (getuid) from 20 bytes
to 36 bytes.
> >> > syscall arg registers still preserved? If not, this is a major cost on
> >> > the userspace side, since any call point that has to loop-and-retry
> >> > (e.g. futex) now needs to make its own place to store the original
> >> > values.)
> >>
> >> Powerpc system calls never did. We could have scv preserve them, but
> >> you'd still need to restore r3. We could make an ABI which does not
> >> clobber r3 but puts the return value in r9, say. I'd like to see what
> >> the user side code looks like to take advantage of such a thing though.
> >
> > Oh wow, I hadn't realized that, but indeed the code we have now is
> > allowing for the kernel to clobber them all. So at least this isn't
> > getting any worse I guess. I think it was a very poor choice of
> > behavior though and a disadvantage vs what other archs do (some of
> > them preserve all registers; others preserve only normally call-saved
> > ones plus the syscall arg ones and possibly a few other specials).
>
> Well, we could change it. Does the generated code improve significantly
> we take those clobbers away?
I'd have to experiment a bit more to see. It's not going to help at
all in functions which are pure syscall wrappers that just do the
syscall and return, since the arg regs are dead after the syscall
anyway (the caller must assume they were clobbered). But where
syscalls are inlined and used in a loop, like a futex wait, it might
make a nontrivial difference.
Unfortunately even if you did change it for the new scv mechanism, it
would be hard to take advantage of the change while also supporting
sc, unless we used a helper function that just did scv directly, but
saved/restored all the arg regs when using the legacy sc mechanism.
Just inlining the hwcap conditional and clobbering more regs in one
code path than in the other likely would not help; gcc won't
shrink-wrap the clobbered/non-clobbered paths separately, and even if
it did, when this were inlined somewhere like a futex loop, it'd end
up having to lift the conditional out of the loop to be very
advantageous, then making the code much larger by producing two copies
of the loop. So I think just behaving similarly to the old sc method
is probably the best option we have...
Rich
^ permalink raw reply
* [PATCH v7 6/7] powerpc/vdso: Switch VDSO to generic C implementation.
From: Christophe Leroy @ 2020-04-20 16:56 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman, nathanl
Cc: arnd, linux-kernel, luto, tglx, vincenzo.frascino, linuxppc-dev
In-Reply-To: <cover.1587401492.git.christophe.leroy@c-s.fr>
For VDSO32 on PPC64, we create a fake 32 bits config, on the same
principle as MIPS architecture, in order to get the correct parts of
the different asm header files.
With the C VDSO, the performance is slightly lower, but it is worth
it as it will ease maintenance and evolution, and also brings clocks
that are not supported with the ASM VDSO.
On an 8xx at 132 MHz, vdsotest with the ASM VDSO:
gettimeofday: vdso: 828 nsec/call
clock-getres-realtime-coarse: vdso: 391 nsec/call
clock-gettime-realtime-coarse: vdso: 614 nsec/call
clock-getres-realtime: vdso: 460 nsec/call
clock-gettime-realtime: vdso: 876 nsec/call
clock-getres-monotonic-coarse: vdso: 399 nsec/call
clock-gettime-monotonic-coarse: vdso: 691 nsec/call
clock-getres-monotonic: vdso: 460 nsec/call
clock-gettime-monotonic: vdso: 1026 nsec/call
On an 8xx at 132 MHz, vdsotest with the C VDSO:
gettimeofday: vdso: 955 nsec/call
clock-getres-realtime-coarse: vdso: 545 nsec/call
clock-gettime-realtime-coarse: vdso: 592 nsec/call
clock-getres-realtime: vdso: 545 nsec/call
clock-gettime-realtime: vdso: 941 nsec/call
clock-getres-monotonic-coarse: vdso: 545 nsec/call
clock-gettime-monotonic-coarse: vdso: 591 nsec/call
clock-getres-monotonic: vdso: 545 nsec/call
clock-gettime-monotonic: vdso: 940 nsec/call
It is even better for gettime with monotonic clocks.
Unsupported clocks with ASM VDSO:
clock-gettime-boottime: vdso: 3851 nsec/call
clock-gettime-tai: vdso: 3852 nsec/call
clock-gettime-monotonic-raw: vdso: 3396 nsec/call
Same clocks with C VDSO:
clock-gettime-tai: vdso: 941 nsec/call
clock-gettime-monotonic-raw: vdso: 1001 nsec/call
clock-gettime-monotonic-coarse: vdso: 591 nsec/call
On an 8321E at 333 MHz, vdsotest with the ASM VDSO:
gettimeofday: vdso: 220 nsec/call
clock-getres-realtime-coarse: vdso: 102 nsec/call
clock-gettime-realtime-coarse: vdso: 178 nsec/call
clock-getres-realtime: vdso: 129 nsec/call
clock-gettime-realtime: vdso: 235 nsec/call
clock-getres-monotonic-coarse: vdso: 105 nsec/call
clock-gettime-monotonic-coarse: vdso: 208 nsec/call
clock-getres-monotonic: vdso: 129 nsec/call
clock-gettime-monotonic: vdso: 274 nsec/call
On an 8321E at 333 MHz, vdsotest with the C VDSO:
gettimeofday: vdso: 272 nsec/call
clock-getres-realtime-coarse: vdso: 160 nsec/call
clock-gettime-realtime-coarse: vdso: 184 nsec/call
clock-getres-realtime: vdso: 166 nsec/call
clock-gettime-realtime: vdso: 281 nsec/call
clock-getres-monotonic-coarse: vdso: 160 nsec/call
clock-gettime-monotonic-coarse: vdso: 184 nsec/call
clock-getres-monotonic: vdso: 169 nsec/call
clock-gettime-monotonic: vdso: 275 nsec/call
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
v7:
- Split out preparatory changes in a new preceding patch
- Added -fasynchronous-unwind-tables to CC flags.
v6:
- Added missing prototypes in asm/vdso/gettimeofday.h for __c_kernel_ functions.
- Using STACK_FRAME_OVERHEAD instead of INT_FRAME_SIZE
- Rebased on powerpc/merge as of 7 Apr 2020
- Fixed build failure with gcc 9
- Added a patch to create asm/vdso/processor.h and more cpu_relax() in it
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
arch/powerpc/Kconfig | 2 +
arch/powerpc/include/asm/vdso/vsyscall.h | 25 ++
arch/powerpc/include/asm/vdso_datapage.h | 40 +--
arch/powerpc/kernel/asm-offsets.c | 49 +---
arch/powerpc/kernel/time.c | 91 +------
arch/powerpc/kernel/vdso.c | 5 +-
arch/powerpc/kernel/vdso32/Makefile | 32 ++-
arch/powerpc/kernel/vdso32/config-fake32.h | 34 +++
arch/powerpc/kernel/vdso32/gettimeofday.S | 291 +--------------------
arch/powerpc/kernel/vdso64/Makefile | 23 +-
arch/powerpc/kernel/vdso64/gettimeofday.S | 243 +----------------
11 files changed, 144 insertions(+), 691 deletions(-)
create mode 100644 arch/powerpc/include/asm/vdso/vsyscall.h
create mode 100644 arch/powerpc/kernel/vdso32/config-fake32.h
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 924c541a9260..4e51454a3e7b 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -166,6 +166,7 @@ config PPC
select GENERIC_STRNCPY_FROM_USER
select GENERIC_STRNLEN_USER
select GENERIC_TIME_VSYSCALL
+ select GENERIC_GETTIMEOFDAY
select HAVE_ARCH_AUDITSYSCALL
select HAVE_ARCH_HUGE_VMAP if PPC_BOOK3S_64 && PPC_RADIX_MMU
select HAVE_ARCH_JUMP_LABEL
@@ -197,6 +198,7 @@ config PPC
select HAVE_FUNCTION_GRAPH_TRACER
select HAVE_FUNCTION_TRACER
select HAVE_GCC_PLUGINS if GCC_VERSION >= 50200 # plugin support on gcc <= 5.1 is buggy on PPC
+ select HAVE_GENERIC_VDSO
select HAVE_HW_BREAKPOINT if PERF_EVENTS && (PPC_BOOK3S || PPC_8xx)
select HAVE_IDE
select HAVE_IOREMAP_PROT
diff --git a/arch/powerpc/include/asm/vdso/vsyscall.h b/arch/powerpc/include/asm/vdso/vsyscall.h
new file mode 100644
index 000000000000..c56a030c0623
--- /dev/null
+++ b/arch/powerpc/include/asm/vdso/vsyscall.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_VDSO_VSYSCALL_H
+#define __ASM_VDSO_VSYSCALL_H
+
+#ifndef __ASSEMBLY__
+
+#include <linux/timekeeper_internal.h>
+#include <asm/vdso_datapage.h>
+
+/*
+ * Update the vDSO data page to keep in sync with kernel timekeeping.
+ */
+static __always_inline
+struct vdso_data *__arch_get_k_vdso_data(void)
+{
+ return vdso_data->data;
+}
+#define __arch_get_k_vdso_data __arch_get_k_vdso_data
+
+/* The asm-generic header needs to be included after the definitions above */
+#include <asm-generic/vdso/vsyscall.h>
+
+#endif /* !__ASSEMBLY__ */
+
+#endif /* __ASM_VDSO_VSYSCALL_H */
diff --git a/arch/powerpc/include/asm/vdso_datapage.h b/arch/powerpc/include/asm/vdso_datapage.h
index 78b8f67e9873..de7221be8de4 100644
--- a/arch/powerpc/include/asm/vdso_datapage.h
+++ b/arch/powerpc/include/asm/vdso_datapage.h
@@ -36,6 +36,7 @@
#include <linux/unistd.h>
#include <linux/time.h>
+#include <vdso/datapage.h>
#define SYSCALL_MAP_SIZE ((NR_syscalls + 31) / 32)
@@ -45,7 +46,7 @@
#ifdef CONFIG_PPC64
-struct vdso_data {
+struct vdso_arch_data {
__u8 eye_catcher[16]; /* Eyecatcher: SYSTEMCFG:PPC64 0x00 */
struct { /* Systemcfg version numbers */
__u32 major; /* Major number 0x10 */
@@ -59,13 +60,13 @@ struct vdso_data {
__u32 processor; /* Processor type 0x1C */
__u64 processorCount; /* # of physical processors 0x20 */
__u64 physicalMemorySize; /* Size of real memory(B) 0x28 */
- __u64 tb_orig_stamp; /* Timebase at boot 0x30 */
+ __u64 tb_orig_stamp; /* (NU) Timebase at boot 0x30 */
__u64 tb_ticks_per_sec; /* Timebase tics / sec 0x38 */
- __u64 tb_to_xs; /* Inverse of TB to 2^20 0x40 */
- __u64 stamp_xsec; /* 0x48 */
- __u64 tb_update_count; /* Timebase atomicity ctr 0x50 */
- __u32 tz_minuteswest; /* Minutes west of Greenwich 0x58 */
- __u32 tz_dsttime; /* Type of dst correction 0x5C */
+ __u64 tb_to_xs; /* (NU) Inverse of TB to 2^20 0x40 */
+ __u64 stamp_xsec; /* (NU) 0x48 */
+ __u64 tb_update_count; /* (NU) Timebase atomicity ctr 0x50 */
+ __u32 tz_minuteswest; /* (NU) Min. west of Greenwich 0x58 */
+ __u32 tz_dsttime; /* (NU) Type of dst correction 0x5C */
__u32 dcache_size; /* L1 d-cache size 0x60 */
__u32 dcache_line_size; /* L1 d-cache line size 0x64 */
__u32 icache_size; /* L1 i-cache size 0x68 */
@@ -78,14 +79,10 @@ struct vdso_data {
__u32 icache_block_size; /* L1 i-cache block size */
__u32 dcache_log_block_size; /* L1 d-cache log block size */
__u32 icache_log_block_size; /* L1 i-cache log block size */
- __u32 stamp_sec_fraction; /* fractional seconds of stamp_xtime */
- __s32 wtom_clock_nsec; /* Wall to monotonic clock nsec */
- __s64 wtom_clock_sec; /* Wall to monotonic clock sec */
- __s64 stamp_xtime_sec; /* xtime secs as at tb_orig_stamp */
- __s64 stamp_xtime_nsec; /* xtime nsecs as at tb_orig_stamp */
- __u32 hrtimer_res; /* hrtimer resolution */
__u32 syscall_map_64[SYSCALL_MAP_SIZE]; /* map of syscalls */
__u32 syscall_map_32[SYSCALL_MAP_SIZE]; /* map of syscalls */
+
+ struct vdso_data data[CS_BASES];
};
#else /* CONFIG_PPC64 */
@@ -93,26 +90,15 @@ struct vdso_data {
/*
* And here is the simpler 32 bits version
*/
-struct vdso_data {
- __u64 tb_orig_stamp; /* Timebase at boot 0x30 */
+struct vdso_arch_data {
__u64 tb_ticks_per_sec; /* Timebase tics / sec 0x38 */
- __u64 tb_to_xs; /* Inverse of TB to 2^20 0x40 */
- __u64 stamp_xsec; /* 0x48 */
- __u32 tb_update_count; /* Timebase atomicity ctr 0x50 */
- __u32 tz_minuteswest; /* Minutes west of Greenwich 0x58 */
- __u32 tz_dsttime; /* Type of dst correction 0x5C */
- __s32 wtom_clock_sec; /* Wall to monotonic clock */
- __s32 wtom_clock_nsec;
- __s32 stamp_xtime_sec; /* xtime seconds as at tb_orig_stamp */
- __s32 stamp_xtime_nsec; /* xtime nsecs as at tb_orig_stamp */
- __u32 stamp_sec_fraction; /* fractional seconds of stamp_xtime */
- __u32 hrtimer_res; /* hrtimer resolution */
__u32 syscall_map_32[SYSCALL_MAP_SIZE]; /* map of syscalls */
+ struct vdso_data data[CS_BASES];
};
#endif /* CONFIG_PPC64 */
-extern struct vdso_data *vdso_data;
+extern struct vdso_arch_data *vdso_data;
#else /* __ASSEMBLY__ */
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index fcf24a365fc0..9a9b4a9353ff 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -394,47 +394,16 @@ int main(void)
#endif /* ! CONFIG_PPC64 */
/* datapage offsets for use by vdso */
- OFFSET(CFG_TB_ORIG_STAMP, vdso_data, tb_orig_stamp);
- OFFSET(CFG_TB_TICKS_PER_SEC, vdso_data, tb_ticks_per_sec);
- OFFSET(CFG_TB_TO_XS, vdso_data, tb_to_xs);
- OFFSET(CFG_TB_UPDATE_COUNT, vdso_data, tb_update_count);
- OFFSET(CFG_TZ_MINUTEWEST, vdso_data, tz_minuteswest);
- OFFSET(CFG_TZ_DSTTIME, vdso_data, tz_dsttime);
- OFFSET(CFG_SYSCALL_MAP32, vdso_data, syscall_map_32);
- OFFSET(WTOM_CLOCK_SEC, vdso_data, wtom_clock_sec);
- OFFSET(WTOM_CLOCK_NSEC, vdso_data, wtom_clock_nsec);
- OFFSET(STAMP_XTIME_SEC, vdso_data, stamp_xtime_sec);
- OFFSET(STAMP_XTIME_NSEC, vdso_data, stamp_xtime_nsec);
- OFFSET(STAMP_SEC_FRAC, vdso_data, stamp_sec_fraction);
- OFFSET(CLOCK_HRTIMER_RES, vdso_data, hrtimer_res);
+ OFFSET(VDSO_DATA_OFFSET, vdso_arch_data, data);
+ OFFSET(CFG_TB_TICKS_PER_SEC, vdso_arch_data, tb_ticks_per_sec);
+ OFFSET(CFG_SYSCALL_MAP32, vdso_arch_data, syscall_map_32);
#ifdef CONFIG_PPC64
- OFFSET(CFG_ICACHE_BLOCKSZ, vdso_data, icache_block_size);
- OFFSET(CFG_DCACHE_BLOCKSZ, vdso_data, dcache_block_size);
- OFFSET(CFG_ICACHE_LOGBLOCKSZ, vdso_data, icache_log_block_size);
- OFFSET(CFG_DCACHE_LOGBLOCKSZ, vdso_data, dcache_log_block_size);
- OFFSET(CFG_SYSCALL_MAP64, vdso_data, syscall_map_64);
- OFFSET(TVAL64_TV_SEC, __kernel_old_timeval, tv_sec);
- OFFSET(TVAL64_TV_USEC, __kernel_old_timeval, tv_usec);
-#endif
- OFFSET(TSPC64_TV_SEC, __kernel_timespec, tv_sec);
- OFFSET(TSPC64_TV_NSEC, __kernel_timespec, tv_nsec);
- OFFSET(TVAL32_TV_SEC, old_timeval32, tv_sec);
- OFFSET(TVAL32_TV_USEC, old_timeval32, tv_usec);
- OFFSET(TSPC32_TV_SEC, old_timespec32, tv_sec);
- OFFSET(TSPC32_TV_NSEC, old_timespec32, tv_nsec);
- /* timeval/timezone offsets for use by vdso */
- OFFSET(TZONE_TZ_MINWEST, timezone, tz_minuteswest);
- OFFSET(TZONE_TZ_DSTTIME, timezone, tz_dsttime);
-
- /* Other bits used by the vdso */
- DEFINE(CLOCK_REALTIME, CLOCK_REALTIME);
- DEFINE(CLOCK_MONOTONIC, CLOCK_MONOTONIC);
- DEFINE(CLOCK_REALTIME_COARSE, CLOCK_REALTIME_COARSE);
- DEFINE(CLOCK_MONOTONIC_COARSE, CLOCK_MONOTONIC_COARSE);
- DEFINE(CLOCK_MAX, CLOCK_TAI);
- DEFINE(NSEC_PER_SEC, NSEC_PER_SEC);
- DEFINE(EINVAL, EINVAL);
- DEFINE(KTIME_LOW_RES, KTIME_LOW_RES);
+ OFFSET(CFG_ICACHE_BLOCKSZ, vdso_arch_data, icache_block_size);
+ OFFSET(CFG_DCACHE_BLOCKSZ, vdso_arch_data, dcache_block_size);
+ OFFSET(CFG_ICACHE_LOGBLOCKSZ, vdso_arch_data, icache_log_block_size);
+ OFFSET(CFG_DCACHE_LOGBLOCKSZ, vdso_arch_data, dcache_log_block_size);
+ OFFSET(CFG_SYSCALL_MAP64, vdso_arch_data, syscall_map_64);
+#endif
#ifdef CONFIG_BUG
DEFINE(BUG_ENTRY_SIZE, sizeof(struct bug_entry));
diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
index 6fcae436ae51..b63b1f97a1b3 100644
--- a/arch/powerpc/kernel/time.c
+++ b/arch/powerpc/kernel/time.c
@@ -91,6 +91,7 @@ static struct clocksource clocksource_timebase = {
.flags = CLOCK_SOURCE_IS_CONTINUOUS,
.mask = CLOCKSOURCE_MASK(64),
.read = timebase_read,
+ .vdso_clock_mode = VDSO_CLOCKMODE_ARCHTIMER,
};
#define DECREMENTER_DEFAULT_MAX 0x7FFFFFFF
@@ -855,95 +856,6 @@ static notrace u64 timebase_read(struct clocksource *cs)
return (u64)get_tb();
}
-
-void update_vsyscall(struct timekeeper *tk)
-{
- struct timespec64 xt;
- struct clocksource *clock = tk->tkr_mono.clock;
- u32 mult = tk->tkr_mono.mult;
- u32 shift = tk->tkr_mono.shift;
- u64 cycle_last = tk->tkr_mono.cycle_last;
- u64 new_tb_to_xs, new_stamp_xsec;
- u64 frac_sec;
-
- if (clock != &clocksource_timebase)
- return;
-
- xt.tv_sec = tk->xtime_sec;
- xt.tv_nsec = (long)(tk->tkr_mono.xtime_nsec >> tk->tkr_mono.shift);
-
- /* Make userspace gettimeofday spin until we're done. */
- ++vdso_data->tb_update_count;
- smp_mb();
-
- /*
- * This computes ((2^20 / 1e9) * mult) >> shift as a
- * 0.64 fixed-point fraction.
- * The computation in the else clause below won't overflow
- * (as long as the timebase frequency is >= 1.049 MHz)
- * but loses precision because we lose the low bits of the constant
- * in the shift. Note that 19342813113834067 ~= 2^(20+64) / 1e9.
- * For a shift of 24 the error is about 0.5e-9, or about 0.5ns
- * over a second. (Shift values are usually 22, 23 or 24.)
- * For high frequency clocks such as the 512MHz timebase clock
- * on POWER[6789], the mult value is small (e.g. 32768000)
- * and so we can shift the constant by 16 initially
- * (295147905179 ~= 2^(20+64-16) / 1e9) and then do the
- * remaining shifts after the multiplication, which gives a
- * more accurate result (e.g. with mult = 32768000, shift = 24,
- * the error is only about 1.2e-12, or 0.7ns over 10 minutes).
- */
- if (mult <= 62500000 && clock->shift >= 16)
- new_tb_to_xs = ((u64) mult * 295147905179ULL) >> (clock->shift - 16);
- else
- new_tb_to_xs = (u64) mult * (19342813113834067ULL >> clock->shift);
-
- /*
- * Compute the fractional second in units of 2^-32 seconds.
- * The fractional second is tk->tkr_mono.xtime_nsec >> tk->tkr_mono.shift
- * in nanoseconds, so multiplying that by 2^32 / 1e9 gives
- * it in units of 2^-32 seconds.
- * We assume shift <= 32 because clocks_calc_mult_shift()
- * generates shift values in the range 0 - 32.
- */
- frac_sec = tk->tkr_mono.xtime_nsec << (32 - shift);
- do_div(frac_sec, NSEC_PER_SEC);
-
- /*
- * Work out new stamp_xsec value for any legacy users of systemcfg.
- * stamp_xsec is in units of 2^-20 seconds.
- */
- new_stamp_xsec = frac_sec >> 12;
- new_stamp_xsec += tk->xtime_sec * XSEC_PER_SEC;
-
- /*
- * tb_update_count is used to allow the userspace gettimeofday code
- * to assure itself that it sees a consistent view of the tb_to_xs and
- * stamp_xsec variables. It reads the tb_update_count, then reads
- * tb_to_xs and stamp_xsec and then reads tb_update_count again. If
- * the two values of tb_update_count match and are even then the
- * tb_to_xs and stamp_xsec values are consistent. If not, then it
- * loops back and reads them again until this criteria is met.
- */
- vdso_data->tb_orig_stamp = cycle_last;
- vdso_data->stamp_xsec = new_stamp_xsec;
- vdso_data->tb_to_xs = new_tb_to_xs;
- vdso_data->wtom_clock_sec = tk->wall_to_monotonic.tv_sec;
- vdso_data->wtom_clock_nsec = tk->wall_to_monotonic.tv_nsec;
- vdso_data->stamp_xtime_sec = xt.tv_sec;
- vdso_data->stamp_xtime_nsec = xt.tv_nsec;
- vdso_data->stamp_sec_fraction = frac_sec;
- vdso_data->hrtimer_res = hrtimer_resolution;
- smp_wmb();
- ++(vdso_data->tb_update_count);
-}
-
-void update_vsyscall_tz(void)
-{
- vdso_data->tz_minuteswest = sys_tz.tz_minuteswest;
- vdso_data->tz_dsttime = sys_tz.tz_dsttime;
-}
-
static void __init clocksource_init(void)
{
struct clocksource *clock;
@@ -1113,7 +1025,6 @@ void __init time_init(void)
sys_tz.tz_dsttime = 0;
}
- vdso_data->tb_update_count = 0;
vdso_data->tb_ticks_per_sec = tb_ticks_per_sec;
/* initialise and enable the large decrementer (if we have one) */
diff --git a/arch/powerpc/kernel/vdso.c b/arch/powerpc/kernel/vdso.c
index d33fa22ddbed..6642dbe92946 100644
--- a/arch/powerpc/kernel/vdso.c
+++ b/arch/powerpc/kernel/vdso.c
@@ -17,6 +17,7 @@
#include <linux/elf.h>
#include <linux/security.h>
#include <linux/memblock.h>
+#include <vdso/datapage.h>
#include <asm/pgtable.h>
#include <asm/processor.h>
@@ -71,10 +72,10 @@ static int vdso_ready;
* with it, it will become dynamically allocated
*/
static union {
- struct vdso_data data;
+ struct vdso_arch_data data;
u8 page[PAGE_SIZE];
} vdso_data_store __page_aligned_data;
-struct vdso_data *vdso_data = &vdso_data_store.data;
+struct vdso_arch_data *vdso_data = &vdso_data_store.data;
/* Format of the patch table */
struct vdso_patch_def
diff --git a/arch/powerpc/kernel/vdso32/Makefile b/arch/powerpc/kernel/vdso32/Makefile
index e147bbdc12cd..b46c21ed9316 100644
--- a/arch/powerpc/kernel/vdso32/Makefile
+++ b/arch/powerpc/kernel/vdso32/Makefile
@@ -2,7 +2,23 @@
# List of files in the vdso, has to be asm only for now
-obj-vdso32 = sigtramp.o gettimeofday.o datapage.o cacheflush.o note.o getcpu.o
+ARCH_REL_TYPE_ABS := R_PPC_JUMP_SLOT|R_PPC_GLOB_DAT|R_PPC_ADDR32|R_PPC_ADDR24|R_PPC_ADDR16|R_PPC_ADDR16_LO|R_PPC_ADDR16_HI|R_PPC_ADDR16_HA|R_PPC_ADDR14|R_PPC_ADDR14_BRTAKEN|R_PPC_ADDR14_BRNTAKEN
+include $(srctree)/lib/vdso/Makefile
+
+obj-vdso32 = sigtramp.o datapage.o cacheflush.o note.o getcpu.o $(obj-vdso32-y)
+obj-vdso32 += gettimeofday.o
+
+ifneq ($(c-gettimeofday-y),)
+ ifdef CONFIG_PPC64
+ CFLAGS_vgettimeofday.o += -include $(srctree)/$(src)/config-fake32.h
+ endif
+ CFLAGS_vgettimeofday.o += -include $(c-gettimeofday-y)
+ CFLAGS_vgettimeofday.o += $(DISABLE_LATENT_ENTROPY_PLUGIN)
+ CFLAGS_vgettimeofday.o += $(call cc-option, -fno-stack-protector)
+ CFLAGS_vgettimeofday.o += -DDISABLE_BRANCH_PROFILING
+ CFLAGS_vgettimeofday.o += -ffreestanding -fasynchronous-unwind-tables
+ CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE)
+endif
# Build rules
@@ -15,6 +31,7 @@ endif
CC32FLAGS :=
ifdef CONFIG_PPC64
CC32FLAGS += -m32
+KBUILD_CFLAGS := $(filter-out -mcmodel=medium,$(KBUILD_CFLAGS))
endif
targets := $(obj-vdso32) vdso32.so vdso32.so.dbg
@@ -23,6 +40,7 @@ obj-vdso32 := $(addprefix $(obj)/, $(obj-vdso32))
GCOV_PROFILE := n
KCOV_INSTRUMENT := n
UBSAN_SANITIZE := n
+KASAN_SANITIZE := n
ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
-Wl,-soname=linux-vdso32.so.1 -Wl,--hash-style=both
@@ -36,8 +54,8 @@ CPPFLAGS_vdso32.lds += -P -C -Upowerpc
$(obj)/vdso32_wrapper.o : $(obj)/vdso32.so
# link rule for the .so file, .lds has to be first
-$(obj)/vdso32.so.dbg: $(src)/vdso32.lds $(obj-vdso32) FORCE
- $(call if_changed,vdso32ld)
+$(obj)/vdso32.so.dbg: $(src)/vdso32.lds $(obj-vdso32) $(obj)/vgettimeofday.o FORCE
+ $(call if_changed,vdso32ld_and_check)
# strip rule for the .so file
$(obj)/%.so: OBJCOPYFLAGS := -S
@@ -47,12 +65,16 @@ $(obj)/%.so: $(obj)/%.so.dbg FORCE
# assembly rules for the .S files
$(obj-vdso32): %.o: %.S FORCE
$(call if_changed_dep,vdso32as)
+$(obj)/vgettimeofday.o: %.o: %.c FORCE
+ $(call if_changed_dep,vdso32cc)
# actual build commands
-quiet_cmd_vdso32ld = VDSO32L $@
- cmd_vdso32ld = $(VDSOCC) $(c_flags) $(CC32FLAGS) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^)
+quiet_cmd_vdso32ld_and_check = VDSO32L $@
+ cmd_vdso32ld_and_check = $(VDSOCC) $(c_flags) $(CC32FLAGS) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^) ; $(cmd_vdso_check)
quiet_cmd_vdso32as = VDSO32A $@
cmd_vdso32as = $(VDSOCC) $(a_flags) $(CC32FLAGS) -c -o $@ $<
+quiet_cmd_vdso32cc = VDSO32C $@
+ cmd_vdso32cc = $(VDSOCC) $(c_flags) $(CC32FLAGS) -c -o $@ $<
# install commands for the unstripped file
quiet_cmd_vdso_install = INSTALL $@
diff --git a/arch/powerpc/kernel/vdso32/config-fake32.h b/arch/powerpc/kernel/vdso32/config-fake32.h
new file mode 100644
index 000000000000..e16041fc15c9
--- /dev/null
+++ b/arch/powerpc/kernel/vdso32/config-fake32.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * In case of a 32 bit VDSO for a 64 bit kernel fake a 32 bit kernel
+ * configuration.
+ */
+
+#undef CONFIG_PPC64
+#undef CONFIG_64BIT
+#define CONFIG_PPC32
+#define CONFIG_32BIT 1
+#define CONFIG_GENERIC_ATOMIC64 1
+
+#ifdef CONFIG_PPC_BOOK3S_64
+#undef CONFIG_PPC_BOOK3S_64
+#undef CONFIG_PPC_PSERIES
+#define CONFIG_PPC_BOOK3S_32
+#else
+#define CONFIG_PPC_MMU_NOHASH_32
+#define CONFIG_FSL_BOOKE
+#endif
+
+#define CONFIG_TASK_SIZE 0
+#undef CONFIG_MMIOWB
+#undef CONFIG_PPC_SPLPAR
+#undef CONFIG_SPARSEMEM
+#undef CONFIG_PGTABLE_LEVELS
+#define CONFIG_PGTABLE_LEVELS 2
+#undef CONFIG_TRANSPARENT_HUGEPAGE
+#undef CONFIG_SPARSEMEM_VMEMMAP
+#undef CONFIG_FLATMEM
+#define CONFIG_FLATMEM
+#undef CONFIG_PPC_INDIRECT_MMIO
+#undef CONFIG_PPC_INDIRECT_PIO
+#undef CONFIG_EEH
diff --git a/arch/powerpc/kernel/vdso32/gettimeofday.S b/arch/powerpc/kernel/vdso32/gettimeofday.S
index 0bbdce0f2a9c..fd7b01c51281 100644
--- a/arch/powerpc/kernel/vdso32/gettimeofday.S
+++ b/arch/powerpc/kernel/vdso32/gettimeofday.S
@@ -12,13 +12,7 @@
#include <asm/vdso_datapage.h>
#include <asm/asm-offsets.h>
#include <asm/unistd.h>
-
-/* Offset for the low 32-bit part of a field of long type */
-#ifdef CONFIG_PPC64
-#define LOPART 4
-#else
-#define LOPART 0
-#endif
+#include <asm/vdso/gettimeofday.h>
.text
/*
@@ -28,32 +22,7 @@
*
*/
V_FUNCTION_BEGIN(__kernel_gettimeofday)
- .cfi_startproc
- mflr r12
- .cfi_register lr,r12
-
- mr. r10,r3 /* r10 saves tv */
- mr r11,r4 /* r11 saves tz */
- get_datapage r9
- beq 3f
- LOAD_REG_IMMEDIATE(r7, 1000000) /* load up USEC_PER_SEC */
- bl __do_get_tspec@local /* get sec/usec from tb & kernel */
- stw r3,TVAL32_TV_SEC(r10)
- stw r4,TVAL32_TV_USEC(r10)
-
-3: cmplwi r11,0 /* check if tz is NULL */
- mtlr r12
- crclr cr0*4+so
- li r3,0
- beqlr
-
- lwz r4,CFG_TZ_MINUTEWEST(r9)/* fill tz */
- lwz r5,CFG_TZ_DSTTIME(r9)
- stw r4,TZONE_TZ_MINWEST(r11)
- stw r5,TZONE_TZ_DSTTIME(r11)
-
- blr
- .cfi_endproc
+ cvdso_call __c_kernel_gettimeofday
V_FUNCTION_END(__kernel_gettimeofday)
/*
@@ -63,127 +32,7 @@ V_FUNCTION_END(__kernel_gettimeofday)
*
*/
V_FUNCTION_BEGIN(__kernel_clock_gettime)
- .cfi_startproc
- /* Check for supported clock IDs */
- cmpli cr0,r3,CLOCK_REALTIME
- cmpli cr1,r3,CLOCK_MONOTONIC
- cror cr0*4+eq,cr0*4+eq,cr1*4+eq
-
- cmpli cr5,r3,CLOCK_REALTIME_COARSE
- cmpli cr6,r3,CLOCK_MONOTONIC_COARSE
- cror cr5*4+eq,cr5*4+eq,cr6*4+eq
-
- cror cr0*4+eq,cr0*4+eq,cr5*4+eq
- bne cr0, .Lgettime_fallback
-
- mflr r12 /* r12 saves lr */
- .cfi_register lr,r12
- mr r11,r4 /* r11 saves tp */
- get_datapage r9
- LOAD_REG_IMMEDIATE(r7, NSEC_PER_SEC) /* load up NSEC_PER_SEC */
- beq cr5, .Lcoarse_clocks
-.Lprecise_clocks:
- bl __do_get_tspec@local /* get sec/nsec from tb & kernel */
- bne cr1, .Lfinish /* not monotonic -> all done */
-
- /*
- * CLOCK_MONOTONIC
- */
-
- /* now we must fixup using wall to monotonic. We need to snapshot
- * that value and do the counter trick again. Fortunately, we still
- * have the counter value in r8 that was returned by __do_get_xsec.
- * At this point, r3,r4 contain our sec/nsec values, r5 and r6
- * can be used, r7 contains NSEC_PER_SEC.
- */
-
- lwz r5,(WTOM_CLOCK_SEC+LOPART)(r9)
- lwz r6,WTOM_CLOCK_NSEC(r9)
-
- /* We now have our offset in r5,r6. We create a fake dependency
- * on that value and re-check the counter
- */
- or r0,r6,r5
- xor r0,r0,r0
- add r9,r9,r0
- lwz r0,(CFG_TB_UPDATE_COUNT+LOPART)(r9)
- cmpl cr0,r8,r0 /* check if updated */
- bne- .Lprecise_clocks
- b .Lfinish_monotonic
-
- /*
- * For coarse clocks we get data directly from the vdso data page, so
- * we don't need to call __do_get_tspec, but we still need to do the
- * counter trick.
- */
-.Lcoarse_clocks:
- lwz r8,(CFG_TB_UPDATE_COUNT+LOPART)(r9)
- andi. r0,r8,1 /* pending update ? loop */
- bne- .Lcoarse_clocks
- add r9,r9,r0 /* r0 is already 0 */
-
- /*
- * CLOCK_REALTIME_COARSE, below values are needed for MONOTONIC_COARSE
- * too
- */
- lwz r3,STAMP_XTIME_SEC+LOPART(r9)
- lwz r4,STAMP_XTIME_NSEC+LOPART(r9)
- bne cr6,1f
-
- /* CLOCK_MONOTONIC_COARSE */
- lwz r5,(WTOM_CLOCK_SEC+LOPART)(r9)
- lwz r6,WTOM_CLOCK_NSEC(r9)
-
- /* check if counter has updated */
- or r0,r6,r5
-1: or r0,r0,r3
- or r0,r0,r4
- xor r0,r0,r0
- add r3,r3,r0
- lwz r0,CFG_TB_UPDATE_COUNT+LOPART(r9)
- cmpl cr0,r0,r8 /* check if updated */
- bne- .Lcoarse_clocks
-
- /* Counter has not updated, so continue calculating proper values for
- * sec and nsec if monotonic coarse, or just return with the proper
- * values for realtime.
- */
- bne cr6, .Lfinish
-
- /* Calculate and store result. Note that this mimics the C code,
- * which may cause funny results if nsec goes negative... is that
- * possible at all ?
- */
-.Lfinish_monotonic:
- add r3,r3,r5
- add r4,r4,r6
- cmpw cr0,r4,r7
- cmpwi cr1,r4,0
- blt 1f
- subf r4,r7,r4
- addi r3,r3,1
-1: bge cr1, .Lfinish
- addi r3,r3,-1
- add r4,r4,r7
-
-.Lfinish:
- stw r3,TSPC32_TV_SEC(r11)
- stw r4,TSPC32_TV_NSEC(r11)
-
- mtlr r12
- crclr cr0*4+so
- li r3,0
- blr
-
- /*
- * syscall fallback
- */
-.Lgettime_fallback:
- li r0,__NR_clock_gettime
- .cfi_restore lr
- sc
- blr
- .cfi_endproc
+ cvdso_call __c_kernel_clock_gettime
V_FUNCTION_END(__kernel_clock_gettime)
@@ -194,37 +43,7 @@ V_FUNCTION_END(__kernel_clock_gettime)
*
*/
V_FUNCTION_BEGIN(__kernel_clock_getres)
- .cfi_startproc
- /* Check for supported clock IDs */
- cmplwi cr0, r3, CLOCK_MAX
- cmpwi cr1, r3, CLOCK_REALTIME_COARSE
- cmpwi cr7, r3, CLOCK_MONOTONIC_COARSE
- bgt cr0, 99f
- LOAD_REG_IMMEDIATE(r5, KTIME_LOW_RES)
- beq cr1, 1f
- beq cr7, 1f
-
- mflr r12
- .cfi_register lr,r12
- get_datapage r3
- lwz r5, CLOCK_HRTIMER_RES(r3)
- mtlr r12
-1: li r3,0
- cmpli cr0,r4,0
- crclr cr0*4+so
- beqlr
- stw r3,TSPC32_TV_SEC(r4)
- stw r5,TSPC32_TV_NSEC(r4)
- blr
-
- /*
- * invalid clock
- */
-99:
- li r3, EINVAL
- crset so
- blr
- .cfi_endproc
+ cvdso_call __c_kernel_clock_getres
V_FUNCTION_END(__kernel_clock_getres)
@@ -235,105 +54,5 @@ V_FUNCTION_END(__kernel_clock_getres)
*
*/
V_FUNCTION_BEGIN(__kernel_time)
- .cfi_startproc
- mflr r12
- .cfi_register lr,r12
-
- mr r11,r3 /* r11 holds t */
- get_datapage r9
-
- lwz r3,STAMP_XTIME_SEC+LOPART(r9)
-
- cmplwi r11,0 /* check if t is NULL */
- mtlr r12
- crclr cr0*4+so
- beqlr
- stw r3,0(r11) /* store result at *t */
- blr
- .cfi_endproc
+ cvdso_call_time __c_kernel_time
V_FUNCTION_END(__kernel_time)
-
-/*
- * This is the core of clock_gettime() and gettimeofday(),
- * it returns the current time in r3 (seconds) and r4.
- * On entry, r7 gives the resolution of r4, either USEC_PER_SEC
- * or NSEC_PER_SEC, giving r4 in microseconds or nanoseconds.
- * It expects the datapage ptr in r9 and doesn't clobber it.
- * It clobbers r0, r5 and r6.
- * On return, r8 contains the counter value that can be reused.
- * This clobbers cr0 but not any other cr field.
- */
-__do_get_tspec:
- .cfi_startproc
- /* Check for update count & load values. We use the low
- * order 32 bits of the update count
- */
-1: lwz r8,(CFG_TB_UPDATE_COUNT+LOPART)(r9)
- andi. r0,r8,1 /* pending update ? loop */
- bne- 1b
- xor r0,r8,r8 /* create dependency */
- add r9,r9,r0
-
- /* Load orig stamp (offset to TB) */
- lwz r5,CFG_TB_ORIG_STAMP(r9)
- lwz r6,(CFG_TB_ORIG_STAMP+4)(r9)
-
- /* Get a stable TB value */
-2: MFTBU(r3)
- MFTBL(r4)
- MFTBU(r0)
- cmplw cr0,r3,r0
- bne- 2b
-
- /* Subtract tb orig stamp and shift left 12 bits.
- */
- subfc r4,r6,r4
- subfe r0,r5,r3
- slwi r0,r0,12
- rlwimi. r0,r4,12,20,31
- slwi r4,r4,12
-
- /*
- * Load scale factor & do multiplication.
- * We only use the high 32 bits of the tb_to_xs value.
- * Even with a 1GHz timebase clock, the high 32 bits of
- * tb_to_xs will be at least 4 million, so the error from
- * ignoring the low 32 bits will be no more than 0.25ppm.
- * The error will just make the clock run very very slightly
- * slow until the next time the kernel updates the VDSO data,
- * at which point the clock will catch up to the kernel's value,
- * so there is no long-term error accumulation.
- */
- lwz r5,CFG_TB_TO_XS(r9) /* load values */
- mulhwu r4,r4,r5
- li r3,0
-
- beq+ 4f /* skip high part computation if 0 */
- mulhwu r3,r0,r5
- mullw r5,r0,r5
- addc r4,r4,r5
- addze r3,r3
-4:
- /* At this point, we have seconds since the xtime stamp
- * as a 32.32 fixed-point number in r3 and r4.
- * Load & add the xtime stamp.
- */
- lwz r5,STAMP_XTIME_SEC+LOPART(r9)
- lwz r6,STAMP_SEC_FRAC(r9)
- addc r4,r4,r6
- adde r3,r3,r5
-
- /* We create a fake dependency on the result in r3/r4
- * and re-check the counter
- */
- or r6,r4,r3
- xor r0,r6,r6
- add r9,r9,r0
- lwz r0,(CFG_TB_UPDATE_COUNT+LOPART)(r9)
- cmplw cr0,r8,r0 /* check if updated */
- bne- 1b
-
- mulhwu r4,r4,r7 /* convert to micro or nanoseconds */
-
- blr
- .cfi_endproc
diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile
index 32ebb3522ea1..29bffd4863f2 100644
--- a/arch/powerpc/kernel/vdso64/Makefile
+++ b/arch/powerpc/kernel/vdso64/Makefile
@@ -1,8 +1,20 @@
# SPDX-License-Identifier: GPL-2.0
# List of files in the vdso, has to be asm only for now
+ARCH_REL_TYPE_ABS := R_PPC_JUMP_SLOT|R_PPC_GLOB_DAT|R_PPC_ADDR32|R_PPC_ADDR24|R_PPC_ADDR16|R_PPC_ADDR16_LO|R_PPC_ADDR16_HI|R_PPC_ADDR16_HA|R_PPC_ADDR14|R_PPC_ADDR14_BRTAKEN|R_PPC_ADDR14_BRNTAKEN
+include $(srctree)/lib/vdso/Makefile
+
obj-vdso64 = sigtramp.o gettimeofday.o datapage.o cacheflush.o note.o getcpu.o
+ifneq ($(c-gettimeofday-y),)
+ CFLAGS_vgettimeofday.o += -include $(c-gettimeofday-y)
+ CFLAGS_vgettimeofday.o += $(DISABLE_LATENT_ENTROPY_PLUGIN)
+ CFLAGS_vgettimeofday.o += $(call cc-option, -fno-stack-protector)
+ CFLAGS_vgettimeofday.o += -DDISABLE_BRANCH_PROFILING
+ CFLAGS_vgettimeofday.o += -ffreestanding -fasynchronous-unwind-tables
+ CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE)
+endif
+
# Build rules
targets := $(obj-vdso64) vdso64.so vdso64.so.dbg
@@ -11,6 +23,7 @@ obj-vdso64 := $(addprefix $(obj)/, $(obj-vdso64))
GCOV_PROFILE := n
KCOV_INSTRUMENT := n
UBSAN_SANITIZE := n
+KASAN_SANITIZE := n
ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
-Wl,-soname=linux-vdso64.so.1 -Wl,--hash-style=both
@@ -20,12 +33,14 @@ obj-y += vdso64_wrapper.o
extra-y += vdso64.lds
CPPFLAGS_vdso64.lds += -P -C -U$(ARCH)
+$(obj)/vgettimeofday.o: %.o: %.c FORCE
+
# Force dependency (incbin is bad)
$(obj)/vdso64_wrapper.o : $(obj)/vdso64.so
# link rule for the .so file, .lds has to be first
-$(obj)/vdso64.so.dbg: $(src)/vdso64.lds $(obj-vdso64) FORCE
- $(call if_changed,vdso64ld)
+$(obj)/vdso64.so.dbg: $(src)/vdso64.lds $(obj-vdso64) $(obj)/vgettimeofday.o FORCE
+ $(call if_changed,vdso64ld_and_check)
# strip rule for the .so file
$(obj)/%.so: OBJCOPYFLAGS := -S
@@ -33,8 +48,8 @@ $(obj)/%.so: $(obj)/%.so.dbg FORCE
$(call if_changed,objcopy)
# actual build commands
-quiet_cmd_vdso64ld = VDSO64L $@
- cmd_vdso64ld = $(CC) $(c_flags) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^)
+quiet_cmd_vdso64ld_and_check = VDSO64L $@
+ cmd_vdso64ld_and_check = $(CC) $(c_flags) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^) ; $(cmd_vdso_check)
# install commands for the unstripped file
quiet_cmd_vdso_install = INSTALL $@
diff --git a/arch/powerpc/kernel/vdso64/gettimeofday.S b/arch/powerpc/kernel/vdso64/gettimeofday.S
index 275f031d0bf1..d7a7bfb51081 100644
--- a/arch/powerpc/kernel/vdso64/gettimeofday.S
+++ b/arch/powerpc/kernel/vdso64/gettimeofday.S
@@ -9,8 +9,10 @@
#include <asm/processor.h>
#include <asm/ppc_asm.h>
#include <asm/vdso.h>
+#include <asm/vdso_datapage.h>
#include <asm/asm-offsets.h>
#include <asm/unistd.h>
+#include <asm/vdso/gettimeofday.h>
.text
/*
@@ -20,31 +22,7 @@
*
*/
V_FUNCTION_BEGIN(__kernel_gettimeofday)
- .cfi_startproc
- mflr r12
- .cfi_register lr,r12
-
- mr r11,r3 /* r11 holds tv */
- mr r10,r4 /* r10 holds tz */
- get_datapage r3
- cmpldi r11,0 /* check if tv is NULL */
- beq 2f
- lis r7,1000000@ha /* load up USEC_PER_SEC */
- addi r7,r7,1000000@l
- bl V_LOCAL_FUNC(__do_get_tspec) /* get sec/us from tb & kernel */
- std r4,TVAL64_TV_SEC(r11) /* store sec in tv */
- std r5,TVAL64_TV_USEC(r11) /* store usec in tv */
-2: cmpldi r10,0 /* check if tz is NULL */
- beq 1f
- lwz r4,CFG_TZ_MINUTEWEST(r3)/* fill tz */
- lwz r5,CFG_TZ_DSTTIME(r3)
- stw r4,TZONE_TZ_MINWEST(r10)
- stw r5,TZONE_TZ_DSTTIME(r10)
-1: mtlr r12
- crclr cr0*4+so
- li r3,0 /* always success */
- blr
- .cfi_endproc
+ cvdso_call __c_kernel_gettimeofday
V_FUNCTION_END(__kernel_gettimeofday)
@@ -55,120 +33,7 @@ V_FUNCTION_END(__kernel_gettimeofday)
*
*/
V_FUNCTION_BEGIN(__kernel_clock_gettime)
- .cfi_startproc
- /* Check for supported clock IDs */
- cmpwi cr0,r3,CLOCK_REALTIME
- cmpwi cr1,r3,CLOCK_MONOTONIC
- cror cr0*4+eq,cr0*4+eq,cr1*4+eq
-
- cmpwi cr5,r3,CLOCK_REALTIME_COARSE
- cmpwi cr6,r3,CLOCK_MONOTONIC_COARSE
- cror cr5*4+eq,cr5*4+eq,cr6*4+eq
-
- cror cr0*4+eq,cr0*4+eq,cr5*4+eq
- bne cr0,99f
-
- mflr r12 /* r12 saves lr */
- .cfi_register lr,r12
- mr r11,r4 /* r11 saves tp */
- get_datapage r3
- lis r7,NSEC_PER_SEC@h /* want nanoseconds */
- ori r7,r7,NSEC_PER_SEC@l
- beq cr5,70f
-50: bl V_LOCAL_FUNC(__do_get_tspec) /* get time from tb & kernel */
- bne cr1,80f /* if not monotonic, all done */
-
- /*
- * CLOCK_MONOTONIC
- */
-
- /* now we must fixup using wall to monotonic. We need to snapshot
- * that value and do the counter trick again. Fortunately, we still
- * have the counter value in r8 that was returned by __do_get_tspec.
- * At this point, r4,r5 contain our sec/nsec values.
- */
-
- ld r6,WTOM_CLOCK_SEC(r3)
- lwa r9,WTOM_CLOCK_NSEC(r3)
-
- /* We now have our result in r6,r9. We create a fake dependency
- * on that result and re-check the counter
- */
- or r0,r6,r9
- xor r0,r0,r0
- add r3,r3,r0
- ld r0,CFG_TB_UPDATE_COUNT(r3)
- cmpld cr0,r0,r8 /* check if updated */
- bne- 50b
- b 78f
-
- /*
- * For coarse clocks we get data directly from the vdso data page, so
- * we don't need to call __do_get_tspec, but we still need to do the
- * counter trick.
- */
-70: ld r8,CFG_TB_UPDATE_COUNT(r3)
- andi. r0,r8,1 /* pending update ? loop */
- bne- 70b
- add r3,r3,r0 /* r0 is already 0 */
-
- /*
- * CLOCK_REALTIME_COARSE, below values are needed for MONOTONIC_COARSE
- * too
- */
- ld r4,STAMP_XTIME_SEC(r3)
- ld r5,STAMP_XTIME_NSEC(r3)
- bne cr6,75f
-
- /* CLOCK_MONOTONIC_COARSE */
- ld r6,WTOM_CLOCK_SEC(r3)
- lwa r9,WTOM_CLOCK_NSEC(r3)
-
- /* check if counter has updated */
- or r0,r6,r9
-75: or r0,r0,r4
- or r0,r0,r5
- xor r0,r0,r0
- add r3,r3,r0
- ld r0,CFG_TB_UPDATE_COUNT(r3)
- cmpld cr0,r0,r8 /* check if updated */
- bne- 70b
-
- /* Counter has not updated, so continue calculating proper values for
- * sec and nsec if monotonic coarse, or just return with the proper
- * values for realtime.
- */
- bne cr6,80f
-
- /* Add wall->monotonic offset and check for overflow or underflow */
-78: add r4,r4,r6
- add r5,r5,r9
- cmpd cr0,r5,r7
- cmpdi cr1,r5,0
- blt 79f
- subf r5,r7,r5
- addi r4,r4,1
-79: bge cr1,80f
- addi r4,r4,-1
- add r5,r5,r7
-
-80: std r4,TSPC64_TV_SEC(r11)
- std r5,TSPC64_TV_NSEC(r11)
-
- mtlr r12
- crclr cr0*4+so
- li r3,0
- blr
-
- /*
- * syscall fallback
- */
-99:
- li r0,__NR_clock_gettime
- .cfi_restore lr
- sc
- blr
- .cfi_endproc
+ cvdso_call __c_kernel_clock_gettime
V_FUNCTION_END(__kernel_clock_gettime)
@@ -179,34 +44,7 @@ V_FUNCTION_END(__kernel_clock_gettime)
*
*/
V_FUNCTION_BEGIN(__kernel_clock_getres)
- .cfi_startproc
- /* Check for supported clock IDs */
- cmpwi cr0,r3,CLOCK_REALTIME
- cmpwi cr1,r3,CLOCK_MONOTONIC
- cror cr0*4+eq,cr0*4+eq,cr1*4+eq
- bne cr0,99f
-
- mflr r12
- .cfi_register lr,r12
- get_datapage r3
- lwz r5, CLOCK_HRTIMER_RES(r3)
- mtlr r12
- li r3,0
- cmpldi cr0,r4,0
- crclr cr0*4+so
- beqlr
- std r3,TSPC64_TV_SEC(r4)
- std r5,TSPC64_TV_NSEC(r4)
- blr
-
- /*
- * syscall fallback
- */
-99:
- li r0,__NR_clock_getres
- sc
- blr
- .cfi_endproc
+ cvdso_call __c_kernel_clock_getres
V_FUNCTION_END(__kernel_clock_getres)
/*
@@ -216,74 +54,5 @@ V_FUNCTION_END(__kernel_clock_getres)
*
*/
V_FUNCTION_BEGIN(__kernel_time)
- .cfi_startproc
- mflr r12
- .cfi_register lr,r12
-
- mr r11,r3 /* r11 holds t */
- get_datapage r3
-
- ld r4,STAMP_XTIME_SEC(r3)
-
- cmpldi r11,0 /* check if t is NULL */
- beq 2f
- std r4,0(r11) /* store result at *t */
-2: mtlr r12
- crclr cr0*4+so
- mr r3,r4
- blr
- .cfi_endproc
+ cvdso_call_time __c_kernel_time
V_FUNCTION_END(__kernel_time)
-
-
-/*
- * This is the core of clock_gettime() and gettimeofday(),
- * it returns the current time in r4 (seconds) and r5.
- * On entry, r7 gives the resolution of r5, either USEC_PER_SEC
- * or NSEC_PER_SEC, giving r5 in microseconds or nanoseconds.
- * It expects the datapage ptr in r3 and doesn't clobber it.
- * It clobbers r0, r6 and r9.
- * On return, r8 contains the counter value that can be reused.
- * This clobbers cr0 but not any other cr field.
- */
-V_FUNCTION_BEGIN(__do_get_tspec)
- .cfi_startproc
- /* check for update count & load values */
-1: ld r8,CFG_TB_UPDATE_COUNT(r3)
- andi. r0,r8,1 /* pending update ? loop */
- bne- 1b
- xor r0,r8,r8 /* create dependency */
- add r3,r3,r0
-
- /* Get TB & offset it. We use the MFTB macro which will generate
- * workaround code for Cell.
- */
- MFTB(r6)
- ld r9,CFG_TB_ORIG_STAMP(r3)
- subf r6,r9,r6
-
- /* Scale result */
- ld r5,CFG_TB_TO_XS(r3)
- sldi r6,r6,12 /* compute time since stamp_xtime */
- mulhdu r6,r6,r5 /* in units of 2^-32 seconds */
-
- /* Add stamp since epoch */
- ld r4,STAMP_XTIME_SEC(r3)
- lwz r5,STAMP_SEC_FRAC(r3)
- or r0,r4,r5
- or r0,r0,r6
- xor r0,r0,r0
- add r3,r3,r0
- ld r0,CFG_TB_UPDATE_COUNT(r3)
- cmpld r0,r8 /* check if updated */
- bne- 1b /* reload if so */
-
- /* convert to seconds & nanoseconds and add to stamp */
- add r6,r6,r5 /* add on fractional seconds of xtime */
- mulhwu r5,r6,r7 /* compute micro or nanoseconds and */
- srdi r6,r6,32 /* seconds since stamp_xtime */
- clrldi r5,r5,32
- add r4,r4,r6
- blr
- .cfi_endproc
-V_FUNCTION_END(__do_get_tspec)
--
2.25.0
^ permalink raw reply related
* [PATCH v7 5/7] powerpc/vdso: Prepare for switching VDSO to generic C implementation.
From: Christophe Leroy @ 2020-04-20 16:56 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman, nathanl
Cc: arnd, linux-kernel, luto, tglx, vincenzo.frascino, linuxppc-dev
In-Reply-To: <cover.1587401492.git.christophe.leroy@c-s.fr>
Prepare for switching VDSO to generic C implementation in following
patch. Here, we:
- Modify __get_datapage() to take an offset
- Prepare the helpers to call the C VDSO functions
- Prepare the required callbacks for the C VDSO functions
- Prepare the clocksource.h files to define VDSO_ARCH_CLOCKMODES
- Add the C trampolines to the generic C VDSO functions
powerpc is a bit special for VDSO as well as system calls in the
way that it requires setting CR SO bit which cannot be done in C.
Therefore, entry/exit needs to be performed in ASM.
Implementing __arch_get_vdso_data() would clobber the link register,
requiring the caller to save it. As the ASM calling function already
has to set a stack frame and saves the link register before calling
the C vdso function, retriving the vdso data pointer there is lighter.
Implement __arch_vdso_capable() and:
- When the timebase is used, make it always return true.
- When the RTC clock is used, make it always return false.
Provide vdso_shift_ns(), as the generic x >> s gives the following
bad result:
18: 35 25 ff e0 addic. r9,r5,-32
1c: 41 80 00 10 blt 2c <shift+0x14>
20: 7c 64 4c 30 srw r4,r3,r9
24: 38 60 00 00 li r3,0
...
2c: 54 69 08 3c rlwinm r9,r3,1,0,30
30: 21 45 00 1f subfic r10,r5,31
34: 7c 84 2c 30 srw r4,r4,r5
38: 7d 29 50 30 slw r9,r9,r10
3c: 7c 63 2c 30 srw r3,r3,r5
40: 7d 24 23 78 or r4,r9,r4
In our case the shift is always <= 32. In addition, the upper 32 bits
of the result are likely nul. Lets GCC know it, it also optimises the
following calculations.
With the patch, we get:
0: 21 25 00 20 subfic r9,r5,32
4: 7c 69 48 30 slw r9,r3,r9
8: 7c 84 2c 30 srw r4,r4,r5
c: 7d 24 23 78 or r4,r9,r4
10: 7c 63 2c 30 srw r3,r3,r5
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
v8:
- New, splitted out of last patch of the series
---
arch/powerpc/include/asm/clocksource.h | 7 +
arch/powerpc/include/asm/vdso/clocksource.h | 7 +
arch/powerpc/include/asm/vdso/gettimeofday.h | 175 +++++++++++++++++++
arch/powerpc/include/asm/vdso_datapage.h | 6 +-
arch/powerpc/kernel/vdso32/vgettimeofday.c | 29 +++
arch/powerpc/kernel/vdso64/vgettimeofday.c | 29 +++
6 files changed, 250 insertions(+), 3 deletions(-)
create mode 100644 arch/powerpc/include/asm/clocksource.h
create mode 100644 arch/powerpc/include/asm/vdso/clocksource.h
create mode 100644 arch/powerpc/include/asm/vdso/gettimeofday.h
create mode 100644 arch/powerpc/kernel/vdso32/vgettimeofday.c
create mode 100644 arch/powerpc/kernel/vdso64/vgettimeofday.c
diff --git a/arch/powerpc/include/asm/clocksource.h b/arch/powerpc/include/asm/clocksource.h
new file mode 100644
index 000000000000..482185566b0c
--- /dev/null
+++ b/arch/powerpc/include/asm/clocksource.h
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_CLOCKSOURCE_H
+#define _ASM_CLOCKSOURCE_H
+
+#include <asm/vdso/clocksource.h>
+
+#endif
diff --git a/arch/powerpc/include/asm/vdso/clocksource.h b/arch/powerpc/include/asm/vdso/clocksource.h
new file mode 100644
index 000000000000..ec5d672d2569
--- /dev/null
+++ b/arch/powerpc/include/asm/vdso/clocksource.h
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_VDSOCLOCKSOURCE_H
+#define __ASM_VDSOCLOCKSOURCE_H
+
+#define VDSO_ARCH_CLOCKMODES VDSO_CLOCKMODE_ARCHTIMER
+
+#endif
diff --git a/arch/powerpc/include/asm/vdso/gettimeofday.h b/arch/powerpc/include/asm/vdso/gettimeofday.h
new file mode 100644
index 000000000000..4452897f9bd8
--- /dev/null
+++ b/arch/powerpc/include/asm/vdso/gettimeofday.h
@@ -0,0 +1,175 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_VDSO_GETTIMEOFDAY_H
+#define __ASM_VDSO_GETTIMEOFDAY_H
+
+#include <asm/ptrace.h>
+
+#ifdef __ASSEMBLY__
+
+.macro cvdso_call funct
+ .cfi_startproc
+ PPC_STLU r1, -STACK_FRAME_OVERHEAD(r1)
+ mflr r0
+ .cfi_register lr, r0
+ PPC_STL r0, STACK_FRAME_OVERHEAD + PPC_LR_STKOFF(r1)
+ get_datapage r5, VDSO_DATA_OFFSET
+ bl \funct
+ PPC_LL r0, STACK_FRAME_OVERHEAD + PPC_LR_STKOFF(r1)
+ cmpwi r3, 0
+ mtlr r0
+ .cfi_restore lr
+ addi r1, r1, STACK_FRAME_OVERHEAD
+ crclr so
+ beqlr+
+ crset so
+ neg r3, r3
+ blr
+ .cfi_endproc
+.endm
+
+.macro cvdso_call_time funct
+ .cfi_startproc
+ PPC_STLU r1, -STACK_FRAME_OVERHEAD(r1)
+ mflr r0
+ .cfi_register lr, r0
+ PPC_STL r0, STACK_FRAME_OVERHEAD + PPC_LR_STKOFF(r1)
+ get_datapage r4, VDSO_DATA_OFFSET
+ bl \funct
+ PPC_LL r0, STACK_FRAME_OVERHEAD + PPC_LR_STKOFF(r1)
+ crclr so
+ mtlr r0
+ .cfi_restore lr
+ addi r1, r1, STACK_FRAME_OVERHEAD
+ blr
+ .cfi_endproc
+.endm
+
+#else
+
+#include <asm/time.h>
+#include <asm/unistd.h>
+#include <uapi/linux/time.h>
+
+#define VDSO_HAS_CLOCK_GETRES 1
+
+#define VDSO_HAS_TIME 1
+
+static __always_inline int do_syscall_2(const unsigned long _r0, const unsigned long _r3,
+ const unsigned long _r4)
+{
+ register long r0 asm("r0") = _r0;
+ register unsigned long r3 asm("r3") = _r3;
+ register unsigned long r4 asm("r4") = _r4;
+ register int ret asm ("r3");
+
+ asm volatile(
+ " sc\n"
+ " bns+ 1f\n"
+ " neg %0, %0\n"
+ "1:\n"
+ : "=r" (ret), "+r" (r4), "+r" (r0)
+ : "r" (r3)
+ : "memory", "r5", "r6", "r7", "r8", "r9", "r10", "r11", "r12", "cr0", "ctr");
+
+ return ret;
+}
+
+static __always_inline
+int gettimeofday_fallback(struct __kernel_old_timeval *_tv, struct timezone *_tz)
+{
+ return do_syscall_2(__NR_gettimeofday, (unsigned long)_tv, (unsigned long)_tz);
+}
+
+static __always_inline
+int clock_gettime_fallback(clockid_t _clkid, struct __kernel_timespec *_ts)
+{
+ return do_syscall_2(__NR_clock_gettime, _clkid, (unsigned long)_ts);
+}
+
+static __always_inline
+int clock_getres_fallback(clockid_t _clkid, struct __kernel_timespec *_ts)
+{
+ return do_syscall_2(__NR_clock_getres, _clkid, (unsigned long)_ts);
+}
+
+#ifdef CONFIG_VDSO32
+
+#define BUILD_VDSO32 1
+
+static __always_inline
+int clock_gettime32_fallback(clockid_t _clkid, struct old_timespec32 *_ts)
+{
+ return do_syscall_2(__NR_clock_gettime, _clkid, (unsigned long)_ts);
+}
+
+static __always_inline
+int clock_getres32_fallback(clockid_t _clkid, struct old_timespec32 *_ts)
+{
+ return do_syscall_2(__NR_clock_getres, _clkid, (unsigned long)_ts);
+}
+#endif
+
+static __always_inline u64 __arch_get_hw_counter(s32 clock_mode)
+{
+ return get_tb();
+}
+
+const struct vdso_data *__arch_get_vdso_data(void);
+
+static inline bool vdso_clocksource_ok(const struct vdso_data *vd)
+{
+ return !__USE_RTC();
+}
+#define vdso_clocksource_ok vdso_clocksource_ok
+
+/*
+ * powerpc specific delta calculation.
+ *
+ * This variant removes the masking of the subtraction because the
+ * clocksource mask of all VDSO capable clocksources on powerpc is U64_MAX
+ * which would result in a pointless operation. The compiler cannot
+ * optimize it away as the mask comes from the vdso data and is not compile
+ * time constant.
+ */
+static __always_inline u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult)
+{
+ return (cycles - last) * mult;
+}
+#define vdso_calc_delta vdso_calc_delta
+
+#ifndef __powerpc64__
+static __always_inline u64 vdso_shift_ns(u64 ns, unsigned long shift)
+{
+ u32 hi = ns >> 32;
+ u32 lo = ns;
+
+ lo >>= shift;
+ lo |= hi << (32 - shift);
+ hi >>= shift;
+
+ if (likely(hi == 0))
+ return lo;
+
+ return ((u64)hi << 32) | lo;
+}
+#define vdso_shift_ns vdso_shift_ns
+#endif
+
+#ifdef __powerpc64__
+int __c_kernel_clock_gettime(clockid_t clock, struct __kernel_timespec *ts,
+ const struct vdso_data *vd);
+int __c_kernel_clock_getres(clockid_t clock_id, struct __kernel_timespec *res,
+ const struct vdso_data *vd);
+#else
+int __c_kernel_clock_gettime(clockid_t clock, struct old_timespec32 *ts,
+ const struct vdso_data *vd);
+int __c_kernel_clock_getres(clockid_t clock_id, struct old_timespec32 *res,
+ const struct vdso_data *vd);
+#endif
+int __c_kernel_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz,
+ const struct vdso_data *vd);
+__kernel_old_time_t __c_kernel_time(__kernel_old_time_t *time,
+ const struct vdso_data *vd);
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_VDSO_GETTIMEOFDAY_H */
diff --git a/arch/powerpc/include/asm/vdso_datapage.h b/arch/powerpc/include/asm/vdso_datapage.h
index f3d7e4e2a45b..78b8f67e9873 100644
--- a/arch/powerpc/include/asm/vdso_datapage.h
+++ b/arch/powerpc/include/asm/vdso_datapage.h
@@ -116,14 +116,14 @@ extern struct vdso_data *vdso_data;
#else /* __ASSEMBLY__ */
-.macro get_datapage ptr
+.macro get_datapage ptr, offset=0
bcl 20, 31, .+4
999:
mflr \ptr
#if CONFIG_PPC_PAGE_SHIFT > 14
- addis \ptr, \ptr, (_vdso_datapage - 999b)@ha
+ addis \ptr, \ptr, (_vdso_datapage + \offset - 999b)@ha
#endif
- addi \ptr, \ptr, (_vdso_datapage - 999b)@l
+ addi \ptr, \ptr, (_vdso_datapage + \offset - 999b)@l
.endm
#endif /* __ASSEMBLY__ */
diff --git a/arch/powerpc/kernel/vdso32/vgettimeofday.c b/arch/powerpc/kernel/vdso32/vgettimeofday.c
new file mode 100644
index 000000000000..0b9ab4c22ef2
--- /dev/null
+++ b/arch/powerpc/kernel/vdso32/vgettimeofday.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Powerpc userspace implementations of gettimeofday() and similar.
+ */
+#include <linux/time.h>
+#include <linux/types.h>
+
+int __c_kernel_clock_gettime(clockid_t clock, struct old_timespec32 *ts,
+ const struct vdso_data *vd)
+{
+ return __cvdso_clock_gettime32_data(vd, clock, ts);
+}
+
+int __c_kernel_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz,
+ const struct vdso_data *vd)
+{
+ return __cvdso_gettimeofday_data(vd, tv, tz);
+}
+
+int __c_kernel_clock_getres(clockid_t clock_id, struct old_timespec32 *res,
+ const struct vdso_data *vd)
+{
+ return __cvdso_clock_getres_time32_data(vd, clock_id, res);
+}
+
+__kernel_old_time_t __c_kernel_time(__kernel_old_time_t *time, const struct vdso_data *vd)
+{
+ return __cvdso_time_data(vd, time);
+}
diff --git a/arch/powerpc/kernel/vdso64/vgettimeofday.c b/arch/powerpc/kernel/vdso64/vgettimeofday.c
new file mode 100644
index 000000000000..5b5500058344
--- /dev/null
+++ b/arch/powerpc/kernel/vdso64/vgettimeofday.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Powerpc userspace implementations of gettimeofday() and similar.
+ */
+#include <linux/time.h>
+#include <linux/types.h>
+
+int __c_kernel_clock_gettime(clockid_t clock, struct __kernel_timespec *ts,
+ const struct vdso_data *vd)
+{
+ return __cvdso_clock_gettime_data(vd, clock, ts);
+}
+
+int __c_kernel_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz,
+ const struct vdso_data *vd)
+{
+ return __cvdso_gettimeofday_data(vd, tv, tz);
+}
+
+int __c_kernel_clock_getres(clockid_t clock_id, struct __kernel_timespec *res,
+ const struct vdso_data *vd)
+{
+ return __cvdso_clock_getres_data(vd, clock_id, res);
+}
+
+__kernel_old_time_t __c_kernel_time(__kernel_old_time_t *time, const struct vdso_data *vd)
+{
+ return __cvdso_time_data(vd, time);
+}
--
2.25.0
^ permalink raw reply related
* [PATCH v7 7/7] [NOT TO BE MERGED] Export sysrq_mask
From: Christophe Leroy @ 2020-04-20 16:56 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman, nathanl
Cc: arnd, linux-kernel, luto, tglx, vincenzo.frascino, linuxppc-dev
In-Reply-To: <cover.1587401492.git.christophe.leroy@c-s.fr>
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
drivers/tty/sysrq.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
index 5e0d0813da55..a0760bcd7a97 100644
--- a/drivers/tty/sysrq.c
+++ b/drivers/tty/sysrq.c
@@ -74,6 +74,7 @@ int sysrq_mask(void)
return 1;
return sysrq_enabled;
}
+EXPORT_SYMBOL_GPL(sysrq_mask);
/*
* A value of 1 means 'all', other nonzero values are an op mask:
--
2.25.0
^ permalink raw reply related
* [PATCH v7 4/7] powerpc/processor: Move cpu_relax() into asm/vdso/processor.h
From: Christophe Leroy @ 2020-04-20 16:56 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman, nathanl
Cc: arnd, linux-kernel, luto, tglx, vincenzo.frascino, linuxppc-dev
In-Reply-To: <cover.1587401492.git.christophe.leroy@c-s.fr>
cpu_relax() need to be in asm/vdso/processor.h to be used by
the C VDSO generic library.
Move it there.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
arch/powerpc/include/asm/processor.h | 10 ++--------
arch/powerpc/include/asm/vdso/processor.h | 23 +++++++++++++++++++++++
2 files changed, 25 insertions(+), 8 deletions(-)
create mode 100644 arch/powerpc/include/asm/vdso/processor.h
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index eedcbfb9a6ff..89c9eab8e45c 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -6,6 +6,8 @@
* Copyright (C) 2001 PPC 64 Team, IBM Corp
*/
+#include <vdso/processor.h>
+
#include <asm/reg.h>
#ifdef CONFIG_VSX
@@ -63,14 +65,6 @@ extern int _chrp_type;
#endif /* defined(__KERNEL__) && defined(CONFIG_PPC32) */
-/* Macros for adjusting thread priority (hardware multi-threading) */
-#define HMT_very_low() asm volatile("or 31,31,31 # very low priority")
-#define HMT_low() asm volatile("or 1,1,1 # low priority")
-#define HMT_medium_low() asm volatile("or 6,6,6 # medium low priority")
-#define HMT_medium() asm volatile("or 2,2,2 # medium priority")
-#define HMT_medium_high() asm volatile("or 5,5,5 # medium high priority")
-#define HMT_high() asm volatile("or 3,3,3 # high priority")
-
#ifdef __KERNEL__
#ifdef CONFIG_PPC64
diff --git a/arch/powerpc/include/asm/vdso/processor.h b/arch/powerpc/include/asm/vdso/processor.h
new file mode 100644
index 000000000000..39b9beace9ca
--- /dev/null
+++ b/arch/powerpc/include/asm/vdso/processor.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_VDSO_PROCESSOR_H
+#define __ASM_VDSO_PROCESSOR_H
+
+#ifndef __ASSEMBLY__
+
+/* Macros for adjusting thread priority (hardware multi-threading) */
+#define HMT_very_low() asm volatile("or 31, 31, 31 # very low priority")
+#define HMT_low() asm volatile("or 1, 1, 1 # low priority")
+#define HMT_medium_low() asm volatile("or 6, 6, 6 # medium low priority")
+#define HMT_medium() asm volatile("or 2, 2, 2 # medium priority")
+#define HMT_medium_high() asm volatile("or 5, 5, 5 # medium high priority")
+#define HMT_high() asm volatile("or 3, 3, 3 # high priority")
+
+#ifdef CONFIG_PPC64
+#define cpu_relax() do { HMT_low(); HMT_medium(); barrier(); } while (0)
+#else
+#define cpu_relax() barrier()
+#endif
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_VDSO_PROCESSOR_H */
--
2.25.0
^ permalink raw reply related
* [PATCH v7 3/7] powerpc/vdso: Remove unused \tmp param in __get_datapage()
From: Christophe Leroy @ 2020-04-20 16:56 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman, nathanl
Cc: arnd, linux-kernel, luto, tglx, vincenzo.frascino, linuxppc-dev
In-Reply-To: <cover.1587401492.git.christophe.leroy@c-s.fr>
The \tmp param is not used anymore, remove it.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
v7: New patch, splitted out of preceding patch
---
arch/powerpc/include/asm/vdso_datapage.h | 2 +-
arch/powerpc/kernel/vdso32/cacheflush.S | 2 +-
arch/powerpc/kernel/vdso32/datapage.S | 4 ++--
arch/powerpc/kernel/vdso32/gettimeofday.S | 8 ++++----
arch/powerpc/kernel/vdso64/cacheflush.S | 2 +-
arch/powerpc/kernel/vdso64/datapage.S | 4 ++--
arch/powerpc/kernel/vdso64/gettimeofday.S | 8 ++++----
7 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/arch/powerpc/include/asm/vdso_datapage.h b/arch/powerpc/include/asm/vdso_datapage.h
index 11886467dfdf..f3d7e4e2a45b 100644
--- a/arch/powerpc/include/asm/vdso_datapage.h
+++ b/arch/powerpc/include/asm/vdso_datapage.h
@@ -116,7 +116,7 @@ extern struct vdso_data *vdso_data;
#else /* __ASSEMBLY__ */
-.macro get_datapage ptr, tmp
+.macro get_datapage ptr
bcl 20, 31, .+4
999:
mflr \ptr
diff --git a/arch/powerpc/kernel/vdso32/cacheflush.S b/arch/powerpc/kernel/vdso32/cacheflush.S
index 3440ddf21c8b..017843bf5382 100644
--- a/arch/powerpc/kernel/vdso32/cacheflush.S
+++ b/arch/powerpc/kernel/vdso32/cacheflush.S
@@ -27,7 +27,7 @@ V_FUNCTION_BEGIN(__kernel_sync_dicache)
#ifdef CONFIG_PPC64
mflr r12
.cfi_register lr,r12
- get_datapage r10, r0
+ get_datapage r10
mtlr r12
#endif
diff --git a/arch/powerpc/kernel/vdso32/datapage.S b/arch/powerpc/kernel/vdso32/datapage.S
index 5513a4f8253e..0513a2eabec8 100644
--- a/arch/powerpc/kernel/vdso32/datapage.S
+++ b/arch/powerpc/kernel/vdso32/datapage.S
@@ -28,7 +28,7 @@ V_FUNCTION_BEGIN(__kernel_get_syscall_map)
mflr r12
.cfi_register lr,r12
mr. r4,r3
- get_datapage r3, r0
+ get_datapage r3
mtlr r12
addi r3,r3,CFG_SYSCALL_MAP32
beqlr
@@ -49,7 +49,7 @@ V_FUNCTION_BEGIN(__kernel_get_tbfreq)
.cfi_startproc
mflr r12
.cfi_register lr,r12
- get_datapage r3, r0
+ get_datapage r3
lwz r4,(CFG_TB_TICKS_PER_SEC + 4)(r3)
lwz r3,CFG_TB_TICKS_PER_SEC(r3)
mtlr r12
diff --git a/arch/powerpc/kernel/vdso32/gettimeofday.S b/arch/powerpc/kernel/vdso32/gettimeofday.S
index a3951567118a..0bbdce0f2a9c 100644
--- a/arch/powerpc/kernel/vdso32/gettimeofday.S
+++ b/arch/powerpc/kernel/vdso32/gettimeofday.S
@@ -34,7 +34,7 @@ V_FUNCTION_BEGIN(__kernel_gettimeofday)
mr. r10,r3 /* r10 saves tv */
mr r11,r4 /* r11 saves tz */
- get_datapage r9, r0
+ get_datapage r9
beq 3f
LOAD_REG_IMMEDIATE(r7, 1000000) /* load up USEC_PER_SEC */
bl __do_get_tspec@local /* get sec/usec from tb & kernel */
@@ -79,7 +79,7 @@ V_FUNCTION_BEGIN(__kernel_clock_gettime)
mflr r12 /* r12 saves lr */
.cfi_register lr,r12
mr r11,r4 /* r11 saves tp */
- get_datapage r9, r0
+ get_datapage r9
LOAD_REG_IMMEDIATE(r7, NSEC_PER_SEC) /* load up NSEC_PER_SEC */
beq cr5, .Lcoarse_clocks
.Lprecise_clocks:
@@ -206,7 +206,7 @@ V_FUNCTION_BEGIN(__kernel_clock_getres)
mflr r12
.cfi_register lr,r12
- get_datapage r3, r0
+ get_datapage r3
lwz r5, CLOCK_HRTIMER_RES(r3)
mtlr r12
1: li r3,0
@@ -240,7 +240,7 @@ V_FUNCTION_BEGIN(__kernel_time)
.cfi_register lr,r12
mr r11,r3 /* r11 holds t */
- get_datapage r9, r0
+ get_datapage r9
lwz r3,STAMP_XTIME_SEC+LOPART(r9)
diff --git a/arch/powerpc/kernel/vdso64/cacheflush.S b/arch/powerpc/kernel/vdso64/cacheflush.S
index cab14324242b..61985de5758f 100644
--- a/arch/powerpc/kernel/vdso64/cacheflush.S
+++ b/arch/powerpc/kernel/vdso64/cacheflush.S
@@ -25,7 +25,7 @@ V_FUNCTION_BEGIN(__kernel_sync_dicache)
.cfi_startproc
mflr r12
.cfi_register lr,r12
- get_datapage r10, r0
+ get_datapage r10
mtlr r12
lwz r7,CFG_DCACHE_BLOCKSZ(r10)
diff --git a/arch/powerpc/kernel/vdso64/datapage.S b/arch/powerpc/kernel/vdso64/datapage.S
index 03bb72c440dc..00760dc69d68 100644
--- a/arch/powerpc/kernel/vdso64/datapage.S
+++ b/arch/powerpc/kernel/vdso64/datapage.S
@@ -28,7 +28,7 @@ V_FUNCTION_BEGIN(__kernel_get_syscall_map)
mflr r12
.cfi_register lr,r12
mr r4,r3
- get_datapage r3, r0
+ get_datapage r3
mtlr r12
addi r3,r3,CFG_SYSCALL_MAP64
cmpldi cr0,r4,0
@@ -50,7 +50,7 @@ V_FUNCTION_BEGIN(__kernel_get_tbfreq)
.cfi_startproc
mflr r12
.cfi_register lr,r12
- get_datapage r3, r0
+ get_datapage r3
ld r3,CFG_TB_TICKS_PER_SEC(r3)
mtlr r12
crclr cr0*4+so
diff --git a/arch/powerpc/kernel/vdso64/gettimeofday.S b/arch/powerpc/kernel/vdso64/gettimeofday.S
index e54c4ce4d356..275f031d0bf1 100644
--- a/arch/powerpc/kernel/vdso64/gettimeofday.S
+++ b/arch/powerpc/kernel/vdso64/gettimeofday.S
@@ -26,7 +26,7 @@ V_FUNCTION_BEGIN(__kernel_gettimeofday)
mr r11,r3 /* r11 holds tv */
mr r10,r4 /* r10 holds tz */
- get_datapage r3, r0
+ get_datapage r3
cmpldi r11,0 /* check if tv is NULL */
beq 2f
lis r7,1000000@ha /* load up USEC_PER_SEC */
@@ -71,7 +71,7 @@ V_FUNCTION_BEGIN(__kernel_clock_gettime)
mflr r12 /* r12 saves lr */
.cfi_register lr,r12
mr r11,r4 /* r11 saves tp */
- get_datapage r3, r0
+ get_datapage r3
lis r7,NSEC_PER_SEC@h /* want nanoseconds */
ori r7,r7,NSEC_PER_SEC@l
beq cr5,70f
@@ -188,7 +188,7 @@ V_FUNCTION_BEGIN(__kernel_clock_getres)
mflr r12
.cfi_register lr,r12
- get_datapage r3, r0
+ get_datapage r3
lwz r5, CLOCK_HRTIMER_RES(r3)
mtlr r12
li r3,0
@@ -221,7 +221,7 @@ V_FUNCTION_BEGIN(__kernel_time)
.cfi_register lr,r12
mr r11,r3 /* r11 holds t */
- get_datapage r3, r0
+ get_datapage r3
ld r4,STAMP_XTIME_SEC(r3)
--
2.25.0
^ permalink raw reply related
* [PATCH v7 0/7] powerpc: switch VDSO to C implementation
From: Christophe Leroy @ 2020-04-20 16:56 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman, nathanl
Cc: arnd, linux-kernel, luto, tglx, vincenzo.frascino, linuxppc-dev
This is the seventh version of a series to switch powerpc VDSO to
generic C implementation.
Main changes since v6 are:
- Added -fasynchronous-unwind-tables in CFLAGS
- Split patch 2 in two parts
- Split patch 5 (which was patch 4) in two parts
This series applies on today's powerpc/merge branch.
See the last two patches for details on changes and performance.
Christophe Leroy (7):
powerpc/vdso64: Switch from __get_datapage() to get_datapage inline
macro
powerpc/vdso: Remove __kernel_datapage_offset and simplify
__get_datapage()
powerpc/vdso: Remove unused \tmp param in __get_datapage()
powerpc/processor: Move cpu_relax() into asm/vdso/processor.h
powerpc/vdso: Prepare for switching VDSO to generic C implementation.
powerpc/vdso: Switch VDSO to generic C implementation.
[NOT TO BE MERGED] Export sysrq_mask
arch/powerpc/Kconfig | 2 +
arch/powerpc/include/asm/clocksource.h | 7 +
arch/powerpc/include/asm/processor.h | 10 +-
arch/powerpc/include/asm/vdso/clocksource.h | 7 +
arch/powerpc/include/asm/vdso/gettimeofday.h | 175 +++++++++++
arch/powerpc/include/asm/vdso/processor.h | 23 ++
arch/powerpc/include/asm/vdso/vsyscall.h | 25 ++
arch/powerpc/include/asm/vdso_datapage.h | 50 ++--
arch/powerpc/kernel/asm-offsets.c | 49 +---
arch/powerpc/kernel/time.c | 91 +-----
arch/powerpc/kernel/vdso.c | 58 +---
arch/powerpc/kernel/vdso32/Makefile | 32 +-
arch/powerpc/kernel/vdso32/cacheflush.S | 2 +-
arch/powerpc/kernel/vdso32/config-fake32.h | 34 +++
arch/powerpc/kernel/vdso32/datapage.S | 7 +-
arch/powerpc/kernel/vdso32/gettimeofday.S | 291 +------------------
arch/powerpc/kernel/vdso32/vdso32.lds.S | 7 +-
arch/powerpc/kernel/vdso32/vgettimeofday.c | 29 ++
arch/powerpc/kernel/vdso64/Makefile | 23 +-
arch/powerpc/kernel/vdso64/cacheflush.S | 9 +-
arch/powerpc/kernel/vdso64/datapage.S | 31 +-
arch/powerpc/kernel/vdso64/gettimeofday.S | 243 +---------------
arch/powerpc/kernel/vdso64/vdso64.lds.S | 7 +-
arch/powerpc/kernel/vdso64/vgettimeofday.c | 29 ++
drivers/tty/sysrq.c | 1 +
25 files changed, 444 insertions(+), 798 deletions(-)
create mode 100644 arch/powerpc/include/asm/clocksource.h
create mode 100644 arch/powerpc/include/asm/vdso/clocksource.h
create mode 100644 arch/powerpc/include/asm/vdso/gettimeofday.h
create mode 100644 arch/powerpc/include/asm/vdso/processor.h
create mode 100644 arch/powerpc/include/asm/vdso/vsyscall.h
create mode 100644 arch/powerpc/kernel/vdso32/config-fake32.h
create mode 100644 arch/powerpc/kernel/vdso32/vgettimeofday.c
create mode 100644 arch/powerpc/kernel/vdso64/vgettimeofday.c
--
2.25.0
^ permalink raw reply
* [PATCH v7 2/7] powerpc/vdso: Remove __kernel_datapage_offset and simplify __get_datapage()
From: Christophe Leroy @ 2020-04-20 16:56 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman, nathanl
Cc: arnd, linux-kernel, luto, tglx, vincenzo.frascino, linuxppc-dev
In-Reply-To: <cover.1587401492.git.christophe.leroy@c-s.fr>
The VDSO datapage and the text pages are always located immediately
next to each other, so it can be hardcoded without an indirection
through __kernel_datapage_offset
In order to ease things, move the data page in front like other
arches, that way there is no need to know the size of the library
to locate the data page.
Before:
clock-getres-realtime-coarse: vdso: 714 nsec/call
clock-gettime-realtime-coarse: vdso: 792 nsec/call
clock-gettime-realtime: vdso: 1243 nsec/call
After:
clock-getres-realtime-coarse: vdso: 699 nsec/call
clock-gettime-realtime-coarse: vdso: 784 nsec/call
clock-gettime-realtime: vdso: 1231 nsec/call
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
v7:
- Moved the removal of the tmp param of __get_datapage()
in a subsequent patch
- Included the addition of the offset param to __get_datapage()
in the further preparatory patch
---
arch/powerpc/include/asm/vdso_datapage.h | 8 ++--
arch/powerpc/kernel/vdso.c | 53 ++++--------------------
arch/powerpc/kernel/vdso32/datapage.S | 3 --
arch/powerpc/kernel/vdso32/vdso32.lds.S | 7 +---
arch/powerpc/kernel/vdso64/datapage.S | 3 --
arch/powerpc/kernel/vdso64/vdso64.lds.S | 7 +---
6 files changed, 16 insertions(+), 65 deletions(-)
diff --git a/arch/powerpc/include/asm/vdso_datapage.h b/arch/powerpc/include/asm/vdso_datapage.h
index b9ef6cf50ea5..11886467dfdf 100644
--- a/arch/powerpc/include/asm/vdso_datapage.h
+++ b/arch/powerpc/include/asm/vdso_datapage.h
@@ -118,10 +118,12 @@ extern struct vdso_data *vdso_data;
.macro get_datapage ptr, tmp
bcl 20, 31, .+4
+999:
mflr \ptr
- addi \ptr, \ptr, (__kernel_datapage_offset - (.-4))@l
- lwz \tmp, 0(\ptr)
- add \ptr, \tmp, \ptr
+#if CONFIG_PPC_PAGE_SHIFT > 14
+ addis \ptr, \ptr, (_vdso_datapage - 999b)@ha
+#endif
+ addi \ptr, \ptr, (_vdso_datapage - 999b)@l
.endm
#endif /* __ASSEMBLY__ */
diff --git a/arch/powerpc/kernel/vdso.c b/arch/powerpc/kernel/vdso.c
index f38f26e844b6..d33fa22ddbed 100644
--- a/arch/powerpc/kernel/vdso.c
+++ b/arch/powerpc/kernel/vdso.c
@@ -190,7 +190,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
* install_special_mapping or the perf counter mmap tracking code
* will fail to recognise it as a vDSO (since arch_vma_name fails).
*/
- current->mm->context.vdso_base = vdso_base;
+ current->mm->context.vdso_base = vdso_base + PAGE_SIZE;
/*
* our vma flags don't have VM_WRITE so by default, the process isn't
@@ -482,42 +482,6 @@ static __init void vdso_setup_trampolines(struct lib32_elfinfo *v32,
vdso32_rt_sigtramp = find_function32(v32, "__kernel_sigtramp_rt32");
}
-static __init int vdso_fixup_datapage(struct lib32_elfinfo *v32,
- struct lib64_elfinfo *v64)
-{
-#ifdef CONFIG_VDSO32
- Elf32_Sym *sym32;
-#endif
-#ifdef CONFIG_PPC64
- Elf64_Sym *sym64;
-
- sym64 = find_symbol64(v64, "__kernel_datapage_offset");
- if (sym64 == NULL) {
- printk(KERN_ERR "vDSO64: Can't find symbol "
- "__kernel_datapage_offset !\n");
- return -1;
- }
- *((int *)(vdso64_kbase + sym64->st_value - VDSO64_LBASE)) =
- (vdso64_pages << PAGE_SHIFT) -
- (sym64->st_value - VDSO64_LBASE);
-#endif /* CONFIG_PPC64 */
-
-#ifdef CONFIG_VDSO32
- sym32 = find_symbol32(v32, "__kernel_datapage_offset");
- if (sym32 == NULL) {
- printk(KERN_ERR "vDSO32: Can't find symbol "
- "__kernel_datapage_offset !\n");
- return -1;
- }
- *((int *)(vdso32_kbase + (sym32->st_value - VDSO32_LBASE))) =
- (vdso32_pages << PAGE_SHIFT) -
- (sym32->st_value - VDSO32_LBASE);
-#endif
-
- return 0;
-}
-
-
static __init int vdso_fixup_features(struct lib32_elfinfo *v32,
struct lib64_elfinfo *v64)
{
@@ -618,9 +582,6 @@ static __init int vdso_setup(void)
if (vdso_do_find_sections(&v32, &v64))
return -1;
- if (vdso_fixup_datapage(&v32, &v64))
- return -1;
-
if (vdso_fixup_features(&v32, &v64))
return -1;
@@ -761,26 +722,26 @@ static int __init vdso_init(void)
vdso32_pagelist = kcalloc(vdso32_pages + 2, sizeof(struct page *),
GFP_KERNEL);
BUG_ON(vdso32_pagelist == NULL);
+ vdso32_pagelist[0] = virt_to_page(vdso_data);
for (i = 0; i < vdso32_pages; i++) {
struct page *pg = virt_to_page(vdso32_kbase + i*PAGE_SIZE);
get_page(pg);
- vdso32_pagelist[i] = pg;
+ vdso32_pagelist[i + 1] = pg;
}
- vdso32_pagelist[i++] = virt_to_page(vdso_data);
- vdso32_pagelist[i] = NULL;
+ vdso32_pagelist[i + 1] = NULL;
#endif
#ifdef CONFIG_PPC64
vdso64_pagelist = kcalloc(vdso64_pages + 2, sizeof(struct page *),
GFP_KERNEL);
BUG_ON(vdso64_pagelist == NULL);
+ vdso64_pagelist[0] = virt_to_page(vdso_data);
for (i = 0; i < vdso64_pages; i++) {
struct page *pg = virt_to_page(vdso64_kbase + i*PAGE_SIZE);
get_page(pg);
- vdso64_pagelist[i] = pg;
+ vdso64_pagelist[i + 1] = pg;
}
- vdso64_pagelist[i++] = virt_to_page(vdso_data);
- vdso64_pagelist[i] = NULL;
+ vdso64_pagelist[i + 1] = NULL;
#endif /* CONFIG_PPC64 */
get_page(virt_to_page(vdso_data));
diff --git a/arch/powerpc/kernel/vdso32/datapage.S b/arch/powerpc/kernel/vdso32/datapage.S
index 217bb630f8f9..5513a4f8253e 100644
--- a/arch/powerpc/kernel/vdso32/datapage.S
+++ b/arch/powerpc/kernel/vdso32/datapage.S
@@ -13,9 +13,6 @@
#include <asm/vdso_datapage.h>
.text
- .global __kernel_datapage_offset;
-__kernel_datapage_offset:
- .long 0
/*
* void *__kernel_get_syscall_map(unsigned int *syscall_count) ;
diff --git a/arch/powerpc/kernel/vdso32/vdso32.lds.S b/arch/powerpc/kernel/vdso32/vdso32.lds.S
index 5206c2eb2a1d..6cf729612268 100644
--- a/arch/powerpc/kernel/vdso32/vdso32.lds.S
+++ b/arch/powerpc/kernel/vdso32/vdso32.lds.S
@@ -4,6 +4,7 @@
* library
*/
#include <asm/vdso.h>
+#include <asm/page.h>
#ifdef __LITTLE_ENDIAN__
OUTPUT_FORMAT("elf32-powerpcle", "elf32-powerpcle", "elf32-powerpcle")
@@ -15,6 +16,7 @@ ENTRY(_start)
SECTIONS
{
+ PROVIDE(_vdso_datapage = . - PAGE_SIZE);
. = VDSO32_LBASE + SIZEOF_HEADERS;
.hash : { *(.hash) } :text
@@ -138,11 +140,6 @@ VERSION
{
VDSO_VERSION_STRING {
global:
- /*
- * Has to be there for the kernel to find
- */
- __kernel_datapage_offset;
-
__kernel_get_syscall_map;
#ifndef CONFIG_PPC_BOOK3S_601
__kernel_gettimeofday;
diff --git a/arch/powerpc/kernel/vdso64/datapage.S b/arch/powerpc/kernel/vdso64/datapage.S
index 067247d3efb9..03bb72c440dc 100644
--- a/arch/powerpc/kernel/vdso64/datapage.S
+++ b/arch/powerpc/kernel/vdso64/datapage.S
@@ -13,9 +13,6 @@
#include <asm/vdso_datapage.h>
.text
-.global __kernel_datapage_offset;
-__kernel_datapage_offset:
- .long 0
/*
* void *__kernel_get_syscall_map(unsigned int *syscall_count) ;
diff --git a/arch/powerpc/kernel/vdso64/vdso64.lds.S b/arch/powerpc/kernel/vdso64/vdso64.lds.S
index 256fb9720298..f58c7e2e9cbd 100644
--- a/arch/powerpc/kernel/vdso64/vdso64.lds.S
+++ b/arch/powerpc/kernel/vdso64/vdso64.lds.S
@@ -4,6 +4,7 @@
* library
*/
#include <asm/vdso.h>
+#include <asm/page.h>
#ifdef __LITTLE_ENDIAN__
OUTPUT_FORMAT("elf64-powerpcle", "elf64-powerpcle", "elf64-powerpcle")
@@ -15,6 +16,7 @@ ENTRY(_start)
SECTIONS
{
+ PROVIDE(_vdso_datapage = . - PAGE_SIZE);
. = VDSO64_LBASE + SIZEOF_HEADERS;
.hash : { *(.hash) } :text
@@ -138,11 +140,6 @@ VERSION
{
VDSO_VERSION_STRING {
global:
- /*
- * Has to be there for the kernel to find
- */
- __kernel_datapage_offset;
-
__kernel_get_syscall_map;
__kernel_gettimeofday;
__kernel_clock_gettime;
--
2.25.0
^ permalink raw reply related
* [PATCH v7 1/7] powerpc/vdso64: Switch from __get_datapage() to get_datapage inline macro
From: Christophe Leroy @ 2020-04-20 16:56 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman, nathanl
Cc: arnd, linux-kernel, luto, tglx, vincenzo.frascino, linuxppc-dev
In-Reply-To: <cover.1587401492.git.christophe.leroy@c-s.fr>
On the same way as already done on PPC32, drop __get_datapage()
function and use get_datapage inline macro instead.
See commit ec0895f08f99 ("powerpc/vdso32: inline __get_datapage()")
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
arch/powerpc/kernel/vdso64/cacheflush.S | 9 ++++----
arch/powerpc/kernel/vdso64/datapage.S | 28 +++--------------------
arch/powerpc/kernel/vdso64/gettimeofday.S | 8 +++----
3 files changed, 11 insertions(+), 34 deletions(-)
diff --git a/arch/powerpc/kernel/vdso64/cacheflush.S b/arch/powerpc/kernel/vdso64/cacheflush.S
index 526f5ba2593e..cab14324242b 100644
--- a/arch/powerpc/kernel/vdso64/cacheflush.S
+++ b/arch/powerpc/kernel/vdso64/cacheflush.S
@@ -8,6 +8,7 @@
#include <asm/processor.h>
#include <asm/ppc_asm.h>
#include <asm/vdso.h>
+#include <asm/vdso_datapage.h>
#include <asm/asm-offsets.h>
.text
@@ -24,14 +25,12 @@ V_FUNCTION_BEGIN(__kernel_sync_dicache)
.cfi_startproc
mflr r12
.cfi_register lr,r12
- mr r11,r3
- bl V_LOCAL_FUNC(__get_datapage)
+ get_datapage r10, r0
mtlr r12
- mr r10,r3
lwz r7,CFG_DCACHE_BLOCKSZ(r10)
addi r5,r7,-1
- andc r6,r11,r5 /* round low to line bdy */
+ andc r6,r3,r5 /* round low to line bdy */
subf r8,r6,r4 /* compute length */
add r8,r8,r5 /* ensure we get enough */
lwz r9,CFG_DCACHE_LOGBLOCKSZ(r10)
@@ -48,7 +47,7 @@ V_FUNCTION_BEGIN(__kernel_sync_dicache)
lwz r7,CFG_ICACHE_BLOCKSZ(r10)
addi r5,r7,-1
- andc r6,r11,r5 /* round low to line bdy */
+ andc r6,r3,r5 /* round low to line bdy */
subf r8,r6,r4 /* compute length */
add r8,r8,r5
lwz r9,CFG_ICACHE_LOGBLOCKSZ(r10)
diff --git a/arch/powerpc/kernel/vdso64/datapage.S b/arch/powerpc/kernel/vdso64/datapage.S
index dc84f5ae3802..067247d3efb9 100644
--- a/arch/powerpc/kernel/vdso64/datapage.S
+++ b/arch/powerpc/kernel/vdso64/datapage.S
@@ -10,35 +10,13 @@
#include <asm/asm-offsets.h>
#include <asm/unistd.h>
#include <asm/vdso.h>
+#include <asm/vdso_datapage.h>
.text
.global __kernel_datapage_offset;
__kernel_datapage_offset:
.long 0
-V_FUNCTION_BEGIN(__get_datapage)
- .cfi_startproc
- /* We don't want that exposed or overridable as we want other objects
- * to be able to bl directly to here
- */
- .protected __get_datapage
- .hidden __get_datapage
-
- mflr r0
- .cfi_register lr,r0
-
- bcl 20,31,data_page_branch
-data_page_branch:
- mflr r3
- mtlr r0
- addi r3, r3, __kernel_datapage_offset-data_page_branch
- lwz r0,0(r3)
- .cfi_restore lr
- add r3,r0,r3
- blr
- .cfi_endproc
-V_FUNCTION_END(__get_datapage)
-
/*
* void *__kernel_get_syscall_map(unsigned int *syscall_count) ;
*
@@ -53,7 +31,7 @@ V_FUNCTION_BEGIN(__kernel_get_syscall_map)
mflr r12
.cfi_register lr,r12
mr r4,r3
- bl V_LOCAL_FUNC(__get_datapage)
+ get_datapage r3, r0
mtlr r12
addi r3,r3,CFG_SYSCALL_MAP64
cmpldi cr0,r4,0
@@ -75,7 +53,7 @@ V_FUNCTION_BEGIN(__kernel_get_tbfreq)
.cfi_startproc
mflr r12
.cfi_register lr,r12
- bl V_LOCAL_FUNC(__get_datapage)
+ get_datapage r3, r0
ld r3,CFG_TB_TICKS_PER_SEC(r3)
mtlr r12
crclr cr0*4+so
diff --git a/arch/powerpc/kernel/vdso64/gettimeofday.S b/arch/powerpc/kernel/vdso64/gettimeofday.S
index 1c9a04703250..e54c4ce4d356 100644
--- a/arch/powerpc/kernel/vdso64/gettimeofday.S
+++ b/arch/powerpc/kernel/vdso64/gettimeofday.S
@@ -26,7 +26,7 @@ V_FUNCTION_BEGIN(__kernel_gettimeofday)
mr r11,r3 /* r11 holds tv */
mr r10,r4 /* r10 holds tz */
- bl V_LOCAL_FUNC(__get_datapage) /* get data page */
+ get_datapage r3, r0
cmpldi r11,0 /* check if tv is NULL */
beq 2f
lis r7,1000000@ha /* load up USEC_PER_SEC */
@@ -71,7 +71,7 @@ V_FUNCTION_BEGIN(__kernel_clock_gettime)
mflr r12 /* r12 saves lr */
.cfi_register lr,r12
mr r11,r4 /* r11 saves tp */
- bl V_LOCAL_FUNC(__get_datapage) /* get data page */
+ get_datapage r3, r0
lis r7,NSEC_PER_SEC@h /* want nanoseconds */
ori r7,r7,NSEC_PER_SEC@l
beq cr5,70f
@@ -188,7 +188,7 @@ V_FUNCTION_BEGIN(__kernel_clock_getres)
mflr r12
.cfi_register lr,r12
- bl V_LOCAL_FUNC(__get_datapage)
+ get_datapage r3, r0
lwz r5, CLOCK_HRTIMER_RES(r3)
mtlr r12
li r3,0
@@ -221,7 +221,7 @@ V_FUNCTION_BEGIN(__kernel_time)
.cfi_register lr,r12
mr r11,r3 /* r11 holds t */
- bl V_LOCAL_FUNC(__get_datapage)
+ get_datapage r3, r0
ld r4,STAMP_XTIME_SEC(r3)
--
2.25.0
^ permalink raw reply related
* Re: [PATCH v3 0/4] Clean up hugetlb boot command line processing
From: Qian Cai @ 2020-04-20 15:34 UTC (permalink / raw)
To: Mike Kravetz
Cc: linux-doc, Catalin Marinas, Dave Hansen, Heiko Carstens, Peter Xu,
Linux-MM, Paul Mackerras, sparclinux, linux-riscv, Will Deacon,
Mina Almasry, linux-s390, Jonathan Corbet, Christian Borntraeger,
Ingo Molnar, Longpeng, Albert Ou, Vasily Gorbik, Paul Walmsley,
Thomas Gleixner, linux-arm-kernel, Nitesh Narayan Lal,
Randy Dunlap, LKML, Palmer Dabbelt, Andrew Morton, linuxppc-dev,
David S . Miller
In-Reply-To: <20200417185049.275845-1-mike.kravetz@oracle.com>
> On Apr 17, 2020, at 2:50 PM, Mike Kravetz <mike.kravetz@oracle.com> wrote:
>
> Longpeng(Mike) reported a weird message from hugetlb command line processing
> and proposed a solution [1]. While the proposed patch does address the
> specific issue, there are other related issues in command line processing.
> As hugetlbfs evolved, updates to command line processing have been made to
> meet immediate needs and not necessarily in a coordinated manner. The result
> is that some processing is done in arch specific code, some is done in arch
> independent code and coordination is problematic. Semantics can vary between
> architectures.
>
> The patch series does the following:
> - Define arch specific arch_hugetlb_valid_size routine used to validate
> passed huge page sizes.
> - Move hugepagesz= command line parsing out of arch specific code and into
> an arch independent routine.
> - Clean up command line processing to follow desired semantics and
> document those semantics.
>
> [1] https://lore.kernel.org/linux-mm/20200305033014.1152-1-longpeng2@huawei.com
>
> Mike Kravetz (4):
> hugetlbfs: add arch_hugetlb_valid_size
> hugetlbfs: move hugepagesz= parsing to arch independent code
> hugetlbfs: remove hugetlb_add_hstate() warning for existing hstate
> hugetlbfs: clean up command line processing
Reverted this series fixed many undefined behaviors on arm64 with the config,
https://raw.githubusercontent.com/cailca/linux-mm/master/arm64.config
[ 54.172683][ T1] UBSAN: shift-out-of-bounds in ./include/linux/hugetlb.h:555:34
[ 54.180411][ T1] shift exponent 4294967285 is too large for 64-bit type 'unsigned long'
[ 54.188885][ T1] CPU: 130 PID: 1 Comm: swapper/0 Not tainted 5.7.0-rc2-next-20200420 #1
[ 54.197284][ T1] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.11 06/18/2019
[ 54.207888][ T1] Call trace:
[ 54.211100][ T1] dump_backtrace+0x0/0x224
[ 54.215565][ T1] show_stack+0x20/0x2c
[ 54.219651][ T1] dump_stack+0xfc/0x184
[ 54.223829][ T1] __ubsan_handle_shift_out_of_bounds+0x304/0x344
[ 54.230204][ T1] hugetlb_add_hstate+0x3ec/0x414
huge_page_size at include/linux/hugetlb.h:555
(inlined by) hugetlb_add_hstate at mm/hugetlb.c:3301
[ 54.235191][ T1] hugetlbpage_init+0x14/0x30
[ 54.239824][ T1] do_one_initcall+0x6c/0x144
[ 54.244446][ T1] do_initcall_level+0x158/0x1c4
[ 54.249336][ T1] do_initcalls+0x68/0xb0
[ 54.253597][ T1] do_basic_setup+0x28/0x30
[ 54.258049][ T1] kernel_init_freeable+0x19c/0x228
[ 54.263188][ T1] kernel_init+0x14/0x208
[ 54.267473][ T1] ret_from_fork+0x10/0x18
[ 55.534338][ T1] UBSAN: shift-out-of-bounds in ./include/linux/hugetlb.h:555:34
[ 55.542064][ T1] shift exponent 4294967285 is too large for 64-bit type 'unsigned long'
[ 55.550555][ T1] CPU: 129 PID: 1 Comm: swapper/0 Not tainted 5.7.0-rc2-next-20200420 #1
[ 55.558992][ T1] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.11 06/18/2019
[ 55.569659][ T1] Call trace:
[ 55.572898][ T1] dump_backtrace+0x0/0x224
[ 55.577335][ T1] show_stack+0x20/0x2c
[ 55.581442][ T1] dump_stack+0xfc/0x184
[ 55.585621][ T1] __ubsan_handle_shift_out_of_bounds+0x304/0x344
[ 55.592031][ T1] __hugetlb_cgroup_file_dfl_init+0x37c/0x384
[ 55.598062][ T1] hugetlb_cgroup_file_init+0x9c/0xd8
[ 55.603399][ T1] hugetlb_init+0x248/0x448
[ 55.607840][ T1] do_one_initcall+0x6c/0x144
[ 55.612493][ T1] do_initcall_level+0x158/0x1c4
[ 55.617404][ T1] do_initcalls+0x68/0xb0
[ 55.621664][ T1] do_basic_setup+0x28/0x30
[ 55.626107][ T1] kernel_init_freeable+0x19c/0x228
[ 55.631253][ T1] kernel_init+0x14/0x208
[ 55.635519][ T1] ret_from_fork+0x10/0x18
[ 153.283648][ T1] ================================================================================
[ 153.293078][ T1] UBSAN: shift-out-of-bounds in ./include/linux/hugetlb.h:555:34
[ 153.300841][ T1] shift exponent 4294967285 is too large for 64-bit type 'unsigned long'
[ 153.309185][ T1] CPU: 161 PID: 1 Comm: swapper/0 Tainted: G L 5.7.0-rc2-next-20200420 #1
[ 153.318879][ T1] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.11 06/18/2019
[ 153.329352][ T1] Call trace:
[ 153.332545][ T1] dump_backtrace+0x0/0x224
[ 153.336945][ T1] show_stack+0x20/0x2c
[ 153.341000][ T1] dump_stack+0xfc/0x184
[ 153.345149][ T1] __ubsan_handle_shift_out_of_bounds+0x304/0x344
[ 153.351465][ T1] hugetlbfs_fill_super+0x424/0x43c
[ 153.356560][ T1] vfs_get_super+0xcc/0x170
[ 153.360959][ T1] get_tree_nodev+0x28/0x34
[ 153.365358][ T1] hugetlbfs_get_tree+0xfc/0x128
[ 153.370193][ T1] vfs_get_tree+0x54/0x158
[ 153.374513][ T1] fc_mount+0x1c/0x5c
[ 153.378399][ T1] mount_one_hugetlbfs+0x54/0xc8
[ 153.383233][ T1] init_hugetlbfs_fs+0x18c/0x268
[ 153.388068][ T1] do_one_initcall+0x6c/0x144
[ 153.392647][ T1] do_initcall_level+0x158/0x1c4
[ 153.397480][ T1] do_initcalls+0x68/0xb0
[ 153.401706][ T1] do_basic_setup+0x28/0x30
[ 153.406105][ T1] kernel_init_freeable+0x19c/0x228
[ 153.411208][ T1] kernel_init+0x14/0x208
[ 153.415436][ T1] ret_from_fork+0x10/0x18
[ 194.312926][ T1828] UBSAN: shift-out-of-bounds in ./include/linux/hugetlb.h:584:11
[ 194.320664][ T1828] shift exponent 4294967285 is too large for 32-bit type 'int'
[ 194.328103][ T1828] CPU: 194 PID: 1828 Comm: systemd-journal Tainted: G L 5.7.0-rc2-next-20200420 #1
[ 194.338558][ T1828] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.11 06/18/2019
[ 194.349010][ T1828] Call trace:
[ 194.352183][ T1828] dump_backtrace+0x0/0x224
[ 194.356560][ T1828] show_stack+0x20/0x2c
[ 194.360595][ T1828] dump_stack+0xfc/0x184
[ 194.364724][ T1828] __ubsan_handle_shift_out_of_bounds+0x304/0x344
[ 194.371020][ T1828] hugetlb_total_pages+0x100/0x128
[ 194.376017][ T1828] vm_commit_limit+0x54/0xb0
[ 194.380484][ T1828] meminfo_proc_show+0x8f4/0xc4c
[ 194.385297][ T1828] seq_read+0x380/0x930
[ 194.389353][ T1828] pde_read+0x5c/0x78
[ 194.393232][ T1828] proc_reg_read+0x74/0xc0
[ 194.397528][ T1828] __vfs_read+0x84/0x1d0
[ 194.401646][ T1828] vfs_read+0xec/0x124
[ 194.405588][ T1828] ksys_read+0xb0/0x120
[ 194.409643][ T1828] __arm64_sys_read+0x54/0x88
[ 194.414195][ T1828] do_el0_svc+0x128/0x1dc
[ 194.418405][ T1828] el0_sync_handler+0x150/0x250
[ 194.423157][ T1828] el0_sync+0x164/0x180
[ 194.427425][ T1828] ================================================================================
[ 194.436930][ T1828] ================================================================================
[ 194.446355][ T1828] UBSAN: shift-out-of-bounds in mm/hugetlb.c:3564:23
[ 194.453190][ T1828] shift exponent 4294967285 is too large for 64-bit type 'unsigned long'
[ 194.461752][ T1828] CPU: 194 PID: 1828 Comm: systemd-journal Tainted: G L 5.7.0-rc2-next-20200420 #1
[ 194.472245][ T1828] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.11 06/18/2019
[ 194.482720][ T1828] Call trace:
[ 194.485909][ T1828] dump_backtrace+0x0/0x224
[ 194.490312][ T1828] show_stack+0x20/0x2c
[ 194.494368][ T1828] dump_stack+0xfc/0x184
[ 194.498513][ T1828] __ubsan_handle_shift_out_of_bounds+0x304/0x344
[ 194.504828][ T1828] hugetlb_report_meminfo+0x25c/0x2ac
[ 194.510103][ T1828] meminfo_proc_show+0xc08/0xc4c
[ 194.514938][ T1828] seq_read+0x380/0x930
[ 194.518993][ T1828] pde_read+0x5c/0x78
[ 194.522874][ T1828] proc_reg_read+0x74/0xc0
[ 194.527190][ T1828] __vfs_read+0x84/0x1d0
[ 194.531335][ T1828] vfs_read+0xec/0x124
[ 194.535304][ T1828] ksys_read+0xb0/0x120
[ 194.539371][ T1828] __arm64_sys_read+0x54/0x88
[ 194.543958][ T1828] do_el0_svc+0x128/0x1dc
[ 194.548187][ T1828] el0_sync_handler+0x150/0x250
[ 194.552936][ T1828] el0_sync+0x164/0x180
>
> .../admin-guide/kernel-parameters.txt | 40 ++--
> Documentation/admin-guide/mm/hugetlbpage.rst | 35 ++++
> arch/arm64/mm/hugetlbpage.c | 30 +--
> arch/powerpc/mm/hugetlbpage.c | 30 +--
> arch/riscv/mm/hugetlbpage.c | 24 +--
> arch/s390/mm/hugetlbpage.c | 24 +--
> arch/sparc/mm/init_64.c | 43 +---
> arch/x86/mm/hugetlbpage.c | 23 +--
> include/linux/hugetlb.h | 2 +-
> mm/hugetlb.c | 190 +++++++++++++++---
> 10 files changed, 271 insertions(+), 170 deletions(-)
>
> --
> 2.25.2
>
>
^ permalink raw reply
* Re: [PATCH v2, RESEND] misc: new driver sram_uapi for user level SRAM access
From: Greg KH @ 2020-04-20 14:51 UTC (permalink / raw)
To: Wang Wenhu
Cc: robh, arnd, Randy Dunlap, linux-kernel, Scott Wood, kernel,
linuxppc-dev
In-Reply-To: <20200420030538.101696-1-wenhu.wang@vivo.com>
On Sun, Apr 19, 2020 at 08:05:38PM -0700, Wang Wenhu wrote:
> A generic User-Kernel interface that allows a misc device created
> by it to support file-operations of ioctl and mmap to access SRAM
> memory from user level. Different kinds of SRAM alloction and free
> APIs could be registered by specific SRAM hardware level driver to
> the available list and then be chosen by users to allocate and map
> SRAM memory from user level.
>
> It is extremely helpful for the user space applications that require
> high performance memory accesses, such as embedded networking devices
> that would process data in user space, and PowerPC e500 is a case.
>
> Signed-off-by: Wang Wenhu <wenhu.wang@vivo.com>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Scott Wood <oss@buserror.net>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Randy Dunlap <rdunlap@infradead.org>
> Cc: linuxppc-dev@lists.ozlabs.org
> ---
> Changes since v1: addressed comments from Arnd
> * Changed the ioctl cmd definitions using _IO micros
> * Export interfaces for HW-SRAM drivers to register apis to available list
> * Modified allocation alignment to PAGE_SIZE
> * Use phys_addr_t as type of SRAM resource size and offset
> * Support compat_ioctl
> * Misc device name:sram
>
> Note: From this on, the SRAM_UAPI driver is independent to any hardware
> drivers, so I would only commit the patch itself as v2, while the v1 of
> it was wrapped together with patches for Freescale L2-Cache-SRAM device.
> Then after, I'd create patches for Freescale L2-Cache-SRAM device as
> another series.
>
> Link for v1:
> * https://lore.kernel.org/lkml/20200418162157.50428-5-wenhu.wang@vivo.com/
Why was this a RESEND? What was wrong with the original v2 version?
Anyway, minor review comments:
> ---
> drivers/misc/Kconfig | 11 ++
> drivers/misc/Makefile | 1 +
> drivers/misc/sram_uapi.c | 352 ++++++++++++++++++++++++++++++++++++++
> include/linux/sram_uapi.h | 28 +++
> 4 files changed, 392 insertions(+)
> create mode 100644 drivers/misc/sram_uapi.c
> create mode 100644 include/linux/sram_uapi.h
>
> diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
> index 99e151475d8f..b19c8b24f18e 100644
> --- a/drivers/misc/Kconfig
> +++ b/drivers/misc/Kconfig
> @@ -465,6 +465,17 @@ config PVPANIC
> a paravirtualized device provided by QEMU; it lets a virtual machine
> (guest) communicate panic events to the host.
>
> +config SRAM_UAPI
> + bool "Generic SRAM User Level API driver"
> + help
> + This driver allows you to create a misc device which could be used
> + as an interface to allocate SRAM memory from user level.
> +
> + It is extremely helpful for some user space applications that require
> + high performance memory accesses.
> +
> + If unsure, say N.
Naming is hard, but shouldn't this just be "sram" and drop the whole
"UAPI" stuff everywhere? That doesn't make much sense as drivers are
just interfaces to userspace...
> +
> source "drivers/misc/c2port/Kconfig"
> source "drivers/misc/eeprom/Kconfig"
> source "drivers/misc/cb710/Kconfig"
> diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
> index 9abf2923d831..794447ca07ca 100644
> --- a/drivers/misc/Makefile
> +++ b/drivers/misc/Makefile
> @@ -46,6 +46,7 @@ obj-$(CONFIG_VMWARE_VMCI) += vmw_vmci/
> obj-$(CONFIG_LATTICE_ECP3_CONFIG) += lattice-ecp3-config.o
> obj-$(CONFIG_SRAM) += sram.o
> obj-$(CONFIG_SRAM_EXEC) += sram-exec.o
> +obj-$(CONFIG_SRAM_UAPI) += sram_uapi.o
> obj-y += mic/
> obj-$(CONFIG_GENWQE) += genwqe/
> obj-$(CONFIG_ECHO) += echo/
> diff --git a/drivers/misc/sram_uapi.c b/drivers/misc/sram_uapi.c
> new file mode 100644
> index 000000000000..66c7b56b635f
> --- /dev/null
> +++ b/drivers/misc/sram_uapi.c
> @@ -0,0 +1,352 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (C) 2020 Vivo Communication Technology Co. Ltd.
> + * Copyright (C) 2020 Wang Wenhu <wenhu.wang@vivo.com>
> + * All rights reserved.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/kernel.h>
> +#include <linux/slab.h>
> +#include <linux/mm.h>
> +#include <linux/fs.h>
> +#include <linux/miscdevice.h>
> +#include <linux/uaccess.h>
> +#include <linux/sram_uapi.h>
> +
> +#define DRIVER_NAME "sram_uapi"
If you really need this, just use KBUILD_MODNAME instead. But I don't
think you need it, as most drivers do not.
> +
> +struct res_info {
> + phys_addr_t offset;
> + phys_addr_t size;
> +};
> +
> +struct sram_resource {
> + struct list_head list;
> + struct res_info info;
> + phys_addr_t phys;
> + void *virt;
> + struct vm_area_struct *vma;
> + struct sram_uapi *parent;
> +};
> +
> +struct sram_uapi {
> + struct list_head res_list;
> + struct sram_api *sa;
> +};
> +
> +static DEFINE_MUTEX(sram_api_list_lock);
> +static LIST_HEAD(sram_api_list);
> +
> +long sram_api_register(struct sram_api *sa)
Why are all of these functions global?
And shouldn't this return an 'int'?
> +{
> + struct sram_api *cur;
> +
> + if (!sa || !sa->name || !sa->sram_alloc || !sa->sram_free)
> + return -EINVAL;
How can any of these be true? You control all the callers of this
function.
> +
> + mutex_lock(&sram_api_list_lock);
> + list_for_each_entry(cur, &sram_api_list, list) {
> + if (cur->type == sa->type) {
> + pr_err("error sram %s type %d exists\n", sa->name,
> + sa->type);
> + mutex_unlock(&sram_api_list_lock);
> + return -EEXIST;
> + }
> + }
> +
> + kref_init(&sa->kref);
So the caller has to allocate the space and set some things up, but not
everything? That's a mess, why not just do it either all at once with
an "allocate" like function, or just do it all in one function. Why is
this a separate function anyway?
> + list_add_tail(&sa->list, &sram_api_list);
> + pr_info("sram %s type %d registered\n", sa->name, sa->type);
Don't leave debugging comments in the driver, if all goes well a driver
should be totally silent. Only log errors.
> +
> + mutex_unlock(&sram_api_list_lock);
> +
> + return 0;
> +};
> +EXPORT_SYMBOL(sram_api_register);
Exported for what? This is a stand-alone driver, nothing else uses
these symbols.
> +
> +long sram_api_unregister(struct sram_api *sa)
Why long?
> +{
> + struct sram_api *cur, *tmp;
> + long ret = -ENODEV;
> +
> + if (!sa || !sa->name || !sa->sram_alloc || !sa->sram_free)
> + return -EINVAL;
Again, how can any of this be true?
> +
> + mutex_lock(&sram_api_list_lock);
> + list_for_each_entry_safe(cur, tmp, &sram_api_list, list) {
> + if (cur->type == sa->type && !strcmp(cur->name, sa->name)) {
> + if (kref_read(&cur->kref)) {
> + pr_err("error sram %s type %d is busy\n",
> + sa->name, sa->type);
> + ret = -EBUSY;
> + } else {
> + list_del(&cur->list);
> + pr_info("sram %s type %d unregistered\n",
> + sa->name, sa->type);
> + ret = 0;
> + }
> + break;
> + }
> + }
> + mutex_unlock(&sram_api_list_lock);
> +
> + return ret;
> +};
> +EXPORT_SYMBOL(sram_api_unregister);
> +
> +static struct sram_api *get_sram_api_from_type(__u32 type)
> +{
> + struct sram_api *cur;
> +
> + mutex_lock(&sram_api_list_lock);
> + list_for_each_entry(cur, &sram_api_list, list) {
> + if (cur->type == type) {
> + kref_get(&cur->kref);
> + mutex_unlock(&sram_api_list_lock);
> + return cur;
Document that the reference count is incremented, or someone will forget
that.
> + }
> + }
> + mutex_unlock(&sram_api_list_lock);
> +
> + return NULL;
> +}
> +
> +static void sram_uapi_res_insert(struct sram_uapi *uapi,
> + struct sram_resource *res)
> +{
> + struct sram_resource *cur, *tmp;
> + struct list_head *head = &uapi->res_list;
> +
> + list_for_each_entry_safe(cur, tmp, head, list) {
> + if (&tmp->list != head &&
> + (cur->info.offset + cur->info.size + res->info.size <=
> + tmp->info.offset)) {
> + res->info.offset = cur->info.offset + cur->info.size;
This feels really odd, what are you doing here with this loop to try to
find a space and then put it in and then change the resource???
> + res->parent = uapi;
> + list_add(&res->list, &cur->list);
> + return;
> + }
> + }
> +
> + if (list_empty(head))
> + res->info.offset = 0;
> + else {
> + tmp = list_last_entry(head, struct sram_resource, list);
> + res->info.offset = tmp->info.offset + tmp->info.size;
> + }
> + list_add_tail(&res->list, head);
No locking anywhere?
> +}
> +
> +static struct sram_resource *sram_uapi_res_delete(struct sram_uapi *uapi,
> + struct res_info *info)
> +{
> + struct sram_resource *res, *tmp;
> +
> + list_for_each_entry_safe(res, tmp, &uapi->res_list, list) {
> + if (res->info.offset == info->offset) {
Why is the offset somehow the "key" for this?
> + list_del(&res->list);
> + res->parent = NULL;
> + return res;
> + }
> + }
No locking anywhere?
> +
> + return NULL;
> +}
> +
> +static struct sram_resource *sram_uapi_find_res(struct sram_uapi *uapi,
> + __u32 offset)
> +{
> + struct sram_resource *res;
> +
> + list_for_each_entry(res, &uapi->res_list, list) {
> + if (res->info.offset == offset)
> + return res;
> + }
No lock??? No reference counting?
> +
> + return NULL;
> +}
> +
> +static int sram_uapi_open(struct inode *inode, struct file *filp)
> +{
> + struct sram_uapi *uapi;
> +
> + uapi = kzalloc(sizeof(*uapi), GFP_KERNEL);
> + if (!uapi)
> + return -ENOMEM;
> +
> + INIT_LIST_HEAD(&uapi->res_list);
> + filp->private_data = uapi;
> +
> + return 0;
> +}
> +
> +static long sram_uapi_ioctl(struct file *filp, unsigned int cmd,
> + unsigned long arg)
> +{
> + struct sram_uapi *uapi = filp->private_data;
> + struct sram_resource *res;
> + struct res_info info;
> + long ret = -ENOIOCTLCMD;
-ENOTTY?
> + int size;
> + __u32 type;
> +
> + if (!uapi)
> + return ret;
> +
> + switch (cmd) {
> + case SRAM_UAPI_IOC_SET_SRAM_TYPE:
> + if (uapi->sa)
> + return -EEXIST;
> +
> + get_user(type, (const __u32 __user *)arg);
> + uapi->sa = get_sram_api_from_type(type);
> + if (uapi->sa)
> + ret = 0;
> + else
> + ret = -ENODEV;
> +
> + break;
Why does userspace have to set the type of sram?
> +
> + case SRAM_UAPI_IOC_ALLOC:
> + if (!uapi->sa)
> + return -EINVAL;
> +
> + res = kzalloc(sizeof(*res), GFP_KERNEL);
> + if (!res)
> + return -ENOMEM;
> +
> + size = copy_from_user((void *)&res->info,
> + (const void __user *)arg,
> + sizeof(res->info));
> + if (!PAGE_ALIGNED(res->info.size) || !res->info.size)
> + return -EINVAL;
> +
> + res->virt = (void *)uapi->sa->sram_alloc(res->info.size,
> + &res->phys,
> + PAGE_SIZE);
Look ma, userspace just allocated as much memory as it asked for!
Remember, all input is evil, check everything.
And, as is obvious now, this isn't really a "driver", it's a shim layer
for something else. Who sets up sram_alloc()?
> + if (!res->virt) {
> + kfree(res);
> + return -ENOMEM;
> + }
> +
> + sram_uapi_res_insert(uapi, res);
> + size = copy_to_user((void __user *)arg,
> + (const void *)&res->info,
> + sizeof(res->info));
an alloc function copies a bunch of memory to userspace????
You need to document the heck out of this new interface as it really
does not make sense to me, sorry.
> +
> + ret = 0;
> + break;
> +
> + case SRAM_UAPI_IOC_FREE:
> + if (!uapi->sa)
> + return -EINVAL;
> +
> + size = copy_from_user((void *)&info, (const void __user *)arg,
> + sizeof(info));
> +
> + res = sram_uapi_res_delete(uapi, &info);
> + if (!res) {
> + pr_err("error no sram resource found\n");
> + return -EINVAL;
> + }
> +
> + uapi->sa->sram_free(res->virt);
> + kfree(res);
Two step delete function with locking at all? Ugh :(
> +
> + ret = 0;
> + break;
> +
> + default:
> + pr_err("error no cmd not supported\n");
> + break;
Userspace now can spam the kernel log, do not do that.
> + }
> +
> + return ret;
> +}
> +
> +static int sram_uapi_mmap(struct file *filp, struct vm_area_struct *vma)
> +{
> + struct sram_uapi *uapi = filp->private_data;
> + struct sram_resource *res;
> +
> + res = sram_uapi_find_res(uapi, vma->vm_pgoff);
> + if (!res)
> + return -EINVAL;
> +
> + if (vma->vm_end - vma->vm_start > res->info.size)
> + return -EINVAL;
> +
> + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
> +
> + return remap_pfn_range(vma, vma->vm_start,
> + res->phys >> PAGE_SHIFT,
> + vma->vm_end - vma->vm_start,
> + vma->vm_page_prot);
> +}
> +
> +static void sram_uapi_res_release(struct sram_uapi *uapi)
> +{
> + struct sram_resource *res, *tmp;
> +
> + list_for_each_entry_safe(res, tmp, &uapi->res_list, list) {
> + list_del(&res->list);
> + uapi->sa->sram_free(res->virt);
> + kfree(res);
> + }
> +}
> +
> +static int sram_uapi_release(struct inode *inodp, struct file *filp)
> +{
> + struct sram_uapi *uapi = filp->private_data;
> +
> + sram_uapi_res_release(uapi);
> + if (uapi->sa)
> + kref_put(&uapi->sa->kref, NULL);
Um, hm, how do I put this nicely.
Wow, this is wrong. So so so so so so wrong.
There's documentation in the kernel tree about how wrong this is. The
kernel itself will warn you about how wrong this is. Actually, no, it
will just crash. If this function ever gets called your kernel just
blew up, so you really really really did not test this code at all.
{sigh}
Someone owes me a new bottle of whisky, this one is about to be empty...
> +
> + kfree(uapi);
> +
> + return 0;
> +}
> +
> +static const struct file_operations sram_uapi_ops = {
> + .owner = THIS_MODULE,
> + .open = sram_uapi_open,
> + .unlocked_ioctl = sram_uapi_ioctl,
> + .compat_ioctl = compat_ptr_ioctl,
> + .mmap = sram_uapi_mmap,
> + .release = sram_uapi_release,
> +};
> +
> +static struct miscdevice sram_uapi_miscdev = {
> + MISC_DYNAMIC_MINOR,
> + "sram",
> + &sram_uapi_ops,
named identifiers please, this isn't the 1980's.
> +};
> +
> +static int __init sram_uapi_init(void)
> +{
> + int ret;
> +
> + INIT_LIST_HEAD(&sram_api_list);
Why do you need a static list?
> + mutex_init(&sram_api_list_lock);
Why do you need a static lock?
> +
> + ret = misc_register(&sram_uapi_miscdev);
> + if (ret)
> + pr_err("failed to register sram uapi misc device\n");
No need for the error, don't be noisy.
> +
> + return ret;
> +}
> +
> +static void __exit sram_uapi_exit(void)
> +{
> + misc_deregister(&sram_uapi_miscdev);
> +}
> +
> +module_init(sram_uapi_init);
> +module_exit(sram_uapi_exit);
drop the crazy static list and lock (again, this isn't the 1990's), and
you can get rid of the module_init/exit stuff too and just use a single
line to register the misc device.
Meta-comment here, you are trying to create a driver with a new api, but
you have no real users of this api, so no one knows if this really is
the correct api at all. That's not ok, especially for one that is
purporting to be so "generic" as to claim it is a sram driver for
everyone.
> +
> +MODULE_AUTHOR("Wang Wenhu <wenhu.wang@vivo.com>");
> +MODULE_DESCRIPTION("SRAM User API Driver");
> +MODULE_ALIAS("platform:" DRIVER_NAME);
> +MODULE_LICENSE("GPL v2");
> diff --git a/include/linux/sram_uapi.h b/include/linux/sram_uapi.h
> new file mode 100644
> index 000000000000..50fbf9dc308f
> --- /dev/null
> +++ b/include/linux/sram_uapi.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef __SRAM_UAPI_H
> +#define __SRAM_UAPI_H
> +
> +/* Set SRAM type to be accessed */
> +#define SRAM_UAPI_IOC_SET_SRAM_TYPE _IOW('S', 0, __u32)
> +
> +/* Allocate resource from SRAM */
> +#define SRAM_UAPI_IOC_ALLOC _IOWR('S', 1, struct res_info)
> +
> +/* Free allocated resource of SRAM */
> +#define SRAM_UAPI_IOC_FREE _IOW('S', 2, struct res_info)
> +
> +struct sram_api {
> + struct list_head list;
> + struct kref kref;
> + __u32 type;
> + const char *name;
> +
> + long (*sram_alloc)(__u32 size, phys_addr_t *phys, __u32 align);
> + void (*sram_free)(void *ptr);
> +};
Why is this structure in a uapi header file?
Oh wait, this isn't a uapi header file, this thing doesn't work at all.
ugh, this needs _so_ much work.
And again, someone owes me more whisky....
greg k-h
^ permalink raw reply
* Re: [PATCH v2, RESEND] misc: new driver sram_uapi for user level SRAM access
From: Arnd Bergmann @ 2020-04-20 14:34 UTC (permalink / raw)
To: Wang Wenhu
Cc: Rob Herring, gregkh, Randy Dunlap, linux-kernel@vger.kernel.org,
Scott Wood, kernel, linuxppc-dev
In-Reply-To: <20200420030538.101696-1-wenhu.wang@vivo.com>
On Mon, Apr 20, 2020 at 5:06 AM Wang Wenhu <wenhu.wang@vivo.com> wrote:
>
> A generic User-Kernel interface that allows a misc device created
> by it to support file-operations of ioctl and mmap to access SRAM
> memory from user level. Different kinds of SRAM alloction and free
> APIs could be registered by specific SRAM hardware level driver to
> the available list and then be chosen by users to allocate and map
> SRAM memory from user level.
>
> It is extremely helpful for the user space applications that require
> high performance memory accesses, such as embedded networking devices
> that would process data in user space, and PowerPC e500 is a case.
>
> Signed-off-by: Wang Wenhu <wenhu.wang@vivo.com>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Scott Wood <oss@buserror.net>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Randy Dunlap <rdunlap@infradead.org>
> Cc: linuxppc-dev@lists.ozlabs.org
> ---
> Changes since v1: addressed comments from Arnd
> * Changed the ioctl cmd definitions using _IO micros
> * Export interfaces for HW-SRAM drivers to register apis to available list
> * Modified allocation alignment to PAGE_SIZE
> * Use phys_addr_t as type of SRAM resource size and offset
> * Support compat_ioctl
> * Misc device name:sram
Looks much better already.
> Note: From this on, the SRAM_UAPI driver is independent to any hardware
> drivers, so I would only commit the patch itself as v2, while the v1 of
> it was wrapped together with patches for Freescale L2-Cache-SRAM device.
> Then after, I'd create patches for Freescale L2-Cache-SRAM device as
> another series.
What I meant to suggest was actually to tie it more closely to
the code we already have in drivers/misc/sram.c, which already
has some form of abstraction.
> +static int __init sram_uapi_init(void)
> +{
> + int ret;
> +
> + INIT_LIST_HEAD(&sram_api_list);
> + mutex_init(&sram_api_list_lock);
> +
> + ret = misc_register(&sram_uapi_miscdev);
> + if (ret)
> + pr_err("failed to register sram uapi misc device\n");
The mutex and listhead are already initialized, so this can
be a one-line function
return misc_register(&sram_uapi_miscdev);
> --- /dev/null
> +++ b/include/linux/sram_uapi.h
The ioctl definitions need to be in include/uapi/linux/
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef __SRAM_UAPI_H
> +#define __SRAM_UAPI_H
> +
> +/* Set SRAM type to be accessed */
> +#define SRAM_UAPI_IOC_SET_SRAM_TYPE _IOW('S', 0, __u32)
> +
> +/* Allocate resource from SRAM */
> +#define SRAM_UAPI_IOC_ALLOC _IOWR('S', 1, struct res_info)
> +
> +/* Free allocated resource of SRAM */
> +#define SRAM_UAPI_IOC_FREE _IOW('S', 2, struct res_info)
struct res_info needs to also be defined here, so user applications can
see the definition, and it has to use __u64, not phys_addr_t, to ensure
the API does not depend on kernel configuraiton.
Arnd
^ permalink raw reply
* Re: [PATCH 15/21] mm: memmap_init: iterate over memblock regions rather that check each PFN
From: Qian Cai @ 2020-04-20 14:26 UTC (permalink / raw)
To: Mike Rapoport, Baoquan He
Cc: Rich Felker, linux-ia64, linux-doc, Catalin Marinas,
Heiko Carstens, Michal Hocko, James E.J. Bottomley, Max Filippov,
Guo Ren, linux-csky, linux-parisc, sparclinux, linux-hexagon,
linux-riscv, Greg Ungerer, linux-arch, linux-s390, linux-snps-arc,
linux-c6x-dev, Brian Cain, Jonathan Corbet, linux-sh,
Helge Deller, the arch/x86 maintainers, Russell King,
Ley Foon Tan, Mike Rapoport, Geert Uytterhoeven, linux-arm-kernel,
Mark Salter, Matt Turner, linux-mips, uclinux-h8-devel,
linux-xtensa, linux-alpha, linux-um, linux-m68k, Tony Luck,
Greentime Hu, Paul Walmsley, Stafford Horne, Guan Xuetao,
Hoan Tran, Michal Simek, Thomas Bogendoerfer, Yoshinori Sato,
Nick Hu, Linux-MM, Vineet Gupta, LKML, openrisc,
Richard Weinberger, Andrew Morton, linuxppc-dev, David S. Miller
In-Reply-To: <20200412194859.12663-16-rppt@kernel.org>
> On Apr 12, 2020, at 3:48 PM, Mike Rapoport <rppt@kernel.org> wrote:
>
> From: Baoquan He <bhe@redhat.com>
>
> When called during boot the memmap_init_zone() function checks if each PFN
> is valid and actually belongs to the node being initialized using
> early_pfn_valid() and early_pfn_in_nid().
>
> Each such check may cost up to O(log(n)) where n is the number of memory
> banks, so for large amount of memory overall time spent in early_pfn*()
> becomes substantial.
>
> Since the information is anyway present in memblock, we can iterate over
> memblock memory regions in memmap_init() and only call memmap_init_zone()
> for PFN ranges that are know to be valid and in the appropriate node.
>
> Signed-off-by: Baoquan He <bhe@redhat.com>
> Signed-off-by: Mike Rapoport <rppt@linux.ibm.com>
> ---
> mm/page_alloc.c | 26 ++++++++++++++++----------
> 1 file changed, 16 insertions(+), 10 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 7f6a3081edb8..c43ce8709457 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -5995,14 +5995,6 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
> * function. They do not exist on hotplugged memory.
> */
> if (context == MEMMAP_EARLY) {
> - if (!early_pfn_valid(pfn)) {
> - pfn = next_pfn(pfn);
> - continue;
> - }
> - if (!early_pfn_in_nid(pfn, nid)) {
> - pfn++;
> - continue;
> - }
This causes a compilation warning from Clang,
mm/page_alloc.c:5917:39: warning: unused function 'next_pfn' [-Wunused-function]
static inline __meminit unsigned long next_pfn(unsigned long pfn)
This should fix it,
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index d469384c9ca7..b48336e20bdc 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5912,23 +5912,6 @@ overlap_memmap_init(unsigned long zone, unsigned long *pfn)
return false;
}
-#ifdef CONFIG_SPARSEMEM
-/* Skip PFNs that belong to non-present sections */
-static inline __meminit unsigned long next_pfn(unsigned long pfn)
-{
- const unsigned long section_nr = pfn_to_section_nr(++pfn);
-
- if (present_section_nr(section_nr))
- return pfn;
- return section_nr_to_pfn(next_present_section_nr(section_nr));
-}
-#else
-static inline __meminit unsigned long next_pfn(unsigned long pfn)
-{
- return pfn++;
-}
-#endif
-
/*
* Initially all pages are reserved - free ones are freed
* up by memblock_free_all() once the early boot process is
> if (overlap_memmap_init(zone, &pfn))
> continue;
> if (defer_init(nid, pfn, end_pfn))
> @@ -6118,9 +6110,23 @@ static void __meminit zone_init_free_lists(struct zone *zone)
> }
>
> void __meminit __weak memmap_init(unsigned long size, int nid,
> - unsigned long zone, unsigned long start_pfn)
> + unsigned long zone,
> + unsigned long range_start_pfn)
> {
> - memmap_init_zone(size, nid, zone, start_pfn, MEMMAP_EARLY, NULL);
> + unsigned long start_pfn, end_pfn;
> + unsigned long range_end_pfn = range_start_pfn + size;
> + int i;
> +
> + for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, NULL) {
> + start_pfn = clamp(start_pfn, range_start_pfn, range_end_pfn);
> + end_pfn = clamp(end_pfn, range_start_pfn, range_end_pfn);
> +
> + if (end_pfn > start_pfn) {
> + size = end_pfn - start_pfn;
> + memmap_init_zone(size, nid, zone, start_pfn,
> + MEMMAP_EARLY, NULL);
> + }
> + }
> }
>
> static int zone_batchsize(struct zone *zone)
> --
> 2.25.1
>
>
^ permalink raw reply related
* Re: [PATCH kernel v2 0/7] powerpc/powenv/ioda: Allow huge DMA window at 4GB
From: Oliver O'Halloran @ 2020-04-20 14:04 UTC (permalink / raw)
To: Alexey Kardashevskiy, Russell Currey
Cc: KVM list, Fabiano Rosas, Alistair Popple, kvm-ppc, linuxppc-dev,
David Gibson
In-Reply-To: <9893c4db-057d-8e42-52fe-8241d6d90b5f@ozlabs.ru>
On Fri, 2020-04-17 at 15:47 +1000, Alexey Kardashevskiy wrote:
>
> On 17/04/2020 11:26, Russell Currey wrote:
> >
> > For what it's worth this sounds like a good idea to me, it just sounds
> > tricky to implement. You're adding another layer of complexity on top
> > of EEH (well, making things look simple to the EEH core and doing your
> > own freezing on top of it) in addition to the DMA handling.
> >
> > If it works then great, just has a high potential to become a new bug
> > haven.
>
> imho putting every PCI function to a separate PE is the right thing to
> do here but I've been told it is not that simple, and I believe that.
> Reusing slave PEs seems unreliable - the configuration will depend on
> whether a PE occupied enough segments to give an unique PE to a PCI
> function and my little brain explodes.
You're overthinking it.
If a bus has no m64 MMIO space we're free to assign whatever PE number
we want since the PE for the bus isn't fixed by the m64 segment its
BARs were placed in. For those buses we assign a PE number starting
from the max and counting down (0xff, 0xfe, etc). For example, with
this PHB:
# lspci -s 1:: -v | egrep '0001:|Memory at'
0001:00:00.0 PCI bridge: IBM Device 04c1 (prog-if 00 [Normal decode])
0001:01:00.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
(prog-if 00 [Normal decode])
Memory at 600c081000000 (32-bit, non-prefetchable) [size=256K]
0001:02:01.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
(prog-if 00 [Normal decode])
0001:02:08.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
(prog-if 00 [Normal decode])
0001:02:09.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
(prog-if 00 [Normal decode])
0001:03:00.0 Non-Volatile memory controller: PMC-Sierra Inc. Device
f117 (rev 06) (prog-if 02 [NVM Express])
Memory at 600c080000000 (64-bit, non-prefetchable) [size=16K]
Memory at 6004000000000 (64-bit, prefetchable) [size=1M]
0001:09:00.0 Ethernet controller: Intel Corporation Ethernet Controller
X710/X557-AT 10GBASE-T (rev 02)
Memory at 6004048000000 (64-bit, prefetchable) [size=8M]
Memory at 600404a000000 (64-bit, prefetchable) [size=32K]
(redundant functions removed)
We get these PE assignments:
0001:00:00.0 - 0xfe # Root port
0001:01:00.0 - 0xfc # upstream port
0001:02:01.0 - 0xfd # downstream port bus
0001:02:08.0 - 0xfd
0001:02:09.0 - 0xfd
0001:03:00.0 - 0x0 # NVMe
0001:09:00.0 - 0x1 # Ethernet
All the switch ports either have 32bit BARs or no BARs so they get
assigned PEs starting from the top. The Ethernet and the NVMe have some
prefetchable 64bit BARs so they have to be in PE 0x0 and 0x1
respectively since that's where their m64 BARs are located. For our
DMA-only slave PEs any MMIO space would remain in their master PE so we
can allocate a PE number for the DMA-PE (our iommu context).
I think the key thing to realise is that we'd only be using the DMA-PE
when a crippled DMA mask is set by the driver. In all other cases we
can just use the "native PE" and when the driver unbinds we can de-
allocate our DMA-PE and return the device to the PE containing it's
MMIO BARs. I think we can keep things relatively sane that way and the
real issue is detecting EEH events on the DMA-PE.
On P9 we don't have PHB error interrupts enabled in firmware so we're
completely reliant on seeing a 0xFF response to an MMIO and manually
checking the PE status to see if it's due to a PE freeze. For our DMA-
PE it could be frozen (due to a bad DMA) and we'd never notice unless
we explicitly check the status of the DMA-PE since there's no
corresponding MMIO space to freeze.
On P8 we had PHB Error interrupts so you would notice that *something*
happened, then go check for frozen PEs, at which point the master-slave
grouping logic would see that one PE in the group was frozen and freeze
the rest of them. We can re-use that on that, but we still need
something to actually notice a freeze occured. A background poller
checking for freezes on each PE might do the trick.
> So this is not happening soon.
Oh ye of little faith.
> For the time being, this patchset is good for:
> 1. weird hardware which has limited DMA mask (this is why the patchset
> was written in the first place)
> 2. debug DMA by routing it via IOMMU (even when 4GB hack is not enabled).
Sure, but it's still dependent on having firmware which supports the
4GB hack and I don't think that's in any offical firmware releases yet.
Oliver
^ permalink raw reply
* [PATCH v3 22/22] powerpc/mm/book3s64: Fix MADV_DONTNEED and parallel page fault race
From: Aneesh Kumar K.V @ 2020-04-20 12:44 UTC (permalink / raw)
To: linuxppc-dev, mpe, linux-mm, linux-kernel, kvm-ppc
Cc: leonardo, Aneesh Kumar K.V, npiggin, kirill
In-Reply-To: <20200420124434.47330-1-aneesh.kumar@linux.ibm.com>
MADV_DONTNEED holds mmap_sem in read mode and that implies a
parallel page fault is possible and the kernel can end up with a level 1 PTE
entry (THP entry) converted to a level 0 PTE entry without flushing
the THP TLB entry.
Most architectures including POWER have issues with kernel instantiating a level
0 PTE entry while holding level 1 TLB entries.
The code sequence I am looking at is
down_read(mmap_sem) down_read(mmap_sem)
zap_pmd_range()
zap_huge_pmd()
pmd lock held
pmd_cleared
table details added to mmu_gather
pmd_unlock()
insert a level 0 PTE entry()
tlb_finish_mmu().
Fix this by forcing a tlb flush before releasing pmd lock if this is
not a fullmm invalidate. We can safely skip this invalidate for
task exit case (fullmm invalidate) because in that case we are sure
there can be no parallel fault handlers.
This do change the Qemu guest RAM del/unplug time as below
128 core, 496GB guest:
Without patch:
munmap start: timer = 196449 ms, PID=6681
munmap finish: timer = 196488 ms, PID=6681 - delta = 39ms
With patch:
munmap start: timer = 196345 ms, PID=6879
munmap finish: timer = 196714 ms, PID=6879 - delta = 369ms
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/include/asm/book3s/64/pgtable.h | 5 +++++
arch/powerpc/mm/book3s64/pgtable.c | 18 ++++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index 03521a8b0292..e1f551159f7d 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -1265,6 +1265,11 @@ static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
}
#define pmdp_collapse_flush pmdp_collapse_flush
+#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL
+pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
+ unsigned long addr,
+ pmd_t *pmdp, int full);
+
#define __HAVE_ARCH_PGTABLE_DEPOSIT
static inline void pgtable_trans_huge_deposit(struct mm_struct *mm,
pmd_t *pmdp, pgtable_t pgtable)
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index 127325ead505..54b6d6d103ea 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -112,6 +112,24 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
return __pmd(old_pmd);
}
+pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmdp, int full)
+{
+ pmd_t pmd;
+ VM_BUG_ON(addr & ~HPAGE_PMD_MASK);
+ VM_BUG_ON((pmd_present(*pmdp) && !pmd_trans_huge(*pmdp) &&
+ !pmd_devmap(*pmdp)) || !pmd_present(*pmdp));
+ pmd = pmdp_huge_get_and_clear(vma->vm_mm, addr, pmdp);
+ /*
+ * if it not a fullmm flush, then we can possibly end up converting
+ * this PMD pte entry to a regular level 0 PTE by a parallel page fault.
+ * Make sure we flush the tlb in this case.
+ */
+ if (!full)
+ flush_pmd_tlb_range(vma, addr, addr + HPAGE_PMD_SIZE);
+ return pmd;
+}
+
static pmd_t pmd_set_protbits(pmd_t pmd, pgprot_t pgprot)
{
return __pmd(pmd_val(pmd) | pgprot_val(pgprot));
--
2.25.3
^ permalink raw reply related
* [PATCH v3 21/22] mm: change pmdp_huge_get_and_clear_full take vm_area_struct as arg
From: Aneesh Kumar K.V @ 2020-04-20 12:44 UTC (permalink / raw)
To: linuxppc-dev, mpe, linux-mm, linux-kernel, kvm-ppc
Cc: Aneesh Kumar K.V, npiggin, kirill, leonardo, akpm
In-Reply-To: <20200420124434.47330-1-aneesh.kumar@linux.ibm.com>
We will use this in later patch to do tlb flush when clearing pmd entries.
Cc: kirill@shutemov.name
Cc: akpm@linux-foundation.org
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/s390/include/asm/pgtable.h | 4 ++--
include/asm-generic/pgtable.h | 4 ++--
mm/huge_memory.c | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 6076c8c912d2..e2528e057980 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -1560,7 +1560,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
}
#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL
-static inline pmd_t pmdp_huge_get_and_clear_full(struct mm_struct *mm,
+static inline pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
unsigned long addr,
pmd_t *pmdp, int full)
{
@@ -1569,7 +1569,7 @@ static inline pmd_t pmdp_huge_get_and_clear_full(struct mm_struct *mm,
*pmdp = __pmd(_SEGMENT_ENTRY_EMPTY);
return pmd;
}
- return pmdp_xchg_lazy(mm, addr, pmdp, __pmd(_SEGMENT_ENTRY_EMPTY));
+ return pmdp_xchg_lazy(vma->vm_mm, addr, pmdp, __pmd(_SEGMENT_ENTRY_EMPTY));
}
#define __HAVE_ARCH_PMDP_HUGE_CLEAR_FLUSH
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index 329b8c8ca703..d10be362eafa 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -159,11 +159,11 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL
-static inline pmd_t pmdp_huge_get_and_clear_full(struct mm_struct *mm,
+static inline pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
unsigned long address, pmd_t *pmdp,
int full)
{
- return pmdp_huge_get_and_clear(mm, address, pmdp);
+ return pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
}
#endif
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 6ecd1045113b..16f2bd6f1549 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1852,8 +1852,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
* pgtable_trans_huge_withdraw after finishing pmdp related
* operations.
*/
- orig_pmd = pmdp_huge_get_and_clear_full(tlb->mm, addr, pmd,
- tlb->fullmm);
+ orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd,
+ tlb->fullmm);
tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
if (vma_is_special_huge(vma)) {
if (arch_needs_pgtable_deposit())
--
2.25.3
^ permalink raw reply related
* [PATCH v3 20/22] powerpc/mm/book3s64: Avoid sending IPI on clearing PMD
From: Aneesh Kumar K.V @ 2020-04-20 12:44 UTC (permalink / raw)
To: linuxppc-dev, mpe, linux-mm, linux-kernel, kvm-ppc
Cc: leonardo, Aneesh Kumar K.V, npiggin, kirill
In-Reply-To: <20200420124434.47330-1-aneesh.kumar@linux.ibm.com>
Now that all the lockless page table walk is careful w.r.t the PTE
address returned, we can now revert
commit: 13bd817bb884 ("powerpc/thp: Serialize pmd clear against a linux page table walk.")
We also drop the equivalent IPI from other pte updates routines. We still keep
IPI in hash pmdp collapse and that is to take care of parallel hash page table
insert. The radix pmdp collapse flush can possibly be removed once I am sure
generic code doesn't have the any expectations around parallel gup walk.
This speeds up Qemu guest RAM del/unplug time as below
128 core, 496GB guest:
Without patch:
munmap start: timer = 13162 ms, PID=7684
munmap finish: timer = 95312 ms, PID=7684 - delta = 82150 ms
With patch:
munmap start: timer = 196449 ms, PID=6681
munmap finish: timer = 196488 ms, PID=6681 - delta = 39ms
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/mm/book3s64/hash_pgtable.c | 11 -----------
arch/powerpc/mm/book3s64/pgtable.c | 8 --------
arch/powerpc/mm/book3s64/radix_pgtable.c | 19 +++++++------------
3 files changed, 7 insertions(+), 31 deletions(-)
diff --git a/arch/powerpc/mm/book3s64/hash_pgtable.c b/arch/powerpc/mm/book3s64/hash_pgtable.c
index 64733b9cb20a..64ca375278dc 100644
--- a/arch/powerpc/mm/book3s64/hash_pgtable.c
+++ b/arch/powerpc/mm/book3s64/hash_pgtable.c
@@ -363,17 +363,6 @@ pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm,
* hash fault look at them.
*/
memset(pgtable, 0, PTE_FRAG_SIZE);
- /*
- * Serialize against find_current_mm_pte variants which does lock-less
- * lookup in page tables with local interrupts disabled. For huge pages
- * it casts pmd_t to pte_t. Since format of pte_t is different from
- * pmd_t we want to prevent transit from pmd pointing to page table
- * to pmd pointing to huge page (and back) while interrupts are disabled.
- * We clear pmd to possibly replace it with page table pointer in
- * different code paths. So make sure we wait for the parallel
- * find_curren_mm_pte to finish.
- */
- serialize_against_pte_lookup(mm);
return old_pmd;
}
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index e0bb69c616e4..127325ead505 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -109,14 +109,6 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
old_pmd = pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, _PAGE_INVALID);
flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
- /*
- * This ensures that generic code that rely on IRQ disabling
- * to prevent a parallel THP split work as expected.
- *
- * Marking the entry with _PAGE_INVALID && ~_PAGE_PRESENT requires
- * a special case check in pmd_access_permitted.
- */
- serialize_against_pte_lookup(vma->vm_mm);
return __pmd(old_pmd);
}
diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
index 8f9edf07063a..dfb9fe92aea8 100644
--- a/arch/powerpc/mm/book3s64/radix_pgtable.c
+++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
@@ -962,7 +962,13 @@ pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long addre
pmd = *pmdp;
pmd_clear(pmdp);
- /*FIXME!! Verify whether we need this kick below */
+ /*
+ * pmdp collapse_flush need to ensure that there are no parallel gup
+ * walk after this call. This is needed so that we can have stable
+ * page ref count when collapsing a page. We don't allow a collapse page
+ * if we have gup taken on the page. We can ensure that by sending IPI
+ * because gup walk happens with IRQ disabled.
+ */
serialize_against_pte_lookup(vma->vm_mm);
radix__flush_tlb_collapsed_pmd(vma->vm_mm, address);
@@ -1023,17 +1029,6 @@ pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm,
old = radix__pmd_hugepage_update(mm, addr, pmdp, ~0UL, 0);
old_pmd = __pmd(old);
- /*
- * Serialize against find_current_mm_pte which does lock-less
- * lookup in page tables with local interrupts disabled. For huge pages
- * it casts pmd_t to pte_t. Since format of pte_t is different from
- * pmd_t we want to prevent transit from pmd pointing to page table
- * to pmd pointing to huge page (and back) while interrupts are disabled.
- * We clear pmd to possibly replace it with page table pointer in
- * different code paths. So make sure we wait for the parallel
- * find_current_mm_pte to finish.
- */
- serialize_against_pte_lookup(mm);
return old_pmd;
}
--
2.25.3
^ permalink raw reply related
* [PATCH v3 19/22] powerpc/kvm/book3s: Use pte_present instead of opencoding _PAGE_PRESENT check
From: Aneesh Kumar K.V @ 2020-04-20 12:44 UTC (permalink / raw)
To: linuxppc-dev, mpe, linux-mm, linux-kernel, kvm-ppc
Cc: leonardo, Aneesh Kumar K.V, npiggin, kirill
In-Reply-To: <20200420124434.47330-1-aneesh.kumar@linux.ibm.com>
This adds _PAGE_PTE check and makes sure we validate the pte value returned via
find_kvm_host_pte.
NOTE: this also considers _PAGE_INVALID to the software valid bit.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/include/asm/kvm_book3s_64.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h
index 1ca1f6495012..c58e64a0a74f 100644
--- a/arch/powerpc/include/asm/kvm_book3s_64.h
+++ b/arch/powerpc/include/asm/kvm_book3s_64.h
@@ -435,7 +435,7 @@ static inline pte_t kvmppc_read_update_linux_pte(pte_t *ptep, int writing)
continue;
}
/* If pte is not present return None */
- if (unlikely(!(pte_val(old_pte) & _PAGE_PRESENT)))
+ if (unlikely(!pte_present(old_pte)))
return __pte(0);
new_pte = pte_mkyoung(old_pte);
--
2.25.3
^ permalink raw reply related
* [PATCH v3 18/22] powerpc/kvm/book3s: Use find_kvm_host_pte in kvmppc_get_hpa
From: Aneesh Kumar K.V @ 2020-04-20 12:44 UTC (permalink / raw)
To: linuxppc-dev, mpe, linux-mm, linux-kernel, kvm-ppc
Cc: leonardo, Aneesh Kumar K.V, npiggin, kirill
In-Reply-To: <20200420124434.47330-1-aneesh.kumar@linux.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kvm/book3s_hv_rm_mmu.c | 32 ++++++++++-------------------
1 file changed, 11 insertions(+), 21 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_hv_rm_mmu.c b/arch/powerpc/kvm/book3s_hv_rm_mmu.c
index 83e987fecf97..3b168c69d503 100644
--- a/arch/powerpc/kvm/book3s_hv_rm_mmu.c
+++ b/arch/powerpc/kvm/book3s_hv_rm_mmu.c
@@ -878,8 +878,8 @@ long kvmppc_h_clear_mod(struct kvm_vcpu *vcpu, unsigned long flags,
return ret;
}
-static int kvmppc_get_hpa(struct kvm_vcpu *vcpu, unsigned long gpa,
- int writing, unsigned long *hpa,
+static int kvmppc_get_hpa(struct kvm_vcpu *vcpu, unsigned long mmu_seq,
+ unsigned long gpa, int writing, unsigned long *hpa,
struct kvm_memory_slot **memslot_p)
{
struct kvm *kvm = vcpu->kvm;
@@ -898,7 +898,7 @@ static int kvmppc_get_hpa(struct kvm_vcpu *vcpu, unsigned long gpa,
hva = __gfn_to_hva_memslot(memslot, gfn);
/* Try to find the host pte for that virtual address */
- ptep = __find_linux_pte(vcpu->arch.pgdir, hva, NULL, &shift);
+ ptep = find_kvm_host_pte(kvm, mmu_seq, hva, &shift);
if (!ptep)
return H_TOO_HARD;
pte = kvmppc_read_update_linux_pte(ptep, writing);
@@ -933,16 +933,11 @@ static long kvmppc_do_h_page_init_zero(struct kvm_vcpu *vcpu,
mmu_seq = kvm->mmu_notifier_seq;
smp_rmb();
- ret = kvmppc_get_hpa(vcpu, dest, 1, &pa, &memslot);
- if (ret != H_SUCCESS)
- return ret;
-
- /* Check if we've been invalidated */
arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
- if (mmu_notifier_retry(kvm, mmu_seq)) {
- ret = H_TOO_HARD;
+
+ ret = kvmppc_get_hpa(vcpu, mmu_seq, dest, 1, &pa, &memslot);
+ if (ret != H_SUCCESS)
goto out_unlock;
- }
/* Zero the page */
for (i = 0; i < SZ_4K; i += L1_CACHE_BYTES, pa += L1_CACHE_BYTES)
@@ -966,19 +961,14 @@ static long kvmppc_do_h_page_init_copy(struct kvm_vcpu *vcpu,
mmu_seq = kvm->mmu_notifier_seq;
smp_rmb();
- ret = kvmppc_get_hpa(vcpu, dest, 1, &dest_pa, &dest_memslot);
- if (ret != H_SUCCESS)
- return ret;
- ret = kvmppc_get_hpa(vcpu, src, 0, &src_pa, NULL);
+ arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
+ ret = kvmppc_get_hpa(vcpu, mmu_seq, dest, 1, &dest_pa, &dest_memslot);
if (ret != H_SUCCESS)
- return ret;
+ goto out_unlock;
- /* Check if we've been invalidated */
- arch_spin_lock(&kvm->mmu_lock.rlock.raw_lock);
- if (mmu_notifier_retry(kvm, mmu_seq)) {
- ret = H_TOO_HARD;
+ ret = kvmppc_get_hpa(vcpu, mmu_seq, src, 0, &src_pa, NULL);
+ if (ret != H_SUCCESS)
goto out_unlock;
- }
/* Copy the page */
memcpy((void *)dest_pa, (void *)src_pa, SZ_4K);
--
2.25.3
^ permalink raw reply related
* [PATCH v3 17/22] powerpc/kvm/book3s: use find_kvm_host_pte in kvmppc_book3s_instantiate_page
From: Aneesh Kumar K.V @ 2020-04-20 12:44 UTC (permalink / raw)
To: linuxppc-dev, mpe, linux-mm, linux-kernel, kvm-ppc
Cc: leonardo, Aneesh Kumar K.V, npiggin, kirill
In-Reply-To: <20200420124434.47330-1-aneesh.kumar@linux.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/kvm/book3s_64_mmu_radix.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c
index 7b18b298f513..8ce3191cc801 100644
--- a/arch/powerpc/kvm/book3s_64_mmu_radix.c
+++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c
@@ -813,20 +813,20 @@ int kvmppc_book3s_instantiate_page(struct kvm_vcpu *vcpu,
* Read the PTE from the process' radix tree and use that
* so we get the shift and attribute bits.
*/
- local_irq_disable();
- ptep = __find_linux_pte(vcpu->arch.pgdir, hva, NULL, &shift);
+ spin_lock(&kvm->mmu_lock);
+ ptep = find_kvm_host_pte(kvm, mmu_seq, hva, &shift);
/*
* If the PTE disappeared temporarily due to a THP
* collapse, just return and let the guest try again.
*/
if (!ptep) {
- local_irq_enable();
+ spin_unlock(&kvm->mmu_lock);
if (page)
put_page(page);
return RESUME_GUEST;
}
- pte = *ptep;
- local_irq_enable();
+ pte = READ_ONCE(*ptep);
+ spin_unlock(&kvm->mmu_lock);
/* If we're logging dirty pages, always map single pages */
large_enable = !(memslot->flags & KVM_MEM_LOG_DIRTY_PAGES);
--
2.25.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox