VDSO for arm 32bit

VDSO for arm 32bit

[Kumar, Ganesh]

Hi All,

? I'm working on OMAP5 on greg's LTS kernel 3.4.25, but I see there is
no support for vdso for arm 32 bit.
I'm planning to port it to arm 32 bit, is there any work under progress?
would like to understand why the support for arm 32 bit is not there, any idea?

Thanks for your time.

Thanks,
GK

VDSO for arm 32bit

[Mikael Pettersson]

Kumar, Ganesh writes:
 > Hi All,
 > 
 > ? I'm working on OMAP5 on greg's LTS kernel 3.4.25, but I see there is
 > no support for vdso for arm 32 bit.
 > I'm planning to port it to arm 32 bit, is there any work under progress?
 > would like to understand why the support for arm 32 bit is not there, any idea?

32-bit ARM has a "kuser" page at a fixed virtual address, which exports a
few critical routines to user-space.  With that, a vdso is mostly redundant.

VDSO for arm 32bit

[Nicolas Pitre]

On Mon, 27 May 2013, Kumar, Ganesh wrote:

> Hi All,
> 
> ? I'm working on OMAP5 on greg's LTS kernel 3.4.25, but I see there is
> no support for vdso for arm 32 bit.
> I'm planning to port it to arm 32 bit, is there any work under progress?
> would like to understand why the support for arm 32 bit is not there, any idea?

Please have a look at Documentation/arm/kernel_user_helpers.txt.


Nicolas

VDSO for arm 32bit

[Kumar, Ganesh]

> Please have a look at Documentation/arm/kernel_user_helpers.txt.

Thanks Nicolas, will start with that and look into the code further.
So the kuser helper are not yet utilized to get the gettimeofday 
as of now on arm32,is my understanding correct?
I'm a newbie in this area and trying to get a fair understanding 
on the topic.

VDSO for arm 32bit

[Kumar, Ganesh]

Hi Mikel,

Kumar, Ganesh writes:
 > Hi All,
 >
 > ? I'm working on OMAP5 on greg's LTS kernel 3.4.25, but I see there is  > no support for vdso for arm 32 bit.
 > I'm planning to port it to arm 32 bit, is there any work under progress?
 > would like to understand why the support for arm 32 bit is not there, any idea?

> 32-bit ARM has a "kuser" page at a fixed virtual address, which exports a few critical routines to user-space.  With that, a vdso is mostly redundant.

  Thanks for the info, my main idea is to get the gettimeofday exported through the kuser helper,tried looking at the sources
but I could not find anything related to that seems done in the mainline kernel tree for arm cortex, am I missing something here?
I plan to do that if its not available yet, right now I've started with the kernel doc for the kuser, any pointers further
to get the gettimeofday up and running using the kuser, Thanks in advance for your time.

--GK

VDSO for arm 32bit

[Will Deacon]

On Tue, May 28, 2013 at 08:48:33AM +0100, Kumar, Ganesh wrote:
> Hi Mikel,
> 
> Kumar, Ganesh writes:
>  > Hi All,
>  >
>  > ? I'm working on OMAP5 on greg's LTS kernel 3.4.25, but I see there is  > no support for vdso for arm 32 bit.
>  > I'm planning to port it to arm 32 bit, is there any work under progress?
>  > would like to understand why the support for arm 32 bit is not there, any idea?
> 
> > 32-bit ARM has a "kuser" page at a fixed virtual address, which exports a few critical routines to user-space.  With that, a vdso is mostly redundant.
> 
>   Thanks for the info, my main idea is to get the gettimeofday exported through the kuser helper,tried looking at the sources
> but I could not find anything related to that seems done in the mainline kernel tree for arm cortex, am I missing something here?
> I plan to do that if its not available yet, right now I've started with the kernel doc for the kuser, any pointers further
> to get the gettimeofday up and running using the kuser, Thanks in advance for your time.

Be careful here. I had a crack at a gtod implementation for the vectors page
in the past, but decided to drop it because it quickly became a maintenance
nightmare. Since the layout of the vectors page must be fixed (as userspace
jumps to the absolute addresses for the helpers), having large, complicated
functions (which gtod does become as you quickly start using the stack to
store all of the shared state with the kernel) means that you can quickly
back yourself into a corner where the layout of the code cannot change
without breaking the ABI.

On top of that, new clock types are added more often than you might think
(not to mention bug fixes in the implementation itself), and you end up in
a horrible situation where you'd end up deprecating one implementation and
adding gettimeofday2 or something equally nasty at a different offset.

Maybe this can be solved by adding loads of padding between the different
functions and hoping for the best, but I think a better solution would be to
try and add something which is actually a vdso but preserves the function
offsets of the current helpers, thus preserving the ABI. That could then
replace the vectors page and newer C libraries could make use of the vdso
functionality. Of course, that's a lot more work and people might have
objections to the approach.

Finally, I assume you're only interested in the co-processor backed
architected timers for this? I know some people were toying with exposing
memory-mapped timers to userspace, but that really scares me.

Will

VDSO for arm 32bit

[Nicolas Pitre]

On Tue, 28 May 2013, Will Deacon wrote:

> On Tue, May 28, 2013 at 08:48:33AM +0100, Kumar, Ganesh wrote:
> > Hi Mikel,
> > 
> > Kumar, Ganesh writes:
> >  > Hi All,
> >  >
> >  > ? I'm working on OMAP5 on greg's LTS kernel 3.4.25, but I see there is  > no support for vdso for arm 32 bit.
> >  > I'm planning to port it to arm 32 bit, is there any work under progress?
> >  > would like to understand why the support for arm 32 bit is not there, any idea?
> > 
> > > 32-bit ARM has a "kuser" page at a fixed virtual address, which exports a few critical routines to user-space.  With that, a vdso is mostly redundant.
> > 
> >   Thanks for the info, my main idea is to get the gettimeofday exported through the kuser helper,tried looking at the sources
> > but I could not find anything related to that seems done in the mainline kernel tree for arm cortex, am I missing something here?
> > I plan to do that if its not available yet, right now I've started with the kernel doc for the kuser, any pointers further
> > to get the gettimeofday up and running using the kuser, Thanks in advance for your time.
> 
> Be careful here. I had a crack at a gtod implementation for the vectors page
> in the past, but decided to drop it because it quickly became a maintenance
> nightmare. Since the layout of the vectors page must be fixed (as userspace
> jumps to the absolute addresses for the helpers), having large, complicated
> functions (which gtod does become as you quickly start using the stack to
> store all of the shared state with the kernel) means that you can quickly
> back yourself into a corner where the layout of the code cannot change
> without breaking the ABI.

Note that only the entry point is fixed.  The layout of the code is not.  
So you may well branch anywhere else within the vector page if the code 
is more complex than a few instructions.

> On top of that, new clock types are added more often than you might think
> (not to mention bug fixes in the implementation itself), and you end up in
> a horrible situation where you'd end up deprecating one implementation and
> adding gettimeofday2 or something equally nasty at a different offset.

That is equally valid for a vdso.


Nicolas

VDSO for arm 32bit

[Kumar, Ganesh]

On Tue, May 28, 2013 at 08:48:33AM +0100, Kumar, Ganesh wrote:
> Hi Mikel,
> 
> Kumar, Ganesh writes:
>  > Hi All,
>  >
>  > ? I'm working on OMAP5 on greg's LTS kernel 3.4.25, but I see there is  > no support for vdso for arm 32 bit.
>  > I'm planning to port it to arm 32 bit, is there any work under progress?
>  > would like to understand why the support for arm 32 bit is not there, any idea?
> 
> > 32-bit ARM has a "kuser" page at a fixed virtual address, which exports a few critical routines to user-space.  With that, a vdso is mostly redundant.
> 
>   Thanks for the info, my main idea is to get the gettimeofday 
> exported through the kuser helper,tried looking at the sources but I could not find anything related to that seems done in the mainline kernel tree for arm cortex, am I missing something here?
> I plan to do that if its not available yet, right now I've started 
> with the kernel doc for the kuser, any pointers further to get the gettimeofday up and running using the kuser, Thanks in advance for your time.

>>>Be careful here. I had a crack at a gtod implementation for the vectors page in the past, but decided to drop it because it quickly became a maintenance nightmare. Since the layout of >>>the vectors page must be fixed (as userspace jumps to the absolute addresses for the helpers), having large, complicated functions (which gtod does become as you quickly start using >>>the stack to store all of the shared state with the kernel) means that you can quickly back yourself into a corner where the layout of the code cannot change without breaking the ABI.
>>>On top of that, new clock types are added more often than you might think (not to mention bug fixes in the implementation itself), and you end up in a horrible situation where you'd >>>end up deprecating one implementation and adding gettimeofday2 or something equally nasty at a different offset.

>>>Maybe this can be solved by adding loads of padding between the different functions and hoping for the best, but I think a better solution would be to try and add something which is >>>actually a vdso but preserves the function offsets of the current helpers, thus preserving the ABI. That could then replace the vectors page and newer C libraries could make use of >>>the vdso functionality. Of course, that's a lot more work and people might have objections to the approach.

   Hmm, it doesn't seem to be quite straight forward, but thanks for the inputs and caution.
Now that I got a direction, I'm planning to proceed as below:
1. As a first step, get the implementation up and running using the kuser helper (understand the situation and plan to limit my implementation for the ARM Cortex's arch timer clock type)
    On the user land, does this approach needs a change to the libc, kindly confirm. I read the kernel documentation for kuser helper, it gave a nice overview of 
    how things can be done, but not so clear on how actually the helper functions can be called from the user land (the big picture), does it needs a tweak in the libc in order to enable     
    the new helper functions defined.

2. Plan to get it up through the VDSO, study in progress. One question here, was the vdso implementation for 32 bit is not available only because of the frequent addition in new clock types or there is any strong reason or something I need to take care about, kindly clarify.
   I'll start looking into the arm64 vdso code as a starting point, is there anything else I can refer/read to get going? kindly help. 

>>> Finally, I assume you're only interested in the co-processor backed architected timers for this?

   Yes, correct. 

>>> I know some people were toying with exposing memory-mapped timers to userspace, but >>> that really scares me.

   Just curious, what are the side effects of exposing the memory-mapped timers to user space.

Will

GK

VDSO for arm 32bit

[Gilles Chanteperdrix]

On 05/28/2013 11:23 AM, Will Deacon wrote:

> On Tue, May 28, 2013 at 08:48:33AM +0100, Kumar, Ganesh wrote:
>> Hi Mikel,
>>
>> Kumar, Ganesh writes:
>>  > Hi All,
>>  >
>>  >   I'm working on OMAP5 on greg's LTS kernel 3.4.25, but I see there is  > no support for vdso for arm 32 bit.
>>  > I'm planning to port it to arm 32 bit, is there any work under progress?
>>  > would like to understand why the support for arm 32 bit is not there, any idea?
>>
>>> 32-bit ARM has a "kuser" page at a fixed virtual address, which exports a few critical routines to user-space.  With that, a vdso is mostly redundant.
>>
>>   Thanks for the info, my main idea is to get the gettimeofday exported through the kuser helper,tried looking at the sources
>> but I could not find anything related to that seems done in the mainline kernel tree for arm cortex, am I missing something here?
>> I plan to do that if its not available yet, right now I've started with the kernel doc for the kuser, any pointers further
>> to get the gettimeofday up and running using the kuser, Thanks in advance for your time.
> 
> Be careful here. I had a crack at a gtod implementation for the vectors page
> in the past, but decided to drop it because it quickly became a maintenance
> nightmare. Since the layout of the vectors page must be fixed (as userspace
> jumps to the absolute addresses for the helpers), having large, complicated
> functions (which gtod does become as you quickly start using the stack to
> store all of the shared state with the kernel) means that you can quickly
> back yourself into a corner where the layout of the code cannot change
> without breaking the ABI.
> 
> On top of that, new clock types are added more often than you might think
> (not to mention bug fixes in the implementation itself), and you end up in
> a horrible situation where you'd end up deprecating one implementation and
> adding gettimeofday2 or something equally nasty at a different offset.
> 
> Maybe this can be solved by adding loads of padding between the different
> functions and hoping for the best, but I think a better solution would be to
> try and add something which is actually a vdso but preserves the function
> offsets of the current helpers, thus preserving the ABI. That could then
> replace the vectors page and newer C libraries could make use of the vdso
> functionality. Of course, that's a lot more work and people might have
> objections to the approach.
> 
> Finally, I assume you're only interested in the co-processor backed
> architected timers for this? I know some people were toying with exposing
> memory-mapped timers to userspace, but that really scares me.


Hi,

We have been doing this for some time with Xenomai on ARM. What we do is
that:
- the function to read the clocksource is in the kuser area, at a fixed
address, its precise implementation (depending on the clocksource type)
is copied to the vector page when the clocksource is chosen, it expects
the user-space virtual address of the MMIO register in r0;
- for MMIO registers, the library maps the register with /dev/mem;
- the data to handle gettimeofday (so namely, the contents of the
"struct timekeeper" structure) are copied to the equivalent of a vdso.

Regards.

-- 
                                                                Gilles.

VDSO for arm 32bit

[Will Deacon]

On Tue, May 28, 2013 at 07:19:00PM +0100, Nicolas Pitre wrote:
> On Tue, 28 May 2013, Will Deacon wrote:
> > Be careful here. I had a crack at a gtod implementation for the vectors page
> > in the past, but decided to drop it because it quickly became a maintenance
> > nightmare. Since the layout of the vectors page must be fixed (as userspace
> > jumps to the absolute addresses for the helpers), having large, complicated
> > functions (which gtod does become as you quickly start using the stack to
> > store all of the shared state with the kernel) means that you can quickly
> > back yourself into a corner where the layout of the code cannot change
> > without breaking the ABI.
> 
> Note that only the entry point is fixed.  The layout of the code is not.  
> So you may well branch anywhere else within the vector page if the code 
> is more complex than a few instructions.

Sure, it may be possible simply to have branches in the kuser slots and then
reserve a chunk at the other end of the vectors page where the actual
implementation can go.

> > On top of that, new clock types are added more often than you might think
> > (not to mention bug fixes in the implementation itself), and you end up in
> > a horrible situation where you'd end up deprecating one implementation and
> > adding gettimeofday2 or something equally nasty at a different offset.
> 
> That is equally valid for a vdso.

Sorry, I wasn't clear. My point was that you can change the layout of a vdso
without breaking anything, since the dynamic loader resolves the entry
points. It relates back to what I said above -- you need to be careful about
how you lay the stuff out in memory.

Perhaps the gtod code won't end up being that big. It's around ~500 bytes on
arm64, but that's without any stack usage. Come to think of it, is there any
reason why we can't just map an additional vectors page after the one we
currently have?

Will

VDSO for arm 32bit

[Will Deacon]

On Wed, May 29, 2013 at 05:00:09AM +0100, Kumar, Ganesh wrote:
>    Hmm, it doesn't seem to be quite straight forward, but thanks for the
>    inputs and caution.  Now that I got a direction, I'm planning to
>    proceed as below: 1. As a first step, get the implementation up and
>    running using the kuser helper (understand the situation and plan to
>    limit my implementation for the ARM Cortex's arch timer clock type) On
>    the user land, does this approach needs a change to the libc, kindly
>    confirm. I read the kernel documentation for kuser helper, it gave a
>    nice overview of how things can be done, but not so clear on how
>    actually the helper functions can be called from the user land (the big
>    picture), does it needs a tweak in the libc in order to enable     the
>    new helper functions defined.

Yes, libc would need updating to direct the various clock system calls via
the vectors page if the kernel is recent enough. Your implementation will
need to fall back to issuing the system call if the arch timer isn't the
current clocksource.

> 2. Plan to get it up through the VDSO, study in progress. One question
> here, was the vdso implementation for 32 bit is not available only because
> of the frequent addition in new clock types or there is any strong reason
> or something I need to take care about, kindly clarify.  I'll start
> looking into the arm64 vdso code as a starting point, is there anything
> else I can refer/read to get going? kindly help. 

This might not be needed after the feedback from Nicolas.

> >>> I know some people were toying with exposing memory-mapped timers to
> >>> userspace, but >>> that really scares me.
> 
>    Just curious, what are the side effects of exposing the memory-mapped
>    timers to user space.

Just that you need to be careful about what you expose and how do you expose
it. For example, control registers may lie in the same page as the actual
counter, and you don't want userspace having access to that if possible.

You'd probably also need to copy code into place, since there are many timer
implementations that could be used.

Will

VDSO for arm 32bit

[Nicolas Pitre]

On Wed, 29 May 2013, Will Deacon wrote:

> On Tue, May 28, 2013 at 07:19:00PM +0100, Nicolas Pitre wrote:
> > On Tue, 28 May 2013, Will Deacon wrote:
> > > Be careful here. I had a crack at a gtod implementation for the vectors page
> > > in the past, but decided to drop it because it quickly became a maintenance
> > > nightmare. Since the layout of the vectors page must be fixed (as userspace
> > > jumps to the absolute addresses for the helpers), having large, complicated
> > > functions (which gtod does become as you quickly start using the stack to
> > > store all of the shared state with the kernel) means that you can quickly
> > > back yourself into a corner where the layout of the code cannot change
> > > without breaking the ABI.
> > 
> > Note that only the entry point is fixed.  The layout of the code is not.  
> > So you may well branch anywhere else within the vector page if the code 
> > is more complex than a few instructions.
> 
> Sure, it may be possible simply to have branches in the kuser slots and then
> reserve a chunk at the other end of the vectors page where the actual
> implementation can go.
> 
> > > On top of that, new clock types are added more often than you might think
> > > (not to mention bug fixes in the implementation itself), and you end up in
> > > a horrible situation where you'd end up deprecating one implementation and
> > > adding gettimeofday2 or something equally nasty at a different offset.
> > 
> > That is equally valid for a vdso.
> 
> Sorry, I wasn't clear. My point was that you can change the layout of a vdso
> without breaking anything, since the dynamic loader resolves the entry
> points. It relates back to what I said above -- you need to be careful about
> how you lay the stuff out in memory.
> 
> Perhaps the gtod code won't end up being that big. It's around ~500 bytes on
> arm64, but that's without any stack usage.

That's still fairly large.

> Come to think of it, is there any
> reason why we can't just map an additional vectors page after the one we
> currently have?

That certainly can be done if necessary.  Some copy_page implementations 
do use some of the address space above the vector page for temporary 
page mapping so some care not to create conflicts would be required.


Nicolas

VDSO for arm 32bit

[Dave Martin]

On Wed, May 29, 2013 at 10:01:40AM -0400, Nicolas Pitre wrote:
> On Wed, 29 May 2013, Will Deacon wrote:
> 
> > On Tue, May 28, 2013 at 07:19:00PM +0100, Nicolas Pitre wrote:
> > > On Tue, 28 May 2013, Will Deacon wrote:
> > > > Be careful here. I had a crack at a gtod implementation for the vectors page
> > > > in the past, but decided to drop it because it quickly became a maintenance
> > > > nightmare. Since the layout of the vectors page must be fixed (as userspace
> > > > jumps to the absolute addresses for the helpers), having large, complicated
> > > > functions (which gtod does become as you quickly start using the stack to
> > > > store all of the shared state with the kernel) means that you can quickly
> > > > back yourself into a corner where the layout of the code cannot change
> > > > without breaking the ABI.
> > > 
> > > Note that only the entry point is fixed.  The layout of the code is not.  
> > > So you may well branch anywhere else within the vector page if the code 
> > > is more complex than a few instructions.
> > 
> > Sure, it may be possible simply to have branches in the kuser slots and then
> > reserve a chunk at the other end of the vectors page where the actual
> > implementation can go.
> > 
> > > > On top of that, new clock types are added more often than you might think
> > > > (not to mention bug fixes in the implementation itself), and you end up in
> > > > a horrible situation where you'd end up deprecating one implementation and
> > > > adding gettimeofday2 or something equally nasty at a different offset.
> > > 
> > > That is equally valid for a vdso.
> > 
> > Sorry, I wasn't clear. My point was that you can change the layout of a vdso
> > without breaking anything, since the dynamic loader resolves the entry
> > points. It relates back to what I said above -- you need to be careful about
> > how you lay the stuff out in memory.
> > 
> > Perhaps the gtod code won't end up being that big. It's around ~500 bytes on
> > arm64, but that's without any stack usage.
> 
> That's still fairly large.
> 
> > Come to think of it, is there any
> > reason why we can't just map an additional vectors page after the one we
> > currently have?
> 
> That certainly can be done if necessary.  Some copy_page implementations 
> do use some of the address space above the vector page for temporary 
> page mapping so some care not to create conflicts would be required.

In theory, we could have a VDSO wrapping the vectors page.  This could allow
the existing kuser helpers to be looked up by ELF symbol as well as the well-
known address.  New functions don't necessarily need to have a well-known
address at all -- they might only be visible as ELF symbols, since there is
no need to offer a backwards-compatible ABI for new functions.

Alternatively, we could have a VDSO which is separate from the vectors
page, exporting only the new functionality, with the vectors page remaining
as before.

Whether or not adding a VDSO brings enough benefit to make it worthwhile
is a separate discussion, though.

Cheers
---Dave