[Xenomai-core] RTDM fd support.

[Xenomai-core] RTDM fd support.

[Gilles Chanteperdrix]

Hi Jan,

from discussions on the mailing list, it seems that we are going to need
that unified file descriptors thing. However, since everybody wants
2.5.0 to be released ASAP, we should try to think about any changes for
this support which would break the ABI, do them now, and keep the rest
for later.

One such problem is the translation which currently exists between rtdm
file descriptors and descriptors passed to the posix skin, by adding
1024 - 128. So, I propose to fix this issue.

The idea Philippe had, and which I tend to agree to, was, in case of
open/socket/accept, to open("/dev/null"), and use an association table
somewhere to associate with the kernel-space descriptor number. Since
we are at it, this association table could in fact be the file
descriptor table, which we would put in the core skin ppd. The actual
data structure should be sparse, a linked list does not scale, so,
probably a hash would do. (I could also propose a solution based on avl
trees, but their implementation is not nearly as simple).

Additionnally, this would allow our open/socket to conform to posix
which states that open should return the lowest free file descriptor.

Should I propose a patch in that direction? Do you see any other
possible cause of ABI breakage when we migrate to an unified file
descriptors structure?

TIA,
regards.

-- 
					    Gilles.

Re: [Xenomai-core] RTDM fd support.

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]

Gilles Chanteperdrix wrote:
> Hi Jan,
> 
> from discussions on the mailing list, it seems that we are going to need
> that unified file descriptors thing. However, since everybody wants
> 2.5.0 to be released ASAP, we should try to think about any changes for
> this support which would break the ABI, do them now, and keep the rest
> for later.
> 
> One such problem is the translation which currently exists between rtdm
> file descriptors and descriptors passed to the posix skin, by adding
> 1024 - 128. So, I propose to fix this issue.
> 
> The idea Philippe had, and which I tend to agree to, was, in case of
> open/socket/accept, to open("/dev/null"), and use an association table
> somewhere to associate with the kernel-space descriptor number. Since
> we are at it, this association table could in fact be the file
> descriptor table, which we would put in the core skin ppd. The actual
> data structure should be sparse, a linked list does not scale, so,
> probably a hash would do. (I could also propose a solution based on avl
> trees, but their implementation is not nearly as simple).
> 
> Additionnally, this would allow our open/socket to conform to posix
> which states that open should return the lowest free file descriptor.
> 
> Should I propose a patch in that direction? Do you see any other
> possible cause of ABI breakage when we migrate to an unified file
> descriptors structure?

Right now this sounds like a plan - but I don't feel 100% comfortable to
predict that we will get along with it. Converting some skin-specific
service to a generic one involves quite a lot of refactoring. It is not
really unlikely that we define some ABI now that will later turn out to
be insufficient for what we want to achieve.

The only way around this is to implement at least a prototype. That
takes some time. And until this isn't done, we cannot release 2.5. Or we
have to set this limitation in stone and live with it until 3.x does it
correctly.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Xenomai-core] RTDM fd support.

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> Hi Jan,
>>
>> from discussions on the mailing list, it seems that we are going to need
>> that unified file descriptors thing. However, since everybody wants
>> 2.5.0 to be released ASAP, we should try to think about any changes for
>> this support which would break the ABI, do them now, and keep the rest
>> for later.
>>
>> One such problem is the translation which currently exists between rtdm
>> file descriptors and descriptors passed to the posix skin, by adding
>> 1024 - 128. So, I propose to fix this issue.
>>
>> The idea Philippe had, and which I tend to agree to, was, in case of
>> open/socket/accept, to open("/dev/null"), and use an association table
>> somewhere to associate with the kernel-space descriptor number. Since
>> we are at it, this association table could in fact be the file
>> descriptor table, which we would put in the core skin ppd. The actual
>> data structure should be sparse, a linked list does not scale, so,
>> probably a hash would do. (I could also propose a solution based on avl
>> trees, but their implementation is not nearly as simple).
>>
>> Additionnally, this would allow our open/socket to conform to posix
>> which states that open should return the lowest free file descriptor.
>>
>> Should I propose a patch in that direction? Do you see any other
>> possible cause of ABI breakage when we migrate to an unified file
>> descriptors structure?
> 
> Right now this sounds like a plan - but I don't feel 100% comfortable to
> predict that we will get along with it. Converting some skin-specific
> service to a generic one involves quite a lot of refactoring. It is not
> really unlikely that we define some ABI now that will later turn out to
> be insufficient for what we want to achieve.

For the posix skin, I can live with a two hops solution right now, and
implement the one-hop solution later.

user-space fd
      |
      | core skin fd table
      v
kernel-space fd
      |
      | posix skin registry
      v
actual object

If we do this, there is not much re-factoring involved.

The question really boils down to whether you accept this solution for
the rtdm skin too.

-- 
					    Gilles.

Re: [Xenomai-core] RTDM fd support.

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2756 bytes --]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Gilles Chanteperdrix wrote:
>>> Hi Jan,
>>>
>>> from discussions on the mailing list, it seems that we are going to need
>>> that unified file descriptors thing. However, since everybody wants
>>> 2.5.0 to be released ASAP, we should try to think about any changes for
>>> this support which would break the ABI, do them now, and keep the rest
>>> for later.
>>>
>>> One such problem is the translation which currently exists between rtdm
>>> file descriptors and descriptors passed to the posix skin, by adding
>>> 1024 - 128. So, I propose to fix this issue.
>>>
>>> The idea Philippe had, and which I tend to agree to, was, in case of
>>> open/socket/accept, to open("/dev/null"), and use an association table
>>> somewhere to associate with the kernel-space descriptor number. Since
>>> we are at it, this association table could in fact be the file
>>> descriptor table, which we would put in the core skin ppd. The actual
>>> data structure should be sparse, a linked list does not scale, so,
>>> probably a hash would do. (I could also propose a solution based on avl
>>> trees, but their implementation is not nearly as simple).
>>>
>>> Additionnally, this would allow our open/socket to conform to posix
>>> which states that open should return the lowest free file descriptor.
>>>
>>> Should I propose a patch in that direction? Do you see any other
>>> possible cause of ABI breakage when we migrate to an unified file
>>> descriptors structure?
>> Right now this sounds like a plan - but I don't feel 100% comfortable to
>> predict that we will get along with it. Converting some skin-specific
>> service to a generic one involves quite a lot of refactoring. It is not
>> really unlikely that we define some ABI now that will later turn out to
>> be insufficient for what we want to achieve.
> 
> For the posix skin, I can live with a two hops solution right now, and
> implement the one-hop solution later.
> 
> user-space fd
>       |
>       | core skin fd table
>       v
> kernel-space fd
>       |
>       | posix skin registry
>       v
> actual object
> 
> If we do this, there is not much re-factoring involved.
> 
> The question really boils down to whether you accept this solution for
> the rtdm skin too.

Let me check if I got the idea: the first change would only modify the
librtdm and the rtdm part of libpthread_rt, right? The second step would
then, later on in the 2.5.x series, refactor the core to a generic fd
service, change the kernel ABI and adapt the libraries accordingly while
keeping the library ABI? That would only break statically linked Xenomai
apps, not sure if this is a problem, though.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Xenomai-core] RTDM fd support.

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Hi Jan,
>>>>
>>>> from discussions on the mailing list, it seems that we are going to need
>>>> that unified file descriptors thing. However, since everybody wants
>>>> 2.5.0 to be released ASAP, we should try to think about any changes for
>>>> this support which would break the ABI, do them now, and keep the rest
>>>> for later.
>>>>
>>>> One such problem is the translation which currently exists between rtdm
>>>> file descriptors and descriptors passed to the posix skin, by adding
>>>> 1024 - 128. So, I propose to fix this issue.
>>>>
>>>> The idea Philippe had, and which I tend to agree to, was, in case of
>>>> open/socket/accept, to open("/dev/null"), and use an association table
>>>> somewhere to associate with the kernel-space descriptor number. Since
>>>> we are at it, this association table could in fact be the file
>>>> descriptor table, which we would put in the core skin ppd. The actual
>>>> data structure should be sparse, a linked list does not scale, so,
>>>> probably a hash would do. (I could also propose a solution based on avl
>>>> trees, but their implementation is not nearly as simple).
>>>>
>>>> Additionnally, this would allow our open/socket to conform to posix
>>>> which states that open should return the lowest free file descriptor.
>>>>
>>>> Should I propose a patch in that direction? Do you see any other
>>>> possible cause of ABI breakage when we migrate to an unified file
>>>> descriptors structure?
>>> Right now this sounds like a plan - but I don't feel 100% comfortable to
>>> predict that we will get along with it. Converting some skin-specific
>>> service to a generic one involves quite a lot of refactoring. It is not
>>> really unlikely that we define some ABI now that will later turn out to
>>> be insufficient for what we want to achieve.
>> For the posix skin, I can live with a two hops solution right now, and
>> implement the one-hop solution later.
>>
>> user-space fd
>>       |
>>       | core skin fd table
>>       v
>> kernel-space fd
>>       |
>>       | posix skin registry
>>       v
>> actual object
>>
>> If we do this, there is not much re-factoring involved.
>>
>> The question really boils down to whether you accept this solution for
>> the rtdm skin too.
> 
> Let me check if I got the idea: the first change would only modify the
> librtdm and the rtdm part of libpthread_rt, right?

The idea was to also modify the kernel-space skins. Only make it
superficial: only modify the part which communicates the file
descriptors to user-space, to include a lookup through the fd table.

So, this means that when a user-space applications calls read() for
instance, the fd table is used to get the kernel-space RTDM descriptor,
and then the internal functions of the RTDM skin, go through their own
lookup mechanim to get the actual object.

So, there are two lookups, that is the two hops I was talking about. I
was worried that you would not like the impact on performance.

-- 
					    Gilles.

Re: [Xenomai-core] RTDM fd support.

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3364 bytes --]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Hi Jan,
>>>>>
>>>>> from discussions on the mailing list, it seems that we are going to need
>>>>> that unified file descriptors thing. However, since everybody wants
>>>>> 2.5.0 to be released ASAP, we should try to think about any changes for
>>>>> this support which would break the ABI, do them now, and keep the rest
>>>>> for later.
>>>>>
>>>>> One such problem is the translation which currently exists between rtdm
>>>>> file descriptors and descriptors passed to the posix skin, by adding
>>>>> 1024 - 128. So, I propose to fix this issue.
>>>>>
>>>>> The idea Philippe had, and which I tend to agree to, was, in case of
>>>>> open/socket/accept, to open("/dev/null"), and use an association table
>>>>> somewhere to associate with the kernel-space descriptor number. Since
>>>>> we are at it, this association table could in fact be the file
>>>>> descriptor table, which we would put in the core skin ppd. The actual
>>>>> data structure should be sparse, a linked list does not scale, so,
>>>>> probably a hash would do. (I could also propose a solution based on avl
>>>>> trees, but their implementation is not nearly as simple).
>>>>>
>>>>> Additionnally, this would allow our open/socket to conform to posix
>>>>> which states that open should return the lowest free file descriptor.
>>>>>
>>>>> Should I propose a patch in that direction? Do you see any other
>>>>> possible cause of ABI breakage when we migrate to an unified file
>>>>> descriptors structure?
>>>> Right now this sounds like a plan - but I don't feel 100% comfortable to
>>>> predict that we will get along with it. Converting some skin-specific
>>>> service to a generic one involves quite a lot of refactoring. It is not
>>>> really unlikely that we define some ABI now that will later turn out to
>>>> be insufficient for what we want to achieve.
>>> For the posix skin, I can live with a two hops solution right now, and
>>> implement the one-hop solution later.
>>>
>>> user-space fd
>>>       |
>>>       | core skin fd table
>>>       v
>>> kernel-space fd
>>>       |
>>>       | posix skin registry
>>>       v
>>> actual object
>>>
>>> If we do this, there is not much re-factoring involved.
>>>
>>> The question really boils down to whether you accept this solution for
>>> the rtdm skin too.
>> Let me check if I got the idea: the first change would only modify the
>> librtdm and the rtdm part of libpthread_rt, right?
> 
> The idea was to also modify the kernel-space skins. Only make it
> superficial: only modify the part which communicates the file
> descriptors to user-space, to include a lookup through the fd table.
> 
> So, this means that when a user-space applications calls read() for
> instance, the fd table is used to get the kernel-space RTDM descriptor,
> and then the internal functions of the RTDM skin, go through their own
> lookup mechanim to get the actual object.
> 
> So, there are two lookups, that is the two hops I was talking about. I
> was worried that you would not like the impact on performance.

Yes, I'm a bit. And I do not get the significant advantage of this
approach over the final one-hop approach anymore.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Xenomai-core] RTDM fd support.

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Jan Kiszka wrote:
>>>>> Gilles Chanteperdrix wrote:
>>>>>> Hi Jan,
>>>>>>
>>>>>> from discussions on the mailing list, it seems that we are going to need
>>>>>> that unified file descriptors thing. However, since everybody wants
>>>>>> 2.5.0 to be released ASAP, we should try to think about any changes for
>>>>>> this support which would break the ABI, do them now, and keep the rest
>>>>>> for later.
>>>>>>
>>>>>> One such problem is the translation which currently exists between rtdm
>>>>>> file descriptors and descriptors passed to the posix skin, by adding
>>>>>> 1024 - 128. So, I propose to fix this issue.
>>>>>>
>>>>>> The idea Philippe had, and which I tend to agree to, was, in case of
>>>>>> open/socket/accept, to open("/dev/null"), and use an association table
>>>>>> somewhere to associate with the kernel-space descriptor number. Since
>>>>>> we are at it, this association table could in fact be the file
>>>>>> descriptor table, which we would put in the core skin ppd. The actual
>>>>>> data structure should be sparse, a linked list does not scale, so,
>>>>>> probably a hash would do. (I could also propose a solution based on avl
>>>>>> trees, but their implementation is not nearly as simple).
>>>>>>
>>>>>> Additionnally, this would allow our open/socket to conform to posix
>>>>>> which states that open should return the lowest free file descriptor.
>>>>>>
>>>>>> Should I propose a patch in that direction? Do you see any other
>>>>>> possible cause of ABI breakage when we migrate to an unified file
>>>>>> descriptors structure?
>>>>> Right now this sounds like a plan - but I don't feel 100% comfortable to
>>>>> predict that we will get along with it. Converting some skin-specific
>>>>> service to a generic one involves quite a lot of refactoring. It is not
>>>>> really unlikely that we define some ABI now that will later turn out to
>>>>> be insufficient for what we want to achieve.
>>>> For the posix skin, I can live with a two hops solution right now, and
>>>> implement the one-hop solution later.
>>>>
>>>> user-space fd
>>>>       |
>>>>       | core skin fd table
>>>>       v
>>>> kernel-space fd
>>>>       |
>>>>       | posix skin registry
>>>>       v
>>>> actual object
>>>>
>>>> If we do this, there is not much re-factoring involved.
>>>>
>>>> The question really boils down to whether you accept this solution for
>>>> the rtdm skin too.
>>> Let me check if I got the idea: the first change would only modify the
>>> librtdm and the rtdm part of libpthread_rt, right?
>> The idea was to also modify the kernel-space skins. Only make it
>> superficial: only modify the part which communicates the file
>> descriptors to user-space, to include a lookup through the fd table.
>>
>> So, this means that when a user-space applications calls read() for
>> instance, the fd table is used to get the kernel-space RTDM descriptor,
>> and then the internal functions of the RTDM skin, go through their own
>> lookup mechanim to get the actual object.
>>
>> So, there are two lookups, that is the two hops I was talking about. I
>> was worried that you would not like the impact on performance.
> 
> Yes, I'm a bit. And I do not get the significant advantage of this
> approach over the final one-hop approach anymore.

More superficial modifications, we do not break the posix and rtdm skins
internal registries.

-- 
					    Gilles.

Re: [Xenomai-core] RTDM fd support.

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3909 bytes --]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Jan Kiszka wrote:
>>>>>> Gilles Chanteperdrix wrote:
>>>>>>> Hi Jan,
>>>>>>>
>>>>>>> from discussions on the mailing list, it seems that we are going to need
>>>>>>> that unified file descriptors thing. However, since everybody wants
>>>>>>> 2.5.0 to be released ASAP, we should try to think about any changes for
>>>>>>> this support which would break the ABI, do them now, and keep the rest
>>>>>>> for later.
>>>>>>>
>>>>>>> One such problem is the translation which currently exists between rtdm
>>>>>>> file descriptors and descriptors passed to the posix skin, by adding
>>>>>>> 1024 - 128. So, I propose to fix this issue.
>>>>>>>
>>>>>>> The idea Philippe had, and which I tend to agree to, was, in case of
>>>>>>> open/socket/accept, to open("/dev/null"), and use an association table
>>>>>>> somewhere to associate with the kernel-space descriptor number. Since
>>>>>>> we are at it, this association table could in fact be the file
>>>>>>> descriptor table, which we would put in the core skin ppd. The actual
>>>>>>> data structure should be sparse, a linked list does not scale, so,
>>>>>>> probably a hash would do. (I could also propose a solution based on avl
>>>>>>> trees, but their implementation is not nearly as simple).
>>>>>>>
>>>>>>> Additionnally, this would allow our open/socket to conform to posix
>>>>>>> which states that open should return the lowest free file descriptor.
>>>>>>>
>>>>>>> Should I propose a patch in that direction? Do you see any other
>>>>>>> possible cause of ABI breakage when we migrate to an unified file
>>>>>>> descriptors structure?
>>>>>> Right now this sounds like a plan - but I don't feel 100% comfortable to
>>>>>> predict that we will get along with it. Converting some skin-specific
>>>>>> service to a generic one involves quite a lot of refactoring. It is not
>>>>>> really unlikely that we define some ABI now that will later turn out to
>>>>>> be insufficient for what we want to achieve.
>>>>> For the posix skin, I can live with a two hops solution right now, and
>>>>> implement the one-hop solution later.
>>>>>
>>>>> user-space fd
>>>>>       |
>>>>>       | core skin fd table
>>>>>       v
>>>>> kernel-space fd
>>>>>       |
>>>>>       | posix skin registry
>>>>>       v
>>>>> actual object
>>>>>
>>>>> If we do this, there is not much re-factoring involved.
>>>>>
>>>>> The question really boils down to whether you accept this solution for
>>>>> the rtdm skin too.
>>>> Let me check if I got the idea: the first change would only modify the
>>>> librtdm and the rtdm part of libpthread_rt, right?
>>> The idea was to also modify the kernel-space skins. Only make it
>>> superficial: only modify the part which communicates the file
>>> descriptors to user-space, to include a lookup through the fd table.
>>>
>>> So, this means that when a user-space applications calls read() for
>>> instance, the fd table is used to get the kernel-space RTDM descriptor,
>>> and then the internal functions of the RTDM skin, go through their own
>>> lookup mechanim to get the actual object.
>>>
>>> So, there are two lookups, that is the two hops I was talking about. I
>>> was worried that you would not like the impact on performance.
>> Yes, I'm a bit. And I do not get the significant advantage of this
>> approach over the final one-hop approach anymore.
> 
> More superficial modifications, we do not break the posix and rtdm skins
> internal registries.

Then please summarize again what you want change from the user's POV (fd
range and arbitration, I guess, but also their scope?) and what impact
that will have on the library and kernel ABIs. What would be part of
step 1, what of step 2 later in 2.5.x?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Xenomai-core] RTDM fd support.

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Then please summarize again what you want change from the user's POV (fd
> range and arbitration, I guess, but also their scope?)

Basically, when going from user-space to kernel-space, instead of a
simple translation, currently done with an addition in user-space for
most services, in kernel-space for select, there is a hash lookup, done
in kernel-space.

 and what impact
> that will have on the library and kernel ABIs.

The impact on the ABI is that we no longer do the translation using
addition/substraction. If we broke all at once in version 2.5.x, we
could not run a 2.5.x-1 user-space support that does the
addition/substraction, with a 2.5.x kernel-space support which has the
unified file descriptors abstraction.

 What would be part of
> step 1, what of step 2 later in 2.5.x?

Step 1 is adding the fdtable, use it in the rtdm and posix skin before
returning a fd to user-space, and when getting an fd from user-space.
Remove the translation by addition/substraction in the rtdm part of
libpthread_rt.

Step 2 is replacing the rtdm and posix registries with an unified fd
support based on the fdtable, the real kernel-space file descriptors (I
mean, the one which really belong to a kernel-space app) would be
implemented using a global fdtable. Duplicate the fdtable at fork, etc...

-- 
					    Gilles.

Re: [Xenomai-core] RTDM fd support.

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2196 bytes --]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Then please summarize again what you want change from the user's POV (fd
>> range and arbitration, I guess, but also their scope?)
> 
> Basically, when going from user-space to kernel-space, instead of a
> simple translation, currently done with an addition in user-space for
> most services, in kernel-space for select, there is a hash lookup, done
> in kernel-space.
> 
>  and what impact
>> that will have on the library and kernel ABIs.
> 
> The impact on the ABI is that we no longer do the translation using
> addition/substraction. If we broke all at once in version 2.5.x, we
> could not run a 2.5.x-1 user-space support that does the
> addition/substraction, with a 2.5.x kernel-space support which has the
> unified file descriptors abstraction.
> 
>  What would be part of
>> step 1, what of step 2 later in 2.5.x?
> 
> Step 1 is adding the fdtable, use it in the rtdm and posix skin before
> returning a fd to user-space, and when getting an fd from user-space.
> Remove the translation by addition/substraction in the rtdm part of
> libpthread_rt.

I assume this mapping will already make user-visible fds process-local,
right?

BTW, one downside of pushing the translation into the kernel part of the
skins is that POSIX borderline threads will pay via an additional
syscall + a futile lookup in the Xenomai fdtable. So far we can dispatch
 very cheaply. On the other hand, POSIX users who care about performance
could also user __real_* or avoid the automatic wrapping.

> 
> Step 2 is replacing the rtdm and posix registries with an unified fd
> support based on the fdtable, the real kernel-space file descriptors (I
> mean, the one which really belong to a kernel-space app) would be
> implemented using a global fdtable. Duplicate the fdtable at fork, etc...
> 

OK, if you are willing to draft a prototype for step 1, I'm open to
discuss it. My requirements would be:
 - per-process fdtables (and locks) for the first level lookup
 - the second level lookup should be limited to device creation; simply
   keep a permanent reference to the rtdm_context in the fdtable

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]