[RFCLUE2] 64 bit driver 32 bit app ioctl

[RFCLUE2] 64 bit driver 32 bit app ioctl

[William D Waddington]

Apologies for dashing this off without the proper homework.  My
customer is out of country doing an installation, and didn't test
this configuration first :(

Customer is running RHEL3 on a 64 bit PC.  Running the 64 bit kernel
and my 64 bit driver.  They are calling the driver from their 32 bit
app.  The driver supports a whole mess of ioctls.

It seems that the kernel is trapping the 32-bit ioctl call and returning
an error to the app w/out calling the driver.  It looks like
register_ioctl32_conversion() can convice the kernel that the driver can
handle 32-bit calls, but it has to be called for each ioctl cmd (??)

Putting aside (please) discussion of whether the kernel should presume
to hijack private ioctls, and whether I should be using the ioctl
interface at all (compatibility with app interface going back to 2.0
and SunOS) is there some way to make _one_ register call to indicate
that all my cmds are safe, or maybe an alternate ioctl entry point
that the  kernel won't trap?

Yours in desperation,
Bill

-- 
--------------------------------------------
William D Waddington
Bainbridge Island, WA, USA
william.waddington@beezmo.com
--------------------------------------------
"Even bugs...are unexpected signposts on
the long road of creativity..." - Ken Burtch

Re: [RFCLUE2] 64 bit driver 32 bit app ioctl

[Mikael Pettersson]

William D Waddington writes:
 > Apologies for dashing this off without the proper homework.  My
 > customer is out of country doing an installation, and didn't test
 > this configuration first :(
 > 
 > Customer is running RHEL3 on a 64 bit PC.  Running the 64 bit kernel
 > and my 64 bit driver.  They are calling the driver from their 32 bit
 > app.  The driver supports a whole mess of ioctls.
 > 
 > It seems that the kernel is trapping the 32-bit ioctl call and returning
 > an error to the app w/out calling the driver.  It looks like
 > register_ioctl32_conversion() can convice the kernel that the driver can
 > handle 32-bit calls, but it has to be called for each ioctl cmd (??)

In these old pre-compat_ioctl kernels you have to register each
ioctl command individually. Yes that sucks. Live with it.

 > Putting aside (please) discussion of whether the kernel should presume
 > to hijack private ioctls, and whether I should be using the ioctl
 > interface at all (compatibility with app interface going back to 2.0
 > and SunOS) is there some way to make _one_ register call to indicate
 > that all my cmds are safe, or maybe an alternate ioctl entry point
 > that the  kernel won't trap?

Not as long as you're stuck with old 2.4 kernels. 2.6 kernels since
2.6.11-rc2 allow you to set up a single ->compat_ioctl() method,
but not even RHEL4 has that yet.

Re: [RFCLUE2] 64 bit driver 32 bit app ioctl

[William D Waddington]

Mikael Pettersson wrote:
> William D Waddington writes:
>  > Apologies for dashing this off without the proper homework.  My
>  > customer is out of country doing an installation, and didn't test
>  > this configuration first :(
>  > 
>  > Customer is running RHEL3 on a 64 bit PC.  Running the 64 bit kernel
>  > and my 64 bit driver.  They are calling the driver from their 32 bit
>  > app.  The driver supports a whole mess of ioctls.
>  > 
>  > It seems that the kernel is trapping the 32-bit ioctl call and returning
>  > an error to the app w/out calling the driver.  It looks like
>  > register_ioctl32_conversion() can convice the kernel that the driver can
>  > handle 32-bit calls, but it has to be called for each ioctl cmd (??)
> 
> In these old pre-compat_ioctl kernels you have to register each
> ioctl command individually. Yes that sucks. Live with it.
> 
>  > Putting aside (please) discussion of whether the kernel should presume
>  > to hijack private ioctls, and whether I should be using the ioctl
>  > interface at all (compatibility with app interface going back to 2.0
>  > and SunOS) is there some way to make _one_ register call to indicate
>  > that all my cmds are safe, or maybe an alternate ioctl entry point
>  > that the  kernel won't trap?
> 
> Not as long as you're stuck with old 2.4 kernels. 2.6 kernels since
> 2.6.11-rc2 allow you to set up a single ->compat_ioctl() method,
> but not even RHEL4 has that yet.

Thanks,

It's working OK in my test cases: FC1/64 and the customer's RHEL3.  I
just #include <asm/ioctl32.h> and register all my ioctls.  Ugh.

The location of ioctl32.h seems to move from 2.4 kernel to kernel (and
distro to distro??).  Any suggestion how to include in a universal way
and how to detect all the appropriate 64 bit configs for conditional
inclusion w/a 2.4 kernel?  I detest #ifdef'd code but I guess I have to
do that too, or just keep one version around for this specific case :(

Thanks again,
Bill
--------------------------------------------
William D Waddington
Bainbridge Island, WA, USA
william.waddington@beezmo.com
--------------------------------------------
"Even bugs...are unexpected signposts on
the long road of creativity..." - Ken Burtch

Re: [RFCLUE2] 64 bit driver 32 bit app ioctl

[Arjan van de Ven]

On Thu, 2006-03-23 at 07:06 -0800, William D Waddington wrote:
> Apologies for dashing this off without the proper homework.  My
> customer is out of country doing an installation, and didn't test
> this configuration first :(
> 
> Customer is running RHEL3 on a 64 bit PC.  Running the 64 bit kernel
> and my 64 bit driver.  They are calling the driver from their 32 bit
> app.  The driver supports a whole mess of ioctls.
> 
> It seems that the kernel is trapping the 32-bit ioctl call and returning
> an error to the app w/out calling the driver.  It looks like
> register_ioctl32_conversion() can convice the kernel that the driver can
> handle 32-bit calls, but it has to be called for each ioctl cmd (??)

you forgot to attach you code btw or post the url to it..

Re: [RFCLUE2] 64 bit driver 32 bit app ioctl

[Avi Kivity]

Arjan van de Ven wrote:
> On Thu, 2006-03-23 at 07:06 -0800, William D Waddington wrote:
>   
> [...]
> you forgot to attach you code btw or post the url to it..
>
>   
is that your new .signature?

 :)

-- 
error compiling committee.c: too many arguments to function

Re: [RFCLUE2] 64 bit driver 32 bit app ioctl

[William D Waddington]

Arjan van de Ven wrote:
> On Thu, 2006-03-23 at 07:06 -0800, William D Waddington wrote:
> 
>>Apologies for dashing this off without the proper homework.  My
>>customer is out of country doing an installation, and didn't test
>>this configuration first :(
>>
>>Customer is running RHEL3 on a 64 bit PC.  Running the 64 bit kernel
>>and my 64 bit driver.  They are calling the driver from their 32 bit
>>app.  The driver supports a whole mess of ioctls.
>>
>>It seems that the kernel is trapping the 32-bit ioctl call and returning
>>an error to the app w/out calling the driver.  It looks like
>>register_ioctl32_conversion() can convice the kernel that the driver can
>>handle 32-bit calls, but it has to be called for each ioctl cmd (??)
> 
> 
> you forgot to attach you code btw or post the url to it..

No I didn't :) It's just too ugly for public view.  And I notice it
needs some other fix-ups like fixed width data types in the ioctl
routine...

Thanks to all for the info.  I've got some typing to do.

Bill