[BUG] Btattach, ioctl fails to set flags from 32-bit user space to 64-bit

[BUG] Btattach, ioctl fails to set flags from 32-bit user space to 64-bit

[Jaap Jan Meijer]

Hi,

WIth Btattach, ioctl fails to set flags from 32-bit user space to 64-bit
kernel. ioctl fails with ENOTTY.

static int attach_proto(const char *path, unsigned int proto,
unsigned int flags)
{
int fd, dev_id;

fd = open_serial(path);
if (fd < 0)
return -1;

if (ioctl(fd, HCIUARTSETFLAGS, flags) < 0) {
perror("Failed to set flags");
close(fd);
return -1;
}
.....

Kind regards,

J.J.

Re: [BUG] Btattach, ioctl fails to set flags from 32-bit user space to 64-bit

[Marcel Holtmann]

Hi Jaap,

> WIth Btattach, ioctl fails to set flags from 32-bit user space to 64-bit
> kernel. ioctl fails with ENOTTY.
> 
> static int attach_proto(const char *path, unsigned int proto,
> unsigned int flags)
> {
> int fd, dev_id;
> 
> fd = open_serial(path);
> if (fd < 0)
> return -1;
> 
> if (ioctl(fd, HCIUARTSETFLAGS, flags) < 0) {
> perror("Failed to set flags");
> close(fd);
> return -1;
> }
> .....

I am a little bit lost on how you get ENOTTY. The only way I can see this happening if the kernel falls through towards n_tty_ioctl_helper and that means that cmd is messed up.

If you go into drivers/bluetooth/hci_ldisc.c and add this line

	hci_uart_ldisc.compat_ioctl = hci_uart_tty_ioctl;

it might fix it. It is a guess right now until you figure out where the ENOTTY is coming from.

Regards

Marcel

Re: [BUG] Btattach, ioctl fails to set flags from 32-bit user space to 64-bit

[Jaap Jan Meijer]

Hi Marcel,

Thank you, that was very good guess work. Btattach is now able to
attach by bcm43241 in the mixed arch situation. Should I supply a
patch or will you just add it? I'll will send-email a patch anyway.

Kind regards,

J.J.




2016-01-21 12:12 GMT+01:00 Marcel Holtmann <marcel@holtmann.org>:
> Hi Jaap,
>
>> WIth Btattach, ioctl fails to set flags from 32-bit user space to 64-bit
>> kernel. ioctl fails with ENOTTY.
>>
>> static int attach_proto(const char *path, unsigned int proto,
>> unsigned int flags)
>> {
>> int fd, dev_id;
>>
>> fd = open_serial(path);
>> if (fd < 0)
>> return -1;
>>
>> if (ioctl(fd, HCIUARTSETFLAGS, flags) < 0) {
>> perror("Failed to set flags");
>> close(fd);
>> return -1;
>> }
>> .....
>
> I am a little bit lost on how you get ENOTTY. The only way I can see this happening if the kernel falls through towards n_tty_ioctl_helper and that means that cmd is messed up.
>
> If you go into drivers/bluetooth/hci_ldisc.c and add this line
>
>         hci_uart_ldisc.compat_ioctl = hci_uart_tty_ioctl;
>
> it might fix it. It is a guess right now until you figure out where the ENOTTY is coming from.
>
> Regards
>
> Marcel
>

Re: [BUG] Btattach, ioctl fails to set flags from 32-bit user space to 64-bit

[Marcel Holtmann]

Hi Jaap,

> Thank you, that was very good guess work. Btattach is now able to
> attach by bcm43241 in the mixed arch situation. Should I supply a
> patch or will you just add it? I'll will send-email a patch anyway.

please send a patch for it.

Regards

Marcel

Re: [BUG] Btattach, ioctl fails to set flags from 32-bit user space to 64-bit

[Marcel Holtmann]

Hi Jaap,

> Thank you, that was very good guess work. Btattach is now able to
> attach by bcm43241 in the mixed arch situation. Should I supply a
> patch or will you just add it? I'll will send-email a patch anyway.

I just send a patch that should be more correct on all platforms. Can you try that one out please.

Regards

Marcel

Re: [BUG] Btattach, ioctl fails to set flags from 32-bit user space to 64-bit

[Marcel Holtmann]

Hi Jaap,

>> Thank you, that was very good guess work. Btattach is now able to
>> attach by bcm43241 in the mixed arch situation. Should I supply a
>> patch or will you just add it? I'll will send-email a patch anyway.
> 
> I just send a patch that should be more correct on all platforms. Can you try that one out please.

actually it might be just some missing COMPATIBLE_IOCTL declarations. I just send a patch for that. The patch is correct by itself, but it might actually fix your issue and no .compat_ioctl is needed.

Regards

Marcel