Re: [regression] ENOTTY returned for tty fds

[Johan Hovold <johan@kernel.org>]

[ For some reason I never received Herbert's replies so I didn't see
  them until today. ]

On Wed, Jul 31, 2024 at 12:25:43PM +0200, Greg Kroah-Hartman wrote:
> On Wed, Jul 24, 2024 at 06:58:22PM +0800, Herbert Xu wrote:
> > On Wed, Jul 24, 2024 at 06:53:00PM +0800, Herbert Xu wrote:
> > > On Wed, Jul 24, 2024 at 12:15:39PM +0200, Greg KH wrote:
> > > >
> > > > -ENOTTY is the documented result of invalid ioctl arguments sent, I am
> > > > pretty sure POSIX requires this somewhere.  So this was fixing a
> > > > requirement here...
> > > 
> > > POSIX does not specify this at all:
> > > 
> > > https://pubs.opengroup.org/onlinepubs/9699919799/functions/ioctl.html
> > 
> > In fact it says:
> > 
> > If an underlying device driver detects an error, then ioctl() shall fail if:
> > 
> > [EINVAL]
> > The request or arg argument is not valid for this device.
> > 
> > [ENOTTY]
> > The file associated with the fildes argument is not a STREAMS device that accepts control functions.
> > 
> > Of course this is all moot since POSIX only specifies ioctl(2)
> > for STREAMS devices, but this patch in question is literally
> > going against the woring here.
> 
> That's going to be going against a lot of existing kernel code then.
> -ENOTTY is the default action of almost all ioctl handlers when the
> command is not a valid one, sorry.

This has been discussed at length before, and Linus made it clear that
ENOTTY is the right errno for unsupported ioctls, for example, here:

	https://lore.kernel.org/lkml/BANLkTi=6W0quy1M71UapwKDe97E67b4EiA@mail.gmail.com/

The tty code has been returning this since 2012 and commit bbb63c514a34
("drivers:tty:fix up ENOIOCTLCMD error handling") except for a few cases
that were missed and that I fixed up in 2021.

If this breaks real applications me may need to revert to EINVAL for the
affected ioctls however. Apparently, dosemu hasn't been
updated since 2013, but it looks like it indeed expects EINVAL for
unsupported TIOCMBIC.

Johan