Re: [regression] ENOTTY returned for tty fds

[Greg KH <gregkh@linuxfoundation.org>]

On Sun, Jul 21, 2024 at 02:52:31PM +0300, stsp wrote:
> Hi!
> 
> I've noticed that one of my progs failed
> to work with serial ports. To distinguish
> between real serial ports and PTS nodes,
> it tried TIOCMBIC, checking errno for EINVAL.
> 
> Obviously that behaviour was changed
> and now ENOTTY is returned. Besides
> being backward-incompatible, I think this
> is also wrong because isatty() returns 1
> on such FDs.
> 
> Here's the test-case:
> socat PTY,link=/tmp/ttyS20,raw,echo=0,b9600
> PTY,link=/tmp/ttyS21,raw,echo=0,b9600
> 
> Then run this program:
> 
> #include <stdio.h>
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <fcntl.h>
> #include <errno.h>
> #include <sys/ioctl.h>
> #include <unistd.h>
> 
> int main() {
>   int fd, err;
>   int data = TIOCM_DTR | TIOCM_RTS;
>   fd = open("/tmp/ttyS20", O_NONBLOCK);
>   if (fd == -1) {
>     perror("open()");
>     return 1;
>   }
>   err = ioctl(fd, TIOCMBIC, &data);
>   if (err && errno == ENOTTY && isatty(fd))
>     printf("Test FAILED: ENOTTY for tty fd\n");
>   return 0;
> }
> 
> 

What caused this change/regression?  And does any real-world programs
rely on this?  What exactly are you trying to determine with this ioctl
test?  Is there a different way to determine that?

thanks,

greg k-h