[PATCH] dup2: Fix return value with oldfd == newfd and invalid fd

[PATCH] dup2: Fix return value with oldfd == newfd and invalid fd

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

 The return value of dup2 when oldfd == newfd and the fd isn't valid is not
 getting properly sign extended. We end up with 4294967287 instead of -EBADF.

 I've reproduced this on SLE11 (2.6.27.21), openSUSE Factory (2.6.29-rc5),
 and Ubuntu 9.04 (2.6.28).

 This patch uses a signed int for the error value so it is properly extended.

 Commit 6c5d0512a091480c9f981162227fdb1c9d70e555 introduced this regression.

Reported-by: Jiri Dluhos <jdluhos@novell.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
- ---
 fs/fcntl.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

- --- a/fs/fcntl.c
+++ b/fs/fcntl.c
@@ -115,13 +115,14 @@ out_unlock:
 
 SYSCALL_DEFINE2(dup2, unsigned int, oldfd, unsigned int, newfd)
 {
+	int ret = oldfd;
 	if (unlikely(newfd == oldfd)) { /* corner case */
 		struct files_struct *files = current->files;
 		rcu_read_lock();
 		if (!fcheck_files(files, oldfd))
- -			oldfd = -EBADF;
+			ret = -EBADF;
 		rcu_read_unlock();
- -		return oldfd;
+		return ret;
 	}
 	return sys_dup3(oldfd, newfd, 0);
 }

- -- 
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkoIbZcACgkQLPWxlyuTD7JBVACgnNFiWRb4lhW9JgqR36BnT6SD
4uQAoJDcfqV2jsjCV340HlQLkk585Yw6
=IBqW
-----END PGP SIGNATURE-----

Re: [PATCH] dup2: Fix return value with oldfd == newfd and invalid fd

[Linus Torvalds]

On Mon, 11 May 2009, Jeff Mahoney wrote:
> 
>  The return value of dup2 when oldfd == newfd and the fd isn't valid is not
>  getting properly sign extended. We end up with 4294967287 instead of -EBADF.

Indeed.

>  This patch uses a signed int for the error value so it is properly extended.

However, I'd rather move the new variable into the block where it is used, 
and keep the whole corner-case thing self-contained.

So can you verify that this trivial variation on the patch is ok by you, 
and I'll commit it as yours with your message? (I realize it's really 
trivial, and I could just do this myself, but good to get the change 
ack'ed anyway).

		Linus

---
 fs/fcntl.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/fcntl.c b/fs/fcntl.c
index cc8e4de..1ad7031 100644
--- a/fs/fcntl.c
+++ b/fs/fcntl.c
@@ -117,11 +117,13 @@ SYSCALL_DEFINE2(dup2, unsigned int, oldfd, unsigned int, newfd)
 {
 	if (unlikely(newfd == oldfd)) { /* corner case */
 		struct files_struct *files = current->files;
+		int retval = oldfd;
+
 		rcu_read_lock();
 		if (!fcheck_files(files, oldfd))
-			oldfd = -EBADF;
+			retval = -EBADF;
 		rcu_read_unlock();
-		return oldfd;
+		return retval;
 	}
 	return sys_dup3(oldfd, newfd, 0);
 }

Re: [PATCH] dup2: Fix return value with oldfd == newfd and invalid fd

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Linus Torvalds wrote:
> 
> On Mon, 11 May 2009, Jeff Mahoney wrote:
>>  The return value of dup2 when oldfd == newfd and the fd isn't valid is not
>>  getting properly sign extended. We end up with 4294967287 instead of -EBADF.
> 
> Indeed.
> 
>>  This patch uses a signed int for the error value so it is properly extended.
> 
> However, I'd rather move the new variable into the block where it is used, 
> and keep the whole corner-case thing self-contained.
> 
> So can you verify that this trivial variation on the patch is ok by you, 
> and I'll commit it as yours with your message? (I realize it's really 
> trivial, and I could just do this myself, but good to get the change 
> ack'ed anyway).

Oh, of course. That's fine with me.

- -Jeff

> 
> 		Linus
> 
> ---
>  fs/fcntl.c |    6 ++++--
>  1 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/fcntl.c b/fs/fcntl.c
> index cc8e4de..1ad7031 100644
> --- a/fs/fcntl.c
> +++ b/fs/fcntl.c
> @@ -117,11 +117,13 @@ SYSCALL_DEFINE2(dup2, unsigned int, oldfd, unsigned int, newfd)
>  {
>  	if (unlikely(newfd == oldfd)) { /* corner case */
>  		struct files_struct *files = current->files;
> +		int retval = oldfd;
> +
>  		rcu_read_lock();
>  		if (!fcheck_files(files, oldfd))
> -			oldfd = -EBADF;
> +			retval = -EBADF;
>  		rcu_read_unlock();
> -		return oldfd;
> +		return retval;
>  	}
>  	return sys_dup3(oldfd, newfd, 0);
>  }


- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkoIdeUACgkQLPWxlyuTD7J8AACgmhqZUznCzrN25qBX6CzexXBu
NkcAn0qOA99rP6c72k+xs7ZRCbhLKKnR
=/rzV
-----END PGP SIGNATURE-----

Re: [PATCH] dup2: Fix return value with oldfd == newfd and invalid fd

[Al Viro]

On Mon, May 11, 2009 at 11:40:56AM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 11 May 2009, Jeff Mahoney wrote:
> > 
> >  The return value of dup2 when oldfd == newfd and the fd isn't valid is not
> >  getting properly sign extended. We end up with 4294967287 instead of -EBADF.
> 
> Indeed.
> 
> >  This patch uses a signed int for the error value so it is properly extended.
> 
> However, I'd rather move the new variable into the block where it is used, 
> and keep the whole corner-case thing self-contained.
> 
> So can you verify that this trivial variation on the patch is ok by you, 
> and I'll commit it as yours with your message? (I realize it's really 
> trivial, and I could just do this myself, but good to get the change 
> ack'ed anyway).

I'm not sure that it's a right fix, actually.  Note that userland declaration
of that sucker is int dup2(int, int); so should we really take unsigned int
as arguments?

Re: [PATCH] dup2: Fix return value with oldfd == newfd and invalid fd

[Linus Torvalds]

On Mon, 11 May 2009, Al Viro wrote:
> 
> I'm not sure that it's a right fix, actually.  Note that userland declaration
> of that sucker is int dup2(int, int); so should we really take unsigned int
> as arguments?

Hmm. They've been "unsigned int" for as long as our history goes back 
(including BK), but yes, making them "int" would have hidden this issue as 
well. 

That said, I think we had reasons to do our fd's as unsigned, ie the whole 
"compare against MAX" thing that doesn't take negative values into 
account.

In fact, I think we should do more of those. Right now we literally depend 
on things like "max_fds" being "unsigned int", and that the compiler then 
turns all the

	if (fd < fdt->max_fds)

tests silently into unsigned tests even when 'fd' is 'int'.

So I suspect we should probably make fs/file.c use _more_ "unsigned int" 
rather than having less of them.

			Linus

Re: [PATCH] dup2: Fix return value with oldfd == newfd and invalid fd

[Al Viro]

On Mon, May 11, 2009 at 12:26:59PM -0700, Linus Torvalds wrote:

> Hmm. They've been "unsigned int" for as long as our history goes back 
> (including BK), but yes, making them "int" would have hidden this issue as 
> well. 
> 
> That said, I think we had reasons to do our fd's as unsigned, ie the whole 
> "compare against MAX" thing that doesn't take negative values into 
> account.
> 
> In fact, I think we should do more of those. Right now we literally depend 
> on things like "max_fds" being "unsigned int", and that the compiler then 
> turns all the
> 
> 	if (fd < fdt->max_fds)
> 
> tests silently into unsigned tests even when 'fd' is 'int'.
> 
> So I suspect we should probably make fs/file.c use _more_ "unsigned int" 
> rather than having less of them.

What we should do is a careful review of the propagation paths of file
descriptors ;-/  As it is, we have an interesting mix of int/unsigned/long
used to carry those around, and quite a few of those are used for -E...
as well.  Note, BTW, that for userland code this bug mostly isn't - libc will
convert that value to int before returning to caller, so sign expansion or
not, we won't notice.  The things like strace will, though...