[PATCH] devpts: fix pty count limit off by one

[PATCH] devpts: fix pty count limit off by one

[Yichong Chen]

devpts_new_index() increments the global pty count before checking it
against the effective global limit. The check currently rejects a new pty
when the incremented count is equal to the limit.

This makes kernel.pty.max allow only max - 1 ptys. For example, if
kernel.pty.nr is 3 and kernel.pty.max is set to 4, opening /dev/ptmx
fails with -ENOSPC even though one more pty should be allowed.

Allow the incremented count to be equal to the effective limit and reject
only counts above it.

Fixes: 0f0a0e54a2a1 ("devpts: Convert to new IDA API")
Signed-off-by: Yichong Chen <chenyichong@uniontech.com>
---
 fs/devpts/inode.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/devpts/inode.c b/fs/devpts/inode.c
index 9844dcf354ee..853cc4c03e73 100644
--- a/fs/devpts/inode.c
+++ b/fs/devpts/inode.c
@@ -467,7 +467,7 @@ int devpts_new_index(struct pts_fs_info *fsi)
 {
 	int index = -ENOSPC;
 
-	if (atomic_inc_return(&pty_count) >= (pty_limit -
+	if (atomic_inc_return(&pty_count) > (pty_limit -
 			  (fsi->mount_opts.reserve ? 0 : pty_reserve)))
 		goto out;
 
-- 
2.51.0

Re: [PATCH] devpts: fix pty count limit off by one

[Greg KH]

On Tue, Aug 04, 2026 at 04:23:21PM +0800, Yichong Chen wrote:
> devpts_new_index() increments the global pty count before checking it
> against the effective global limit. The check currently rejects a new pty
> when the incremented count is equal to the limit.
> 
> This makes kernel.pty.max allow only max - 1 ptys. For example, if
> kernel.pty.nr is 3 and kernel.pty.max is set to 4, opening /dev/ptmx
> fails with -ENOSPC even though one more pty should be allowed.
> 
> Allow the incremented count to be equal to the effective limit and reject
> only counts above it.
> 
> Fixes: 0f0a0e54a2a1 ("devpts: Convert to new IDA API")

As this is a user-visible change, what is now going to break because of
this?  It's been this way for a very long time now.

thanks,

greg k-h

Re: [PATCH] devpts: fix pty count limit off by one

[Yichong Chen]

Hi Greg,

Yes, this is user-visible.

The reason I sent the patch is that this used to allow pty_count to reach
the configured limit.  Before commit 0f0a0e54a2a1 ("devpts: Convert to new
IDA API"), devpts_new_index() checked the old pty_count first and only
incremented it after the allocation succeeded:

  if (pty_count >= limit)
          return -ENOSPC;
  ...
  pty_count++;

After that commit, the code started using atomic_inc_return(), but kept the
>= comparison:

  if (atomic_inc_return(&pty_count) >= limit)
          goto out;

So a count equal to the configured limit became rejected.  That makes
kernel.pty.max behave as an effective max - 1 limit.

I thought allowing the count to reach kernel.pty.max again would better
match the usual interpretation of a "max" sysctl.  But I agree this also
changes visible behavior again, and the current max - 1 behavior has been
around since 2018.

So if you prefer to preserve the existing behavior for compatibility, I am
fine with dropping the patch.

Thanks,
Yichong

Re: [PATCH] devpts: fix pty count limit off by one

[Greg KH]

On Tue, Aug 04, 2026 at 05:52:52PM +0800, Yichong Chen wrote:
> Hi Greg,
> 
> Yes, this is user-visible.

Sorry, always include the proper context, and respond after email text.
Remember, some of us get 1000+ emails a day to deal with.

For specifics:

A: http://en.wikipedia.org/wiki/Top_post
Q: Were do I find info about this thing called top-posting?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I include quotations after my reply?

http://daringfireball.net/2007/07/on_top

> The reason I sent the patch is that this used to allow pty_count to reach
> the configured limit.  Before commit 0f0a0e54a2a1 ("devpts: Convert to new
> IDA API"), devpts_new_index() checked the old pty_count first and only
> incremented it after the allocation succeeded:
> 
>   if (pty_count >= limit)
>           return -ENOSPC;
>   ...
>   pty_count++;
> 
> After that commit, the code started using atomic_inc_return(), but kept the
> >= comparison:
> 
>   if (atomic_inc_return(&pty_count) >= limit)
>           goto out;
> 
> So a count equal to the configured limit became rejected.  That makes
> kernel.pty.max behave as an effective max - 1 limit.
> 
> I thought allowing the count to reach kernel.pty.max again would better
> match the usual interpretation of a "max" sysctl.  But I agree this also
> changes visible behavior again, and the current max - 1 behavior has been
> around since 2018.
> 
> So if you prefer to preserve the existing behavior for compatibility, I am
> fine with dropping the patch.

What is currently breaking that was working just fine with the way the
code is today?  If nothing, then we should probably leave it alone,
right?

thanks,

greg k-h