Re: [PATCH v3 09/14] vfs: add O_CREAT|O_DIRECTORY to open*(2)

[Pedro Falcato <pfalcato@suse.de>]

On Tue, Jul 07, 2026 at 12:06:44PM +0200, Jori Koolstra wrote:
> Hi Pedro,
> 
> > Op 07-07-2026 10:35 CEST schreef Pedro Falcato <pfalcato@suse.de>:
> > 
> > Ok so I genuinely don't see how users are supposed to use O_CREAT | O_DIRECTORY.
> > Lets think about the portability here. This is undefined by POSIX (meaning you
> > cannot rely on a O_CREAT | O_DIRECTORY having particular semantics). FreeBSD
> > and NetBSD each have different semantics (my investigation stopped there
> > ~3 years ago; I vaguely remember that macOS/Darwin does not/did not support
> > O_DIRECTORY. I don't remember what's the deal with OpenBSD/Solaris/Illumos
> > here).
> > 
> > So, essentially you can't rely on _any_ common behavior between platforms.
> > Ok. Now, you may say "it's 2026 lol who cares about BSD". So lets look at Linux:
> > - Linux for most of its lifetime had "return open directory, or create
> >   a regular file".
> > - It later had the above, with the added caveat that it always returned
> >   ENOTDIR.
> > - In 2023 I then found the bug and Christian changed it to always return
> >   EINVAL on that combination.
> > 
> > Ok. So Linux had 3 behaviors and 2 of them do not make any kind of sense. But
> > that's fine, we return EINVAL now. Except that the other behaviors existed.
> > 
> > So, lets say this feature lands as-is. If you're writing something *for Linux*,
> > you have to contend with the previous *3 behaviors* for a grand total of 4
> > behaviors. Including bizarre runtime tests, lest you are running on something
> > pre-6.4 (system calls can be similarly problematic, but at least there its a
> > simple ENOSYS test).
> > 
> 
> But isn't this a problem with all UAPI design? You introduce a new feature that
> does not exist on old kernels, then userspace can use that feature but must
> provide fallback behaviors for any older kernel that the userspace program wants
> to support. Ideally, this is abstracted away by some library. Then, at some point
> this library raises its minimal supported kernel version bar, and it can clean-up
> the now obsolete fallback code.

Correct, but usually most new UAPIs have very simple checks:

if (syscall(SYS_ponies) < 0 && errno == ENOSYS) {
	fallback();
}

or

if (ponies(NEW_FLAG) < 0 && errno == EINVAL) {
	fallback();
}

(in fact, many system calls out there can even be silently emulated by the
libc if need be; this one can't, but that's besides the point)

My problem isn't with adding a new uapi, it's that the flag combination has
so horrifically overloaded with weird semantics that it's pretty non-trivial
to find out conclusively whether or not you have the feature, across the recent
lifetime of Linux. I'm not even concerned with repurposing the flags because the
old semantics were so obviously BS.

> 
> I am fairly positive for instance that libpathrs would like to use O_CREAT | O_DIRECTORY,
> but it is not going to demand kernel > 7.x anytime soon.

so it will handroll its own tests for the feature which may or may not be
wrong.

> 
> 
> > If you're writing something that wants to be portable, or that intends to
> > standardize on something, you have some 5 different behaviors all across
> > the FOSS UNIX landscape, not considering everything else. It's also something
> > that will, FWIW, probably never be included in POSIX because no one can agree
> > on the semantics here.
> > 
> 
> I have no idea how POSIX standardizes things. Does a committee get together and
> revisit old syscalls every now and then?

In theory: System X thinks up a useful feature, systems Y, Z think it's
useful and also implement it. Once there is some sort of consensus, it's
added to POSIX. Remaining systems implement it.

(this is how stuff like O_DIRECTORY and OFD locks landed from Linux to POSIX to
every other UNIX-like)

> 
> But yeah, I think POSIX compliance isn't something we are really doing anymore.

The vast majority of the kernel is compliant, and a great portion of software
uses mostly/only portable interfaces.

> Afaik there's nothing in POSIX about openat2(2) either, and we definitely wish
> people would move to that instead of keep relying on open(2) (or even better,
> use libpathrs!)

Hah, good luck! :)

> 
> > TL;DR I don't see, given the reasons above, how users are supposed to use
> > this without it being a total minefield.
> > 
> > If you're intent on the open() path, I would suggest that perhaps openat2()
> > would be a nice alternative, although definitely not as beautiful as
> > open(O_CREAT | O_DIRECTORY).
> > 
> 
> Christian told me (somewhere on this list) that at LSFMMBPF all vfs maintainers
> agreed on the O_CREAT | O_DIRECTORY direction. It allows to use all the neat
> options from openat2(2) at the cost of a bit of multiplexing in lookup_open().
> The question remains whether we also want the other things like mknodat_fd().


FWIW, thinking about this a bit more, it's possible that openat2() could serve
as a stricter check for the UAPI in open() in general. Making this less
footgunny to use.

> 
> > Just my 2c. And I feel horrible typing this out because you clearly spent
> > a great amount of time and effort between v2 and v3 as you redid this :/
> > 
> 
> It's been a lot of work, yes :)
> 
> But luckily it has allowed me to study path resolution and the open()/mkdir()/etc.
> syscalls in a fun and educative way.

Yay!

-- 
Pedro