[Buildroot] bash: /dev/fd/62: No such file or directory

[Buildroot] bash: /dev/fd/62: No such file or directory

[Jörg Krause]

Hi,

I am facing an issue with busybox init system, bash and mdev. When a
bash script uses process substitution, bash ends with an error:

   """
   # bash -c 'read x< <(echo 1); echo $x'
   bash: /dev/fd/62: No such file or directory
   """

The issue is addressed is some threads, e.g. [1]. The problem is that
/dev/fd does not exist when using mdev:

   """
   # ls -l /dev/fd
   ls: /dev/fd: No such file or directory
   """

As addressed here [2], the issue can be fixed by adding:

   """
   ln -snf /proc/self/fd /dev/fd
   """

... to /etc/inittab or /etc/init.d/S10mdev:

   """
   # ls -l /dev/fd
   lrwxrwxrwx    1 root     root            13 Mar 15 22:39 /dev/fd -> /proc/self/fd
   # ls -l /dev/fd/
   total 0
   lrwx------    1 root     root            64 Mar 15 22:40 0 -> /dev/pts/0
   lrwx------    1 root     root            64 Mar 15 22:40 1 -> /dev/pts/0
   lrwx------    1 root     root            64 Mar 15 22:40 2 -> /dev/pts/0
   ls: /dev/fd/3: cannot read link: No such file or directory
   lr-x------    1 root     root            64 Mar 15 22:40 3
   """

The process substitution works now as expected:

   """
   # bash -c 'read x< <(echo 1); echo $x'
   1
   """

I found some other threads which additionally add the following lines:

   """
   ln -snf /proc/self/fd/0 /dev/stdin
   ln -snf /proc/self/fd/1 /dev/stdout
   ln -snf /proc/self/fd/2 /dev/stderr
   """

... as /dev/std{in,out,err} do not exist when using mdev

My question is how to best fix this issue in Buildroot? Any
suggestions?

[1] http://gnu-bash.2382.n7.nabble.com/dev-fd-62-No-such-file-or-directory-td13233.html
[2] https://bugs.alpinelinux.org/issues/1465

Best regards
J?rg Krause

[Buildroot] bash: /dev/fd/62: No such file or directory

[Peter Korsgaard]

>>>>> "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:

 > Hi,
 > I am facing an issue with busybox init system, bash and mdev. When a
 > bash script uses process substitution, bash ends with an error:

 >    """
 >    # bash -c 'read x< <(echo 1); echo $x'
 >    bash: /dev/fd/62: No such file or directory
 >    """

 > The issue is addressed is some threads, e.g. [1]. The problem is that
 > /dev/fd does not exist when using mdev:

So that is really strictly seen a bug in bash. The /dev/fd convenience
symlink is afaik a convention from udev/systemd.

But OK, bash might not be alone and the pragmatic solution would indeed
be to create these symlinks ourselves in the busybox and sysvinit
inittabs, similar to how we create /dev/pts and /dev/shm.

It is only really needed if (e)udev isn't used, but it doesn't really
hurt to create them even if eudev will recreate them afterwards.

Keep in mind that we still support a static /dev, so we should also add
them to system/device_table_dev.txt and redirect errors to /dev/null so
you don't get a bunch of warnings about these symlinks when booting with
a static /dev.

Notice that it isn't only about /dev/fd. From
eudev-3.2.5/src/shared/dev-setup.c:

        static const char symlinks[] =
                "-/proc/kcore\0"     "/dev/core\0"
                "/proc/self/fd\0"    "/dev/fd\0"
                "/proc/self/fd/0\0"  "/dev/stdin\0"
                "/proc/self/fd/1\0"  "/dev/stdout\0"
                "/proc/self/fd/2\0"  "/dev/stderr\0";

There is some logic to only create /dev/core if /proc/kcore exists, but
I think we can just create it unconditionally.

Care to send patches doing this?

-- 
Bye, Peter Korsgaard

[Buildroot] bash: /dev/fd/62: No such file or directory

[Jörg Krause]

Hi Peter,

On Fri, 2018-03-16 at 15:19 +0100, Peter Korsgaard wrote:
> > > > > > "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:
> 
>  > Hi,
>  > I am facing an issue with busybox init system, bash and mdev. When
> a
>  > bash script uses process substitution, bash ends with an error:
> 
>  >    """
>  >    # bash -c 'read x< <(echo 1); echo $x'
>  >    bash: /dev/fd/62: No such file or directory
>  >    """
> 
>  > The issue is addressed is some threads, e.g. [1]. The problem is
> that
>  > /dev/fd does not exist when using mdev:
> 
> So that is really strictly seen a bug in bash. The /dev/fd
> convenience
> symlink is afaik a convention from udev/systemd.
> 
> But OK, bash might not be alone and the pragmatic solution would
> indeed
> be to create these symlinks ourselves in the busybox and sysvinit
> inittabs, similar to how we create /dev/pts and /dev/shm.
> 
> It is only really needed if (e)udev isn't used, but it doesn't really
> hurt to create them even if eudev will recreate them afterwards.
> 
> Keep in mind that we still support a static /dev, so we should also
> add
> them to system/device_table_dev.txt and redirect errors to /dev/null
> so
> you don't get a bunch of warnings about these symlinks when booting
> with
> a static /dev.
> 
> Notice that it isn't only about /dev/fd. From
> eudev-3.2.5/src/shared/dev-setup.c:
> 
>         static const char symlinks[] =
>                 "-/proc/kcore\0"     "/dev/core\0"
>                 "/proc/self/fd\0"    "/dev/fd\0"
>                 "/proc/self/fd/0\0"  "/dev/stdin\0"
>                 "/proc/self/fd/1\0"  "/dev/stdout\0"
>                 "/proc/self/fd/2\0"  "/dev/stderr\0";
> 
> There is some logic to only create /dev/core if /proc/kcore exists,
> but
> I think we can just create it unconditionally.

Thanks for looking into this!

> Care to send patches doing this?

Yes, I can do that.

J?rg

[Buildroot] bash: /dev/fd/62: No such file or directory

[Jörg Krause]

Hi Peter,

On Fri, 2018-03-16 at 15:19 +0100, Peter Korsgaard wrote:
> > > > > > "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:
> 
>  > Hi,
>  > I am facing an issue with busybox init system, bash and mdev. When a
>  > bash script uses process substitution, bash ends with an error:
> 
>  >    """
>  >    # bash -c 'read x< <(echo 1); echo $x'
>  >    bash: /dev/fd/62: No such file or directory
>  >    """
> 
>  > The issue is addressed is some threads, e.g. [1]. The problem is that
>  > /dev/fd does not exist when using mdev:
> 
> So that is really strictly seen a bug in bash. The /dev/fd convenience
> symlink is afaik a convention from udev/systemd.
> 
> But OK, bash might not be alone and the pragmatic solution would indeed
> be to create these symlinks ourselves in the busybox and sysvinit
> inittabs, similar to how we create /dev/pts and /dev/shm.
> 
> It is only really needed if (e)udev isn't used, but it doesn't really
> hurt to create them even if eudev will recreate them afterwards.
> 
> Keep in mind that we still support a static /dev, so we should also add
> them to system/device_table_dev.txt and redirect errors to /dev/null so
> you don't get a bunch of warnings about these symlinks when booting with
> a static /dev.

I am trying to get this working. I managed to add the entries to
inittab for busybox and sysvinit, but I am unsure how to add those
symlinks to the static /dev. makedevs does not support symbolic links.

> Notice that it isn't only about /dev/fd. From
> eudev-3.2.5/src/shared/dev-setup.c:
> 
>         static const char symlinks[] =
>                 "-/proc/kcore\0"     "/dev/core\0"
>                 "/proc/self/fd\0"    "/dev/fd\0"
>                 "/proc/self/fd/0\0"  "/dev/stdin\0"
>                 "/proc/self/fd/1\0"  "/dev/stdout\0"
>                 "/proc/self/fd/2\0"  "/dev/stderr\0";
> 
> There is some logic to only create /dev/core if /proc/kcore exists, but
> I think we can just create it unconditionally.
> 
> Care to send patches doing this?

Best regards
J?rg Krause

[Buildroot] bash: /dev/fd/62: No such file or directory

[Peter Korsgaard]

>>>>> "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:

Hi,

 >> Keep in mind that we still support a static /dev, so we should also add
 >> them to system/device_table_dev.txt and redirect errors to /dev/null so
 >> you don't get a bunch of warnings about these symlinks when booting with
 >> a static /dev.

 > I am trying to get this working. I managed to add the entries to
 > inittab for busybox and sysvinit

Great!

 > but I am unsure how to add those symlinks to the static
 > /dev. makedevs does not support symbolic links.

Heh, you are right. We can either:

- Add a BASH_ROOTFS_PRE_CMD_HOOKS hook in bash.mk if static /dev is used
  to create these symlinks

- Extend the makedevs code to also support symlinks

- Completely ignore this issue for static /dev

The first option is probably the easiest/nicest.

-- 
Bye, Peter Korsgaard

[Buildroot] bash: /dev/fd/62: No such file or directory

[Jörg Krause]

On Fri, 2018-03-30 at 13:11 +0200, Peter Korsgaard wrote:
> > > > > > "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:
> 
> Hi,
> 
>  >> Keep in mind that we still support a static /dev, so we should also add
>  >> them to system/device_table_dev.txt and redirect errors to /dev/null so
>  >> you don't get a bunch of warnings about these symlinks when booting with
>  >> a static /dev.
> 
>  > I am trying to get this working. I managed to add the entries to
>  > inittab for busybox and sysvinit
> 
> Great!

Thanks! I'm adding these symlinks regardless if bash is selected or
not. Agreed?

>  > but I am unsure how to add those symlinks to the static
>  > /dev. makedevs does not support symbolic links.
> 
> Heh, you are right. We can either:
> 
> - Add a BASH_ROOTFS_PRE_CMD_HOOKS hook in bash.mk if static /dev is used
>   to create these symlinks
> 
> - Extend the makedevs code to also support symlinks
> 
> - Completely ignore this issue for static /dev
> 
> The first option is probably the easiest/nicest.
> 

Agreed! Let's take the first option :-)

J?rg

[Buildroot] bash: /dev/fd/62: No such file or directory

[Peter Korsgaard]

>>>>> "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:

Hi,

 >> > I am trying to get this working. I managed to add the entries to
 >> > inittab for busybox and sysvinit
 >> 
 >> Great!

 > Thanks! I'm adding these symlinks regardless if bash is selected or
 > not. Agreed?

Yes, I think that's fine.

 >> The first option is probably the easiest/nicest.

 > Agreed! Let's take the first option :-)

Great!

-- 
Bye, Peter Korsgaard

[Buildroot] bash: /dev/fd/62: No such file or directory

[Jörg Krause]

Hi,

On Fri, 2018-03-30 at 13:11 +0200, Peter Korsgaard wrote:
> > > > > > "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:
> 
> Hi,
> 
>  >> Keep in mind that we still support a static /dev, so we should also add
>  >> them to system/device_table_dev.txt and redirect errors to /dev/null so
>  >> you don't get a bunch of warnings about these symlinks when booting with
>  >> a static /dev.
> 
>  > I am trying to get this working. I managed to add the entries to
>  > inittab for busybox and sysvinit
> 
> Great!
> 
>  > but I am unsure how to add those symlinks to the static
>  > /dev. makedevs does not support symbolic links.
> 
> Heh, you are right. We can either:
> 
> - Add a BASH_ROOTFS_PRE_CMD_HOOKS hook in bash.mk if static /dev is used
>   to create these symlinks
> 
> - Extend the makedevs code to also support symlinks
> 
> - Completely ignore this issue for static /dev
> 
> The first option is probably the easiest/nicest.

On second thought, bash might not be the only application using
/dev/fd. Maybe fs/common.mk is the better place?

There is one issue about /proc/kcore. It is not available on ARM
systems, and maybe others. Apparently, this feature is not much
appreciated [1], so I'm gonna skip this symlink.

[1] https://lwn.net/Articles/45315/

J?rg

[Buildroot] bash: /dev/fd/62: No such file or directory

[Peter Korsgaard]

>>>>> "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:

Hi,

 >> The first option is probably the easiest/nicest.

 > On second thought, bash might not be the only application using
 > /dev/fd. Maybe fs/common.mk is the better place?

Possible, yes. I'm not aware of any other tools needing it though.

 > There is one issue about /proc/kcore. It is not available on ARM
 > systems, and maybe others. Apparently, this feature is not much
 > appreciated [1], so I'm gonna skip this symlink.

Yes, lets just skip it.

-- 
Bye, Peter Korsgaard

[Buildroot] bash: /dev/fd/62: No such file or directory

[Jörg Krause]

Hi Peter,

On Fri, 2018-03-30 at 13:11 +0200, Peter Korsgaard wrote:
> > > > > > "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:
> 
> Hi,
> 
>  >> Keep in mind that we still support a static /dev, so we should also add
>  >> them to system/device_table_dev.txt and redirect errors to /dev/null so
>  >> you don't get a bunch of warnings about these symlinks when booting with
>  >> a static /dev.
> 
>  > I am trying to get this working. I managed to add the entries to
>  > inittab for busybox and sysvinit
> 
> Great!

I just proposed the patches...

>  > but I am unsure how to add those symlinks to the static
>  > /dev. makedevs does not support symbolic links.
> 
> Heh, you are right. We can either:
> 
> - Add a BASH_ROOTFS_PRE_CMD_HOOKS hook in bash.mk if static /dev is used
>   to create these symlinks
> 
> - Extend the makedevs code to also support symlinks
> 
> - Completely ignore this issue for static /dev
> 
> The first option is probably the easiest/nicest.

It turned out that nothing needs to be done for static /dev as the
symlinks are created now by the init system.

Best regards
J?rg Krause

[Buildroot] bash: /dev/fd/62: No such file or directory

[Peter Korsgaard]

>>>>> "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:

Hi,

 >> > but I am unsure how to add those symlinks to the static
 >> > /dev. makedevs does not support symbolic links.
 >> 
 >> Heh, you are right. We can either:
 >> 
 >> - Add a BASH_ROOTFS_PRE_CMD_HOOKS hook in bash.mk if static /dev is used
 >> to create these symlinks
 >> 
 >> - Extend the makedevs code to also support symlinks
 >> 
 >> - Completely ignore this issue for static /dev
 >> 
 >> The first option is probably the easiest/nicest.

 > It turned out that nothing needs to be done for static /dev as the
 > symlinks are created now by the init system.

But how can that work? static /dev normally means readonly /dev.

-- 
Bye, Peter Korsgaard

[Buildroot] bash: /dev/fd/62: No such file or directory

[Jörg Krause]

Hi,

On Wed, 2018-04-04 at 20:58 +0200, Peter Korsgaard wrote:
> > > > > > "J?rg" == J?rg Krause <joerg.krause@embedded.rocks> writes:
> 
> Hi,
> 
>  >> > but I am unsure how to add those symlinks to the static
>  >> > /dev. makedevs does not support symbolic links.
>  >> 
>  >> Heh, you are right. We can either:
>  >> 
>  >> - Add a BASH_ROOTFS_PRE_CMD_HOOKS hook in bash.mk if static /dev is used
>  >> to create these symlinks
>  >> 
>  >> - Extend the makedevs code to also support symlinks
>  >> 
>  >> - Completely ignore this issue for static /dev
>  >> 
>  >> The first option is probably the easiest/nicest.
> 
>  > It turned out that nothing needs to be done for static /dev as the
>  > symlinks are created now by the init system.
> 
> But how can that work? static /dev normally means readonly /dev.

I've tested this with the following config fragment:

   BR2_ROOTFS_DEVICE_CREATION_STATIC=y
   BR2_SYSTEM_BIN_SH_BASH=y
   BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y

Running on the target:

   # ls -l /dev/fd*
   lrwxrwxrwx    1 root     root            13 Jan  1 00:00 /dev/fd -> /proc/self/fd
   # ls -l /dev/fd*/**
   ls: /dev/fd/255: No such file or directory
   ls: /dev/fd/3: No such file or directory
   lrwx------    1 root     root            64 Jan  1 00:01 /dev/fd/0 -> /dev/ttyS0
   lrwx------    1 root     root            64 Jan  1 00:01 /dev/fd/1 -> /dev/ttyS0
   lrwx------    1 root     root            64 Jan  1 00:01 /dev/fd/2 -> /dev/ttyS0
   # bash -c 'read x< <(echo 1); echo $x'
   1

Not sure it is really readonly. I thought that it means no dynamic device nodes are created...

J?rg

[Buildroot] bash: /dev/fd/62: No such file or directory

[Peter Korsgaard]

>>>>> "Peter" == Peter Korsgaard <peter@korsgaard.com> writes:

Hi,

 >> It turned out that nothing needs to be done for static /dev as the
 >> symlinks are created now by the init system.

 > But how can that work? static /dev normally means readonly /dev.

That didn't come out very clear. What I wanted to say is that static
/dev should also work on a readonly fs, so these symlinks should be made
at build time, not at runtime.

-- 
Bye, Peter Korsgaard

[Buildroot] bash: /dev/fd/62: No such file or directory

[Jörg Krause]

On Wed, 2018-04-04 at 22:23 +0200, Peter Korsgaard wrote:
> > > > > > "Peter" == Peter Korsgaard <peter@korsgaard.com> writes:
> 
> Hi,
> 
>  >> It turned out that nothing needs to be done for static /dev as the
>  >> symlinks are created now by the init system.
> 
>  > But how can that work? static /dev normally means readonly /dev.
> 
> That didn't come out very clear. What I wanted to say is that static
> /dev should also work on a readonly fs, so these symlinks should be made
> at build time, not at runtime.

I see!

I've rebuild the system with BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW=n.

Snippet of the inittab file:

   # Startup the system
   ::sysinit:/bin/mount -t proc proc /proc
   #::sysinit:/bin/mount -o remount,rw /
   ::sysinit:/bin/mkdir -p /dev/pts
   ::sysinit:/bin/mkdir -p /dev/shm
   ::sysinit:/bin/mount -a
   ::sysinit:/bin/ln -sf /proc/self/fd /dev/fd
   ::sysinit:/bin/ln -sf /proc/self/fd/0 /dev/stdin
   ::sysinit:/bin/ln -sf /proc/self/fd/1 /dev/stdout
   ::sysinit:/bin/ln -sf /proc/self/fd/2 /dev/stderr
   ::sysinit:/bin/hostname -F /etc/hostname

On the system /dev is still writable:

   # echo 1 > /dev/1.txt
   # echo 1 > /root/1.txt
   -sh: /root/1.txt: Read-only file system

   # ls -l /dev/fd/*
   ls: /dev/fd/255: No such file or directory
   ls: /dev/fd/3: No such file or directory
   lrwx------    1 root     root            64 Jan  1 00:09 /dev/fd/0 -> /dev/ttyS0
   lrwx------    1 root     root            64 Jan  1 00:09 /dev/fd/1 -> /dev/ttyS0
   lrwx------    1 root     root            64 Jan  1 00:09 /dev/fd/2 -> /dev/ttyS0

   # bash -c "cat <(date)"
   Thu Jan  1 00:10:07 UTC 1970

J?rg