[Buildroot] Where is /dev/console created when using devtmpfs?

[Buildroot] Where is /dev/console created when using devtmpfs?

[Steve Kenton]

I'm currently running with devtmpfs only but am looking into using mdev
so I spent some time looking at my device usage and for the life of me
I can't find where /dev/console gets created during the build process.
Looking at the root filesystem when it's not running shows this in /dev

drwxrwxr-x 2 root root 4096 Mar  1 15:26 pts
lrwxrwxrwx 1 root root   10 Mar  1 15:26 log -> ../tmp/log
crw--w--w- 1 root root 5, 1 Mar 22 21:52 console

and I see that pts and log come from the skeleton but I can't find console.
I'm running an x86_64 system using the default vga console if that matters.
I'm obviously missing something but I can't find a mknod or mkdevs for it.

Where is /dev/console created when using devtmpfs?

Thanks,

Steve

[Buildroot] Where is /dev/console created when using devtmpfs?

[Rob Landley]

On Wed, Mar 25, 2015 at 12:44 PM, Steve Kenton <skenton@ou.edu> wrote:
> I'm currently running with devtmpfs only but am looking into using mdev

As the original author of mdev and the guy who did:

http://lkml.iu.edu/hypermail/linux/kernel/1306.3/04204.html

I should probably poke my nose in here. :)

> so I spent some time looking at my device usage and for the life of me
> I can't find where /dev/console gets created during the build process.

The console PID 1 inherits at process launch is the /dev/console in
initramfs. When you don't have initramfs, the init/noinitramfs.c code
literally does:

        err = sys_mkdir((const char __user __force *) "/dev", 0755);
        if (err < 0)
                goto out;

        err = sys_mknod((const char __user __force *) "/dev/console",
                        S_IFCHR | S_IRUSR | S_IWUSR,
                        new_encode_dev(MKDEV(5, 1)));

And then init/main.c does:

        /* Open the /dev/console on the rootfs, this should never fail */
        if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0)
                printk(KERN_WARNING "Warning: unable to open an
initial console.\n");

        (void) sys_dup(0);
        (void) sys_dup(0);

The devtmpfs mount happens later in the boot, and in fact I have a
pending todo item to submit a patch to make the kernel DEVTMPFS_MOUNT
config option actually _apply_ to initramfs. (Last I checked it would
only mount devtmpfs on /dev in the fallback root that root= switches
you to when you didn't have an executable /init in initramfs. I should
make it so that if it's about to exec /init out of rootfs, it does the
above mknod /dev trick (ignore failure if it's already there) and then
mounts devtmpfs on it (and if _that_ fails some crazy person probably
made a file called /dev and they get to keep the pieces).

> Looking at the root filesystem when it's not running shows this in /dev
>
> drwxrwxr-x 2 root root 4096 Mar  1 15:26 pts
> lrwxrwxrwx 1 root root   10 Mar  1 15:26 log -> ../tmp/log
> crw--w--w- 1 root root 5, 1 Mar 22 21:52 console
>
> and I see that pts and log come from the skeleton but I can't find console.

Possibly I'm answering the wrong question.

devtmpfs is a synthetic filesystem, like /proc or /sys. You're not
talking to any sort of backing store when you mount it, instead the
driver is hallucinating contents for you to read and write from on the
fly, and when you interact with it you talk to functions in the driver
that send/receive some data they just made up to humor your request.

I wrote a thing about the four different types of filesystems a while back:

http://landley.livejournal.com/52326.html

> I'm running an x86_64 system using the default vga console if that matters.
> I'm obviously missing something but I can't find a mknod or mkdevs for it.
>
> Where is /dev/console created when using devtmpfs?

The same place /proc/mounts is created on procfs?

The name "console" comes from drivers/tty/tty_io.c:

        consdev = device_create(tty_class, NULL, MKDEV(TTYAUX_MAJOR, 1), NULL,
                              "console");

That sticks an entry into some kernel linked list (or possibly tree),
which is interpreted by devtmpfs to give you a console device. It's
also interpreted by sysfs to give you a "console" directory under
/sys/class/tty. Both filesystem drivers are parsing the same kernel
internal data to produce different types of output. They do so on the
fly, when you open the relevant directory and list its contents.

> Thanks,
>
> Steve

Rob

[Buildroot] Where is /dev/console created when using devtmpfs?

[Steve Kenton]

On 03/26/2015 02:51 PM, Rob Landley wrote:
> On Wed, Mar 25, 2015 at 12:44 PM, Steve Kenton <skenton@ou.edu> wrote:
>> I'm currently running with devtmpfs only but am looking into using mdev
> 
> As the original author of mdev and the guy who did:
> 
> http://lkml.iu.edu/hypermail/linux/kernel/1306.3/04204.html
> 
> I should probably poke my nose in here. :)
Thank you very much for taking the time!

> 
>> so I spent some time looking at my device usage and for the life of me
>> I can't find where /dev/console gets created during the build process.
> 
> The console PID 1 inherits at process launch is the /dev/console in
> initramfs. When you don't have initramfs, the init/noinitramfs.c code
> literally does:
Yes, this is my configuration - initramfs not linked into kernel by buildroot
# BR2_TARGET_ROOTFS_INITRAMFS is not set

> 
>         err = sys_mkdir((const char __user __force *) "/dev", 0755);
>         if (err < 0)
>                 goto out;
> 
>         err = sys_mknod((const char __user __force *) "/dev/console",
>                         S_IFCHR | S_IRUSR | S_IWUSR,
>                         new_encode_dev(MKDEV(5, 1)));
> 
> And then init/main.c does:
> 
>         /* Open the /dev/console on the rootfs, this should never fail */
>         if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0)
>                 printk(KERN_WARNING "Warning: unable to open an
> initial console.\n");
If I understand correctly all of which occurs on rootfs in ram and does not
touch whatever disk based filesystem may eventually get mounted on /?
> 
>         (void) sys_dup(0);
>         (void) sys_dup(0);
> 
> The devtmpfs mount happens later in the boot, and in fact I have a
> pending todo item to submit a patch to make the kernel DEVTMPFS_MOUNT
> config option actually _apply_ to initramfs. (Last I checked it would
> only mount devtmpfs on /dev in the fallback root that root= switches
> you to when you didn't have an executable /init in initramfs. I should
Apparently still the case, at least buildroot 2015.02 supplied this as /init:

#!/bin/sh
# devtmpfs does not get automounted for initramfs
/bin/mount -t devtmpfs devtmpfs /dev
exec 0</dev/console
exec 1>/dev/console
exec 2>/dev/console
exec /sbin/init $*

> make it so that if it's about to exec /init out of rootfs, it does the
> above mknod /dev trick (ignore failure if it's already there) and then
> mounts devtmpfs on it (and if _that_ fails some crazy person probably
> made a file called /dev and they get to keep the pieces).
grin
> 
>> Looking at the root filesystem when it's not running shows this in /dev
>>
>> drwxrwxr-x 2 root root 4096 Mar  1 15:26 pts
>> lrwxrwxrwx 1 root root   10 Mar  1 15:26 log -> ../tmp/log
>> crw--w--w- 1 root root 5, 1 Mar 22 21:52 console
>>
>> and I see that pts and log come from the skeleton but I can't find console.
> 
> Possibly I'm answering the wrong question.
Since I'm trying to understand what's going on it's a very good answer
and I've read it multiple time while perusing the kernel code, but yes
I was originally wondering where /dev/console was created on the target
filesystem as part of the buildroot make process.

> 
> devtmpfs is a synthetic filesystem, like /proc or /sys. You're not
> talking to any sort of backing store when you mount it, instead the
> driver is hallucinating contents for you to read and write from on the
> fly, and when you interact with it you talk to functions in the driver
> that send/receive some data they just made up to humor your request.
Again if I under stand correctly the contents of the on-disk  /dev on the
root filesystem in my case (initramfs not linked into kernel) are unused
and normally inaccessible because of devtmpfs mount on top?

> 
> I wrote a thing about the four different types of filesystems a while back:
> 
> http://landley.livejournal.com/52326.html
Another very good read - thanks for that too

> 
>> I'm running an x86_64 system using the default vga console if that matters.
>> I'm obviously missing something but I can't find a mknod or mkdevs for it.
>>
>> Where is /dev/console created when using devtmpfs?
I was thinking in terms of the buildroot "use devtmpfs only" as opposed with mdev,
eudev, or systemd

> 
> The same place /proc/mounts is created on procfs?
> 
> The name "console" comes from drivers/tty/tty_io.c:
> 
>         consdev = device_create(tty_class, NULL, MKDEV(TTYAUX_MAJOR, 1), NULL,
>                               "console");
OK
> 
> That sticks an entry into some kernel linked list (or possibly tree),
> which is interpreted by devtmpfs to give you a console device. It's
> also interpreted by sysfs to give you a "console" directory under
> /sys/class/tty. Both filesystem drivers are parsing the same kernel
> internal data to produce different types of output. They do so on the
> fly, when you open the relevant directory and list its contents.
Thanks again for your time and detailed explaination.

Steve

> 
>> Thanks,
>>
>> Steve
> 
> Rob
> 

[Buildroot] Where is /dev/console created when using devtmpfs?

[Arnout Vandecappelle]

On 27/03/15 05:14, Steve Kenton wrote:
> On 03/26/2015 02:51 PM, Rob Landley wrote:
>> On Wed, Mar 25, 2015 at 12:44 PM, Steve Kenton <skenton@ou.edu> wrote:
[snip]
>>> so I spent some time looking at my device usage and for the life of me
>>> I can't find where /dev/console gets created during the build process.
>>
>> The console PID 1 inherits at process launch is the /dev/console in
>> initramfs. When you don't have initramfs, the init/noinitramfs.c code
>> literally does:
> Yes, this is my configuration - initramfs not linked into kernel by buildroot
> # BR2_TARGET_ROOTFS_INITRAMFS is not set

 Just to be clear "initramfs not linked into kernel" is BR2_TARGET_ROOTFS_CPIO,
"initramfs linked into the kernel" is BR2_TARGET_ROOTFS_INITRAMFS. But for the
purpose of the contents of the rootfs, that doesn't make a difference.

 But so in your case, Steve, I believe you _do_ have an initramfs, i.e. you have
selected BR2_TARGET_ROOTFS_CPIO. Otherwise /dev/console wouldn't have been
created :-).

> 
>>
>>         err = sys_mkdir((const char __user __force *) "/dev", 0755);
>>         if (err < 0)
>>                 goto out;
>>
>>         err = sys_mknod((const char __user __force *) "/dev/console",
>>                         S_IFCHR | S_IRUSR | S_IWUSR,
>>                         new_encode_dev(MKDEV(5, 1)));

 Note again that this stuff is only done when there is no initramfs. In other
words, when there is an initramfs, this initramfs must contain /dev/console or
you have no console.

 That's why buildroot adds it explicitly in fs/cpio/cpio.mk:

PACKAGES_PERMISSIONS_TABLE += /dev/console c 622 0 0 5 1 - - -$(sep)

>>
>> And then init/main.c does:
>>
>>         /* Open the /dev/console on the rootfs, this should never fail */
>>         if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0)
>>                 printk(KERN_WARNING "Warning: unable to open an
>> initial console.\n");
> If I understand correctly all of which occurs on rootfs in ram and does not
> touch whatever disk based filesystem may eventually get mounted on /?

 Indeed.

[snip]
>>
>> devtmpfs is a synthetic filesystem, like /proc or /sys. You're not
>> talking to any sort of backing store when you mount it, instead the
>> driver is hallucinating contents for you to read and write from on the
>> fly, and when you interact with it you talk to functions in the driver
>> that send/receive some data they just made up to humor your request.
> Again if I under stand correctly the contents of the on-disk  /dev on the
> root filesystem in my case (initramfs not linked into kernel) are unused
> and normally inaccessible because of devtmpfs mount on top?

 Indeed, its contents becomes inaccessible as soon as devtmpfs is mounted on
top. In case of initramfs, that is when /init is executed and it is mounted
explicitly; in case of non-initramfs, it is mounted just after / is mounted and
before /sbin/init is executed (Rob, correct me if I'm wrong).

 So with initramfs, you will have two versions of /dev/console: one real
mknod'ed one in the initramfs itself, and a virtual one in the devtmpfs. After
devtmpfs is mounted, the one in initramfs is no longer accessible.

> 
>>
>> I wrote a thing about the four different types of filesystems a while back:
>>
>> http://landley.livejournal.com/52326.html
> Another very good read - thanks for that too
> 
>>
>>> I'm running an x86_64 system using the default vga console if that matters.
>>> I'm obviously missing something but I can't find a mknod or mkdevs for it.
>>>
>>> Where is /dev/console created when using devtmpfs?
> I was thinking in terms of the buildroot "use devtmpfs only" as opposed with mdev,
> eudev, or systemd

 Actually, all four of these use devtmpfs. The only thing that mdev, eudev, and
systemd-udev add is the possibility to do more processing (e.g. creating
symlinks) when the device is created. But the kernel itself will always create
the device node in devtmpfs.


 Regards,
 Arnout

[snip]


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Where is /dev/console created when using devtmpfs?

[Steve Kenton]

On 03/27/2015 05:14 PM, Arnout Vandecappelle wrote:
> On 27/03/15 05:14, Steve Kenton wrote:
>> On 03/26/2015 02:51 PM, Rob Landley wrote:
>>> On Wed, Mar 25, 2015 at 12:44 PM, Steve Kenton <skenton@ou.edu> wrote:
> [snip]
>>>> so I spent some time looking at my device usage and for the life of me
>>>> I can't find where /dev/console gets created during the build process.
>>>
>>> The console PID 1 inherits at process launch is the /dev/console in
>>> initramfs. When you don't have initramfs, the init/noinitramfs.c code
>>> literally does:
>> Yes, this is my configuration - initramfs not linked into kernel by buildroot
>> # BR2_TARGET_ROOTFS_INITRAMFS is not set
> 
>  Just to be clear "initramfs not linked into kernel" is BR2_TARGET_ROOTFS_CPIO,
> "initramfs linked into the kernel" is BR2_TARGET_ROOTFS_INITRAMFS. But for the
> purpose of the contents of the rootfs, that doesn't make a difference.
> 
>  But so in your case, Steve, I believe you _do_ have an initramfs, i.e. you have
> selected BR2_TARGET_ROOTFS_CPIO. Otherwise /dev/console wouldn't have been
> created :-).
I've been known to be confused once or twice :-)

Yes, BR2_TARGET_ROOTFS_CPIO was still explicitly set in my config from previous testing
and experimentation with initramfs even though BR2_TARGET_ROOTFS_INITRAMFS was not set.
Since BR2_TARGET_ROOTFS_INITRAMFS selects BR2_TARGET_ROOTFS_CPIO I must have been confused :-(
So even though I am currently using the tar version of the rootfs there is indeed
also a cpio version getting created and that process is where /dev/console comes from in
my on-disk rootfs. It should disappear when I clear the unneeded BR2_TARGET_ROOTFS_CPIO.


.../buildroot-2015.02/output/images$ ls -l
total 548980
-rw-rw-r-- 1 steve steve   5896640 Mar 26 22:44 bzImage
-rw-rw-r-- 1 steve steve    139560 Mar 22 19:59 grub.img
-rw-rw-r-- 1 steve steve 183162368 Mar 26 22:44 rootfs.cpio
-rw-rw-r-- 1 steve steve 213949440 Mar 26 22:44 rootfs.ext2
lrwxrwxrwx 1 steve steve        11 Mar 26 22:44 rootfs.ext4 -> rootfs.ext2
-rw-rw-r-- 1 steve steve 185610240 Mar 26 22:44 rootfs.tar
drwxr-xr-x 2 steve steve      4096 Mar 22 20:01 syslinux

Oops!

.../dev$ ls -l
total 4
crw--w--w- 1 root root 5, 1 Mar 25 22:54 console
lrwxrwxrwx 1 root root   10 Mar  1 15:26 log -> ../tmp/log
drwxrwxr-x 2 root root 4096 Mar  1 15:26 pts

And it makes sense now.


> 
>>
>>>
>>>         err = sys_mkdir((const char __user __force *) "/dev", 0755);
>>>         if (err < 0)
>>>                 goto out;
>>>
>>>         err = sys_mknod((const char __user __force *) "/dev/console",
>>>                         S_IFCHR | S_IRUSR | S_IWUSR,
>>>                         new_encode_dev(MKDEV(5, 1)));
> 
>  Note again that this stuff is only done when there is no initramfs. In other
> words, when there is an initramfs, this initramfs must contain /dev/console or
> you have no console.
> 
>  That's why buildroot adds it explicitly in fs/cpio/cpio.mk:
> 
> PACKAGES_PERMISSIONS_TABLE += /dev/console c 622 0 0 5 1 - - -$(sep)

Ahhh! and that answers my original question of where the on-disk /dev/console come from.

Thanks everyone for the information.

Rob, my 2 cents worth says that todo item of yours is a good idea.

Steve

[snip]

[Buildroot] Something odd about /dev/console creation

[Steve Kenton]

It looks like there is something odd going on with the creation of /dev/console
There also seems to be some unintended mixing of creation of the cpio filesystem
vs installing it as an initramfs as controlled by these settings.

BR2_TARGET_ROOTFS_CPIO
BR2_TARGET_ROOTFS_INITRAMFS

Which can result in /init being created and populated with the script for an initramfs
boot when in fact it is not needed if BR2_TARGET_ROOTFS_CPIO os set and BR2_TARGET_ROOTFS_INITRAMFS
is not. It does not seem to hurt anything but is odd to see.

# Anyway, back to /dev/console apparently always being created in the on-disk filesystem.
# As far as I can tell this line always gets executed during any buildroot make
# independent of the above settings for actually creating a cpio filesystem or initramfs
./fs/cpio/cpio.mk:PACKAGES_PERMISSIONS_TABLE += /dev/console c 622 0 0 5 1 - - -$(sep)

# Which means that this line:
./fs/common.mk:	printf '$$(subst $$(sep),\n,$$(PACKAGES_PERMISSIONS_TABLE))' >> $$(FULL_DEVICE_TABLE)

# will always add /dev/console to _device_table.txt for creation during the build (example from my usual build)
printf '	/bin/busybox                     f 4755 0  0 - - - - -\n	/usr/share/udhcpc/default.script f 755  0  0 - - - - -\n \  # split for email readability
/dev/console c 622 0 0 5 1 - - -\n' >> /data/bcdist/buildroot-2015.02/output/build/_device_table.txt


# I got lost in the maze of twisty windy make rules, all different, so I
# added a syntactically invalid 'puke here' to the fs/cpio/cpio.mk as shown here
PACKAGES_PERMISSIONS_TABLE += /dev/console c 622 0 0 5 1 - - -$(sep)
puke here
# Then did 'make defconfig; make -d' in a clean buildroot tree and generated this output when it died
# which definitely shows it's being read, even when no kernel is being built.
...
Reading makefile 'fs/cpio/cpio.mk' (search path) (no ~ expansion)...
fs/cpio/cpio.mk:25: *** missing separator (did you mean TAB instead of 8 spaces?).  Stop.

# Without the 'puke here' the minimal build results in a dev directory containing these files:
# .../buildroot-2015.02$ ls -l /output/target/dev
total 4
-rw--w--w- 1 steve steve    0 Mar 28 14:32 console
lrwxrwxrwx 1 steve steve   10 Mar  1 15:26 log -> ../tmp/log
drwxrwxr-x 2 steve steve 4096 Mar  1 15:26 pts


I'm not sure how to proceed from here but I don't think the current situation is 'correct'

Steve Kenton

[Buildroot] Where is /dev/console created when using devtmpfs?

[Rob Landley]

On 03/27/2015 05:14 PM, Arnout Vandecappelle wrote:
> On 27/03/15 05:14, Steve Kenton wrote:
>> On 03/26/2015 02:51 PM, Rob Landley wrote:
>>> On Wed, Mar 25, 2015 at 12:44 PM, Steve Kenton <skenton@ou.edu> wrote:
> [snip]
>>>> so I spent some time looking at my device usage and for the life of me
>>>> I can't find where /dev/console gets created during the build process.
>>>
>>> The console PID 1 inherits at process launch is the /dev/console in
>>> initramfs. When you don't have initramfs, the init/noinitramfs.c code
>>> literally does:
>> Yes, this is my configuration - initramfs not linked into kernel by buildroot
>> # BR2_TARGET_ROOTFS_INITRAMFS is not set
> 
>  Just to be clear "initramfs not linked into kernel" is BR2_TARGET_ROOTFS_CPIO,
> "initramfs linked into the kernel" is BR2_TARGET_ROOTFS_INITRAMFS. But for the
> purpose of the contents of the rootfs, that doesn't make a difference.
> 
>  But so in your case, Steve, I believe you _do_ have an initramfs, i.e. you have
> selected BR2_TARGET_ROOTFS_CPIO. Otherwise /dev/console wouldn't have been
> created :-).

I note that these are buildroot symbols and I was talking about kernel
config symbols, so you may have to trace through the build process a bit
to see what the kernel .config actually winds up with and thus what
happens at boot time. But it looks like you covered that.

>>> devtmpfs is a synthetic filesystem, like /proc or /sys. You're not
>>> talking to any sort of backing store when you mount it, instead the
>>> driver is hallucinating contents for you to read and write from on the
>>> fly, and when you interact with it you talk to functions in the driver
>>> that send/receive some data they just made up to humor your request.
>> Again if I under stand correctly the contents of the on-disk  /dev on the
>> root filesystem in my case (initramfs not linked into kernel) are unused
>> and normally inaccessible because of devtmpfs mount on top?
> 
>  Indeed, its contents becomes inaccessible as soon as devtmpfs is mounted on
> top. In case of initramfs, that is when /init is executed and it is mounted
> explicitly; in case of non-initramfs, it is mounted just after / is mounted and
> before /sbin/init is executed (Rob, correct me if I'm wrong).

I mentioned I've been meaning to send a patch to make initramfs honor
the kernel's auto-mount-devtmpfs config option. Last I checked it was
only happening for root=/dev/hda style mounts, not for initramfs mounts.
(So your /init script had to mount devtmpfs, I don't think the kernel
does it for you on initramfs. But it's quite possible somebody else
fixed it in the last year or so and I didn't notice. I've been off doing
toybox and banging on sh2 nommu stuff.)

Either way, if you mount another filesystem on top of /dev, the original
contents of /dev are inaccessable until you unmount the filesystem
blocking it. This is pretty much true of the contents of any directory
you mount stuff on: the mount renders the original contents
inaccessable. (Nobody said the directory you mounted on was empty...)

(Ok, that's not _quite_ true. You can mount a filesystem on $PWD but
still access the old contents through "." until you cd out of it. A new
traversal from root wouldn't get you _back_ there, but your current
position still links to the otherwise inaccessable directory. And yes,
back in the day people hacking into 1980's unix systems would hide their
files from the sysadmin by overmounting them. Each process has "/" and
"." as two process-local symlinks indicating your current chroot inode
and your current chdir inode, all paths start from one or the other.
This leads to a fun trick for breaking out of chroots where you make a
subdirectory, call chroot(2) to point / into that subdirectory but _not_
cd into it, then chroot ../../../.. until you stop moving, then chroot /
to get access to the full tree. There's only one chroot symlink you see,
and the test for "can't cd .. past it" is an == test which won't trigger
if your cwd is currently _above_ it.)

Sorry, digression. :)

>  So with initramfs, you will have two versions of /dev/console: one real
> mknod'ed one in the initramfs itself, and a virtual one in the devtmpfs. After
> devtmpfs is mounted, the one in initramfs is no longer accessible.

Neither is "real" or "virtual". Both are device nodes with "block/char",
"major", and "minor" telling the system which driver to talk to you when
you open it.

(If you loopback mount a squashfs that contains device nodes, is that
not a "real" filesystem? How about if rhw dilw you loopback mounted it
from is on a network mount?)

Rob