PWS 433au (Miata) recovery update

PWS 433au (Miata) recovery update

[Bob Tracy]

Gentlemen,

Figured it was past time for an update, now that I actually have the
Alpha back on-line and functioning in its pre-meltdown capacity as my
IPv6 router and Linux kernel git repository.  The following narrative is
going to necessarily be somewhat long-winded, seeing as it's intended to
be a modern synthesis of knowledge gleaned from out-of-date Debian and
Gentoo installation documents, mailing list archives, bitter experience,
and source code examination.  Make whatever use of it you will.  My
intent is to have this written down *somewhere* for "the next time".

The ability to recover the machine in a somewhat timely fashion was
predicated on having reasonably current backups and a way to get them
onto the Alpha.  I never figured the latter consideration would be the
most difficult part of the job.  Not to put too fine a point on it, but
functional boot media for Alpha is more scarce than it should be.
Special thanks to the people at Gentoo (and Matt Turner in particular)
for being responsive and fixing the "qla1280" firmware issue that was
preventing the effective use of Gentoo's "install-alpha-minimal" image
as a recovery tool.  After a few off-line conversations with Michael,
I'm cautiously optimistic we'll eventually see a useful Debian NETINST
image at some point in the not-too-distant future.

The Gentoo image had neither the requisite USB drivers nor "ntfs-3g"
filesystem support, so I had to mount my external USB drive remotely and
copy my backups across the network.  Not too much pain, even over a
relatively slow 10 Mbit/s link.  Perhaps somewhat fortuitously, I used
a 36 GB disk in the PWS with a layout something like this:

(reserved for aboot)
/boot	(about 75 MB)
/	(about 4 GB)
swap	(about 2 GB)
/tmp	(about 3 GB)
/usr	(about 13.5 GB)
/opt	(about 13.5 GB)

Out of the lot, the real contents of "/opt" didn't have to be present
for the system to function when I initially booted off the hard drive, so
that was my staging area for the backups.  Once I was running off the hard
drive, the plan was to hook up the USB drive and restore "/opt".

Odd thing about the disk partitioning scheme.  The disk label definitely
has to be "bsd" for SRM to be happy, but if Linux is the only OS on the
disk, all the rest of the BSD partitioning conventions don't have to be
observed as far as slice "c" spanning the entire disk, slice "a" being
the "boot" slice, slice "b" being "swap", and so forth.  I doubt dual-
booting with Digital UNIX or one of the *BSD variants is a practical
possibility for most people, particularly those with PWS systems having
limited disk space.

A brief note about partitioning programs: "fdisk" is NOT your friend
on the Alpha, especially in "modern" times.  Use "parted" and save
yourself much frustration.

Using "parted", I set the default units to "cyl" and created a
"sacrificial" first partition beginning on cylinder 0 and ending on
cylinder 1.  This is detected by Linux as, for example "sda1" and should
not be used for anything on the off chance "aboot" installation
overwrites it.  So, the sequence of partition creation commands was:

mkpart 0 1
mkpart 1 a
mkpart a b
mkpart b c
mkpart c d
mkpart d e
mkpart e f

where the letters "a" through "f" represent starting and ending cylinder
numbers for each partition, and the starting cylinder for each partition
is the ending cylinder of the preceding partition, and yes, "parted"
makes sure things don't overlap.

Bonus: when it comes time to do "swriteboot", you don't have to specify
"-f3" because there's no slice 3 spanning the entire disk to prevent
"swriteboot" from writing the boot sectors.

Once I copied my backups into place (with the exception of "/opt" as
mentioned earlier) and wrote the boot sector, I ran into an interesting
show-stopper.  I had evidently upgraded the "initramfs-tools" package
prior to creating my backups, and, long story short, I was getting
dropped into an interactive shell with an "(initramfs)" prompt due to
the following braindamage:

(1) "systemd" (and "udevd" by extension) don't play well with "/usr"
being on a separate partition from "/".  If I have *any* advice to offer
both the battle-scarred veteran and the newbie, it would be to consider
consolidating those two partitions into a single partition.  Me?  I'd
prefer the younger generation of system programmers consider the
perfectly valid reasons why those filesystems might have been separate
to begin with, and respect those reasons.  (Hint: much smaller disks.)

(2) Perhaps as a consequence of (1), "/lib/systemd/systemd-udevd" refuses
to start/run on the initramfs, in spite of the appropriate support being
enabled in the kernel configuration per systemd's README file.  The
error messages appearing on the console are:

error getting socket: Address family not supported by protocol
error initializing udev control socket
could not listen on fds: Invalid argument

This isn't necessarily a fatal error, EXCEPT...

(3) The brain-dead "init-bottom/udev" script doesn't perform the
needed
    mount -n -o move /dev "${rootmnt:?}/dev"
because a prior "udevadm control --exit" fails because "udevd" isn't
running.  So...

(4) validate_init() fails with the cryptic error message
"run-init: opening console: No such file or directory" due to there
being no populated "/dev" directory on the real root fs.

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=811479
Ben thought he had this fixed back in 2016, and I'm pretty sure he did.

My current workaround is to edit
"/usr/share/initramfs-tools/script/init-bottom/udev" and change the
"udevadm" line to read something like
udevadm control --exit || echo "warning: udevd not running"
then run "update-initramfs" as appropriate.  The point being, without
the logical OR clause in that statement, the script exits with an error
at that point, and the subsequent needed mount commands don't get
executed :-(.  The *real* fix is for some bright person to figure out
why "udevd" won't start.  It's running just fine on the hard disk once
the pivot from initramfs gets made.

--Bob

Re: PWS 433au (Miata) recovery update

[Bob Tracy]

On Wed, Jan 16, 2019 at 01:10:14AM -0600, Bob Tracy wrote:
> (initramfs / systemd / udevd issue)

I think I may have this one painted into a corner...

CONFIG_UNIX (at least) needs to be "y" instead of "m".  This is a
relatively new dependency.

For userland applications used in the boot process to have such
dependencies on kernel configuration options is "unfortunate" to
put it mildly.

--Bob

Re: PWS 433au (Miata) recovery update

[Maciej W. Rozycki]

On Wed, 16 Jan 2019, Bob Tracy wrote:

> Odd thing about the disk partitioning scheme.  The disk label definitely
> has to be "bsd" for SRM to be happy, but if Linux is the only OS on the
> disk, all the rest of the BSD partitioning conventions don't have to be
> observed as far as slice "c" spanning the entire disk, slice "a" being
> the "boot" slice, slice "b" being "swap", and so forth.  I doubt dual-
> booting with Digital UNIX or one of the *BSD variants is a practical
> possibility for most people, particularly those with PWS systems having
> limited disk space.

 SRM doesn't actually care about or know the disk partitioning scheme, 
however it does require a signature and a pointer to the location of the 
boot loader (with `aboot' being the usual choice for Linux) in the first 
physical sector, and these structures happen to clash with the location of 
the IBM PC's DOS-style partition table.

 As you observe the partitions for Linux use can be arbitrary.  I used to 
install `aboot' outside any partition by just leaving a number of sectors 
at the beginning of the disk unassigned; `aboot' takes some 80KiB only. 
This is analogous to disks using a DOS-style partition table where you'd 
often leave the whole (simulated) first track of first cylinder unassigned 
for boot loader use.  This saves a partition table entry for a data area 
that can hardly be used through the corresponding block device anyway.

 NB be aware if you ever boot Windows NT on an Alpha machine that has an 
SRM signature on some disks, then the first sector of these disks will get 
corrupted under NT's assumption that these disks have no data and need to 
have a DOS-style partition table installed in the first sector.  Any OSF/1 
disk label will be retained, however the SRM signature will be lost making 
such disks unbootable and will have to be restored to make them bootable 
again.  Also the presence of the DOS-style partition table marker may 
confuse the Linux kernel if support for such partition tables has been 
compiled in in the addition to support for OSF/1 disk labels.

> A brief note about partitioning programs: "fdisk" is NOT your friend
> on the Alpha, especially in "modern" times.  Use "parted" and save
> yourself much frustration.

 I had a look at the GIT history of `util-linux' and it looks like in the 
conversion of the tool to use `libfdisk', also included with `util-linux', 
support has been lost, which I find rather unfortunate.

 It should be pretty straightforward to resurrect with the aid of old 
code, however it was never particularly clean (as has not been what still 
remains in `fdisk') due to the use of `#ifdef __alpha__' to select the 
label type, meaning you could not work with a disk intended for an Alpha 
system using fdisk compiled for a different host architecture.  It looks 
to me like the current framework would make it easier to avoid this #ifdef 
hack, but obviously any remains would have to be cleaned up.

 There used to be a tool called `minlabel' too that you could use to 
partition your Alpha disk, but it was rather crude.

> (1) "systemd" (and "udevd" by extension) don't play well with "/usr"
> being on a separate partition from "/".  If I have *any* advice to offer
> both the battle-scarred veteran and the newbie, it would be to consider
> consolidating those two partitions into a single partition.  Me?  I'd
> prefer the younger generation of system programmers consider the
> perfectly valid reasons why those filesystems might have been separate
> to begin with, and respect those reasons.  (Hint: much smaller disks.)

 In the old days it used to be a common practice to have /usr NFS-mounted 
in some installations, often where NIS was also used; perhaps it is not 
anymore.

 FWIW,

  Maciej

Re: PWS 433au (Miata) recovery update

[Bob Tracy]

On Fri, Jan 18, 2019 at 12:19:52AM +0000, Maciej W. Rozycki wrote:
> (much useful information set in the appropriate historical context)

Thank you for your thoughts.

The earlier reported problem with "/lib/systemd/systemd-udevd" evidently
requiring AF_UNIX socket support to be built-in rather than modular has
been confirmed.  Setting "CONFIG_UNIX=y" in the kernel configuration was
enough to get me past that particular problem I was seeing with the
initial ramdisk.  So, per advice I was given a long time ago, *do*
examine the "systemd" README file under "/usr/share/doc": many kernel
configuration requirements are mentioned there.  As far as gleaning the
additional udev-related info, one *might* infer it from the error
messages produced by the executable, *or* one can examine the udev-
related files under "/lib/systemd/system", one of which explicitly
mentions AF_UNIX in the context of a restricted address family.

I also note that the current "initramfs-tools" have evidently forgotten
how to automatically check and mount local file systems *other* than "/"
and "/usr".  Every boot since restoring my PWS thus far has dropped me
into emergency mode with everything mounted read-write and ready to go
(including swap) *other* than the local non-tmpfs file systems.
Manually running the appropriate flavor of "fsck" and mounting the file
systems before exiting emergency mode results in the expected normal
startup of multi-user system services.

"journalctl -xb" has, for the case of one such file system that didn't
get checked/mounted, the following messages:

(...)
-- The job identifier is 271 and the job result is done.
Dec 21 13:02:10 smirkin systemd[1]: Starting of /dev/sda2 not supported.
-- Subject: A start job for unit dev-sda2.device has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit dev-sda2.device has finished with a failure.
-- 
-- The job identifier is 307 and the job result is unsupported.
Dec 21 13:02:10 smirkin systemd[1]: Dependency failed for File System Check on /dev/sda2.
-- Subject: A start job for unit systemd-fsck@dev-sda2.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit systemd-fsck@dev-sda2.service has finished with a failure.
-- 
-- The job identifier is 306 and the job result is dependency.
Dec 21 13:02:10 smirkin systemd[1]: Dependency failed for /boot.
-- Subject: A start job for unit boot.mount has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support