Linux v5.7.1: Ext4-FS and systemd-journald errors after suspend + resume

Linux v5.7.1: Ext4-FS and systemd-journald errors after suspend + resume

[Sedat Dilek]

Hi,

for a long time I did not try suspend + resume.

So, with Linux v5.7.1 I tried it.

As I upgraded my systemd to version 245.6-1 I suspected this change,
see my report to Debian/systemd team.

Second, as I saw read-only filesystem problems in the logs I changed
in /etc/fstab:

-UUID=<UUID-of-rootfs> /   ext4 errors=remount-ro 0 1
+UUID=<UUID-of-rootfs> / ext4 defaults 0 1

That did not help.

I have one single / root-fs partition.

What I still see after suspend (45 secs) and resume:

Ext4: ... unable to read itable block ...
Ext4: ... dx_probe:768 ... error read-only directory block ...
Ext4: ... ext4_get_inode_loc ...
systemd-journald: Failed to write entry - Read-only file system <---
Also kded5 etc.

The system is in an awful and unusable situation.
Typing any command in konsole shows command not known/found.
I have not found a way to debug this.

What informations do you need?
Any hints on how to debug this?

Thanks.

Regards,
- Sedat -

[1] https://alioth-lists.debian.net/pipermail/pkg-systemd-maintainers/2020-June/041057.html

Re: Linux v5.7.1: Ext4-FS and systemd-journald errors after suspend + resume

[Sedat Dilek]

On Mon, Jun 8, 2020 at 3:26 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> Hi,
>
> for a long time I did not try suspend + resume.
>
> So, with Linux v5.7.1 I tried it.
>
> As I upgraded my systemd to version 245.6-1 I suspected this change,
> see my report to Debian/systemd team.
>
> Second, as I saw read-only filesystem problems in the logs I changed
> in /etc/fstab:
>
> -UUID=<UUID-of-rootfs> /   ext4 errors=remount-ro 0 1
> +UUID=<UUID-of-rootfs> / ext4 defaults 0 1
>
> That did not help.
>
> I have one single / root-fs partition.
>
> What I still see after suspend (45 secs) and resume:
>
> Ext4: ... unable to read itable block ...
> Ext4: ... dx_probe:768 ... error read-only directory block ...
> Ext4: ... ext4_get_inode_loc ...
> systemd-journald: Failed to write entry - Read-only file system <---
> Also kded5 etc.
>
> The system is in an awful and unusable situation.
> Typing any command in konsole shows command not known/found.
> I have not found a way to debug this.
>
> What informations do you need?
> Any hints on how to debug this?
>
> Thanks.
>
> Regards,
> - Sedat -
>
> [1] https://alioth-lists.debian.net/pipermail/pkg-systemd-maintainers/2020-June/041057.html

I checked the health of /dev/sdc with:

root# badblocks -sv /dev/sdc2

No errors.

- Sedat -

Re: Linux v5.7.1: Ext4-FS and systemd-journald errors after suspend + resume

[Sedat Dilek]

On Mon, Jun 8, 2020 at 5:46 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Mon, Jun 8, 2020 at 3:26 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > Hi,
> >
> > for a long time I did not try suspend + resume.
> >
> > So, with Linux v5.7.1 I tried it.
> >
> > As I upgraded my systemd to version 245.6-1 I suspected this change,
> > see my report to Debian/systemd team.
> >
> > Second, as I saw read-only filesystem problems in the logs I changed
> > in /etc/fstab:
> >
> > -UUID=<UUID-of-rootfs> /   ext4 errors=remount-ro 0 1
> > +UUID=<UUID-of-rootfs> / ext4 defaults 0 1
> >
> > That did not help.
> >
> > I have one single / root-fs partition.
> >
> > What I still see after suspend (45 secs) and resume:
> >
> > Ext4: ... unable to read itable block ...
> > Ext4: ... dx_probe:768 ... error read-only directory block ...
> > Ext4: ... ext4_get_inode_loc ...
> > systemd-journald: Failed to write entry - Read-only file system <---
> > Also kded5 etc.
> >
> > The system is in an awful and unusable situation.
> > Typing any command in konsole shows command not known/found.
> > I have not found a way to debug this.
> >
> > What informations do you need?
> > Any hints on how to debug this?
> >
> > Thanks.
> >
> > Regards,
> > - Sedat -
> >
> > [1] https://alioth-lists.debian.net/pipermail/pkg-systemd-maintainers/2020-June/041057.html
>
> I checked the health of /dev/sdc with:
>
> root# badblocks -sv /dev/sdc2
>
> No errors.
>

Just as an information:

The external /dev/sdc hdd is attached to an internal usb-3.0
controller from ASMedia.

- Sedat -

Re: Linux v5.7.1: Ext4-FS and systemd-journald errors after suspend + resume

[Theodore Y. Ts'o]

On Mon, Jun 08, 2020 at 03:26:40PM +0200, Sedat Dilek wrote:
> Hi,
> 
> for a long time I did not try suspend + resume.
> 
> So, with Linux v5.7.1 I tried it.
> 
> As I upgraded my systemd to version 245.6-1 I suspected this change,
> see my report to Debian/systemd team.
> 
> Second, as I saw read-only filesystem problems in the logs I changed
> in /etc/fstab:
> 
> -UUID=<UUID-of-rootfs> /   ext4 errors=remount-ro 0 1
> +UUID=<UUID-of-rootfs> / ext4 defaults 0 1
>
> That did not help.

If you didn't update Othe fstab in the initramfs, the root file system
may still be being mounted with errors=remount-ro.

You can check the current status of a file system's mount options
using /proc/mounts.  Or if you want the full set of changes, you can
look at the file /proc/fs/ext4/<device>/options.

When was the last kernel version and systemd where suspend/resume
worked for you?  If the things work fine until you do a
suspend/resume, this could be either a hardware issue, a driver issue
in the kernel, or systemd issue.  It's almost certainly not a file
system issue, however.  It's likely that you'll need to do a
disciplined set of debugging, where you find which versions of
software work, and then try figuring out what was the first version of
the kernel and/or/systemd where thigns stop working.

					- Ted

Re: Linux v5.7.1: Ext4-FS and systemd-journald errors after suspend + resume

[Sedat Dilek]

On Mon, Jun 8, 2020 at 10:21 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Mon, Jun 08, 2020 at 03:26:40PM +0200, Sedat Dilek wrote:
> > Hi,
> >
> > for a long time I did not try suspend + resume.
> >
> > So, with Linux v5.7.1 I tried it.
> >
> > As I upgraded my systemd to version 245.6-1 I suspected this change,
> > see my report to Debian/systemd team.
> >
> > Second, as I saw read-only filesystem problems in the logs I changed
> > in /etc/fstab:
> >
> > -UUID=<UUID-of-rootfs> /   ext4 errors=remount-ro 0 1
> > +UUID=<UUID-of-rootfs> / ext4 defaults 0 1
> >
> > That did not help.
>

Hi Ted,

> If you didn't update Othe fstab in the initramfs, the root file system
> may still be being mounted with
.
>
> You can check the current status of a file system's mount options
> using /proc/mounts.  Or if you want the full set of changes, you can
> look at the file /proc/fs/ext4/<device>/options.
>

Cool.
I was using... cat /etc/mtab

# cat /proc/fs/ext4/sdc2/options
rw
bsddf
nogrpid
block_validity
dioread_nolock
nodiscard
delalloc
nowarn_on_error
journal_checksum
barrier
auto_da_alloc
user_xattr
acl
noquota
resuid=0
resgid=0
errors=continue
commit=5
min_batch_time=0
max_batch_time=15000
stripe=0
data=ordered
inode_readahead_blks=32
init_itable=10
max_dir_size_kb=0

> When was the last kernel version and systemd where suspend/resume
> worked for you?  If the things work fine until you do a
> suspend/resume, this could be either a hardware issue, a driver issue
> in the kernel, or systemd issue.  It's almost certainly not a file
> system issue, however.  It's likely that you'll need to do a
> disciplined set of debugging, where you find which versions of
> software work, and then try figuring out what was the first version of
> the kernel and/or/systemd where thigns stop working.
>

I *decided* to revert all changes as I really do not use
suspend+resume for ages.
I do my daily work and poweroff my machine.

Can you say your opinion to setting...

   errors=remount-ro VS. defaults

...for the Root-FS?

Last I used s+r was on Ubuntu/precise 12.04 LTS installed on an *internal* HDD.
I used precise for the period of LTS aka 5 years - until 2017.

As said I am 99% sure this is due my Root-FS is installed on an
*external* USB-3.0 HDD connected to an ASMedia USB-3.0 controller.

For a "disciplined" debugging I need to understand how the suspend and
resume works under Debian/testing.

BTW, this is a Samsung Ultrabook 2nd generation with SandyBridge
CPU/GPU with known "broken" ACPI and UEFI?
I am using BIOS-legacy not UEFI.

[    0.000000] DMI: SAMSUNG ELECTRONICS CO., LTD.
530U3BI/530U4BI/530U4BH/530U3BI/530U4BI/530U4BH, BIOS 13XK 03/28/2013

Booting yesterday ArchLinux ISO 2020.06.01 (UEFI) showed me I can use UEFI.
Yes, I like ArchLinux's chroot script and use the ISO to rescue my
Debian system :-).

I gasped over some cool hints in the ArchLinux wikis concerning
power-management and USB autosuspend/wakeup hacks (see [2], [3]).

The real history is I realized a powertop.service (systemd) and
contributed on how to the build+install dance on Debian/testing [4].
Yesterday, in my experiments I stopped and disabled powertop.service
as it triggers systemd-udev-trigger.service (see P.S.).

The most important thing to me was to use an intelligent
power-management while I am working at my notebook.

Thanks for your precious time and your feedback.

Regards,
- Sedat -

[1] http://ftp.halifax.rwth-aachen.de/archlinux/iso/2020.06.01/
[2] https://wiki.archlinux.org/index.php/Powertop
[3] https://wiki.archlinux.org/index.php/Power_management
[4] https://github.com/fenrus75/powertop/commit/a92d6b15600335f0004d2b7b93e21ab3a84e15f9

P.S.: powertop.service with powertop v2.13-rc1

# systemctl cat powertop.service
# /etc/systemd/system/powertop.service
[Unit]
Description=Powertop tunings

[Service]
Type=oneshot
##ExecStart=/usr/sbin/powertop --auto-tune
ExecStart=/opt/powertop/sbin/powertop --auto-tune

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/powertop.service.d/override.conf
[Service]
# Reload /etc/udev/rules.d/50-usb-autosuspend.rules
ExecStart=/bin/systemctl restart systemd-udev-trigger.service

# cat /etc/udev/rules.d/50-usb-autosuspend.rules
# blacklist/whitelist for usb autosuspend
#
# blacklist Logitech, Inc. M-BJ58/M-BJ69 Optical Wheel Mouse
ACTION=="add", SUBSYSTEM=="usb", TEST=="power/control",
ATTR{idVendor}=="046d", ATTR{idProduct}=="c00e",
ATTR{power/control}="on"

- EOT -