Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Martin Steigerwald]

Hi Holger!

CC'ing BCacheFS mailing list.

My original mail is here:

https://lore.kernel.org/linux-usb/5264d425-fc13-6a77-2dbf-6853479051a0@applied-asynchrony.com/T/
#m5ec9ecad1240edfbf41ad63c7aeeb6aa6ea38a5e

Holger Hoffstätte - 11.02.24, 17:02:29 CET:
> On 2024-02-11 16:42, Martin Steigerwald wrote:
> > Hi!
> > I am trying to put data on an external Kingston XS-2000 4 TB SSD using
> > self-compiled Linux 6.7.4 kernel and encrypted BCacheFS. I do not
> > think BCacheFS has any part in the errors I see, but if you disagree 
> > feel free to CC the BCacheFS mailing list as you reply.
> 
> This is indeed a known bug with bcachefs on USB-connected devices.
> Apply the following commit:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi
> t/fs/bcachefs?id=3e44f325f6f75078cdcd44cd337f517ba3650d05
> 
> This and some other commits are already scheduled for -stable.

Thanks!

Oh my. I was aware of some bug fixes coming for stable. I briefly looked 
through them, but now I did not make a connection.

I will wait for 6.7.5 and retry then I bet.

Best,
-- 
Martin

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Kent Overstreet]

On Sun, Feb 11, 2024 at 06:06:27PM +0100, Martin Steigerwald wrote:
> Hi Holger!
> 
> CC'ing BCacheFS mailing list.
> 
> My original mail is here:
> 
> https://lore.kernel.org/linux-usb/5264d425-fc13-6a77-2dbf-6853479051a0@applied-asynchrony.com/T/
> #m5ec9ecad1240edfbf41ad63c7aeeb6aa6ea38a5e
> 
> Holger Hoffstätte - 11.02.24, 17:02:29 CET:
> > On 2024-02-11 16:42, Martin Steigerwald wrote:
> > > Hi!
> > > I am trying to put data on an external Kingston XS-2000 4 TB SSD using
> > > self-compiled Linux 6.7.4 kernel and encrypted BCacheFS. I do not
> > > think BCacheFS has any part in the errors I see, but if you disagree 
> > > feel free to CC the BCacheFS mailing list as you reply.
> > 
> > This is indeed a known bug with bcachefs on USB-connected devices.
> > Apply the following commit:
> > 
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi
> > t/fs/bcachefs?id=3e44f325f6f75078cdcd44cd337f517ba3650d05
> > 
> > This and some other commits are already scheduled for -stable.
> 
> Thanks!
> 
> Oh my. I was aware of some bug fixes coming for stable. I briefly looked 
> through them, but now I did not make a connection.
> 
> I will wait for 6.7.5 and retry then I bet.

That doesn't look related - the device claims to not support flush or
fua, and the bug resulted in us not sending flush/fua devices; the main
thing people would see without that patch, on 6.8, would be an immediate
-EOPNOTSUP on the first flush journal write.

He only got errors after an hour or so, or 10 minutes with UAS disabled;
we send flushes once a second. Sounds like a screwy device.

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Martin Steigerwald]

Kent Overstreet - 11.02.24, 19:51:32 CET:
> On Sun, Feb 11, 2024 at 06:06:27PM +0100, Martin Steigerwald wrote:
[…]
> > CC'ing BCacheFS mailing list.
> > 
> > My original mail is here:
> > 
> > https://lore.kernel.org/linux-usb/5264d425-fc13-6a77-2dbf-6853479051a0
> > @applied-asynchrony.com/T/ #m5ec9ecad1240edfbf41ad63c7aeeb6aa6ea38a5e
> > 
> > Holger Hoffstätte - 11.02.24, 17:02:29 CET:
> > > On 2024-02-11 16:42, Martin Steigerwald wrote:
> > > > Hi!
> > > > I am trying to put data on an external Kingston XS-2000 4 TB SSD
> > > > using
> > > > self-compiled Linux 6.7.4 kernel and encrypted BCacheFS. I do not
> > > > think BCacheFS has any part in the errors I see, but if you
> > > > disagree
> > > > feel free to CC the BCacheFS mailing list as you reply.
> > > 
> > > This is indeed a known bug with bcachefs on USB-connected devices.
> > > Apply the following commit:
> > > 
> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/c
> > > ommi t/fs/bcachefs?id=3e44f325f6f75078cdcd44cd337f517ba3650d05
> > > 
> > > This and some other commits are already scheduled for -stable.
> > 
> > Thanks!
> > 
> > Oh my. I was aware of some bug fixes coming for stable. I briefly
> > looked through them, but now I did not make a connection.
> > 
> > I will wait for 6.7.5 and retry then I bet.
> 
> That doesn't look related - the device claims to not support flush or
> fua, and the bug resulted in us not sending flush/fua devices; the main
> thing people would see without that patch, on 6.8, would be an immediate
> -EOPNOTSUP on the first flush journal write.
> 
> He only got errors after an hour or so, or 10 minutes with UAS disabled;
> we send flushes once a second. Sounds like a screwy device.

Thanks for that explanation, Kent.

I am the one with that external Transcend XS 2000 4 TB SSD and I
specifically did not CC bcachefs mailing list at the beginning as after
seeing things like

[33963.462694] sd 0:0:0:0: [sda] tag#10 uas_zap_pending 0 uas-tag 1 inflight: CMD 
[33963.462708] sd 0:0:0:0: [sda] tag#10 CDB: Write(16) 8a 00 00 00 00 00 82 c1 bc 00 00 00 04 00 00 00
[…]
[33963.592872] sd 0:0:0:0: [sda] tag#10 FAILED Result: hostbyte=DID_RESET driverbyte=DRIVER_OK cmd_age=182s

I thought some quirks in the device to be at fault.

However while Sandisk Extreme Pro 2 TB claims to support DPO and FUA I see

Write cache: disabled, read cache: enabled, doesn't support DPO or FUA

also with other devices like external Toshiba Canvio 4 TB hard disks. Using
LUKS encrypted BTRFS on those I never saw any timeout while writing out
data issue with any of those hard disks. Also with disabled write cache
any cache flush / FUA request should be a no-op anyway? These hard disks
have been doing a ton of backup workloads without any issues, but so far
only with BTRFS.

I may test the Transcend XS2000 with BTRFS to see whether it makes a
difference, however I really like to use it with BCacheFS and I do not really
like to use LUKS for external devices. According to the kernel log I still
don't really think those errors at the block layer were about anything
filesystem specific, but what  do I know?

With UAS enabled for Transcend XS2000 I see:

Write cache: disabled, read cache: enabled, doesn't support DPO or FUA

This sounds about right: Without cache flush / FUA request disable write
cache.

With UAS disabled, using only usb-storage, however I see:

Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Which appears to be broken to me: If it cannot do cache flush / FUA it
should have write cache disabled.

Thus I removed the quirk to disable UAS again. It did not help anyway.

However when I look at the output of "hdparm -I" for that Transcend XS2000
none of this makes sense. Cause it blatantly advertises to support

[…]
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
[…]
           *    WRITE_{DMA|MULTIPLE}_FUA_EXT
[…]

It has firmware revision S9K00107. I see whether I can get this updated
in case any update is available. Which is not obvious to me as Kingston
only offers to download a Windows application to update the firmware.

I asked them how to do an update on Linux. But am also prepared to run to
a friend with Windows system to do the update.

There is no urgency in this, so let's see whether a firmware update may
fix anything. In case someone has any additional insight, feel free to add
it. Otherwise I consider it case closed unless I retest with either Linux
kernel 6.7.5 or 6.8-rc4 and/or after having made a firmware update
if available.

Maybe also some other quirks would need to be enabled for that
device? I tested it with:

% cat /etc/modprobe.d/disable-uas.conf
# Does not work with external SSD Transcend XS2000 4TB
options usb-storage quirks=0951:176b:u

but as explained that did not help and thus I disabled UAS disabling
quirk again.

Best,
-- 
Martin

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Kent Overstreet]

On Mon, Feb 12, 2024 at 04:52:09PM +0100, Martin Steigerwald wrote:
> Kent Overstreet - 11.02.24, 19:51:32 CET:
> > On Sun, Feb 11, 2024 at 06:06:27PM +0100, Martin Steigerwald wrote:
> […]
> > > CC'ing BCacheFS mailing list.
> > > 
> > > My original mail is here:
> > > 
> > > https://lore.kernel.org/linux-usb/5264d425-fc13-6a77-2dbf-6853479051a0
> > > @applied-asynchrony.com/T/ #m5ec9ecad1240edfbf41ad63c7aeeb6aa6ea38a5e
> > > 
> > > Holger Hoffstätte - 11.02.24, 17:02:29 CET:
> > > > On 2024-02-11 16:42, Martin Steigerwald wrote:
> > > > > Hi!
> > > > > I am trying to put data on an external Kingston XS-2000 4 TB SSD
> > > > > using
> > > > > self-compiled Linux 6.7.4 kernel and encrypted BCacheFS. I do not
> > > > > think BCacheFS has any part in the errors I see, but if you
> > > > > disagree
> > > > > feel free to CC the BCacheFS mailing list as you reply.
> > > > 
> > > > This is indeed a known bug with bcachefs on USB-connected devices.
> > > > Apply the following commit:
> > > > 
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/c
> > > > ommi t/fs/bcachefs?id=3e44f325f6f75078cdcd44cd337f517ba3650d05
> > > > 
> > > > This and some other commits are already scheduled for -stable.
> > > 
> > > Thanks!
> > > 
> > > Oh my. I was aware of some bug fixes coming for stable. I briefly
> > > looked through them, but now I did not make a connection.
> > > 
> > > I will wait for 6.7.5 and retry then I bet.
> > 
> > That doesn't look related - the device claims to not support flush or
> > fua, and the bug resulted in us not sending flush/fua devices; the main
> > thing people would see without that patch, on 6.8, would be an immediate
> > -EOPNOTSUP on the first flush journal write.
> > 
> > He only got errors after an hour or so, or 10 minutes with UAS disabled;
> > we send flushes once a second. Sounds like a screwy device.
> 
> Thanks for that explanation, Kent.
> 
> I am the one with that external Transcend XS 2000 4 TB SSD and I
> specifically did not CC bcachefs mailing list at the beginning as after
> seeing things like
> 
> [33963.462694] sd 0:0:0:0: [sda] tag#10 uas_zap_pending 0 uas-tag 1 inflight: CMD 
> [33963.462708] sd 0:0:0:0: [sda] tag#10 CDB: Write(16) 8a 00 00 00 00 00 82 c1 bc 00 00 00 04 00 00 00
> […]
> [33963.592872] sd 0:0:0:0: [sda] tag#10 FAILED Result: hostbyte=DID_RESET driverbyte=DRIVER_OK cmd_age=182s
> 
> I thought some quirks in the device to be at fault.
> 
> However while Sandisk Extreme Pro 2 TB claims to support DPO and FUA I see
> 
> Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
> 
> also with other devices like external Toshiba Canvio 4 TB hard disks. Using
> LUKS encrypted BTRFS on those I never saw any timeout while writing out
> data issue with any of those hard disks. Also with disabled write cache
> any cache flush / FUA request should be a no-op anyway? These hard disks
> have been doing a ton of backup workloads without any issues, but so far
> only with BTRFS.
> 
> I may test the Transcend XS2000 with BTRFS to see whether it makes a
> difference, however I really like to use it with BCacheFS and I do not really
> like to use LUKS for external devices. According to the kernel log I still
> don't really think those errors at the block layer were about anything
> filesystem specific, but what  do I know?

It's definitely not unheard of for one specific filesystem to be
tickling driver/device bugs and not others.

I wonder what it would take to dump the outstanding requests on device
timeout.

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Martin Steigerwald]

Kent Overstreet - 12.02.24, 21:42:26 CET:

[thoughts about whether a cache flush / FUA request with write caches 
disabled would be a no-op anyway]

> > I may test the Transcend XS2000 with BTRFS to see whether it makes a
> > difference, however I really like to use it with BCacheFS and I do not
> > really like to use LUKS for external devices. According to the kernel
> > log I still don't really think those errors at the block layer were
> > about anything filesystem specific, but what  do I know?
> 
> It's definitely not unheard of for one specific filesystem to be
> tickling driver/device bugs and not others.
> 
> I wonder what it would take to dump the outstanding requests on device
> timeout.

I got some reply back from Transcend support.

They brought up two possible issues:

1) Copied to many files at once. I am not going to accept that one. An 
external 4 TB SSD should handle writing 1,4 TB in about 215000 files, 
coming from a slower Toshiba Canvio Basics external HD, just fine. About 
90000 files was larger files like sound and video files or installation 
archives. The rest is from a Linux system backup, so smaller files. I 
likely move those elsewhere before I try again as I do not need these on 
flash anyway. However if the amount of files or data matters I could never 
know what amount of data I could write safely in one go. That is not 
acceptable to me.

2) Power management related to USB port. Cause I am using a laptop. It may 
have been that the Linux kernel decided to put the USB port the SSD was 
connected to into some kind of sleep state. However it was a constant 
rsync based copy workload. Yes, the kernel buffers data and the reads from 
Toshiba HD should be quite a bit slower than the Transcend SSD could 
handle the writes. I saw now more than 80-90 MiB/s coming from the hard 
disk. However I would doubt this lead to pauses of write activity of more 
than 30 seconds. Still it could be a thing.

Regarding further testing I am unsure whether to first test with BTRFS on 
top of LUKS – I do not like to store clear text data on the SSD – or with 
BCacheFS plus fixes which are 6.7.5 or 6.8-rc4 in just in the case the flush 
handling fixes would still have an influence on the issue at hand.

First I will have a look on how to see what USB power management options 
may be in place and how to tell Linux to keep the USB port the SSD is 
connected to at all times.

Let's see how this story unfolds. At least I am in no hurry about it.

Best,
-- 
Martin

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Alan Stern]

On Thu, Feb 15, 2024 at 12:09:20PM +0100, Martin Steigerwald wrote:
> Kent Overstreet - 12.02.24, 21:42:26 CET:
> 
> [thoughts about whether a cache flush / FUA request with write caches 
> disabled would be a no-op anyway]
> 
> > > I may test the Transcend XS2000 with BTRFS to see whether it makes a
> > > difference, however I really like to use it with BCacheFS and I do not
> > > really like to use LUKS for external devices. According to the kernel
> > > log I still don't really think those errors at the block layer were
> > > about anything filesystem specific, but what  do I know?
> > 
> > It's definitely not unheard of for one specific filesystem to be
> > tickling driver/device bugs and not others.
> > 
> > I wonder what it would take to dump the outstanding requests on device
> > timeout.
> 
> I got some reply back from Transcend support.
> 
> They brought up two possible issues:
> 
> 1) Copied to many files at once. I am not going to accept that one. An 
> external 4 TB SSD should handle writing 1,4 TB in about 215000 files, 
> coming from a slower Toshiba Canvio Basics external HD, just fine. About 
> 90000 files was larger files like sound and video files or installation 
> archives. The rest is from a Linux system backup, so smaller files. I 
> likely move those elsewhere before I try again as I do not need these on 
> flash anyway. However if the amount of files or data matters I could never 
> know what amount of data I could write safely in one go. That is not 
> acceptable to me.
> 
> 2) Power management related to USB port. Cause I am using a laptop. It may 
> have been that the Linux kernel decided to put the USB port the SSD was 
> connected to into some kind of sleep state. However it was a constant 
> rsync based copy workload. Yes, the kernel buffers data and the reads from 
> Toshiba HD should be quite a bit slower than the Transcend SSD could 
> handle the writes. I saw now more than 80-90 MiB/s coming from the hard 
> disk. However I would doubt this lead to pauses of write activity of more 
> than 30 seconds. Still it could be a thing.
> 
> Regarding further testing I am unsure whether to first test with BTRFS on 
> top of LUKS – I do not like to store clear text data on the SSD – or with 
> BCacheFS plus fixes which are 6.7.5 or 6.8-rc4 in just in the case the flush 
> handling fixes would still have an influence on the issue at hand.
> 
> First I will have a look on how to see what USB power management options 
> may be in place and how to tell Linux to keep the USB port the SSD is 
> connected to at all times.
> 
> Let's see how this story unfolds. At least I am in no hurry about it.

This may not be an issue of power management but rather one of 
insufficient power.  A laptop may not provide enough power through its 
USB ports for the Transcend SSD to work properly under load.

You can test this by connecting a powered UBS-3 hub between the laptop 
and the drive.

Alan Stern

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Martin Steigerwald]

Alan Stern - 15.02.24, 16:19:54 CET:
> > First I will have a look on how to see what USB power management
> > options may be in place and how to tell Linux to keep the USB port
> > the SSD is connected to at all times.
> > 
> > Let's see how this story unfolds. At least I am in no hurry about it.
> 
> This may not be an issue of power management but rather one of
> insufficient power.  A laptop may not provide enough power through its
> USB ports for the Transcend SSD to work properly under load.
> 
> You can test this by connecting a powered UBS-3 hub between the laptop
> and the drive.

Interesting idea. Maybe the Transcend XS-2000 4TB needs more power than 
the Sandisk Extreme Pro 2TB.

Not sure whether I have one at hand with USB-C here, cause my regular USB 
hub only has USB-A connectors. Need to look for one with enough USB-A and 
USB-C connectors as I use an USB hub as replacement for a docking station. 
But I do have at least optionally powered hub with USB-C one at another 
place. It does not have many ports. But for the task ahead one USB-C port 
is sufficient.

I will try this as well. Thanks.

Best,
-- 
Martin

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Martin Steigerwald]

Hi!

Kent Overstreet - 11.02.24, 19:51:32 CET:
> He only got errors after an hour or so, or 10 minutes with UAS disabled;
> we send flushes once a second. Sounds like a screwy device.

Kingston support intends to RMA the XS-2000 4 TB SSD with a variant with a 
newer firmware version, in case they have it available, while they work on 
a newer firmware version for the device variant the error happened on.

So it appears the device has a bug. I will keep you posted, once I either 
receive that other variant or a firmware upgrade for the existing one.

I am happy with Kingston support so far. It takes quite a while, but they 
are taking the issue for real instead of writing use Windows instead of 
Linux or something like that :) - like I read before in other occasions 
with hardware from other suppliers. Thanks!

Best,
-- 
Martin