Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Kent Overstreet]

On Mon, Feb 12, 2024 at 04:52:09PM +0100, Martin Steigerwald wrote:
> Kent Overstreet - 11.02.24, 19:51:32 CET:
> > On Sun, Feb 11, 2024 at 06:06:27PM +0100, Martin Steigerwald wrote:
> […]
> > > CC'ing BCacheFS mailing list.
> > > 
> > > My original mail is here:
> > > 
> > > https://lore.kernel.org/linux-usb/5264d425-fc13-6a77-2dbf-6853479051a0
> > > @applied-asynchrony.com/T/ #m5ec9ecad1240edfbf41ad63c7aeeb6aa6ea38a5e
> > > 
> > > Holger Hoffstätte - 11.02.24, 17:02:29 CET:
> > > > On 2024-02-11 16:42, Martin Steigerwald wrote:
> > > > > Hi!
> > > > > I am trying to put data on an external Kingston XS-2000 4 TB SSD
> > > > > using
> > > > > self-compiled Linux 6.7.4 kernel and encrypted BCacheFS. I do not
> > > > > think BCacheFS has any part in the errors I see, but if you
> > > > > disagree
> > > > > feel free to CC the BCacheFS mailing list as you reply.
> > > > 
> > > > This is indeed a known bug with bcachefs on USB-connected devices.
> > > > Apply the following commit:
> > > > 
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/c
> > > > ommi t/fs/bcachefs?id=3e44f325f6f75078cdcd44cd337f517ba3650d05
> > > > 
> > > > This and some other commits are already scheduled for -stable.
> > > 
> > > Thanks!
> > > 
> > > Oh my. I was aware of some bug fixes coming for stable. I briefly
> > > looked through them, but now I did not make a connection.
> > > 
> > > I will wait for 6.7.5 and retry then I bet.
> > 
> > That doesn't look related - the device claims to not support flush or
> > fua, and the bug resulted in us not sending flush/fua devices; the main
> > thing people would see without that patch, on 6.8, would be an immediate
> > -EOPNOTSUP on the first flush journal write.
> > 
> > He only got errors after an hour or so, or 10 minutes with UAS disabled;
> > we send flushes once a second. Sounds like a screwy device.
> 
> Thanks for that explanation, Kent.
> 
> I am the one with that external Transcend XS 2000 4 TB SSD and I
> specifically did not CC bcachefs mailing list at the beginning as after
> seeing things like
> 
> [33963.462694] sd 0:0:0:0: [sda] tag#10 uas_zap_pending 0 uas-tag 1 inflight: CMD 
> [33963.462708] sd 0:0:0:0: [sda] tag#10 CDB: Write(16) 8a 00 00 00 00 00 82 c1 bc 00 00 00 04 00 00 00
> […]
> [33963.592872] sd 0:0:0:0: [sda] tag#10 FAILED Result: hostbyte=DID_RESET driverbyte=DRIVER_OK cmd_age=182s
> 
> I thought some quirks in the device to be at fault.
> 
> However while Sandisk Extreme Pro 2 TB claims to support DPO and FUA I see
> 
> Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
> 
> also with other devices like external Toshiba Canvio 4 TB hard disks. Using
> LUKS encrypted BTRFS on those I never saw any timeout while writing out
> data issue with any of those hard disks. Also with disabled write cache
> any cache flush / FUA request should be a no-op anyway? These hard disks
> have been doing a ton of backup workloads without any issues, but so far
> only with BTRFS.
> 
> I may test the Transcend XS2000 with BTRFS to see whether it makes a
> difference, however I really like to use it with BCacheFS and I do not really
> like to use LUKS for external devices. According to the kernel log I still
> don't really think those errors at the block layer were about anything
> filesystem specific, but what  do I know?

It's definitely not unheard of for one specific filesystem to be
tickling driver/device bugs and not others.

I wonder what it would take to dump the outstanding requests on device
timeout.

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Martin Steigerwald]

Kent Overstreet - 12.02.24, 21:42:26 CET:

[thoughts about whether a cache flush / FUA request with write caches 
disabled would be a no-op anyway]

> > I may test the Transcend XS2000 with BTRFS to see whether it makes a
> > difference, however I really like to use it with BCacheFS and I do not
> > really like to use LUKS for external devices. According to the kernel
> > log I still don't really think those errors at the block layer were
> > about anything filesystem specific, but what  do I know?
> 
> It's definitely not unheard of for one specific filesystem to be
> tickling driver/device bugs and not others.
> 
> I wonder what it would take to dump the outstanding requests on device
> timeout.

I got some reply back from Transcend support.

They brought up two possible issues:

1) Copied to many files at once. I am not going to accept that one. An 
external 4 TB SSD should handle writing 1,4 TB in about 215000 files, 
coming from a slower Toshiba Canvio Basics external HD, just fine. About 
90000 files was larger files like sound and video files or installation 
archives. The rest is from a Linux system backup, so smaller files. I 
likely move those elsewhere before I try again as I do not need these on 
flash anyway. However if the amount of files or data matters I could never 
know what amount of data I could write safely in one go. That is not 
acceptable to me.

2) Power management related to USB port. Cause I am using a laptop. It may 
have been that the Linux kernel decided to put the USB port the SSD was 
connected to into some kind of sleep state. However it was a constant 
rsync based copy workload. Yes, the kernel buffers data and the reads from 
Toshiba HD should be quite a bit slower than the Transcend SSD could 
handle the writes. I saw now more than 80-90 MiB/s coming from the hard 
disk. However I would doubt this lead to pauses of write activity of more 
than 30 seconds. Still it could be a thing.

Regarding further testing I am unsure whether to first test with BTRFS on 
top of LUKS – I do not like to store clear text data on the SSD – or with 
BCacheFS plus fixes which are 6.7.5 or 6.8-rc4 in just in the case the flush 
handling fixes would still have an influence on the issue at hand.

First I will have a look on how to see what USB power management options 
may be in place and how to tell Linux to keep the USB port the SSD is 
connected to at all times.

Let's see how this story unfolds. At least I am in no hurry about it.

Best,
-- 
Martin

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Alan Stern]

On Thu, Feb 15, 2024 at 12:09:20PM +0100, Martin Steigerwald wrote:
> Kent Overstreet - 12.02.24, 21:42:26 CET:
> 
> [thoughts about whether a cache flush / FUA request with write caches 
> disabled would be a no-op anyway]
> 
> > > I may test the Transcend XS2000 with BTRFS to see whether it makes a
> > > difference, however I really like to use it with BCacheFS and I do not
> > > really like to use LUKS for external devices. According to the kernel
> > > log I still don't really think those errors at the block layer were
> > > about anything filesystem specific, but what  do I know?
> > 
> > It's definitely not unheard of for one specific filesystem to be
> > tickling driver/device bugs and not others.
> > 
> > I wonder what it would take to dump the outstanding requests on device
> > timeout.
> 
> I got some reply back from Transcend support.
> 
> They brought up two possible issues:
> 
> 1) Copied to many files at once. I am not going to accept that one. An 
> external 4 TB SSD should handle writing 1,4 TB in about 215000 files, 
> coming from a slower Toshiba Canvio Basics external HD, just fine. About 
> 90000 files was larger files like sound and video files or installation 
> archives. The rest is from a Linux system backup, so smaller files. I 
> likely move those elsewhere before I try again as I do not need these on 
> flash anyway. However if the amount of files or data matters I could never 
> know what amount of data I could write safely in one go. That is not 
> acceptable to me.
> 
> 2) Power management related to USB port. Cause I am using a laptop. It may 
> have been that the Linux kernel decided to put the USB port the SSD was 
> connected to into some kind of sleep state. However it was a constant 
> rsync based copy workload. Yes, the kernel buffers data and the reads from 
> Toshiba HD should be quite a bit slower than the Transcend SSD could 
> handle the writes. I saw now more than 80-90 MiB/s coming from the hard 
> disk. However I would doubt this lead to pauses of write activity of more 
> than 30 seconds. Still it could be a thing.
> 
> Regarding further testing I am unsure whether to first test with BTRFS on 
> top of LUKS – I do not like to store clear text data on the SSD – or with 
> BCacheFS plus fixes which are 6.7.5 or 6.8-rc4 in just in the case the flush 
> handling fixes would still have an influence on the issue at hand.
> 
> First I will have a look on how to see what USB power management options 
> may be in place and how to tell Linux to keep the USB port the SSD is 
> connected to at all times.
> 
> Let's see how this story unfolds. At least I am in no hurry about it.

This may not be an issue of power management but rather one of 
insufficient power.  A laptop may not provide enough power through its 
USB ports for the Transcend SSD to work properly under load.

You can test this by connecting a powered UBS-3 hub between the laptop 
and the drive.

Alan Stern

Re: I/O errors while writing to external Transcend XS-2000 4TB SSD

[Martin Steigerwald]

Alan Stern - 15.02.24, 16:19:54 CET:
> > First I will have a look on how to see what USB power management
> > options may be in place and how to tell Linux to keep the USB port
> > the SSD is connected to at all times.
> > 
> > Let's see how this story unfolds. At least I am in no hurry about it.
> 
> This may not be an issue of power management but rather one of
> insufficient power.  A laptop may not provide enough power through its
> USB ports for the Transcend SSD to work properly under load.
> 
> You can test this by connecting a powered UBS-3 hub between the laptop
> and the drive.

Interesting idea. Maybe the Transcend XS-2000 4TB needs more power than 
the Sandisk Extreme Pro 2TB.

Not sure whether I have one at hand with USB-C here, cause my regular USB 
hub only has USB-A connectors. Need to look for one with enough USB-A and 
USB-C connectors as I use an USB hub as replacement for a docking station. 
But I do have at least optionally powered hub with USB-C one at another 
place. It does not have many ports. But for the task ahead one USB-C port 
is sufficient.

I will try this as well. Thanks.

Best,
-- 
Martin