SanDisk Extreme Pro w/btrfs Frozen: nodiscard per Western Digital

["John L. Poole" <jlpoole56@gmail.com>]

** Issue: I have 3 SanDisk Extreme Pro micro SD cards using btrfs
which are readable, but not writable; they are frozen.

** Background:  I purchased from Amazon (1) 5 SanDisk Extreme Pro (64GB)
microSDXC(tm) UHS-I cards in August, 2023.  They have been used
to test/build-out the GenPi64 Gentoo Linux for the Raspberry Pi(2).
I selected the Extreme Pro series because they had faster I/O ratings.
This is the first time I have used the Extreme Pro series, previously,
I have used other cards including the SanDisk Ultra and I have not
experienced any problems.

The GenPi64 image uses the btrfs file system.  /etc/fstab has:

PARTUUID="426c28a1-02"  /       btrfs   noatime, [LINE BREAK]
compress=zstd,ssd,discard,x-systemd.growfs      0       0

During my build-out I ran into the problem that my session's file
changes did not persist on reboots.  I'd install a Gentoo package
with "emerge", and/or I'd create/edit a file, and after reboot
those changes were missing.  Although dmesg revealed some warnings,
there was nothing in the start-up post or system to suggests there was
any problem.  So a non-technical user would not be alerted to any
problems and on reboot suddenly and without fanfare learn that
the files from the previous session are not saved.  This would be btrfs
doing its job: not integrating that last set of changes
because it detected a corruption.

My build-out has been on a Raspberry Pi 4B with 8 MB(3) and
the Raspberry Pi forums suggest that the SD card I/O has a
limitation of 50 MBs/second.(4)

What seemed to trigger the problem was when I emerged the package
sys-kernel/raspberrypi-sources.  During the processing, there are over 1 GB
of files that are to be staged and the emerge would fail during the final
transfer of those files.

Here's a lsblk output:

eos /tmp # date; lsblk --output 
NAME,FSSIZE,FSUSED,FSTYPE,FSVER,LABEL,UUID,MODE,STATE,DISC-ALN
Thu Oct 12 10:44:29 PDT 2023
NAME        FSSIZE FSUSED FSTYPE FSVER LABEL 
UUID                                 MODE       STATE DISC-ALN
mmcblk0 brw-rw----              0
+-mmcblk0p1               vfat   FAT16 
932D-ABE2                            brw-rw----        3145728
+-mmcblk0p2  59.2G   2.2G btrfs d15fd42d-7572-4d9f-b5fb-eadbe3d70d5c 
brw-rw----              0

I do not know what DISC-ALN means other than "discard alignment
offset" which "lsblk --help" tells me, but share the following.
The GenPi64 image that I burned was a 4GB file. When I boot up,
there is a script at boot-up which will expand the root (btrfs)
file system to use up any available space on the card, hence a one-time
expansion occurs changing the root file system from its less than
4 GB size to 59.2 GB. The DISC-ALN value for the vfat system,
3145728, concerns me since the vfat is suppose to be 256 MBs.
I'm just wondering if that 3145728 value is somewhat in interference
with the expanded btrfs system and may be contributing to the frozen
read-only mode of the card.

I began to use the "emerge sys-kernel/raspberrypi-sources" as a test
which would cause the SD card to become unwritable.

I opened a ticket with Western Digital and their response was:

     [--- start response ---]

     Please also refer to the following command to run BEFORE doing any
     major writes to card after installing OS.

     Using the appropriate format command can avoid the Read-Only mode
     issue. File system format tools such as mkfs and mke2fs
     have the nodiscard flag.

     Format SanDisk card with ext4 FS:

       Linux Host is needed
       Native microSD/SD support.
       Discard feature supported on Host and card
       Use the nodiscard flag.
       Command: $ mkfs.ext4 -E nodiscard -F /dev/mmcblk0p1
     [--- end response ---]

I altered my /etc/fstab on a newly burned enhanced image and changed
"discard" to "nodiscard" and I successfully installed
"sys-kernel/raspberrypi-sources".  Furthermore, subsequent activity on
the image seems normal and it appears the mount specification of "nodiscard"
has solved my problem... for a micro disk not already frozen.

Western Digital has offered to replace whatever disks with like kind; I had
suggested they simply provide me Ultras and refund the price difference, but
they insist that they can only provide Extreme Pro replacements.
Therefore, I am stuck with five Extreme Pro Cards that have
a propensity to become unusable.

I thought I would try to resurrect the three frozen cards before
resolving my open ticket with Western Digital.  To that end and
working with the principal developer of the GenPi64 project, we devised 
these
steps to test whether the cards could be salvaged:

     # need to be root to write to card
     su
     # create a test file of 512 zeros
     date; time dd if=/dev/zero of=/tmp/1_block-512_zeros.raw count=1
     # write test file to card
     date; time dd if=/tmp/1_block-512_zeros.raw  of=/dev/mmcblk0
     # copy the first 512 bytes out to a file
     date; time dd if=/dev/mmcblk0 of=/tmp/1_block-512_from_card_A.raw 
count=1
     # view the files with hexdump abbreviate with head
     hexdump 1_block-512_from_card_A.raw |head -n 1
     hexdump 1_block-512_zeros.raw | head -n 1

     eos /tmp #      # create a test file of 512 zeros
     date; time dd if=/dev/zero of=/tmp/1_block-512_zeros.raw count=1
     Thu Oct 12 10:26:11 PDT 2023
     1+0 records in
     1+0 records out
     512 bytes copied, 8.6744e-05 s, 5.9 MB/s

     real    0m0.001s
     user    0m0.000s
     sys     0m0.001s
     eos /tmp #      # write test file to card
     date; time dd if=/tmp/1_block-512_zeros.raw  of=/dev/mmcblk0
     Thu Oct 12 10:26:23 PDT 2023
     1+0 records in
     1+0 records out
     512 bytes copied, 0.0211458 s, 24.2 kB/s

     real    0m0.023s
     user    0m0.002s
     sys     0m0.000s
     eos /tmp #      # copy the first 512 bytes out to a file
     date; time dd if=/dev/mmcblk0 of=/tmp/1_block-512_from_card_A.raw 
count=1
     Thu Oct 12 10:26:34 PDT 2023
     1+0 records in
     1+0 records out
     512 bytes copied, 0.0202807 s, 25.2 kB/s

     real    0m0.022s
     user    0m0.002s
     sys     0m0.000s
     eos /tmp #      # view the files with hexdump abbreviate with head
     hexdump 1_block-512_from_card_A.raw |head -n 1
     hexdump 1_block-512_zeros.raw | head -n 1
     0000000 b8fa 1000 d08e 00bc b8b0 0000 d88e c08e
     0000000 0000 0000 0000 0000 0000 0000 0000 0000
     eos /tmp #

The above shows that my attempts to write to the card fail; both
rows should be zeros.

Moreover, I've tried using fdisk and deleting the partitions and
writing my changes and fdisk appears to execute the task without
error or warning, yet the partitions persist and were not deleted.

There are currently pending 3 class action lawsuits against Western Digital
over their SSD drives where customers would experience lost data. (5)

Western Digital has a web page "Format Using Linux, Raspberry Pi,
or Android Host Results in Read Only Mode on a SanDisk SD or
microSD Memory Card "(6).

** Questions:

1) Is there some way to write to the card, or some tools I can use
so I can resurrect the cards and restore them to normal use?
2) It looks like the use "discard" for SanDisk Extreme Pro cards causes
problems?  I am told btrfs uses discard by default.
3) Is there some assay anyone wants to suggest I try to see if I can trigger
the read-only condition such as I have stumbled upon three times?
I have 2 cards that I can still write to.  I am willing to try anything 
someone
suggests that would help shed light onto this problem.  I was hoping to
garner the attention of an engineer from Western Digital.

Before I accept Western Digital's offer to replace the cards, I'd like 
to know
what is happening or what is causing them to lock up.  I do not
want to encounter the problem again. What is troubling is that
an inexperienced user will not detect anything wrong until after a 
reboot when they
discover everything from the previous session did not persist. This
kind of media failure could discourage the use of the btrfs based image.

Thank you,

John Poole

-----------------------

Footnotes
(1) 
https://www.amazon.com/SanDisk-Extreme-microSDTM-Adapter-SDSQXCU-064G-GN6MA/dp/B09X7BYSFG/ref=sr_1_2?crid=20PTT2TLYVAVM&keywords=SanDisk+Extreme+Pro+%2864GB%29+micro+sd&qid=1697127725&sprefix=sandisk+extreme+pro+64gb+micro+sd%2Caps%2C133&sr=8-2
(2) https://github.com/GenPi64
(3) Specs on Raspberry Pi 4B: 
https://www.raspberrypi.com/documentation/computers/raspberry-pi.html.
(4) "SD micro limitations in Raspberry Pi hardware?" at
https://forums.raspberrypi.com/viewtopic.php?t=356431
(5) 
https://arstechnica.com/gadgets/2023/08/sandisk-extreme-ssds-are-worthless-multiple-lawsuits-against-wd-say/
(6) 
https://support-en.wd.com/app/answers/detailweb/a_id/50076/~/format-using-linux%2C-raspberry-pi%2C-or-android-host-results-in-read-only-mode-on