trim not working and irreparable errors from btrfsck

trim not working and irreparable errors from btrfsck

[Christian]

Hi,

I have a btrfs partions on my laptop containing / and /home. Recently I 
noticed that trim (fstrim) didn't work anymore. It always tells me there 
nothing to trim which can not be correct. I then tried to to run btrfsck 
--repair and this is the result:

ubuntu@ubuntu:~$ sudo btrfsck /dev/sda3 --repair
enabling repair mode
Fixed 0 roots.
Checking filesystem on /dev/sda3
UUID: 3d52dc93-c89f-453f-965d-8601d11e7710
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
root 257 inode 353819 errors 400, nbytes wrong
root 2260 inode 353819 errors 400, nbytes wrong
root 2262 inode 353819 errors 400, nbytes wrong
root 2264 inode 353819 errors 400, nbytes wrong
root 2266 inode 353819 errors 400, nbytes wrong
root 2268 inode 353819 errors 400, nbytes wrong
root 2270 inode 353819 errors 400, nbytes wrong
root 2273 inode 353819 errors 400, nbytes wrong
root 2275 inode 353819 errors 400, nbytes wrong
root 2277 inode 353819 errors 400, nbytes wrong
root 2279 inode 353819 errors 400, nbytes wrong
root 2281 inode 353819 errors 400, nbytes wrong
root 2283 inode 353819 errors 400, nbytes wrong
root 2285 inode 353819 errors 400, nbytes wrong
root 2287 inode 353819 errors 400, nbytes wrong
root 2289 inode 353819 errors 400, nbytes wrong
root 2291 inode 353819 errors 400, nbytes wrong
root 2293 inode 353819 errors 400, nbytes wrong
root 2295 inode 353819 errors 400, nbytes wrong
root 2297 inode 353819 errors 400, nbytes wrong
root 2299 inode 353819 errors 400, nbytes wrong
root 2301 inode 353819 errors 400, nbytes wrong
root 2303 inode 353819 errors 400, nbytes wrong
root 2305 inode 353819 errors 400, nbytes wrong
root 2317 inode 353819 errors 400, nbytes wrong
root 2320 inode 353819 errors 400, nbytes wrong
root 2326 inode 353819 errors 400, nbytes wrong
root 2556 inode 353819 errors 400, nbytes wrong
root 2574 inode 353819 errors 400, nbytes wrong
root 2592 inode 353819 errors 400, nbytes wrong
root 2601 inode 353819 errors 400, nbytes wrong
root 2617 inode 353819 errors 400, nbytes wrong
root 2620 inode 353819 errors 400, nbytes wrong
root 2621 inode 353819 errors 400, nbytes wrong
root 2624 inode 353819 errors 400, nbytes wrong
root 2625 inode 353819 errors 400, nbytes wrong
root 2626 inode 353819 errors 400, nbytes wrong
root 2627 inode 353819 errors 400, nbytes wrong
root 2628 inode 353819 errors 400, nbytes wrong
root 2629 inode 353819 errors 400, nbytes wrong
root 2630 inode 353819 errors 400, nbytes wrong
root 2631 inode 353819 errors 400, nbytes wrong
root 2632 inode 353819 errors 400, nbytes wrong
found 49406146650 bytes used err is 1
total csum bytes: 117817456
total tree bytes: 792248320
total fs tree bytes: 604520448
total extent tree bytes: 52101120
btree space waste bytes: 180897891
file data blocks allocated: 195314843648
  referenced 135991529472
Btrfs v3.17

I get the exact same result every time I run btrfsck --repair. If the 
file system irreparable or is there something I can do to save it?


-- 
//Christian

Re: trim not working and irreparable errors from btrfsck

[Chris Murphy]

I experienced the errors 400 problem on an HDD and how I fixed it is
in comment 2 in this bug:
https://bugzilla.kernel.org/show_bug.cgi?id=90071

The gist is to find out what file is affected by finding the
path/filename for the inode with the error and then deleting it.
You'll need to recover the file from backups. Once you do this the
problem is resolved, the file system itself is OK. (At least it was in
my case.)

I don't know that the source of the problem you're having has anything
to do with trim, but I recommend not using discard or fstrim at all
until you isolate  what's causing it. There are some trim bugs that
have been fixed in newer kernels that sound like the problem you're
having; and there are definitely some SSDs out there with trim
problems where the wrong sectors get trimmed. It shows up as files
with 512 byte holes of zeros in them. Of course, that's when the bug
affects data. If it affects metadata it can obliterate the file system
beyond repair.

Chris Murphy

Re: trim not working and irreparable errors from btrfsck

[Chris Murphy]

On Wed, Jun 17, 2015 at 6:56 AM, Christian Dysthe <cdysthe@gmail.com> wrote:
> Hi,
>
> Sorry for asking more about this. I'm not a developer but trying to learn.
> In my case I get several errors like this one:
>
> root 2625 inode 353819 errors 400, nbytes wrong
>
> Is it inode 353819 I should focus on and what is the number after "root", in
> this case 2625?

I'm going to guess it's tree root 2625, which is the same thing as fs
tree, which is the same thing as subvolume. Each subvolume has its own
inodes. So on a given Btrfs volume, an inode number can exist more
than once, but in separate subvolumes. When you use btrfs inspect
inode it will list all files with that inode number, but only the one
in subvol ID 2625 is what you care about deleting and replacing.

-- 
Chris Murphy

Re: trim not working and irreparable errors from btrfsck

[Christian]

On 06/17/2015 10:22 AM, Chris Murphy wrote:
> On Wed, Jun 17, 2015 at 6:56 AM, Christian Dysthe <cdysthe@gmail.com> wrote:
>> Hi,
>>
>> Sorry for asking more about this. I'm not a developer but trying to learn.
>> In my case I get several errors like this one:
>>
>> root 2625 inode 353819 errors 400, nbytes wrong
>>
>> Is it inode 353819 I should focus on and what is the number after "root", in
>> this case 2625?
>
> I'm going to guess it's tree root 2625, which is the same thing as fs
> tree, which is the same thing as subvolume. Each subvolume has its own
> inodes. So on a given Btrfs volume, an inode number can exist more
> than once, but in separate subvolumes. When you use btrfs inspect
> inode it will list all files with that inode number, but only the one
> in subvol ID 2625 is what you care about deleting and replacing.
>
Thanks! Deleting the file for that inode took care of it. No more 
errors. Restored it from a backup.

However, fstrim still gives me "0 B (0 bytes) trimmed, so that may be 
another problem. Is there a way to check if trim works?

-- 
//Christian

Re: trim not working and irreparable errors from btrfsck

[Chris Murphy]

On Wed, Jun 17, 2015 at 8:33 AM, Christian <cdysthe@gmail.com> wrote:
> On 06/17/2015 10:22 AM, Chris Murphy wrote:
>>
>> On Wed, Jun 17, 2015 at 6:56 AM, Christian Dysthe <cdysthe@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> Sorry for asking more about this. I'm not a developer but trying to
>>> learn.
>>> In my case I get several errors like this one:
>>>
>>> root 2625 inode 353819 errors 400, nbytes wrong
>>>
>>> Is it inode 353819 I should focus on and what is the number after "root",
>>> in
>>> this case 2625?
>>
>>
>> I'm going to guess it's tree root 2625, which is the same thing as fs
>> tree, which is the same thing as subvolume. Each subvolume has its own
>> inodes. So on a given Btrfs volume, an inode number can exist more
>> than once, but in separate subvolumes. When you use btrfs inspect
>> inode it will list all files with that inode number, but only the one
>> in subvol ID 2625 is what you care about deleting and replacing.
>>
> Thanks! Deleting the file for that inode took care of it. No more errors.
> Restored it from a backup.
>
> However, fstrim still gives me "0 B (0 bytes) trimmed, so that may be
> another problem. Is there a way to check if trim works?

That sounds like maybe your SSD is blacklisted for trim, is all I can
think of. So trim shouldn't be the cause of the problem if it's being
blacklisted. The recent problems appear to be around newer SSDs that
support queue trim and newer kernels that issue queued trim. There
have been some patches related to trim to the kernel, but the
existence of blacklisting and claims of bugs in firmware make it
difficult to test and isolate.

http://techreport.com/news/28473/some-samsung-ssds-may-suffer-from-a-buggy-trim-implementation

-- 
Chris Murphy

Re: trim not working and irreparable errors from btrfsck

[Christian]

On 06/17/2015 11:28 AM, Chris Murphy wrote:

>>
>> However, fstrim still gives me "0 B (0 bytes) trimmed, so that may be
>> another problem. Is there a way to check if trim works?
>
> That sounds like maybe your SSD is blacklisted for trim, is all I can
> think of. So trim shouldn't be the cause of the problem if it's being
> blacklisted. The recent problems appear to be around newer SSDs that
> support queue trim and newer kernels that issue queued trim. There
> have been some patches related to trim to the kernel, but the
> existence of blacklisting and claims of bugs in firmware make it
> difficult to test and isolate.
>
> http://techreport.com/news/28473/some-samsung-ssds-may-suffer-from-a-buggy-trim-implementation
>
This is an Intel SSD in a Lenovo Thinkpad X1 Carbon. Trim worked until a 
few weeks ago and still works for my small ext4 boot partition (just ran 
it to check). I will keep looking for a solution. Thanks!

-- 
//Christian

Re: trim not working and irreparable errors from btrfsck

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

On 2015-06-17 11:40, Christian wrote:
> On 06/17/2015 11:28 AM, Chris Murphy wrote:
>
>>>
>>> However, fstrim still gives me "0 B (0 bytes) trimmed, so that may be
>>> another problem. Is there a way to check if trim works?
>>
>> That sounds like maybe your SSD is blacklisted for trim, is all I can
>> think of. So trim shouldn't be the cause of the problem if it's being
>> blacklisted. The recent problems appear to be around newer SSDs that
>> support queue trim and newer kernels that issue queued trim. There
>> have been some patches related to trim to the kernel, but the
>> existence of blacklisting and claims of bugs in firmware make it
>> difficult to test and isolate.
>>
>> http://techreport.com/news/28473/some-samsung-ssds-may-suffer-from-a-buggy-trim-implementation
>>
>>
> This is an Intel SSD in a Lenovo Thinkpad X1 Carbon. Trim worked until a
> few weeks ago and still works for my small ext4 boot partition (just ran
> it to check). I will keep looking for a solution. Thanks!
>
I'm seeing the same issue here, but with a Crucial brand SSD.  Somewhat 
interestingly, I don't see any issues like this with BTRFS on top of 
LVM's thin-provisioning volumes, or with any other filesystems, so I 
think it has something to do with how BTRFS is reporting unused space or 
how it is submitting the discard requests.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]

Re: trim not working and irreparable errors from btrfsck

[Duncan]

Austin S Hemmelgarn posted on Wed, 17 Jun 2015 13:17:22 -0400 as
excerpted:

> On 2015-06-17 11:40, Christian wrote:
>> On 06/17/2015 11:28 AM, Chris Murphy wrote:
>>
>>
>>>> However, fstrim still gives me "0 B (0 bytes) trimmed, so that may be
>>>> another problem. Is there a way to check if trim works?
>>>
>>> That sounds like maybe your SSD is blacklisted for trim, is all I can
>>> think of. So trim shouldn't be the cause of the problem if it's being
>>> blacklisted. The recent problems appear to be around newer SSDs that
>>> support queue trim and newer kernels that issue queued trim. There
>>> have been some patches related to trim to the kernel, but the
>>> existence of blacklisting and claims of bugs in firmware make it
>>> difficult to test and isolate.
>>>
>>> http://techreport.com/news/28473/some-samsung-ssds-may-suffer-from-a-
buggy-trim-implementation
>>>
>>>
>> This is an Intel SSD in a Lenovo Thinkpad X1 Carbon. Trim worked until
>> a few weeks ago and still works for my small ext4 boot partition (just
>> ran it to check). I will keep looking for a solution. Thanks!
>>
> I'm seeing the same issue here, but with a Crucial brand SSD.  Somewhat
> interestingly, I don't see any issues like this with BTRFS on top of
> LVM's thin-provisioning volumes, or with any other filesystems, so I
> think it has something to do with how BTRFS is reporting unused space or
> how it is submitting the discard requests.

FWIW, there's a current btrfs patch in progress that relates to problems 
with btrfs trim.

But while I do have SSDs, I purposefully overprovisioned them by nearly 
100% (IOW I partitioned only about 55%, the rest is entirely unused), so 
trim isn't as critical here as it is for many.  I don't use the discard 
mount option, and have a systemd timer job setup to automate my fstrims 
and don't worry about the output too much, so I haven't been following 
the patch progress /that/ closely.

But I do know that recent kernel btrfs trims (either fstrim or discard 
mount option triggered) haven't been working as originally intended due 
to some bug, and this patch is supposed to fix it.

I'd thus conclude that you're very likely hitting this known issue, and 
that either for 4.1 or 4.2 (again, I'm not following progress that 
closely, and don't remember for sure if it's in 4.1, altho I've been 
running the rcs since rc6 or so), the problem should be fixed as that 
patch gets into mainline.

Anyone wishing to investigate further can of course check the list (and/
or possibly the kernel's git log) for discard/trim related patches and 
follow the progress once found.

... Actually, just checked myself.  Looks like the patches were first 
posted on March 30 @ 15:12:17 -0400 or so (that's the time for one of 
them).  There's one for the discard mount option, and another for FITRIM 
(which may or may not be a typo for FSTRIM, I'm not actually sure).  Jeff 
Mahoney <jeffm@suse.com> author.  That should be enough to find the 
threads.  And I don't see the patches in the late 4.1-rc I'm running so 
either my git log search foo is bad or it'll be (at least) 4.2.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: trim not working and irreparable errors from btrfsck

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 6/18/15 1:25 AM, Duncan wrote:
> Austin S Hemmelgarn posted on Wed, 17 Jun 2015 13:17:22 -0400 as 
> excerpted:
> 
>> On 2015-06-17 11:40, Christian wrote:
>>> On 06/17/2015 11:28 AM, Chris Murphy wrote:
>>> 
>>> 
>>>>> However, fstrim still gives me "0 B (0 bytes) trimmed, so
>>>>> that may be another problem. Is there a way to check if
>>>>> trim works?
>>>> 
>>>> That sounds like maybe your SSD is blacklisted for trim, is
>>>> all I can think of. So trim shouldn't be the cause of the
>>>> problem if it's being blacklisted. The recent problems appear
>>>> to be around newer SSDs that support queue trim and newer
>>>> kernels that issue queued trim. There have been some patches
>>>> related to trim to the kernel, but the existence of
>>>> blacklisting and claims of bugs in firmware make it difficult
>>>> to test and isolate.
>>>> 
>>>> http://techreport.com/news/28473/some-samsung-ssds-may-suffer-from-
a-
>
>>>> 
buggy-trim-implementation
>>>> 
>>>> 
>>> This is an Intel SSD in a Lenovo Thinkpad X1 Carbon. Trim
>>> worked until a few weeks ago and still works for my small ext4
>>> boot partition (just ran it to check). I will keep looking for
>>> a solution. Thanks!
>>> 
>> I'm seeing the same issue here, but with a Crucial brand SSD.
>> Somewhat interestingly, I don't see any issues like this with
>> BTRFS on top of LVM's thin-provisioning volumes, or with any
>> other filesystems, so I think it has something to do with how
>> BTRFS is reporting unused space or how it is submitting the
>> discard requests.
> 
> FWIW, there's a current btrfs patch in progress that relates to
> problems with btrfs trim.
> 
> But while I do have SSDs, I purposefully overprovisioned them by
> nearly 100% (IOW I partitioned only about 55%, the rest is entirely
> unused), so trim isn't as critical here as it is for many.  I don't
> use the discard mount option, and have a systemd timer job setup to
> automate my fstrims and don't worry about the output too much, so I
> haven't been following the patch progress /that/ closely.
> 
> But I do know that recent kernel btrfs trims (either fstrim or
> discard mount option triggered) haven't been working as originally
> intended due to some bug, and this patch is supposed to fix it.
> 
> I'd thus conclude that you're very likely hitting this known issue,
> and that either for 4.1 or 4.2 (again, I'm not following progress
> that closely, and don't remember for sure if it's in 4.1, altho
> I've been running the rcs since rc6 or so), the problem should be
> fixed as that patch gets into mainline.
> 
> Anyone wishing to investigate further can of course check the list
> (and/ or possibly the kernel's git log) for discard/trim related
> patches and follow the progress once found.
> 
> ... Actually, just checked myself.  Looks like the patches were
> first posted on March 30 @ 15:12:17 -0400 or so (that's the time
> for one of them).  There's one for the discard mount option, and
> another for FITRIM (which may or may not be a typo for FSTRIM, I'm
> not actually sure).  Jeff Mahoney <jeffm@suse.com> author.  That
> should be enough to find the threads.  And I don't see the patches
> in the late 4.1-rc I'm running so either my git log search foo is
> bad or it'll be (at least) 4.2.

It's not a typo. FITRIM is the name of the ioctl that fstrim calls.

The final version of that patch set is ready to go.  Mostly.  It
probably needs to be re-integrated now.  The reason it was delayed for
inclusion is that it makes other bugs more obvious and irrecoverable
since the data is completely gone.

I'm not sure what Chris's timeline for inclusion is.

- -Jeff

- -- 
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)

iQIcBAEBAgAGBQJVzgU/AAoJEB57S2MheeWyjSwQALOLfzNLiuAdxBXDnP076Pq7
8m2F2DtTRxuDCwBmnlgZtX3QuWK/J1HVRpAO/aC6WQkOo3uRNFrG4xK45EOTA5hH
VBNtAFooMreicFQq5ZE6i+4yEdV8D4YRSoVn7+GrjL40IjiP8u7HXtDGw0x4ugGI
iVNf3yipaTZtlRcjGt91dfW3w3D8RpjUK3z7RwSOEy3C8GP90omRWVkYV/jcFIo4
hqFMZ77hisRLf1aCFxXlO14ERyMpLPtC3HOBMLHRrdpjPp/f4XnXyFmFA0kbOX8S
dwS9qPRmlnS5Lif2XMXK0a6aA0HK7sN/ghMigAh9t4zHwDkuDpAd6OWVEuCMMpCY
uN2KyuNsjam2DxJHQVulNu1xlS/sGedfh8p66lC29fkB8ZpyGp4fnK1N4MVRdk8R
4o/emRb+vg7CTZ3fvss7Af6w+m22GISO43Q1MWr6Hr1Ll2y0DWL1IaB/zky8sr/5
u6E5RI7DOvbFyC31dGqvh5WQDIPrTxRoDMJL+pSOkF4CM5SM1uHak4IgUqfZ85hr
MXVhRHFmH9UXRTFrkxzAV2wmSNpl2ki2pX5ItB6+c4fMMStb5dynThv27R69xxHf
mn8qZBuwc5iXXsPJ9dUAxTRoquOw9Rd/1fz4S/oLH6xOrtlNlLa2HFour4Ofp16h
3e6CvcV+h4/sz0PYpYSQ
=ZiEh
-----END PGP SIGNATURE-----

RE: trim not working and irreparable errors from btrfsck

[Paul Jones]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1801 bytes --]

> -----Original Message-----
> From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
> owner@vger.kernel.org] On Behalf Of Christian
> Sent: Thursday, 18 June 2015 12:34 AM
> To: linux-btrfs@vger.kernel.org
> Subject: Re: trim not working and irreparable errors from btrfsck
> 
> On 06/17/2015 10:22 AM, Chris Murphy wrote:
> > On Wed, Jun 17, 2015 at 6:56 AM, Christian Dysthe <cdysthe@gmail.com>
> wrote:
> >> Hi,
> >>
> >> Sorry for asking more about this. I'm not a developer but trying to learn.
> >> In my case I get several errors like this one:
> >>
> >> root 2625 inode 353819 errors 400, nbytes wrong
> >>
> >> Is it inode 353819 I should focus on and what is the number after
> >> "root", in this case 2625?
> >
> > I'm going to guess it's tree root 2625, which is the same thing as fs
> > tree, which is the same thing as subvolume. Each subvolume has its own
> > inodes. So on a given Btrfs volume, an inode number can exist more
> > than once, but in separate subvolumes. When you use btrfs inspect
> > inode it will list all files with that inode number, but only the one
> > in subvol ID 2625 is what you care about deleting and replacing.
> >
> Thanks! Deleting the file for that inode took care of it. No more errors.
> Restored it from a backup.
> 
> However, fstrim still gives me "0 B (0 bytes) trimmed, so that may be another
> problem. Is there a way to check if trim works?

I've got the same problem. I've got 2 SSDs with 2 partitions in RAID1, fstrim always works on the 2nd partition but not the first. There are no errors on either filesystem that I know of, but the first one is root so I can't take it offline to run btrfs check.

Paul.
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: trim not working and irreparable errors from btrfsck

[Chris Murphy]

File a new bug at bugzilla.kernel.org describing this problem. Include
make/model of all involved SSDs, which you can get from smartctl or
hdparm. And then do a strace fstrim on the working and non-working
volumes, saving the output to separate files and attaching them to the
bug report. And then it's probably best to create a new list thread to
post the URL for the bug, since this thread is really about two
problems that may not be related.

Chris Murphy

Re: trim not working and irreparable errors from btrfsck

[Chris Murphy]

And you might as well just attach a full dmesg to the bug report too.
Who knows there might be something buried in there that's useful. The
cleanest approach is to reboot and then reproduce all of this with
fstrim on each volume, then capture the dmesg to a file. Then do the
strace fstrim for each volume. But a bunch of kernel message clutter
may be better than no messages at all.


Chris Murphy

RE: trim not working and irreparable errors from btrfsck

[Lutz Euler]

Hi Christian, Paul and Austin,

Christian wrote:
> However, fstrim still gives me "0 B (0 bytes) trimmed, so that may be
> another problem. Is there a way to check if trim works?

Paul wrote:
> I've got the same problem. I've got 2 SSDs with 2 partitions in RAID1,
> fstrim always works on the 2nd partition but not the first. There are
> no errors on either filesystem that I know of, but the first one is
> root so I can't take it offline to run btrfs check.

Austin wrote:
> I'm seeing the same issue here, but with a Crucial brand SSD.
> Somewhat interestingly, I don't see any issues like this with BTRFS on
> top of LVM's thin-provisioning volumes, or with any other filesystems,
> so I think it has something to do with how BTRFS is reporting unused
> space or how it is submitting the discard requests.

Probably you all suffer from the same problem I had a few years ago.
It is a bug in how btrfs implements fstrim.

To check whether you are a victim of this bug simply run:

# btrfs-debug-tree /dev/whatever | grep 'FIRST_CHUNK_TREE CHUNK_ITEM'

where /dev/whatever is a device of your filesystem, and interrupt after
the first several output lines with C-c. (Officially the filesystem
should be unmounted when running btrfs-debug-tree, but that is not
necessary as we only read from it and the relevant data doesn't change
very often.)

You get something like:

  item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 0)
  item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 12947816448)
  item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 14021558272)
  ...

(This output is from an old version of btrfs-progs. I understand newer
version are more verbose, but you should nevertheless easily be able to
interpret the output).

If the first number different from 0 (here, the 12947816448) is larger
than the sum of the sizes of the devices the filesystem consists of,
bingo.

This has been discussed already in the past and there is a patch.

Please see for the patch:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg40618.html

and for the background:
http://comments.gmane.org/gmane.comp.file-systems.btrfs/15597

Kind regards,

Lutz Euler
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

RE: trim not working and irreparable errors from btrfsck

[Paul Jones]

> -----Original Message-----
> From: Lutz Euler [mailto:lutz.euler@freenet.de]
> Sent: Sunday, 21 June 2015 12:11 AM
> To: Christian; Paul Jones; Austin S Hemmelgarn
> Cc: linux-btrfs@vger.kernel.org
> Subject: RE: trim not working and irreparable errors from btrfsck
> 
> Hi Christian, Paul and Austin,
> 
> Christian wrote:
> > However, fstrim still gives me "0 B (0 bytes) trimmed, so that may be
> > another problem. Is there a way to check if trim works?
> 
> Paul wrote:
> > I've got the same problem. I've got 2 SSDs with 2 partitions in RAID1,
> > fstrim always works on the 2nd partition but not the first. There are
> > no errors on either filesystem that I know of, but the first one is
> > root so I can't take it offline to run btrfs check.
> 
> Austin wrote:
> > I'm seeing the same issue here, but with a Crucial brand SSD.
> > Somewhat interestingly, I don't see any issues like this with BTRFS on
> > top of LVM's thin-provisioning volumes, or with any other filesystems,
> > so I think it has something to do with how BTRFS is reporting unused
> > space or how it is submitting the discard requests.
> 
> Probably you all suffer from the same problem I had a few years ago.
> It is a bug in how btrfs implements fstrim.
> 
> To check whether you are a victim of this bug simply run:
> 
> # btrfs-debug-tree /dev/whatever | grep 'FIRST_CHUNK_TREE
> CHUNK_ITEM'
> 
> where /dev/whatever is a device of your filesystem, and interrupt after the
> first several output lines with C-c. (Officially the filesystem should be
> unmounted when running btrfs-debug-tree, but that is not necessary as we
> only read from it and the relevant data doesn't change very often.)
> 
> You get something like:
> 
>   item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 0)
>   item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 12947816448)
>   item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 14021558272)
>   ...
> 
> (This output is from an old version of btrfs-progs. I understand newer version
> are more verbose, but you should nevertheless easily be able to interpret
> the output).
> 
> If the first number different from 0 (here, the 12947816448) is larger than the
> sum of the sizes of the devices the filesystem consists of, bingo.
> 
> This has been discussed already in the past and there is a patch.
> 
> Please see for the patch:
> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg40618.html
> 
> and for the background:
> http://comments.gmane.org/gmane.comp.file-systems.btrfs/15597
> 
> Kind regards,
> 
> Lutz Euler

I tried the test and the numbers I was getting seemed reasonable, however I went ahead and applied the patch anyway. Trim now works correctly!

Thanks,
Paul.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

Re: trim not working and irreparable errors from btrfsck

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 4767 bytes --]

Am Sun, 21 Jun 2015 07:21:03 +0000
schrieb Paul Jones <paul@pauljones.id.au>:

> > -----Original Message-----
> > From: Lutz Euler [mailto:lutz.euler@freenet.de]
> > Sent: Sunday, 21 June 2015 12:11 AM
> > To: Christian; Paul Jones; Austin S Hemmelgarn
> > Cc: linux-btrfs@vger.kernel.org
> > Subject: RE: trim not working and irreparable errors from btrfsck
> > 
> > Hi Christian, Paul and Austin,
> > 
> > Christian wrote:
> > > However, fstrim still gives me "0 B (0 bytes) trimmed, so that may be
> > > another problem. Is there a way to check if trim works?
> > 
> > Paul wrote:
> > > I've got the same problem. I've got 2 SSDs with 2 partitions in RAID1,
> > > fstrim always works on the 2nd partition but not the first. There are
> > > no errors on either filesystem that I know of, but the first one is
> > > root so I can't take it offline to run btrfs check.
> > 
> > Austin wrote:
> > > I'm seeing the same issue here, but with a Crucial brand SSD.
> > > Somewhat interestingly, I don't see any issues like this with BTRFS on
> > > top of LVM's thin-provisioning volumes, or with any other filesystems,
> > > so I think it has something to do with how BTRFS is reporting unused
> > > space or how it is submitting the discard requests.
> > 
> > Probably you all suffer from the same problem I had a few years ago.
> > It is a bug in how btrfs implements fstrim.
> > 
> > To check whether you are a victim of this bug simply run:
> > 
> > # btrfs-debug-tree /dev/whatever | grep 'FIRST_CHUNK_TREE
> > CHUNK_ITEM'
> > 
> > where /dev/whatever is a device of your filesystem, and interrupt after the
> > first several output lines with C-c. (Officially the filesystem should be
> > unmounted when running btrfs-debug-tree, but that is not necessary as we
> > only read from it and the relevant data doesn't change very often.)
> > 
> > You get something like:
> > 
> >   item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 0)
> >   item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 12947816448)
> >   item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 14021558272)
> >   ...
> > 
> > (This output is from an old version of btrfs-progs. I understand newer version
> > are more verbose, but you should nevertheless easily be able to interpret
> > the output).
> > 
> > If the first number different from 0 (here, the 12947816448) is larger than the
> > sum of the sizes of the devices the filesystem consists of, bingo.
> > 
> > This has been discussed already in the past and there is a patch.
> > 
> > Please see for the patch:
> > http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg40618.html
> > 
> > and for the background:
> > http://comments.gmane.org/gmane.comp.file-systems.btrfs/15597
> > 
> > Kind regards,
> > 
> > Lutz Euler
> 
> I tried the test and the numbers I was getting seemed reasonable, however I went ahead and applied the patch anyway. Trim now works correctly!
> 
> Thanks,
> Paul.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

Speaking as a user, since "fstrim -av" still always outputs 0 bytes trimmed
on my system: what's the status of this?  Did anybody ever file a bug report?
There was also that other thread, "fstrim not working on one of three BTRFS
filesystems", that also never went anywhere.

I take it from this that my SSD has been running untrimmed for quite a while
now?

(FWIW, queued trim is blocked by my kernel (it's "forced_unqueued"), but fstrim
should still start an unqueued trim, right?)

# uname -a        
Linux thetick 4.1.4-gentoo #1 SMP PREEMPT Tue Aug 4 21:58:41 CEST 2015 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ AuthenticAMD GNU/Linux
# btrfs --version
btrfs-progs v4.1.2
# btrfs filesystem show 
Label: 'MARCEC_ROOT'  uuid: 0267d8b3-a074-460a-832d-5d5fd36bae64
        Total devices 1 FS bytes used 56.59GiB
        devid    1 size 107.79GiB used 69.03GiB path /dev/sda1

Label: 'MARCEC_STORAGE'  uuid: 472c9290-3ff2-4096-9c47-0612d3a52cef
        Total devices 2 FS bytes used 597.75GiB
        devid    1 size 931.51GiB used 600.03GiB path /dev/sdc
        devid    2 size 931.51GiB used 600.03GiB path /dev/sdb

Label: 'MARCEC_BACKUP'  uuid: f97b3cda-15e8-418b-bb9b-235391ef2a38
        Total devices 1 FS bytes used 807.59GiB
        devid    1 size 976.56GiB used 837.06GiB path /dev/sdd2

btrfs-progs v4.1.2
# btrfs filesystem df /
Data, single: total=65.00GiB, used=54.83GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=4.00GiB, used=1.76GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Greetings
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: trim not working and irreparable errors from btrfsck

[Chris Murphy]

On Thu, Aug 13, 2015 at 3:23 AM, Marc Joliet <marcec@gmx.de> wrote:

> Speaking as a user, since "fstrim -av" still always outputs 0 bytes trimmed
> on my system: what's the status of this?  Did anybody ever file a bug report?

Since I'm not having this problem with my SSD, I'm not in a position
to provide any meaningful information for such a report.

The bug should whether this problem is reproducible with ext4 and XFS
on the same device, and the complete details of the stacking (if this
is not the full device or partition of it; e.g. if LVM, md, or
encryption is between fs and physical device). And also the bug should
include full dmesg as attachment, and strace of the fstrim command
that results in 0 bytes trimmed. And probably separate bugs for each
make/model of SSD, with the bug including make/model and firmware
version.

Right now I think there's no status because a.) no bug report and b.)
not enough information.

-- 
Chris Murphy

Re: trim not working and irreparable errors from btrfsck

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 1798 bytes --]

Am Thu, 13 Aug 2015 17:14:36 -0600
schrieb Chris Murphy <lists@colorremedies.com>:

> On Thu, Aug 13, 2015 at 3:23 AM, Marc Joliet <marcec@gmx.de> wrote:
> 
> > Speaking as a user, since "fstrim -av" still always outputs 0 bytes trimmed
> > on my system: what's the status of this?  Did anybody ever file a bug report?
> 
> Since I'm not having this problem with my SSD, I'm not in a position
> to provide any meaningful information for such a report.
> 
> The bug should whether this problem is reproducible with ext4 and XFS
> on the same device, and the complete details of the stacking (if this
> is not the full device or partition of it; e.g. if LVM, md, or
> encryption is between fs and physical device). And also the bug should
> include full dmesg as attachment, and strace of the fstrim command
> that results in 0 bytes trimmed. And probably separate bugs for each
> make/model of SSD, with the bug including make/model and firmware
> version.
> 
> Right now I think there's no status because a.) no bug report and b.)
> not enough information.

I was mainly asking because apparently there *is* a patch that helps some
people affected by this, but nobody ever commented on it.  Perhaps there's a
reason for that, but I found it curious.  (I see now that it was submitted in
early January, in the thread "[PATCH V2] Btrfs: really fix trim 0 bytes after a
device delete".)

I can open a bug (I mean, that's part of being a user of btrfs at this stage),
I'm just surprised that nobody else has.

BTW, is there a way to tell if the "discard" mount option does anything?  I'm
curious about whether it could behave differently.

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: trim not working and irreparable errors from btrfsck

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 373 bytes --]

Am Fri, 14 Aug 2015 10:05:55 +0200
schrieb Marc Joliet <marcec@gmx.de>:

> (I mean, that's part of being a user of btrfs at this stage)

I meant *being prepared* to file a bug report, not that one constantly has to
file bug reports :) .

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

RE: trim not working and irreparable errors from btrfsck

[Paul Jones]

> -----Original Message-----
> From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
> owner@vger.kernel.org] On Behalf Of Marc Joliet
> Sent: Friday, 14 August 2015 6:06 PM
> To: linux-btrfs@vger.kernel.org
> Subject: Re: trim not working and irreparable errors from btrfsck
> 
> Am Thu, 13 Aug 2015 17:14:36 -0600
> schrieb Chris Murphy <lists@colorremedies.com>:
> 
> >
> > Right now I think there's no status because a.) no bug report and b.)
> > not enough information.
> 
> I was mainly asking because apparently there *is* a patch that helps some
> people affected by this, but nobody ever commented on it.  Perhaps there's
> a reason for that, but I found it curious.  (I see now that it was submitted in
> early January, in the thread "[PATCH V2] Btrfs: really fix trim 0 bytes after a
> device delete".)
> 
> I can open a bug (I mean, that's part of being a user of btrfs at this stage), I'm
> just surprised that nobody else has.

I have to use that patch on one of my systems. I just assumed it was never merged because it wasn't quite ready yet. It seems to work fine for me though.


Paul.