Fwd: OS X Time Machine and BTRFS

Fwd: OS X Time Machine and BTRFS

[Ruben Salzgeber]

Hi everyone

I'm reaching out to you because I experience unusually slow read and
write speeds on my Arch Linux server in combination with OS X time
machine. My setup is consists of an Core i3 6300, 16GB Ram, 128GB SSD
for the OS and a 8 drive RAID5 BTRFS volume as archive. The latest
version of Avahi, Netatalk and BTRFS-Progs are installed. On a wired
connection I reach 120MB/s for normal filetransfers from OS X to the
Server. When using Time Machine I measure peak network trafic around
1-2MB/s and long durations of almost no trafic. Could this be in any
relation to BTRFS? Is there a special configuration necessary for
folders containing the Time Machine sparsebundle file?

Hope hear from you, kind regards

Ruben

Re: Fwd: OS X Time Machine and BTRFS

[Sean Greenslade]

On Mon, Sep 26, 2016 at 10:15:11PM +0200, Ruben Salzgeber wrote:
> Hi everyone
> 
> I'm reaching out to you because I experience unusually slow read and
> write speeds on my Arch Linux server in combination with OS X time
> machine. My setup is consists of an Core i3 6300, 16GB Ram, 128GB SSD
> for the OS and a 8 drive RAID5 BTRFS volume as archive. The latest
> version of Avahi, Netatalk and BTRFS-Progs are installed. On a wired
> connection I reach 120MB/s for normal filetransfers from OS X to the
> Server. When using Time Machine I measure peak network trafic around
> 1-2MB/s and long durations of almost no trafic. Could this be in any
> relation to BTRFS? Is there a special configuration necessary for
> folders containing the Time Machine sparsebundle file?

Have you done any inspection with tools like [h]top, iotop, etc.?
Identifying the bottleneck is usually pretty straightforward with those,
and it's nice to try and dismiss things like cpu bottlenecks before
going to more advanced diagnostics.

--Sean

Re: OS X Time Machine and BTRFS

[Chris Murphy]

On Mon, Sep 26, 2016 at 2:15 PM, Ruben Salzgeber
<ruben.salzgeber@gmail.com> wrote:
> Hi everyone
>
> I'm reaching out to you because I experience unusually slow read and
> write speeds on my Arch Linux server in combination with OS X time
> machine. My setup is consists of an Core i3 6300, 16GB Ram, 128GB SSD
> for the OS and a 8 drive RAID5 BTRFS volume as archive. The latest
> version of Avahi, Netatalk and BTRFS-Progs are installed. On a wired
> connection I reach 120MB/s for normal filetransfers from OS X to the
> Server. When using Time Machine I measure peak network trafic around
> 1-2MB/s and long durations of almost no trafic. Could this be in any
> relation to BTRFS? Is there a special configuration necessary for
> folders containing the Time Machine sparsebundle file?

For the likely majority who have no idea what this means: this creates
a file on the server, on which an HFS+ volume is created and then
remotely mounted on the client. Client sees an HFS+ volume. Server
side, this file isn't really just one file, it's actually a directory
containing a bunch of 8MiB files. So it's like a qcow2 file, in that
it grows dynamically, but is made up of 8MiB "extents" that appear as
files.

First question is if the directory containing the sparsebundle file
has xattr +C set on it? If not, individual bundle files won't inherit
nodatacow as they're created. You'll have literally hundreds if not
thousands of these files, many of which are being CoW'd constantly and
simultaneously as they're being modified. Even a tiny metadata "touch"
for an area of the HFS+ file system will result in an 8MiB file being
affected. What I don't know, but suspect, is that each change to one
of these 8MiB files is causing the whole 8MiB to be CoW'd. I have no
idea what kind of optimization is possible here. If Netatalk is
updating one of these 8MiB files, what does that write pattern look
like? If it's making half a dozen small changes, is that half a dozen
CoW of that entire 8MiB file? Or is it just CoW'ing what's changed?
And then to what degree is Netatalk doing fsync at all? It could be a
worse case scenario where it's CoWing 8MiB increments each time and
with lots of fsyncs, which would just obliterate the performance.

xattr +C would probably solve most of this, so long as you're not
taking Btrfs snapshots during a TM backup. Of course if +C is set,
notdatacow implies nodatasum so there's no checksumming of this
sparsebundle.

Maybe someone has an idea how to sample the activity as it happens, to
model the write pattern for this use case? Without knowing more about
the pattern, it's a stab in the dark how to optimize the storage for
it.


-- 
Chris Murphy

Re: Fwd: OS X Time Machine and BTRFS

[Chris Murphy]

On Mon, Sep 26, 2016 at 2:56 PM, Sean Greenslade
<sean@seangreenslade.com> wrote:
> On Mon, Sep 26, 2016 at 10:15:11PM +0200, Ruben Salzgeber wrote:
>> Hi everyone
>>
>> I'm reaching out to you because I experience unusually slow read and
>> write speeds on my Arch Linux server in combination with OS X time
>> machine. My setup is consists of an Core i3 6300, 16GB Ram, 128GB SSD
>> for the OS and a 8 drive RAID5 BTRFS volume as archive. The latest
>> version of Avahi, Netatalk and BTRFS-Progs are installed. On a wired
>> connection I reach 120MB/s for normal filetransfers from OS X to the
>> Server. When using Time Machine I measure peak network trafic around
>> 1-2MB/s and long durations of almost no trafic. Could this be in any
>> relation to BTRFS? Is there a special configuration necessary for
>> folders containing the Time Machine sparsebundle file?
>
> Have you done any inspection with tools like [h]top, iotop, etc.?
> Identifying the bottleneck is usually pretty straightforward with those,
> and it's nice to try and dismiss things like cpu bottlenecks before
> going to more advanced diagnostics.

Good idea.

Another possibly useful one would be using filefrag to get some idea
how badly fragmented the bands (individual files) in the bundle are
getting.

$ filefrag backup.sparsebundle/bands/*

This is likely going to be excessively long. And example output


backup.sparsebundle/bands/0: 43 extents found
backup.sparsebundle/bands/1: 1 extent found
backup.sparsebundle/bands/10: 212 extents found
backup.sparsebundle/bands/11: 1 extent found

Maybe sort -k can sort by the number of extents, and then use head to
just show the top 50 offenders.


-- 
Chris Murphy

Re: OS X Time Machine and BTRFS

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 393 bytes --]

On Mon, 26 Sep 2016 15:06:39 -0600
Chris Murphy <lists@colorremedies.com> wrote:

> First question is if the directory containing the sparsebundle file
> has xattr +C set on it?

It's not xattr, but chattr. "xattr" manages "extended attributes", i.e.
arbitrary name=value pairs you can set on a file or directory. But +C is just
one of the standard ones.

-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: OS X Time Machine and BTRFS

[Chris Murphy]

On Mon, Sep 26, 2016 at 8:16 PM, Roman Mamedov <rm@romanrm.net> wrote:
> On Mon, 26 Sep 2016 15:06:39 -0600
> Chris Murphy <lists@colorremedies.com> wrote:
>
>> First question is if the directory containing the sparsebundle file
>> has xattr +C set on it?
>
> It's not xattr, but chattr. "xattr" manages "extended attributes", i.e.
> arbitrary name=value pairs you can set on a file or directory. But +C is just
> one of the standard ones.

I'm using xattr as a contraction for extended attribute, I don't have
an xattr command on my Fedora system. But macOS does have an xattr
command so I can see how that might be confusing.


-- 
Chris Murphy

Re: Fwd: OS X Time Machine and BTRFS

[Chris Murphy]

readding btrfs list

On Tue, Sep 27, 2016 at 12:05 PM, Ruben Salzgeber
<ruben.salzgeber@gmail.com> wrote:
> Hi Chris
>
> Again thanks for your help here. I've tried to see how badly the files
> in the sparsbundle are fragmented, unfortunatlly I wasn't able to run
> filefrag (FIBMAP unsupported).

What command did you use? It will not work on directories, it only
works on files, or you will get that error message.

>Iotop was showing lots of processes accessing the disks:

Use -o option, it will show only processes producing I/O.



> I'll copy the sparsbundle to the macbook, set chattr +C on the
> timemachine folder and copy all back. This should deactivate CoW on
> all subfolders and files.

Yeah just realize it also disables compression and checksumming as well.



-- 
Chris Murphy

Re: OS X Time Machine and BTRFS

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 4245 bytes --]

On Mon, Sep 26, 2016 at 03:06:39PM -0600, Chris Murphy wrote:
> On Mon, Sep 26, 2016 at 2:15 PM, Ruben Salzgeber
> <ruben.salzgeber@gmail.com> wrote:
> > Hi everyone
> >
> > I'm reaching out to you because I experience unusually slow read and
> > write speeds on my Arch Linux server in combination with OS X time
> > machine. My setup is consists of an Core i3 6300, 16GB Ram, 128GB SSD
> > for the OS and a 8 drive RAID5 BTRFS volume as archive. The latest
> > version of Avahi, Netatalk and BTRFS-Progs are installed. On a wired
> > connection I reach 120MB/s for normal filetransfers from OS X to the
> > Server. When using Time Machine I measure peak network trafic around
> > 1-2MB/s and long durations of almost no trafic. Could this be in any
> > relation to BTRFS? Is there a special configuration necessary for
> > folders containing the Time Machine sparsebundle file?

Ruben, if nobody has told you before now:  don't use btrfs raid5 in
production.

> For the likely majority who have no idea what this means: this creates
> a file on the server, on which an HFS+ volume is created and then
> remotely mounted on the client. Client sees an HFS+ volume. Server
> side, this file isn't really just one file, it's actually a directory
> containing a bunch of 8MiB files. So it's like a qcow2 file, in that
> it grows dynamically, but is made up of 8MiB "extents" that appear as
> files.
> 
> First question is if the directory containing the sparsebundle file
> has xattr +C set on it? If not, individual bundle files won't inherit
> nodatacow as they're created. You'll have literally hundreds if not
> thousands of these files, many of which are being CoW'd constantly and
> simultaneously as they're being modified. Even a tiny metadata "touch"
> for an area of the HFS+ file system will result in an 8MiB file being
> affected. What I don't know, but suspect, is that each change to one
> of these 8MiB files is causing the whole 8MiB to be CoW'd. I have no
> idea what kind of optimization is possible here. If Netatalk is
> updating one of these 8MiB files, what does that write pattern look
> like? If it's making half a dozen small changes, is that half a dozen
> CoW of that entire 8MiB file? Or is it just CoW'ing what's changed?
> And then to what degree is Netatalk doing fsync at all? It could be a
> worse case scenario where it's CoWing 8MiB increments each time and
> with lots of fsyncs, which would just obliterate the performance.

Btrfs will CoW just the parts that are changed, leaving the original 8MB
extent on disk until the last original block is overwritten.  This could
double the amount of disk space used easily (and multiply it by several
thousand times in the worst case).

fsync could be heavy especially with older btrfs versions.  For that
matter, kernels before 4.4 will burn a lot of CPU looking for free space.

Lots of random small writes will drive up the extent count, which would
mean lots of extra seeks, each costing a few milliseconds on spinning
rust.  Over time, free space fragmentation would mean that random block
updates would trigger more raid5 stripe RMW updates (which, in addition to
eating the data when a disk fails, will make random writes much slower).

> xattr +C would probably solve most of this, so long as you're not
> taking Btrfs snapshots during a TM backup. Of course if +C is set,
> notdatacow implies nodatasum so there's no checksumming of this
> sparsebundle.
> 
> Maybe someone has an idea how to sample the activity as it happens, to
> model the write pattern for this use case? Without knowing more about
> the pattern, it's a stab in the dark how to optimize the storage for
> it.

Run 'filefrag -v' on the sparse files at e.g. hourly intervals, and
compare the results.  It will show extents changing physical address
and size as they are split up.  Each one will cost a few milliseconds
to access; if there are millions, you'll be waiting a long time.

> 
> -- 
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: OS X Time Machine and BTRFS

[Chris Murphy]

On Tue, Sep 27, 2016 at 4:57 PM, Zygo Blaxell
<ce3g8jdj@umail.furryterror.org> wrote:
> On Mon, Sep 26, 2016 at 03:06:39PM -0600, Chris Murphy wrote:
>> On Mon, Sep 26, 2016 at 2:15 PM, Ruben Salzgeber
>> <ruben.salzgeber@gmail.com> wrote:
>> > Hi everyone
>> >
>> > I'm reaching out to you because I experience unusually slow read and
>> > write speeds on my Arch Linux server in combination with OS X time
>> > machine. My setup is consists of an Core i3 6300, 16GB Ram, 128GB SSD
>> > for the OS and a 8 drive RAID5 BTRFS volume as archive. The latest
>> > version of Avahi, Netatalk and BTRFS-Progs are installed. On a wired
>> > connection I reach 120MB/s for normal filetransfers from OS X to the
>> > Server. When using Time Machine I measure peak network trafic around
>> > 1-2MB/s and long durations of almost no trafic. Could this be in any
>> > relation to BTRFS? Is there a special configuration necessary for
>> > folders containing the Time Machine sparsebundle file?
>
> Ruben, if nobody has told you before now:  don't use btrfs raid5 in
> production.


I spaced this out, it might be a factor in the performance problem. More below.


>
>> For the likely majority who have no idea what this means: this creates
>> a file on the server, on which an HFS+ volume is created and then
>> remotely mounted on the client. Client sees an HFS+ volume. Server
>> side, this file isn't really just one file, it's actually a directory
>> containing a bunch of 8MiB files. So it's like a qcow2 file, in that
>> it grows dynamically, but is made up of 8MiB "extents" that appear as
>> files.
>>
>> First question is if the directory containing the sparsebundle file
>> has xattr +C set on it? If not, individual bundle files won't inherit
>> nodatacow as they're created. You'll have literally hundreds if not
>> thousands of these files, many of which are being CoW'd constantly and
>> simultaneously as they're being modified. Even a tiny metadata "touch"
>> for an area of the HFS+ file system will result in an 8MiB file being
>> affected. What I don't know, but suspect, is that each change to one
>> of these 8MiB files is causing the whole 8MiB to be CoW'd. I have no
>> idea what kind of optimization is possible here. If Netatalk is
>> updating one of these 8MiB files, what does that write pattern look
>> like? If it's making half a dozen small changes, is that half a dozen
>> CoW of that entire 8MiB file? Or is it just CoW'ing what's changed?
>> And then to what degree is Netatalk doing fsync at all? It could be a
>> worse case scenario where it's CoWing 8MiB increments each time and
>> with lots of fsyncs, which would just obliterate the performance.
>
> Btrfs will CoW just the parts that are changed, leaving the original 8MB
> extent on disk until the last original block is overwritten.

If the parts that are changed are less than the full data stripe size,
in this case 448KiB, then my understanding is it's not CoW, it's RMW
for that stripe.

In this use case, I mostly expect the backup is producing new writes
on HFS+, and thus new writes on Btrfs. But two things could cause a
lot of RMW: the HFS+ journal is ~16MiB, and maybe it's getting hit
with lots of changes during backups, I'm not sure. And there could be
a lot of small block writes for HFS+ metadata, also causing RMW.


-- 
Chris Murphy

Re: OS X Time Machine and BTRFS

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 5008 bytes --]

On Tue, Sep 27, 2016 at 07:31:00PM -0600, Chris Murphy wrote:
> On Tue, Sep 27, 2016 at 4:57 PM, Zygo Blaxell
> <ce3g8jdj@umail.furryterror.org> wrote:
> > On Mon, Sep 26, 2016 at 03:06:39PM -0600, Chris Murphy wrote:
> >> On Mon, Sep 26, 2016 at 2:15 PM, Ruben Salzgeber
> >> <ruben.salzgeber@gmail.com> wrote:
> >> > Hi everyone
> >> >
> >> > I'm reaching out to you because I experience unusually slow read and
> >> > write speeds on my Arch Linux server in combination with OS X time
> >> > machine. My setup is consists of an Core i3 6300, 16GB Ram, 128GB SSD
> >> > for the OS and a 8 drive RAID5 BTRFS volume as archive. The latest
> >> > version of Avahi, Netatalk and BTRFS-Progs are installed. On a wired
> >> > connection I reach 120MB/s for normal filetransfers from OS X to the
> >> > Server. When using Time Machine I measure peak network trafic around
> >> > 1-2MB/s and long durations of almost no trafic. Could this be in any
> >> > relation to BTRFS? Is there a special configuration necessary for
> >> > folders containing the Time Machine sparsebundle file?
> >
> > Ruben, if nobody has told you before now:  don't use btrfs raid5 in
> > production.
> 
> 
> I spaced this out, it might be a factor in the performance problem. More below.
> 
> 
> >
> >> For the likely majority who have no idea what this means: this creates
> >> a file on the server, on which an HFS+ volume is created and then
> >> remotely mounted on the client. Client sees an HFS+ volume. Server
> >> side, this file isn't really just one file, it's actually a directory
> >> containing a bunch of 8MiB files. So it's like a qcow2 file, in that
> >> it grows dynamically, but is made up of 8MiB "extents" that appear as
> >> files.
> >>
> >> First question is if the directory containing the sparsebundle file
> >> has xattr +C set on it? If not, individual bundle files won't inherit
> >> nodatacow as they're created. You'll have literally hundreds if not
> >> thousands of these files, many of which are being CoW'd constantly and
> >> simultaneously as they're being modified. Even a tiny metadata "touch"
> >> for an area of the HFS+ file system will result in an 8MiB file being
> >> affected. What I don't know, but suspect, is that each change to one
> >> of these 8MiB files is causing the whole 8MiB to be CoW'd. I have no
> >> idea what kind of optimization is possible here. If Netatalk is
> >> updating one of these 8MiB files, what does that write pattern look
> >> like? If it's making half a dozen small changes, is that half a dozen
> >> CoW of that entire 8MiB file? Or is it just CoW'ing what's changed?
> >> And then to what degree is Netatalk doing fsync at all? It could be a
> >> worse case scenario where it's CoWing 8MiB increments each time and
> >> with lots of fsyncs, which would just obliterate the performance.
> >
> > Btrfs will CoW just the parts that are changed, leaving the original 8MB
> > extent on disk until the last original block is overwritten.
> 
> If the parts that are changed are less than the full data stripe size,
> in this case 448KiB, then my understanding is it's not CoW, it's RMW
> for that stripe.

There are two levels at work here:

At the *extent* level (the structure you can see with FIEMAP or filefrag
-v), the small data writes are doing CoW in increments as small as 4K.
If the writes are all block-aligned there is no reading of data at all,
the new writes will just be blasted out to disk in big mostly-contiguous
physical (but highly discontiguous logical) bursts.

At the *block* level (inside each raid5 block group) small data writes
are doing RMW on 448K physical stripes.  If there's a burst of blocks
then these get batched up into full-stripe writes and there's no
performance problem (well, not one unique to this use case, anyway).
If the filesystem is partly full then old physical addresses can be
re-used and the RMW rate goes back up again.

In btrfs, the CoW and RMW layers have no knowledge of each other:  the CoW
layer thinks the disk is a contiguous surface of individually-addressable
4K blocks, while the RMW layer can't cope with (or atomically update)
anything smaller than 448KB.  Worse, the allocator (which lives on the
CoW layer) tends to stumble into the bad RMW cases very hard if you do
a lot of fsync().

> In this use case, I mostly expect the backup is producing new writes
> on HFS+, and thus new writes on Btrfs. But two things could cause a
> lot of RMW: the HFS+ journal is ~16MiB, and maybe it's getting hit
> with lots of changes during backups, I'm not sure. And there could be
> a lot of small block writes for HFS+ metadata, also causing RMW.

While raid5 RMW would make this even slower, it's not going to be fast
on any other btrfs raid profile either.  If you're OK with not having
checksums, nodatacow is the way to go for this use case; otherwise,
invest heavily in SSD and RAID1.

> 
> 
> -- 
> Chris Murphy

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]