Working on Btrfs as topic for master thesis

Working on Btrfs as topic for master thesis

[Toggenburger Lukas]

Hi all

I'm a student of ICT currently doing my master's degree besides working as a research assistant. Currently I'm looking for topics for my master thesis. One of my ideas was to work on Btrfs. I studied the list of project ideas at https://btrfs.wiki.kernel.org/index.php/Project_ideas and am especially interested in working on one of the following topics:

1. (Enhanced) hot spare support as described in https://btrfs.wiki.kernel.org/index.php/Project_ideas#Hot_spare_support and https://btrfs.wiki.kernel.org/index.php/Project_ideas#.22Enhanced.22_.28ala_RAID.29_support

2. Adding more checksumming algorithms ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#More_checksumming_algorithms )

3. Improving subvolume handling regarding taking recursive snapshots ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Take_recursive_snapshots ) and taking snapshots of arbitrary directories ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Snapshot_arbitrary_directories )

What is the current state of development concerning these features? I did see Josef Bacik being listed as working on 2. at https://btrfs.wiki.kernel.org/index.php/Roadmap after a quick search but nothing on the other things.

What would I need to know before going about one of these?

Lukas


--
Wissenschaftlicher Mitarbeiter
Bachelor of Science FHO in Elektrotechnik
Hochschule für Technik und Wirtschaft HTW Chur
Institut für Informations- und Kommunikationstechnologien

Pulvermühlestrasse 57, CH-7004 Chur
Tel. +41 (0)81 286 37 22
lukas.toggenburger@htwchur.ch

www.htwchur.ch

Re: Working on Btrfs as topic for master thesis

[David Sterba]

On Thu, Jan 16, 2014 at 07:23:18PM +0000, Toggenburger Lukas wrote:
> Hi all
> 
> I'm a student of ICT currently doing my master's degree besides
> working as a research assistant. Currently I'm looking for topics for
> my master thesis. One of my ideas was to work on Btrfs. I studied the
> list of project ideas at
> https://btrfs.wiki.kernel.org/index.php/Project_ideas and am
> especially interested in working on one of the following topics:

Please beware that the project ideas are unreviewed, sometimes very
short on details or not up-to-date with the people silently implementing
them. (Wiki is not the perfect tool for tracking tasks.)

> 1. (Enhanced) hot spare support as described in
> https://btrfs.wiki.kernel.org/index.php/Project_ideas#Hot_spare_support
> and
> https://btrfs.wiki.kernel.org/index.php/Project_ideas#.22Enhanced.22_.28ala_RAID.29_support

This logically fits to the set of features like 'error statistics',
device replace, so I'm not sure if Stefan is not working on this
already.

> 2. Adding more checksumming algorithms ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#More_checksumming_algorithms )

As you've found, Josef is claiming this, but it's not mentioned on the
idea's page.

> 3. Improving subvolume handling regarding taking recursive snapshots (
> https://btrfs.wiki.kernel.org/index.php/Project_ideas#Take_recursive_snapshots
> ) and taking snapshots of arbitrary directories (
> https://btrfs.wiki.kernel.org/index.php/Project_ideas#Snapshot_arbitrary_directories
> )

Goffredo sent patches for the recursive snapshots, implemented in
userspace. http://www.spinics.net/lists/linux-btrfs/msg29553.html
I have some notes on kernel implementation from the time I reviewed the
patches.

The second project is essentially implementing the
'cp --reflink=always dir newsubvol' workaround.

> What is the current state of development concerning these features?

Unless there are patches in the mailinglist, hard to say. Sometimes the
status is shared on IRC.

> What would I need to know before going about one of these?

Not sure what exactly you're asking about.
Have an idea how to implement it. Then try to get ack from someobody
familiar with the area you're going to touch. Implement, write tests,
write some howto/docs, submit it.

HTH,
david

Re: Working on Btrfs as topic for master thesis

[Tomasz Torcz]

On Thu, Jan 16, 2014 at 07:23:18PM +0000, Toggenburger Lukas wrote:
> Hi all
> I'm a student of ICT currently doing my master's degree besides working as
> a research assistant. Currently I'm looking for topics for my master thesis.
> One of my ideas was to work on Btrfs. I studied the list of project ideas at
> https://btrfs.wiki.kernel.org/index.php/Project_ideas and am especially
> interested in working on one of the following topics: 

  Have you considered per-file/per-directory selection of raid level?

-- 
Tomasz Torcz                 "God, root, what's the difference?"
xmpp: zdzichubg@chrome.pl         "God is more forgiving."

RE: Working on Btrfs as topic for master thesis

[Toggenburger Lukas]

Hello Tomasz

> Have you considered per-file/per-directory selection of raid level?

Sounds great, I haven't thought about it before.

Do you or someone else know what the current state of development is? Is someone working on this?

Lukas


________________________________________
From: linux-btrfs-owner@vger.kernel.org [linux-btrfs-owner@vger.kernel.org] on behalf of Tomasz Torcz [tomek@pipebreaker.pl]
Sent: Friday, January 17, 2014 5:04 PM
To: linux-btrfs@vger.kernel.org
Subject: Re: Working on Btrfs as topic for master thesis

On Thu, Jan 16, 2014 at 07:23:18PM +0000, Toggenburger Lukas wrote:
> Hi all
> I'm a student of ICT currently doing my master's degree besides working as
> a research assistant. Currently I'm looking for topics for my master thesis.
> One of my ideas was to work on Btrfs. I studied the list of project ideas at
> https://btrfs.wiki.kernel.org/index.php/Project_ideas and am especially
> interested in working on one of the following topics:

  Have you considered per-file/per-directory selection of raid level?

--
Tomasz Torcz                 "God, root, what's the difference?"
xmpp: zdzichubg@chrome.pl         "God is more forgiving."

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Working on Btrfs as topic for master thesis

[David Sterba]

On Sat, Jan 18, 2014 at 12:50:54PM +0000, Toggenburger Lukas wrote:
> Hello Tomasz
> 
> > Have you considered per-file/per-directory selection of raid level?
> 
> Sounds great, I haven't thought about it before.
> 
> Do you or someone else know what the current state of development is?
> Is someone working on this?

The feature lacks interface to specify the raid flags per-object. This
is WIP, keyword is 'properties', you'll find some preliminary patches in
the list. This is the ground work for all sorts of fancy tuning.

The filesystem split into areas with different raid levels will bring
interesting problems regarding free space and operations that cross the
raid levels. But I think it's doable.

Re: Working on Btrfs as topic for master thesis

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1249 bytes --]

On Wed, Jan 22, 2014 at 01:05:33PM +0100, David Sterba wrote:
> On Sat, Jan 18, 2014 at 12:50:54PM +0000, Toggenburger Lukas wrote:
> > Hello Tomasz
> > 
> > > Have you considered per-file/per-directory selection of raid level?
> > 
> > Sounds great, I haven't thought about it before.
> > 
> > Do you or someone else know what the current state of development is?
> > Is someone working on this?
> 
> The feature lacks interface to specify the raid flags per-object. This
> is WIP, keyword is 'properties', you'll find some preliminary patches in
> the list. This is the ground work for all sorts of fancy tuning.
> 
> The filesystem split into areas with different raid levels will bring
> interesting problems regarding free space and operations that cross the
> raid levels. But I think it's doable.

   There's some potentially horrible ENOSPC cases with uneven-sized
devices. I need to run some simulations to see how it'll behave...

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
       --- The early bird gets the worm,  but the second mouse ---       
                            gets the cheese.                             

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Working on Btrfs as topic for master thesis

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 16/01/14 11:23, Toggenburger Lukas wrote:
> One of my ideas was to work on Btrfs.

One thing I would like see it automatic backup copies of data.  For
example if you are only using 10% of the total space then make an
additional 9 copies of the data in the free space (low priority in the
background).  As more space is used reclaim that dup/free space.  The
principle especially applies when you have more than one device.

The goal is to ensure that all the space is used, and that there is the
maximum probability of data recovery.

If you are more interested in the theoretical side then looking into
compression would be interesting.  ie how close to the theoretical best
compression are we.  Various filesystems like btrfs and NTFS make all
sorts of compromises in algorithm choices but also especially in the size
of blocks they compress.  How much better could be done?

Roger

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)

iEYEARECAAYFAlLct8cACgkQmOOfHg372QTiNACgs1WNJF1tCM5boWP8L3obD0jU
4qsAnAjANq6TE6Gh5ATNbEGTmoA6/W+s
=YnrI
-----END PGP SIGNATURE-----

Re: Working on Btrfs as topic for master thesis

[David Sterba]

On Sun, Jan 19, 2014 at 09:44:39PM -0800, Roger Binns wrote:
> If you are more interested in the theoretical side then looking into
> compression would be interesting.  ie how close to the theoretical best
> compression are we.  Various filesystems like btrfs and NTFS make all
> sorts of compromises in algorithm choices but also especially in the size
> of blocks they compress.  How much better could be done?

I have done some work here, so far it's stalled due to more important
work.

https://btrfs.wiki.kernel.org/index.php/Project_ideas#Compression_enhancements

Do you have other suggestions beyond what's proposed there?

Re: Working on Btrfs as topic for master thesis

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 22/01/14 04:12, David Sterba wrote:
> I have done some work here, so far it's stalled due to more important 
> work.
> 
> https://btrfs.wiki.kernel.org/index.php/Project_ideas#Compression_enhancements
>
>  Do you have other suggestions beyond what's proposed there?

There was the theoretical side - ie coming up with a way of defining
perfection which then allows measuring against.  For example you have
going up to a 128K block size but without knowing the theoretical best we
don't know if that is a stopgap or very good.

That also feeds into things like if it would be a good idea to go back
afterwards (perhaps as part of defrag) and spend more effort on
(re)compression.

Another consideration is perhaps having the compression dictionary kept
separate from the compressed blocks thereby allowing it to be used across
blocks and potentially files.  Compressors like smaz (very good on short
pieces of text) work by having a precomputed dictionary - perhaps those
can be used too.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)

iEYEARECAAYFAlLgMEQACgkQmOOfHg372QRGDACeI604tw4OZsITHZEY60O6aiQX
GD4AoIj9s2rbVWiRp2W4FR6rkAf+iSsH
=cD4/
-----END PGP SIGNATURE-----

Re: Working on Btrfs as topic for master thesis

[David Sterba]

On Wed, Jan 22, 2014 at 12:55:39PM -0800, Roger Binns wrote:
> There was the theoretical side - ie coming up with a way of defining
> perfection which then allows measuring against.  For example you have
> going up to a 128K block size but without knowing the theoretical best we
> don't know if that is a stopgap or very good.

I got some inspiration from
http://web.archive.org/web/20060511214233/http://www.namesys.com/compression.txt
there's a metric proposed and some analysis of cluster sizes.

Referenced from
http://web.archive.org/web/20070403001821/http://www.namesys.com/cryptcompress_design.html
with the related filesysem details

'Theoretical best' seems too vaguely defined, with compression it's
always some trade-off and compromise (comp/decomp speed vs ratio) and
counting the filesystem requirements like a reasonable read and write
latencies/throughput it's not getting easier.

The main categories I'm targeting are 'real-time, good on average for
random read-write', and 'slow, high compression ratio, expected
streaming reads/writes'.

I'm trying to evaluate the effects of the changes with respect to the
filesystem, eg. when the larger chunk means the ratio improves from 80%
to 70% for some data set type, what's the reduced block count and if
it's worth.

If the interface is flexible enough, it gives the user chance to
experiment with the options and make the choice.

> That also feeds into things like if it would be a good idea to go back
> afterwards (perhaps as part of defrag) and spend more effort on
> (re)compression.

This is a bit different usecase, defrag is triggered by user at the time
he knows the resources are available and usually on known files and
their potential compressibility. More, the userspace defrag process can
do a preliminary analysis of the files, eg. guess the mime type or do a
random sample of the file data and guess compressibility.

> Another consideration is perhaps having the compression dictionary kept
> separate from the compressed blocks thereby allowing it to be used across
> blocks and potentially files.  Compressors like smaz (very good on short
> pieces of text) work by having a precomputed dictionary - perhaps those
> can be used too.

Keeping the dictionary implies more data to be read/written, with small
chunks there's a low chance of actual dictionary reuse for other files.
Also, thinking about the implementaion, it would become too complex to
do in kernel for this particular usecase.

It's interesting theoretically, there's a paper about that

"To Zip or Not to Zip: Effective Resource Usage for Real-Time
Compression"
https://www.usenix.org/conference/fast13/technical-sessions/presentation/harnik

but haven't found it feasible to push the implementation to kernel and
worth the required disk format changes.

Re: Working on Btrfs as topic for master thesis

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23/01/14 10:36, David Sterba wrote:
> 'Theoretical best' seems too vaguely defined,

It seems like a good thing for someone to tackle as part of a master's
thesis :-)

> with compression it's always some trade-off and compromise

Which you can put in context against the theoretical best.  The links you
gave are a good example of trying to do that.

> This is a bit different usecase, defrag is triggered by user at the
> time he knows the resources are available

I'm a user and I use autodefrag :-)  As a developer you are more
interested in making users be aware of what they do, when they do it, and
carefully select the optimum conditions and configuration.

As a user I just want to point to a pile of storage and have btrfs do the
right thing, without me having to babysit it or play admin.  Computers
have billions of processor cycles per second, gigabytes of memory etc.
They should just figure this stuff out and not require me to be versed in
lots of intricate details!

> Keeping the dictionary implies more data to be read/written, with
> small chunks there's a low chance of actual dictionary reuse for other
> files.

I'm willing to bet that there is a good chance of reuse for files with the
same extension as an example.  And it is highly likely for Maildir files.

> Also, thinking about the implementaion, it would become too complex to 
> do in kernel for this particular usecase.

A thesis could study if it is worth doing first.  If it found that was a
good idea, then figuring out how to implement it is a second step.

> "To Zip or Not to Zip: Effective Resource Usage for Real-Time 
> Compression"

Except for systems that are 100% busy all the time, there is no need to
get perfect real time compression.  IMHO it is fine coming back later and
doing a better job of it.  Again this assumes that there is a sufficiently
large difference between what real time does and what a later
recompression does.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)

iEYEARECAAYFAlLhjeUACgkQmOOfHg372QTBKwCgxbWmfwr0MMfAo9bVwThmGTOq
F1EAoIsgVlzfeqPZS9zpKM1mJ3Cdw9LL
=IINU
-----END PGP SIGNATURE-----

Re: Working on Btrfs as topic for master thesis

[Austin S Hemmelgarn]

On 2014-01-16 14:23, Toggenburger Lukas wrote:
> Hi all
> 
> I'm a student of ICT currently doing my master's degree besides working as a research assistant. Currently I'm looking for topics for my master thesis. One of my ideas was to work on Btrfs. I studied the list of project ideas at https://btrfs.wiki.kernel.org/index.php/Project_ideas and am especially interested in working on one of the following topics:
> 
> 1. (Enhanced) hot spare support as described in https://btrfs.wiki.kernel.org/index.php/Project_ideas#Hot_spare_support and https://btrfs.wiki.kernel.org/index.php/Project_ideas#.22Enhanced.22_.28ala_RAID.29_support
> 
> 2. Adding more checksumming algorithms ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#More_checksumming_algorithms )
> 
> 3. Improving subvolume handling regarding taking recursive snapshots ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Take_recursive_snapshots ) and taking snapshots of arbitrary directories ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Snapshot_arbitrary_directories )

Another option that I would personally love to see would be support for
write-mostly devices (that is, devices in a RAID1/RAID10 setup that only
get written too unless the data can't be found elsewhere).  This would
in particular provide an alternative to using bcache/dm-cache (namely an
SSD and HDD in RAID1 with the HDD set to write-mostly).
Based on the current development focus, I don't think anybody is working
on this already (I would be, but I don't have either the time or the
skills with kernel programming that would be needed).

Re: Working on Btrfs as topic for master thesis

[Sandy McArthur]

On Mon, Jan 20, 2014 at 7:20 AM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
>
> On 2014-01-16 14:23, Toggenburger Lukas wrote:
> > 3. Improving subvolume handling regarding taking recursive snapshots ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Take_recursive_snapshots ) and taking snapshots of arbitrary directories ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Snapshot_arbitrary_directories )
>
> Another option that I would personally love to see would be support for
> write-mostly devices (that is, devices in a RAID1/RAID10 setup that only
> get written too unless the data can't be found elsewhere).  This would
> in particular provide an alternative to using bcache/dm-cache (namely an
> SSD and HDD in RAID1 with the HDD set to write-mostly).
> Based on the current development focus, I don't think anybody is working
> on this already (I would be, but I don't have either the time or the
> skills with kernel programming that would be needed).

Maybe this happens already: Might a similar effect be automatically
achieved by tracking per-device I/O load averages and distributing
reads based on the I/O loads of possible read devices?

-- 
Sandy McArthur

Re: Working on Btrfs as topic for master thesis

[Austin S Hemmelgarn]

On 2014-01-21 01:42, Sandy McArthur wrote:
> On Mon, Jan 20, 2014 at 7:20 AM, Austin S Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>>
>> On 2014-01-16 14:23, Toggenburger Lukas wrote:
>>> 3. Improving subvolume handling regarding taking recursive snapshots ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Take_recursive_snapshots ) and taking snapshots of arbitrary directories ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Snapshot_arbitrary_directories )
>>
>> Another option that I would personally love to see would be support for
>> write-mostly devices (that is, devices in a RAID1/RAID10 setup that only
>> get written too unless the data can't be found elsewhere).  This would
>> in particular provide an alternative to using bcache/dm-cache (namely an
>> SSD and HDD in RAID1 with the HDD set to write-mostly).
>> Based on the current development focus, I don't think anybody is working
>> on this already (I would be, but I don't have either the time or the
>> skills with kernel programming that would be needed).
> 
> Maybe this happens already: Might a similar effect be automatically
> achieved by tracking per-device I/O load averages and distributing
> reads based on the I/O loads of possible read devices?
> 
That might be the case, it depends on how the I/O load averages are
calculated.  I actually hadn't realized BTRFS did this, I thought it
behaved more like MD RAID (that is, distributing the reads among devices
in a un-weighted round-robin fashion).

Re: Working on Btrfs as topic for master thesis

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1988 bytes --]

On Tue, Jan 21, 2014 at 07:25:43AM -0500, Austin S Hemmelgarn wrote:
> On 2014-01-21 01:42, Sandy McArthur wrote:
> > On Mon, Jan 20, 2014 at 7:20 AM, Austin S Hemmelgarn
> > <ahferroin7@gmail.com> wrote:
> >>
> >> On 2014-01-16 14:23, Toggenburger Lukas wrote:
> >>> 3. Improving subvolume handling regarding taking recursive snapshots ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Take_recursive_snapshots ) and taking snapshots of arbitrary directories ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Snapshot_arbitrary_directories )
> >>
> >> Another option that I would personally love to see would be support for
> >> write-mostly devices (that is, devices in a RAID1/RAID10 setup that only
> >> get written too unless the data can't be found elsewhere).  This would
> >> in particular provide an alternative to using bcache/dm-cache (namely an
> >> SSD and HDD in RAID1 with the HDD set to write-mostly).
> >> Based on the current development focus, I don't think anybody is working
> >> on this already (I would be, but I don't have either the time or the
> >> skills with kernel programming that would be needed).
> > 
> > Maybe this happens already: Might a similar effect be automatically
> > achieved by tracking per-device I/O load averages and distributing
> > reads based on the I/O loads of possible read devices?
> > 
> That might be the case, it depends on how the I/O load averages are
> calculated.  I actually hadn't realized BTRFS did this, I thought it
> behaved more like MD RAID (that is, distributing the reads among devices
> in a un-weighted round-robin fashion).

   I think David tried that a while ago, and the benchmarks were
actually worse. I'm not sure how much investigation he did into why,
though.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
           --- Quidquid latine dictum sit,  altum videtur. ---           

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Working on Btrfs as topic for master thesis

[Austin S Hemmelgarn]

On 2014-01-21 11:52, Hugo Mills wrote:
> On Tue, Jan 21, 2014 at 07:25:43AM -0500, Austin S Hemmelgarn wrote:
>> On 2014-01-21 01:42, Sandy McArthur wrote:
>>> On Mon, Jan 20, 2014 at 7:20 AM, Austin S Hemmelgarn
>>> <ahferroin7@gmail.com> wrote:
>>>>
>>>> On 2014-01-16 14:23, Toggenburger Lukas wrote:
>>>>> 3. Improving subvolume handling regarding taking recursive snapshots ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Take_recursive_snapshots ) and taking snapshots of arbitrary directories ( https://btrfs.wiki.kernel.org/index.php/Project_ideas#Snapshot_arbitrary_directories )
>>>>
>>>> Another option that I would personally love to see would be support for
>>>> write-mostly devices (that is, devices in a RAID1/RAID10 setup that only
>>>> get written too unless the data can't be found elsewhere).  This would
>>>> in particular provide an alternative to using bcache/dm-cache (namely an
>>>> SSD and HDD in RAID1 with the HDD set to write-mostly).
>>>> Based on the current development focus, I don't think anybody is working
>>>> on this already (I would be, but I don't have either the time or the
>>>> skills with kernel programming that would be needed).
>>>
>>> Maybe this happens already: Might a similar effect be automatically
>>> achieved by tracking per-device I/O load averages and distributing
>>> reads based on the I/O loads of possible read devices?
>>>
>> That might be the case, it depends on how the I/O load averages are
>> calculated.  I actually hadn't realized BTRFS did this, I thought it
>> behaved more like MD RAID (that is, distributing the reads among devices
>> in a un-weighted round-robin fashion).
> 
>    I think David tried that a while ago, and the benchmarks were
> actually worse. I'm not sure how much investigation he did into why,
> though.
> 
>    Hugo.
> 
If he implemented it such that it waited for the write to complete to
all devices before returning from the call to write(), then it is very
easy to see how it could be worse.  The trick (for writes at least) is
to wait synchronously for all of the non-writemostly devices to finish
the write, then return from write() and have a background thread cleanup
if there are write errors on the writemostly devices (at that point, you
already have a (probably) good copy on the other devices that you could
replicate to the device that failed the write).
As for the reads, you would only need to find some way to exclude the
writemostly devices from the read-scheduling code unless the read is
because of a read error on the other devices.

Re: Working on Btrfs as topic for master thesis

[David Sterba]

On Tue, Jan 21, 2014 at 04:52:00PM +0000, Hugo Mills wrote:
> On Tue, Jan 21, 2014 at 07:25:43AM -0500, Austin S Hemmelgarn wrote:
> > > Maybe this happens already: Might a similar effect be automatically
> > > achieved by tracking per-device I/O load averages and distributing
> > > reads based on the I/O loads of possible read devices?
> > > 
> > That might be the case, it depends on how the I/O load averages are
> > calculated.  I actually hadn't realized BTRFS did this, I thought it
> > behaved more like MD RAID (that is, distributing the reads among devices
> > in a un-weighted round-robin fashion).
> 
>    I think David tried that a while ago, and the benchmarks were
> actually worse. I'm not sure how much investigation he did into why,
> though.

I haven't done any extensive testing, only streaming writes, and the
heuristic just batched writes by a given threshold before switching to
another mirror.  Load balancing was done without any logic that would
look at actual IO load of the devices. For me, the result was that
there's room for improvement.

Re: Working on Btrfs as topic for master thesis

[David Sterba]

On Wed, Jan 22, 2014 at 01:20:10PM +0100, David Sterba wrote:
> I haven't done any extensive testing, only streaming writes, and the
> heuristic just batched writes by a given threshold before switching to
> another mirror.  Load balancing was done without any logic that would
> look at actual IO load of the devices. For me, the result was that
> there's room for improvement.

http://www.spinics.net/lists/linux-btrfs/msg12745.html
http://www.spinics.net/lists/linux-btrfs/msg17228.html