Re: [PATCH 4/4] btrfs-progs: change -t option for subvolume list to print a simple space-separated table (making it machine-readable)

[Duncan <1i5t5.duncan@cox.net>]

Goffredo Baroncelli posted on Sat, 03 Oct 2015 19:41:33 +0200 as
excerpted:

> On 2015-10-03 12:09, Axel Burri wrote:
>> 
>> 
>> On 2015-10-03 11:56, Goffredo Baroncelli wrote:
>>> On 2015-10-02 18:41, axel@tty0.ch wrote:
>>>> Old implementation used tabs "\t", and tried to work around problems
>>>> by guessing amount of tabs needed (e.g. "\t\t" after top level", with
>>>> buggy output as soon as empty uuids are printed). This will never
>>>> work correctly, as tab width is a user-defined setting in the
>>>> terminal.
>>>
>>>
>>> Why not use string_table() and table_*() functions  ?
>> 
>> string_table(), as well as all table functions by nature, needs to know
>> the maximum size of all cells in a row before printing, and therefore
>> buffers all the output before printing. It would eat up a lot of memory
>> for large tables (it is not unusual to have 1000+ subvolumes in btrfs
>> if you make heavy use of snapshotting). Furthermore, it would slow down
>> things by not printing the output linewise.
> 
> 
> Assuming 200bytes per row (== subvolume) x 1000 subvolumes = 200kB... I
> don't think that this could be a problem, nor in terms of memory used
> nor in terms of speed: if you have 1000+ subvolumes the most time
> consuming activity is traversing the filesystem looking at the
> snapshot...

Perhaps unfortunately, scaling to millions of snapshots/subvolumes really 
*is* expected by some people.  You'd be surprised at the number of folks 
that setup automated per-minute snapshotting with no automated thinning, 
and expect to be able to keep several years' worth of snapshots, and then 
wonder why btrfs maintenance commands such as balance take weeks/months...

5 years * 365 days/year * 24 hours/day * 60 minutes/hour * 1 snapshot/
minute = ...

(years, days, hours, minutes cancel out, leaving snapshots... yeah, the 
math should be right)

2.6 million plus snapshots.  And that's just one subvolume.  Multiply 
that by a half dozen subvolumes...

Over 15 million snapshots.  Multiply that by 200 bytes/snapshot...

Several gigs of buffer memory, now.

Obviously btrfs doesn't scale to that level now, and if it did, someone 
making the mistake of trying to get a listing of millions of snapshots 
would very likely change their mind before even hitting 10%...

But that's why actually processing line-by-line is important, so they'll 
actually /see/ what happened and ctrl-C it, instead of the program 
aborting as it runs into (for example) the 32-bit user/kernel memory 
barrier, without printing anything useful...

The line-by-line way... "Oops, that's waayyy too many snapshots to be 
practical, maybe I should start thinning..."

The buffer-it-all-way... "Oops, there's a bug in the program; it crashed."

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman