[PATCH 0/4] Add functions to the fsck wrapper to improve standalone operation.

[PATCH 0/4] Add functions to the fsck wrapper to improve standalone operation.

[Frank Mayhar]

This set of patches adds functions that help improve fsck operation in
large installations and when running in unattended or headless mode.  It
adds support for reporting rusage statistics for the individual fsck
runs, for capturing fsck output, for killing fsck runs that take too
long and for running scripts when each fsck completes.

We're currently using these functions to improve our fsck monitoring
capability and to replace some unwieldy and hard-to-maintain shell
scripts.
-- 
Frank Mayhar
fmayhar@google.com

Re: [PATCH 0/4] Add functions to the fsck wrapper to improve standalone operation.

[Pádraig Brady]

On 02/07/2012 09:05 PM, Frank Mayhar wrote:
> This set of patches adds functions that help improve fsck operation in
> large installations and when running in unattended or headless mode.  It
> adds support for reporting rusage statistics for the individual fsck
> runs, for capturing fsck output, for killing fsck runs that take too
> long and for running scripts when each fsck completes.
> 
> We're currently using these functions to improve our fsck monitoring
> capability and to replace some unwieldy and hard-to-maintain shell
> scripts.

Couldn't you do this with separate fsck command runs,
and use standard system utils?

flist="/dev/sda1 /dev/sda2"
for sys in $flist; do
  timeout 10m fsck > /var/log/fsck.$(basename $sys) \
    && success_script $sys \
    || fail_script $sys
done

cheers,
Pádraig.

Re: [PATCH 0/4] Add functions to the fsck wrapper to improve standalone operation.

[Frank Mayhar]

2012/2/15 Pádraig Brady <P@draigbrady.com>:
> On 02/07/2012 09:05 PM, Frank Mayhar wrote:
>> This set of patches adds functions that help improve fsck operation in
>> large installations and when running in unattended or headless mode.  It
>> adds support for reporting rusage statistics for the individual fsck
>> runs, for capturing fsck output, for killing fsck runs that take too
>> long and for running scripts when each fsck completes.
>>
>> We're currently using these functions to improve our fsck monitoring
>> capability and to replace some unwieldy and hard-to-maintain shell
>> scripts.
>
> Couldn't you do this with separate fsck command runs,
> and use standard system utils?

Yes, of course.  That's where the "unwieldy and hard-to-maintain shell
scripts" came in.  Putting the functions in the wrapper itself, on the
other hand, means the scripts don't have to reimplement functions that
already exist there (like parallelizing the fsck runs or tracking exit
status), eliminates some external dependencies and makes the process
quite a bit less fragile.
-- 
Frank Mayhar
fmayhar@google.com

Re: [PATCH 0/4] Add functions to the fsck wrapper to improve standalone operation.

[Pádraig Brady]

On 02/15/2012 03:09 PM, Frank Mayhar wrote:
> 2012/2/15 Pádraig Brady <P@draigbrady.com>:
>> On 02/07/2012 09:05 PM, Frank Mayhar wrote:
>>> This set of patches adds functions that help improve fsck operation in
>>> large installations and when running in unattended or headless mode.  It
>>> adds support for reporting rusage statistics for the individual fsck
>>> runs, for capturing fsck output, for killing fsck runs that take too
>>> long and for running scripts when each fsck completes.
>>>
>>> We're currently using these functions to improve our fsck monitoring
>>> capability and to replace some unwieldy and hard-to-maintain shell
>>> scripts.
>>
>> Couldn't you do this with separate fsck command runs,
>> and use standard system utils?
> 
> Yes, of course.  That's where the "unwieldy and hard-to-maintain shell
> scripts" came in.  Putting the functions in the wrapper itself, on the
> other hand, means the scripts don't have to reimplement functions that
> already exist there (like parallelizing the fsck runs or tracking exit
> status), eliminates some external dependencies and makes the process
> quite a bit less fragile.

OK, thanks for the clarification.

It seems to me that these functions are supported
by quite simple shell scripting as I demonstrated.
I do agree it's a margin call, but I'd be on the
side of not bringing that logic within fsck.

cheers,
Pádraig.

Re: [PATCH 0/4] Add functions to the fsck wrapper to improve standalone operation.

[Karel Zak]

On Wed, Feb 15, 2012 at 04:06:35PM +0000, Pádraig Brady wrote:
> On 02/15/2012 03:09 PM, Frank Mayhar wrote:
> > 2012/2/15 Pádraig Brady <P@draigbrady.com>:
> >> On 02/07/2012 09:05 PM, Frank Mayhar wrote:
> >>> This set of patches adds functions that help improve fsck operation in
> >>> large installations and when running in unattended or headless mode.  It
> >>> adds support for reporting rusage statistics for the individual fsck
> >>> runs, for capturing fsck output, for killing fsck runs that take too
> >>> long and for running scripts when each fsck completes.
> >>>
> >>> We're currently using these functions to improve our fsck monitoring
> >>> capability and to replace some unwieldy and hard-to-maintain shell
> >>> scripts.
> >>
> >> Couldn't you do this with separate fsck command runs,
> >> and use standard system utils?
> > 
> > Yes, of course.  That's where the "unwieldy and hard-to-maintain shell
> > scripts" came in.  Putting the functions in the wrapper itself, on the
> > other hand, means the scripts don't have to reimplement functions that
> > already exist there (like parallelizing the fsck runs or tracking exit

 BTW, the latest fsck supports new -l option (lock disk) for parallel
 fsck processes. So you can start arbitrary number of

    fsck -l /dev/xxx

 without care about performance. We use it for systems with systemd
 where fsck is executed per device (fstab entry).

 If you want to use the same thing for the classic init scripts then 
 you can use something like

    for x in $(findmnt --fstab -n -o SOURCE); do
        fsck -l $x &< /var/log/fsck-$x &
    done

 rather tan fsck -A. Add some extra checks (completion scripts) to this
 for() should be pretty simple.

> > status), eliminates some external dependencies and makes the process
> > quite a bit less fragile.
> 
> OK, thanks for the clarification.
> 
> It seems to me that these functions are supported
> by quite simple shell scripting as I demonstrated.

 I have no problem with proposed -r option (to report memory and
 runtime statistics) and -O option to force-kill fscks that run too
 long.

 ... but I'm still not sure if we really need the completion script.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: [PATCH 0/4] Add functions to the fsck wrapper to improve standalone operation.

[Frank Mayhar]

On Wed, 2012-02-15 at 18:42 +0100, Karel Zak wrote:
> On Wed, Feb 15, 2012 at 04:06:35PM +0000, Pádraig Brady wrote:
> > On 02/15/2012 03:09 PM, Frank Mayhar wrote:
> > > 2012/2/15 Pádraig Brady <P@draigbrady.com>:
> > >> On 02/07/2012 09:05 PM, Frank Mayhar wrote:
> > >>> This set of patches adds functions that help improve fsck operation in
> > >>> large installations and when running in unattended or headless mode.  It
> > >>> adds support for reporting rusage statistics for the individual fsck
> > >>> runs, for capturing fsck output, for killing fsck runs that take too
> > >>> long and for running scripts when each fsck completes.
> > >>>
> > >>> We're currently using these functions to improve our fsck monitoring
> > >>> capability and to replace some unwieldy and hard-to-maintain shell
> > >>> scripts.
> > >>
> > >> Couldn't you do this with separate fsck command runs,
> > >> and use standard system utils?
> > > 
> > > Yes, of course.  That's where the "unwieldy and hard-to-maintain shell
> > > scripts" came in.  Putting the functions in the wrapper itself, on the
> > > other hand, means the scripts don't have to reimplement functions that
> > > already exist there (like parallelizing the fsck runs or tracking exit
> 
>  BTW, the latest fsck supports new -l option (lock disk) for parallel
>  fsck processes. So you can start arbitrary number of
> 
>     fsck -l /dev/xxx
> 
>  without care about performance. We use it for systems with systemd
>  where fsck is executed per device (fstab entry).
> 
>  If you want to use the same thing for the classic init scripts then 
>  you can use something like
> 
>     for x in $(findmnt --fstab -n -o SOURCE); do
>         fsck -l $x &< /var/log/fsck-$x &
>     done
> 
>  rather tan fsck -A. Add some extra checks (completion scripts) to this
>  for() should be pretty simple.

I have to disagree with Pádraig's assertion that the functions are
supported by "quite simple" shell scripting.  As it turned out, doing
everything we needed involved a script that was pretty long and
involved.  (I wish I could share the script with you; it's pretty
impressive, actually, in a grim and painful kind of way, and a real pain
to maintain.)  In particular, the short loop you show here and that
Pádraig used was just not even close to being sufficient.

Further, those of us that are stuck using somewhat older distributions
don't have some of the newer tools such as findmnt and timeout.

> > > status), eliminates some external dependencies and makes the process
> > > quite a bit less fragile.
> > 
> > OK, thanks for the clarification.
> > 
> > It seems to me that these functions are supported
> > by quite simple shell scripting as I demonstrated.
> 
>  I have no problem with proposed -r option (to report memory and
>  runtime statistics) and -O option to force-kill fscks that run too
>  long.
> 
>  ... but I'm still not sure if we really need the completion script.

We have to do special stuff if an fsck fails for a particular file
system.  Without running each fsck individually (something I want to
avoid for a number of reasons), how do you propose we do that "special
stuff" without some command or script that is run by fsck when an
individual check fails?  In other words, what do you propose I use
instead of a completion script if I want to use the existing functions
of fsck for everything else?

I'm not trying to be rude, by the way (just in case this is coming off
that way), I'm just trying to point out that in our case we really need
the completion script and I would bet that there are others out there
that would find it very useful as well.
-- 
Frank Mayhar
fmayhar@google.com

Re: [PATCH 0/4] Add functions to the fsck wrapper to improve standalone operation.

[Pádraig Brady]

On 02/16/2012 08:17 PM, Frank Mayhar wrote:
> On Wed, 2012-02-15 at 18:42 +0100, Karel Zak wrote:
>> On Wed, Feb 15, 2012 at 04:06:35PM +0000, Pádraig Brady wrote:
>>> On 02/15/2012 03:09 PM, Frank Mayhar wrote:
>>>> 2012/2/15 Pádraig Brady <P@draigbrady.com>:
>>>>> On 02/07/2012 09:05 PM, Frank Mayhar wrote:
>>>>>> This set of patches adds functions that help improve fsck operation in
>>>>>> large installations and when running in unattended or headless mode.  It
>>>>>> adds support for reporting rusage statistics for the individual fsck
>>>>>> runs, for capturing fsck output, for killing fsck runs that take too
>>>>>> long and for running scripts when each fsck completes.
>>>>>>
>>>>>> We're currently using these functions to improve our fsck monitoring
>>>>>> capability and to replace some unwieldy and hard-to-maintain shell
>>>>>> scripts.
>>>>>
>>>>> Couldn't you do this with separate fsck command runs,
>>>>> and use standard system utils?
>>>>
>>>> Yes, of course.  That's where the "unwieldy and hard-to-maintain shell
>>>> scripts" came in.  Putting the functions in the wrapper itself, on the
>>>> other hand, means the scripts don't have to reimplement functions that
>>>> already exist there (like parallelizing the fsck runs or tracking exit
>>
>>  BTW, the latest fsck supports new -l option (lock disk) for parallel
>>  fsck processes. So you can start arbitrary number of
>>
>>     fsck -l /dev/xxx
>>
>>  without care about performance. We use it for systems with systemd
>>  where fsck is executed per device (fstab entry).
>>
>>  If you want to use the same thing for the classic init scripts then 
>>  you can use something like
>>
>>     for x in $(findmnt --fstab -n -o SOURCE); do
>>         fsck -l $x &< /var/log/fsck-$x &
>>     done
>>
>>  rather tan fsck -A. Add some extra checks (completion scripts) to this
>>  for() should be pretty simple.
> 
> I have to disagree with Pádraig's assertion that the functions are
> supported by "quite simple" shell scripting.  As it turned out, doing
> everything we needed involved a script that was pretty long and
> involved.  (I wish I could share the script with you; it's pretty
> impressive, actually, in a grim and painful kind of way, and a real pain
> to maintain.)  In particular, the short loop you show here and that
> Pádraig used was just not even close to being sufficient.

I still don't see how running a script dependent on the
exit code of fsck, is hard or any different to doing so internally.
Could you give a quick summary of the advantages of pulling
this logic into fsck. Scratch that, you do so below...

I see timeouts and completion scripts etc. as generic functionality.
Pulling stuff like that within commands is less unixy (more coupled) IMHO.

> Further, those of us that are stuck using somewhat older distributions
> don't have some of the newer tools such as findmnt and timeout.

Or newer fsck ;)

>>>> status), eliminates some external dependencies and makes the process
>>>> quite a bit less fragile.
>>>
>>> OK, thanks for the clarification.
>>>
>>> It seems to me that these functions are supported
>>> by quite simple shell scripting as I demonstrated.
>>
>>  I have no problem with proposed -r option (to report memory and
>>  runtime statistics) and -O option to force-kill fscks that run too
>>  long.
>>
>>  ... but I'm still not sure if we really need the completion script.
> 
> We have to do special stuff if an fsck fails for a particular file
> system.  Without running each fsck individually (something I want to
> avoid for a number of reasons),

please give a couple as this is a crucial point

> how do you propose we do that "special
> stuff" without some command or script that is run by fsck when an
> individual check fails?  In other words, what do you propose I use
> instead of a completion script if I want to use the existing functions
> of fsck for everything else?
> 
> I'm not trying to be rude, by the way (just in case this is coming off
> that way), I'm just trying to point out that in our case we really need
> the completion script and I would bet that there are others out there
> that would find it very useful as well.

I'm not trying to be obstructive BTW.
It just changes like these require appropriate justification.

cheers,
Pádraig.

Re: [PATCH 0/4] Add functions to the fsck wrapper to improve standalone operation.

[Frank Mayhar]

Sorry for the belated reply, I got bad news from the oncologist recently
and have been a bit distracted.

On Fri, 2012-02-17 at 02:53 +0000, Pádraig Brady wrote:
> On 02/16/2012 08:17 PM, Frank Mayhar wrote:
> > We have to do special stuff if an fsck fails for a particular file
> > system.  Without running each fsck individually (something I want to
> > avoid for a number of reasons),
> 
> please give a couple as this is a crucial point

One, memory use.  Running each fsck individually means we use more
memory than allowing fsck itself to do the parallelization.  Not a _lot_
more memory, certainly, but under certain conditions it can become
significant (e.g. when running in a cgroup, among other things).

Two, tracking multiple parallel instances of fsck from a shell script is
a lot less straightforward than allowing the fsck wrapper itself to do
so.  The fsck wrapper already has the code to do the tracking, the
functions I added simply build on that code.  Writing a shell script to
do the same thing (particularly when one has to handle fsck errors
specially, as we do) is redundant, potentially error-prone and (the real
kicker as far as I'm concerned) hard to maintain.  It also (three) adds
complexity to the start-up scripts which are already plenty complex
_and_ it adds a dependency on that script which would not exist
otherwise.  (That is, to do a "proper" fsck by hand one would either
have to set up the environment properly so that the script doesn't fall
over, or provide _another_ script that can be run independently.  It's a
_lot_ easier to be able to just type "fsck".)

Adding a way to allow special handling to fsck itself is easy (the code
is really straightforward), reduces the fsck footprint and reduces
complexity, making things easier to maintain.
-- 
Frank Mayhar
fmayhar@google.com