Re: ext4_clear_journal_err: Filesystem error recorded from previous mount: IO failure

[Bernd Schubert <bschubert@ddn.com>]

[-- Attachment #1: Type: text/plain, Size: 3020 bytes --]

On 10/24/2010 05:49 PM, Ric Wheeler wrote:
>   On 10/24/2010 11:39 AM, Bernd Schubert wrote:
>> On 10/24/2010 05:20 PM, Ric Wheeler wrote:
>>> This still sounds more like a Lustre issue than an ext4 one, Andreas can fill in
>>> the technical details.
>> The underlying device handling is unrelated to Lustre. In that sense it
>> is just a local filesystem.
>>
>>> What ever shared storage sits under ext4 is irrelevant to the fail over case.
>>>
>>> Unless Lustre does other magic, they still need to obey the basic cluster rules
>>> - one mount per cluster.
>> Yes, one mount per cluster.
>>
>>> If Lustre is doing the same trick you would do with active/passive failure over
>>> clusters that export ext4 via NFS, you would still need to clean up the file
>>> system before being able to re-export it from a fail over node.
>> What exactly is your question here? We use pacemaker/stonith to do the
>> fencing job.
>> What exactly do you want to clean up? The device is recovered by
>> journals, Lustre goes into recovery mode, clients reconnect, locks are
>> updated and incomplete transactions resend.
>>
>>
>> Cheers,
>> Bernd
>>
> 
> What I don't get (certainly might just be me) is why this is a unique issue when 
> used by lustre. Normally, any similar type of fail over will clean up the local 
> file system normally before trying to re-export from the second node.

Of course that is not a Lustre specific issue, which is why I also did
not open a Lustre bugzilla, but opened the thread here.

> 
> Why exactly can't you use the same type of recovery here? Is it the fencing 
> agent killing nodes on detection of the file system errors?

But I'm using the same type of recovery! I just rewrote pacemakers
default "Filesystem" agent to a lustre_server agent, to include more
Lustre specific checks. When I then added last week a check for the
dumpe2fs "Filesystem state", I noticed, that sometimes the error state
is only set *after* mounting the filesystem, so difficult to script it.
And as I also wrote, running e2fsck from that script and to do a
complete fs check is not appropriate, as that might simply time out.
Again not Lustre specific. So after some discussion, the proposed
solution is to add a "journal recovery only" option to e2fsck and to do
that before the mount. I will add that to the 'lustre_server' agent
(which is part of Lustre now), but leave it to someone else to that for
the 'Filesystem' agent script (I'm not using that script myself and IMHO
it is already too complex, as it tries to support all filesystems -
shell code is ideal anymore then).

Really, only Lustre specific here is the feature to have a proc file to
see if filesystem errors came up on a node. That is a missing feature in
extX and all other linux filesystems I have worked with. And Lustre
server nodes just means the usage of dozens to hundreds of
ext3/ext4/ldiskfs devices, so bugs are more likely exposed by that high
number.


Cheers,
Bernd


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]