Why is one sync() not enough?

Why is one sync() not enough?

[Nico Schottelius]

[-- Attachment #1: Type: text/plain, Size: 469 bytes --]

Hello again!

When my system shuts down and init calls sync() and after that
umount and then reboot, the filesystem is left in an unclean state.

If I do sync() two times (one before umount, one after umount) it
seems to work.

Can someboy explain that to me?

Architecture is ppc and x86.

Nico

-- 
Keep it simple & stupid, use what's available.
Please use pgp encryption: 8D0E 27A4 is my id.
http://nico.schotteli.us | http://linux.schottelius.org

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: Why is one sync() not enough?

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 471 bytes --]

hoi :)

On Tue, Jun 14, 2005 at 11:41:41AM +0200, Nico Schottelius wrote:
> When my system shuts down and init calls sync() and after that
> umount and then reboot, the filesystem is left in an unclean state.
> 
> If I do sync() two times (one before umount, one after umount) it
> seems to work.

unmounting the filesystem writes to the disk.
If you don't wait for those writes to reach the disk, then
you still have a dirty filesystem.

-- 
Martin Waitz

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Why is one sync() not enough?

[Jesper Juhl]

On 6/14/05, Martin Waitz <tali@admingilde.org> wrote:
> hoi :)
> 
> On Tue, Jun 14, 2005 at 11:41:41AM +0200, Nico Schottelius wrote:
> > When my system shuts down and init calls sync() and after that
> > umount and then reboot, the filesystem is left in an unclean state.
> >
> > If I do sync() two times (one before umount, one after umount) it
> > seems to work.
> 
> unmounting the filesystem writes to the disk.
> If you don't wait for those writes to reach the disk, then
> you still have a dirty filesystem.
> 

If I remember correctly, sync (at least in the past) schedules the
dumping of buffers to disk, but may return before everything is
actually written. Thus it was common in ages past to run  sync ; sync
; halt  when shutting down a system since even though the first sync
might return before writing was done, the second sync wouldn't start
before the first was really done, thus when the second sync returned
you'd know that at least the first one had completed.
I believe sync in recent Linux's actually waits for the data to be
written and never returns early, so I don't think this is relevant any
more - and everyone uses shutdown these days anyway. But go read the
implementation of sync if you really want to know.

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Why is one sync() not enough?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]

On Tue, 14 Jun 2005 17:58:36 +0200, Jesper Juhl said:

> actually written. Thus it was common in ages past to run  sync ; sync
> ; halt  when shutting down a system since even though the first sync
> might return before writing was done

Actually, it was:

# sync
# sync
# sync
# halt

If you enter 'sync;sync;sync' all 3 will get scheduled almost on top of each
other when you hit return.  If you hit return on each one, the first one starts
running when you hit return, and hopefully completes while you're typing the second
and third ones.....

'sync' on recent Linux kernels appears to be synchronous and not return until
the data is out the door.  If you're seeing 'sync; unmount; sync' being needed,
the most likely cause is that the *first* sync is bogus (as unmount will flush
the buffers too) - and the *second* sync is flushing out blocks from some file
that was opened in the directory *before you did the mount*:

# cd /usr/local  (with nothing mounted on it, so it's an empty dir in /usr)
# touch foo
# tail -f foo &
# mount /usr/local

Now that 'foo' is an open but unreachable file on the /usr filesystem, and will
likely cause the need for a sync *for /usr* after unmounting /usr/local at system
shutdown.

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Why is one sync() not enough?

[Nico Schottelius]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

Jesper Juhl [Tue, Jun 14, 2005 at 05:58:36PM +0200]:
> [sync twice, sync waits for sync]
> 
> I believe sync in recent Linux's actually waits for the data to be
> written and never returns early, so I don't think this is relevant any
> more - and everyone uses shutdown these days anyway. But go read the
> implementation of sync if you really want to know.

Thanks for the detailled instructions. I'll have a look at
the sync implementation to see how it really works.

Anyway: As I am writing an init-system, I also have to write a
shutdown procedure ;-)

Greetings,

Nico


-- 
Keep it simple & stupid, use what's available.
Please use pgp encryption: 8D0E 27A4 is my id.
http://nico.schotteli.us | http://linux.schottelius.org

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: Why is one sync() not enough?

[Andrew Morton]

Nico Schottelius <nico-kernel@schottelius.org> wrote:
>
> Hello again!
> 
> When my system shuts down and init calls sync() and after that
> umount and then reboot, the filesystem is left in an unclean state.
> 
> If I do sync() two times (one before umount, one after umount) it
> seems to work.
> 

That's a bug.

The standards say that sync() is supposed to "start" I/O, or something
similarly vague and waffly.  The Linux implementation of sync() has always
started all I/O and then waited upon all of it before returning from
sync().

And umount() itself will sync everything to disk, so the additional sync()
calls should be unnecessary.

That being said, if umount was leaving dirty filesystems then about 1000000
people would be complaining.  So there's something unusual about your
setup.

What filesystem?  What kernel version?  Any unusual bind mounts, loopback
mounts, etc?  There must be something there...

Re: Why is one sync() not enough?

[jmerkey]

There are cases where bitmaps can dirty themselves in the FS's (I have 
had to deal with several of these issues in 2.6) after a sync,
and I have found that:

#
# sync
# sync
#

is required on ext2 in some situations due to some race conditions with 
remote and local clients using device based FS's, so
there are some holes -- and comments in ext2 and several other FS's seem 
to bear this assumption out.

:-)

Jeff

Andrew Morton wrote:

>Nico Schottelius <nico-kernel@schottelius.org> wrote:
>  
>
>>Hello again!
>>
>>When my system shuts down and init calls sync() and after that
>>umount and then reboot, the filesystem is left in an unclean state.
>>
>>If I do sync() two times (one before umount, one after umount) it
>>seems to work.
>>
>>    
>>
>
>That's a bug.
>
>The standards say that sync() is supposed to "start" I/O, or something
>similarly vague and waffly.  The Linux implementation of sync() has always
>started all I/O and then waited upon all of it before returning from
>sync().
>
>And umount() itself will sync everything to disk, so the additional sync()
>calls should be unnecessary.
>
>That being said, if umount was leaving dirty filesystems then about 1000000
>people would be complaining.  So there's something unusual about your
>setup.
>
>What filesystem?  What kernel version?  Any unusual bind mounts, loopback
>mounts, etc?  There must be something there...
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>

Re: Why is one sync() not enough?

[Bernd Eckenfels]

In article <20050614215032.35d44e93.akpm@osdl.org> you wrote:
> What filesystem?  What kernel version?  Any unusual bind mounts, loopback
> mounts, etc?  There must be something there...

most likely unmount fils, or the ro-remount of root was not possible.

bernd

Re: Why is one sync() not enough?

[Nico Schottelius]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]

Hello Andrew,

you wrote:
> What filesystem?

jfs

> What kernel version?

2.6.11.11

> Any unusual bind mounts, loopback
> mounts, etc?  There must be something there...>

Yes, dm-crypt-mounted-jfs.

So if I understood everything correctly, a simple umount() without
a sync() before should be enough?

If so, I'll try that, I am happy about every less function call
I need to do.

Nico

P.S.: Helge, nice hint for debugging, cat'ing /proc/mounts before
      reboot/power-off/etc.!

-- 
Keep it simple & stupid, use what's available.
Please use pgp encryption: 8D0E 27A4 is my id.
http://nico.schotteli.us | http://linux.schottelius.org

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: Why is one sync() not enough?

[Andrew Morton]

Nico Schottelius <nico-kernel@schottelius.org> wrote:
>
> Hello Andrew,
> 
> you wrote:
> > What filesystem?
> 
> jfs

It would be useful to test with a different filesystem (ext3 mounted with
data=writeback is close to equivalent).  That'll help us identify the bug.

> > What kernel version?
> 
> 2.6.11.11
> 
> > Any unusual bind mounts, loopback
> > mounts, etc?  There must be something there...>
> 
> Yes, dm-crypt-mounted-jfs.

Again, if you can temporarily eliminate dm-crypt as well it will help
narrow it down.

> So if I understood everything correctly, a simple umount() without
> a sync() before should be enough?

Yes.

> If so, I'll try that, I am happy about every less function call
> I need to do.

OK.  sync()+umount() shouldn't take any longer than a bare umount() (which
has to do a sync too).  Unless you have multiple disks, in which case
there's no point in syncing the disks which aren't being umounted.  But
then, I have a feeling that standard util-linux umount(8) does a global
sync anyway.

Re: Why is one sync() not enough?

[Helge Hafting]

Nico Schottelius wrote:

>Hello again!
>
>When my system shuts down and init calls sync() and after that
>umount and then reboot, the filesystem is left in an unclean state.
>
>If I do sync() two times (one before umount, one after umount) it
>seems to work.
>
>Can someboy explain that to me?
>  
>
You shouldn't need those syncs, as umount does its own
syncing.  There may be other explanations:

* Your reboot actually powers down (or resets) the disk.
   IDE disks are known for caching stuff, they may indicate
   that data is written slightly before it actually happens.
   (The same applies to scsi - if you enable caching there for
   the little extra performance it buys.)
 
   Rebooting really quickly after umount in such a case can cut
   power to the disk before it finishes writing.  If this is the case,
   then a few seconds of sleep after umount before reboot
   will work just as well as that sync.  I don't recommend this
   as a solution, but it is an easy diagnostic!

* Your startup script accidentally mounted the fs twice.
  (Yes - linux support that, and the first umount won't undo
   both mounts.)  This simply means the fs isn't umounted
  when you reboot, but an extra sync and you might get lucky.
 
  Check to see that nothing at all is mounted as you reboot.

Helge Hafting

Re: Why is one sync() not enough?

[Denis Vlasenko]

On Wednesday 15 June 2005 11:17, Helge Hafting wrote:
> Nico Schottelius wrote:
> 
> >Hello again!
> >
> >When my system shuts down and init calls sync() and after that
> >umount and then reboot, the filesystem is left in an unclean state.
> >
> >If I do sync() two times (one before umount, one after umount) it
> >seems to work.

sync before umount is superfluous.

> >Can someboy explain that to me?
> >  
> >
> You shouldn't need those syncs, as umount does its own
> syncing.  There may be other explanations:
> 
> * Your reboot actually powers down (or resets) the disk.
>    IDE disks are known for caching stuff, they may indicate
>    that data is written slightly before it actually happens.
>    (The same applies to scsi - if you enable caching there for
>    the little extra performance it buys.)
>  
>    Rebooting really quickly after umount in such a case can cut
>    power to the disk before it finishes writing.  If this is the case,
>    then a few seconds of sleep after umount before reboot
>    will work just as well as that sync.  I don't recommend this
>    as a solution, but it is an easy diagnostic!
> 
> * Your startup script accidentally mounted the fs twice.
>   (Yes - linux support that, and the first umount won't undo
>    both mounts.)  This simply means the fs isn't umounted
>   when you reboot, but an extra sync and you might get lucky.

My reboot script is checking (/proc/mounts) for stray rw mounts on reboot,
prints a warning and waits for a keypress. This helps spot such things.
--
vda

Re: Why is one sync() not enough?

[Coywolf Qi Hunt]

On 6/15/05, Denis Vlasenko <vda@ilport.com.ua> wrote:
> On Wednesday 15 June 2005 11:17, Helge Hafting wrote:
> > Nico Schottelius wrote:
> >
> > >Hello again!
> > >
> > >When my system shuts down and init calls sync() and after that
> > >umount and then reboot, the filesystem is left in an unclean state.
> > >
> > >If I do sync() two times (one before umount, one after umount) it
> > >seems to work.
> 
> sync before umount is superfluous.
 
BTW, how about sysrq-s and sysrq-u? The same?
-- 
Coywolf Qi Hunt
http://ahbl.org/~coywolf/