Appropriate use of sync() from user space?

Appropriate use of sync() from user space?

[Mark Mielke]

Hi all:

Quick summary: We have a vendor who is claiming that it is required for 
their userspace program to execute sync(), and I am looking for some 
sort of authoritative document or person to refer them to that will 
state that this belief is incorrect and/or that this architecture is not 
acceptable in a Unix environment.

I checked Google and the archives and didn't find anything appropriate. 
Unfortunately, the word "sync" is very popular. :-)

We have users who have been experiencing 3 to 5 minutes "freezes" for a 
particular command which often times out and fails. I traced this down 
from the commercial userspace program (IBM Rational ClearCase / 
"cleartool mkview") that they are executing to a backend "view_server" 
process (also IBM Rational ClearCase) that is running sync() as a means 
of synchronizing their database to disk before proceeding, and VMware 
using a "large" memory mapped file to back it's virtual "RAM". The 
sync() for my computer normally completes in 7 to 8 seconds. The sync() 
for some of our users is taking 5 minutes or longer. This can be 
demonstrated simply by typing "time sync" from the command line at 
intervals. The time itself is relevant because if it finishes before a 
timeout elapses - the operation works (albeit slowly). If the timeout 
elapses, the operation fails.

The vendor stated that sync() is integral to their synchronization 
process to ensure all files reach disk before they are accessed, and 
that this is not a defect in their product. We have a work around - run 
"sync" before calling their command, and this generally avoids the failures.

I think the use of sync() in this regard is a hack. According to POSIX.1 
and the Linux man pages, it seems clear to me that sync() does not 
guarantee data integrity (bytes guaranteed to have reached disk) - and 
it also seems clear that forcing all system data to flush out in 
response to a minor command is over kill. Like cutting down the forest 
to harvest fruit from a single tree.

I'm wondering what you think.

Thanks!

-- 
Mark Mielke<mark@mielke.cc>

Re: Appropriate use of sync() from user space?

[Chris Friesen]

On 10/05/2011 05:57 PM, Mark Mielke wrote:

> The vendor stated that sync() is integral to their synchronization
> process to ensure all files reach disk before they are accessed, and
> that this is not a defect in their product. We have a work around - run
> "sync" before calling their command, and this generally avoids the
> failures.
>
> I think the use of sync() in this regard is a hack. According to POSIX.1
> and the Linux man pages, it seems clear to me that sync() does not
> guarantee data integrity (bytes guaranteed to have reached disk) - and
> it also seems clear that forcing all system data to flush out in
> response to a minor command is over kill. Like cutting down the forest
> to harvest fruit from a single tree.
>
> I'm wondering what you think.

Totally agree.

The susv3 man page for sync() is pretty clear:  "The writing, although 
scheduled, is not necessarily complete upon return from sync()."

They should probably be using msync()/fsync()/fdatasync() which only 
affect the specified files and are supposed to wait for the data to hit 
the storage device.  Of course this would require them to do something 
for each file they touch rather than once at the end of the whole operation.

Chris

-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com

Re: Appropriate use of sync() from user space?

[Jan Kara]

  Hello,

On Wed 05-10-11 19:57:03, Mark Mielke wrote:
> Quick summary: We have a vendor who is claiming that it is required
> for their userspace program to execute sync(), and I am looking for
> some sort of authoritative document or person to refer them to that
> will state that this belief is incorrect and/or that this
> architecture is not acceptable in a Unix environment.
> 
> I checked Google and the archives and didn't find anything
> appropriate. Unfortunately, the word "sync" is very popular. :-)
> 
> We have users who have been experiencing 3 to 5 minutes "freezes"
> for a particular command which often times out and fails. I traced
> this down from the commercial userspace program (IBM Rational
> ClearCase / "cleartool mkview") that they are executing to a backend
> "view_server" process (also IBM Rational ClearCase) that is running
> sync() as a means of synchronizing their database to disk before
> proceeding, and VMware using a "large" memory mapped file to back
> it's virtual "RAM". The sync() for my computer normally completes in
> 7 to 8 seconds. The sync() for some of our users is taking 5 minutes
> or longer. This can be demonstrated simply by typing "time sync"
> from the command line at intervals. The time itself is relevant
> because if it finishes before a timeout elapses - the operation
> works (albeit slowly). If the timeout elapses, the operation fails.
> 
> The vendor stated that sync() is integral to their synchronization
> process to ensure all files reach disk before they are accessed, and
> that this is not a defect in their product. We have a work around -
> run "sync" before calling their command, and this generally avoids
> the failures.
> 
> I think the use of sync() in this regard is a hack. According to
> POSIX.1 and the Linux man pages, it seems clear to me that sync()
> does not guarantee data integrity (bytes guaranteed to have reached
> disk) - and it also seems clear that forcing all system data to
> flush out in response to a minor command is over kill. Like cutting
> down the forest to harvest fruit from a single tree.
  Actually the manpage is wrong. Linux waits for all data to be safely on
disk before sync returns. So calling sync is a correct way (although
inefficient at times) to achieve data integrity. What kernel version are
you using? Different kernel versions are differently efficient when doing
sync(2) and quite some effort went to make sync less prone to livelocks in
recent kernels...

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: Appropriate use of sync() from user space?

[David Rientjes]

On Tue, 18 Oct 2011, Jan Kara wrote:

> > Quick summary: We have a vendor who is claiming that it is required
> > for their userspace program to execute sync(), and I am looking for
> > some sort of authoritative document or person to refer them to that
> > will state that this belief is incorrect and/or that this
> > architecture is not acceptable in a Unix environment.
> > 
> > I checked Google and the archives and didn't find anything
> > appropriate. Unfortunately, the word "sync" is very popular. :-)
> > 
> > We have users who have been experiencing 3 to 5 minutes "freezes"
> > for a particular command which often times out and fails. I traced
> > this down from the commercial userspace program (IBM Rational
> > ClearCase / "cleartool mkview") that they are executing to a backend
> > "view_server" process (also IBM Rational ClearCase) that is running
> > sync() as a means of synchronizing their database to disk before
> > proceeding, and VMware using a "large" memory mapped file to back
> > it's virtual "RAM". The sync() for my computer normally completes in
> > 7 to 8 seconds. The sync() for some of our users is taking 5 minutes
> > or longer. This can be demonstrated simply by typing "time sync"
> > from the command line at intervals. The time itself is relevant
> > because if it finishes before a timeout elapses - the operation
> > works (albeit slowly). If the timeout elapses, the operation fails.
> > 
> > The vendor stated that sync() is integral to their synchronization
> > process to ensure all files reach disk before they are accessed, and
> > that this is not a defect in their product. We have a work around -
> > run "sync" before calling their command, and this generally avoids
> > the failures.
> > 
> > I think the use of sync() in this regard is a hack. According to
> > POSIX.1 and the Linux man pages, it seems clear to me that sync()
> > does not guarantee data integrity (bytes guaranteed to have reached
> > disk) - and it also seems clear that forcing all system data to
> > flush out in response to a minor command is over kill. Like cutting
> > down the forest to harvest fruit from a single tree.
>   Actually the manpage is wrong. Linux waits for all data to be safely on
> disk before sync returns. So calling sync is a correct way (although
> inefficient at times) to achieve data integrity. What kernel version are
> you using? Different kernel versions are differently efficient when doing
> sync(2) and quite some effort went to make sync less prone to livelocks in
> recent kernels...
> 

Let's make sure to keep Michael Kerrisk cc'd if anything needs to be 
clarified in the manpages.

Re: Appropriate use of sync() from user space?

[david]

On Tue, 18 Oct 2011, David Rientjes wrote:

> On Tue, 18 Oct 2011, Jan Kara wrote:
>
>>> Quick summary: We have a vendor who is claiming that it is required
>>> for their userspace program to execute sync(), and I am looking for
>>> some sort of authoritative document or person to refer them to that
>>> will state that this belief is incorrect and/or that this
>>> architecture is not acceptable in a Unix environment.
>>>
>>> I checked Google and the archives and didn't find anything
>>> appropriate. Unfortunately, the word "sync" is very popular. :-)
>>>
>>> We have users who have been experiencing 3 to 5 minutes "freezes"
>>> for a particular command which often times out and fails. I traced
>>> this down from the commercial userspace program (IBM Rational
>>> ClearCase / "cleartool mkview") that they are executing to a backend
>>> "view_server" process (also IBM Rational ClearCase) that is running
>>> sync() as a means of synchronizing their database to disk before
>>> proceeding, and VMware using a "large" memory mapped file to back
>>> it's virtual "RAM". The sync() for my computer normally completes in
>>> 7 to 8 seconds. The sync() for some of our users is taking 5 minutes
>>> or longer. This can be demonstrated simply by typing "time sync"
>>> from the command line at intervals. The time itself is relevant
>>> because if it finishes before a timeout elapses - the operation
>>> works (albeit slowly). If the timeout elapses, the operation fails.
>>>
>>> The vendor stated that sync() is integral to their synchronization
>>> process to ensure all files reach disk before they are accessed, and
>>> that this is not a defect in their product. We have a work around -
>>> run "sync" before calling their command, and this generally avoids
>>> the failures.
>>>
>>> I think the use of sync() in this regard is a hack. According to
>>> POSIX.1 and the Linux man pages, it seems clear to me that sync()
>>> does not guarantee data integrity (bytes guaranteed to have reached
>>> disk) - and it also seems clear that forcing all system data to
>>> flush out in response to a minor command is over kill. Like cutting
>>> down the forest to harvest fruit from a single tree.
>>   Actually the manpage is wrong. Linux waits for all data to be safely on
>> disk before sync returns. So calling sync is a correct way (although
>> inefficient at times) to achieve data integrity. What kernel version are
>> you using? Different kernel versions are differently efficient when doing
>> sync(2) and quite some effort went to make sync less prone to livelocks in
>> recent kernels...
>>
>
> Let's make sure to keep Michael Kerrisk cc'd if anything needs to be
> clarified in the manpages.

also, you may want to check if they are really doing a 'sync' (syncing the 
entire filesystem) or just a 'fsync' (syncing the file). Depending on the 
technical depth of the people you are talking to, they may say sync when 
what is actually happening is a fsync.

there is little dispute that fsync is correct, but not a complete answer 
to the issue. take a look at the LWN article on the subject at 
http://lwn.net/Articles/457667

Ext3 has a pathalogical condition where a sync to one file can force a 
complete journal flush, which isn't as bad as a sync of the entire 
filesystem, but can still take a long time if there is other ongoing write 
activity on the system (I knwo I've read about fsyncs taking longer than 
30 seconds, and I think I've heard of them taking minutes). As far as I 
know, Ext3 is the only filesystem to suffer this problem, but 
unfortunantly it's the default filesystem on most linux distros.

David Lang

Re: Appropriate use of sync() from user space?

[Mark Mielke]

On 10/18/2011 08:03 PM, david@lang.hm wrote:
> On Tue, 18 Oct 2011, David Rientjes wrote:
>> On Tue, 18 Oct 2011, Jan Kara wrote:
>>>> Quick summary: We have a vendor who is claiming that it is required
>>>> for their userspace program to execute sync(), and I am looking for
>>>> some sort of authoritative document or person to refer them to that
>>>> will state that this belief is incorrect and/or that this
>>>> architecture is not acceptable in a Unix environment.
>>>>
> also, you may want to check if they are really doing a 'sync' (syncing 
> the entire filesystem) or just a 'fsync' (syncing the file). Depending 
> on the technical depth of the people you are talking to, they may say 
> sync when what is actually happening is a fsync.
>
> there is little dispute that fsync is correct, but not a complete 
> answer to the issue. take a look at the LWN article on the subject at 
> http://lwn.net/Articles/457667

Thanks, Dave. Yes, the vendor is doing a real sync() - and is also doing 
a few fsync(). We managed to convince them that this is a defect through 
their sales channel as technical failed. Oh well.

I note that you say "there is little dispute that fsync is correct ..." 
- which I also would have assumed, but I found very little authoritative 
documents on this. I think it is just so "self evident" (which the 
exception that you state in ext3) that people don't tend to do this.

In the mean time, our users are literally doing "sync; cleartool mkview 
..." which forces most of the data out to disk before starting the 
vendor command, so that the IBM Rational "cleartool mkview" is able to 
complete successfully without timing out and failing after 3 to 5 
minutes. *sigh*

The problematic file seems to be a VMware Player mmap()'d file with size 
of 1 GByte or more. ClearCase "cleartool mkview" vs VMware Player normal 
operation. *sigh x 2*

Thanks,
mark

-- 
Mark Mielke<mark@mielke.cc>