Old O_DIRECT story

Old O_DIRECT story

[Leon Pollak]

Hi, all.
There was a discussion here:
https://lkml.org/lkml/2007/1/10/231

Linus wrote in this discussion:
"So don't use O_DIRECT. Use things like madvise() and posix_fadvise()
instead"

After the full week of tests, searches, discussions, I have impudence to
turn to the community - has one tried to implement this approach?

The situation is very simple:
I have the incoming DMA stream using scatter/gather technique. the driver
read() function provides the next ready DMA buffer descriptor with the
virtual address pointer to the acquired data. I need to store this data to
the disk partition as fast as possible, as the incoming stream is too very
fast. According to tests, O_DIRECT/mapping is fast enough, while write() is
not.

I tried in all ways to implement this with mmap(), but it does not success,
because I did not find a way to mmap() file as O_WRONLY. Mapping as O_RDWR
makes kernel to pre-fill mapped memory with partition data. So, kernel and
DMA actually compete on the RAM area to fill it - one with garbage, one
with actual data. Kernel wins.

So, how to implement Linus's advice?
Leon

Re: Old O_DIRECT story

[Theodore Ts'o]

On Sat, Dec 27, 2014 at 03:31:26PM +0200, Leon Pollak wrote:
> Hi, all.
> There was a discussion here:
> https://lkml.org/lkml/2007/1/10/231
> 
> Linus wrote in this discussion:
> "So don't use O_DIRECT. Use things like madvise() and posix_fadvise()
> instead"
> 
> After the full week of tests, searches, discussions, I have impudence to
> turn to the community - has one tried to implement this approach?

As Linus stated in one of the other messages in the thread:

   As a result, our madvise and/or posix_fadvise interfaces may not be all
   that strong, because people sadly don't use them that much. It's a sad
   example of a totally broken interface (O_DIRECT) resulting in better
   interfaces not getting used, and then not getting as much development
   effort put into them.

There are two reasons to use O_DIRECT.  One is controlling the cache
usage, and the other is performance.

> The situation is very simple:
> I have the incoming DMA stream using scatter/gather technique. the driver
> read() function provides the next ready DMA buffer descriptor with the
> virtual address pointer to the acquired data. I need to store this data to
> the disk partition as fast as possible, as the incoming stream is too very
> fast. According to tests, O_DIRECT/mapping is fast enough, while write() is
> not.

Do you understand *why* write is not fast enough?  Is it realy a
matter of memory bandwidth issues, where you are actually limited by
the copy time implied by the write(2).  If you are being constrained
by memory bandwidth issues, then this won't help, but if the issue
with using buffered writes is that you can't control the writeback
precisely enough, you might try using sync_file_range(2).

The perf program should help confirm if you really are getting hit by
memory bandwidth issues.

> I tried in all ways to implement this with mmap(), but it does not success,
> because I did not find a way to mmap() file as O_WRONLY. Mapping as O_RDWR
> makes kernel to pre-fill mapped memory with partition data. So, kernel and
> DMA actually compete on the RAM area to fill it - one with garbage, one
> with actual data. Kernel wins.

I would be *very* surprised that mmap() is fast enough, because the
overhead in dealing with the page tables and TLB flush usually dooms
the mmap() method.

But if in fact the issue is the pre-fill with partition table, if you
are using a file system, and using fallocate so that you are mapping
in a sparse file, then there would be no pre-population.  I'm guessing
though that since you mention "partition data", you're using a raw
block device, right?

> So, how to implement Linus's advice?

Ultimately, if nothing else works, O_DIRECT is still there for a
reason.  Nothing should stop you from using it.  It is a very awkward
interface, yes, but from a design perspective, it is ugly as sin.  But
at the end of the day, you really need the performance, it's there for
you to use.

Cheers,

						- Ted

Re: Old O_DIRECT story

[One Thousand Gnomes]

> I tried in all ways to implement this with mmap(), but it does not success,
> because I did not find a way to mmap() file as O_WRONLY. Mapping as O_RDWR
> makes kernel to pre-fill mapped memory with partition data. So, kernel and
> DMA actually compete on the RAM area to fill it - one with garbage, one
> with actual data. Kernel wins.
> 
> So, how to implement Linus's advice?

Use O_DIRECT. There are lots of problems with the mmap() model, in
particular with how mmu table changes scale to large numbers of CPU
threads (ie they don't).

You would need to modify the kernel to add an madvise type of
soemthing like ZEROFILL (you can't do WRONLY because the x86 CPU can't
really do write only) but then you'd be stuck with only running on the
latest and greatest kernel. A ZEROFILL madvise would at least mean each
time you touched a new page it took a fault and did a 4K clear not a read.

mmap works well if you don't need to change the page permissions or map
new pages all the time. If you have to keep touching the MMU then it gets
ugly because you tend to need to synchronize btween cores.

Alan

Re: Old O_DIRECT story

[Kirill A. Shutemov]

On Mon, Jan 05, 2015 at 03:52:10PM +0000, One Thousand Gnomes wrote:
> > I tried in all ways to implement this with mmap(), but it does not success,
> > because I did not find a way to mmap() file as O_WRONLY. Mapping as O_RDWR
> > makes kernel to pre-fill mapped memory with partition data. So, kernel and
> > DMA actually compete on the RAM area to fill it - one with garbage, one
> > with actual data. Kernel wins.
> > 
> > So, how to implement Linus's advice?
> 
> Use O_DIRECT. There are lots of problems with the mmap() model, in
> particular with how mmu table changes scale to large numbers of CPU
> threads (ie they don't).

They do. Kinda. See split page table lock.
But, yeah, mmap() approach should not be faster anyway.

-- 
 Kirill A. Shutemov

Re: Old O_DIRECT story

[Leon Pollak]

IMHO(!):
1. It will be slower, as pre-fill takes its significant time.
2. DMA-Kernel competition makes this method unusable.

On 6 January 2015 at 04:04, Kirill A. Shutemov <kirill@shutemov.name> wrote:
> On Mon, Jan 05, 2015 at 03:52:10PM +0000, One Thousand Gnomes wrote:
>> > I tried in all ways to implement this with mmap(), but it does not success,
>> > because I did not find a way to mmap() file as O_WRONLY. Mapping as O_RDWR
>> > makes kernel to pre-fill mapped memory with partition data. So, kernel and
>> > DMA actually compete on the RAM area to fill it - one with garbage, one
>> > with actual data. Kernel wins.
>> >
>> > So, how to implement Linus's advice?
>>
>> Use O_DIRECT. There are lots of problems with the mmap() model, in
>> particular with how mmu table changes scale to large numbers of CPU
>> threads (ie they don't).
>
> They do. Kinda. See split page table lock.
> But, yeah, mmap() approach should not be faster anyway.
>
> --
>  Kirill A. Shutemov