Is concurrent file read/write with O_DIRECT flag atomic?

[Leo Chen <liangc8367@gmail.com>]

Hi,

I apologize if this topic is not proper for this mail list. I asked
the question on other channel, but haven't got answers yet (see
https://stackoverflow.com/questions/47245162/is-concurrent-file-read-write-with-o-direct-flag-atomic).

Basically, I have a non-sparse binary file. A writer process opens the
file using O_DIRECT flag, and it keeps calling pwrite() to update the
first 128KB data of the file. Meanwhile, multiple readers also keeps
calling pread() to read the first 128KB data. The readers open the
file using O_DIRECT flag.

Although I could not find any document saying that O_DIRECT guarantee
atomicity of concurrent read/write, I thought that the readers should
read back consistent data. Since the data to read/write from/to the
file is block aligned, I assume that kernel would just submit a single
scatter-gather command for one pwrite() or one pread(). Per my
knowledge of SCSI/SATA driver, HDD should (correct me if I'm wrong)
process each scatter-gather command atomically. Therefore, I thought
read/write operations in this scenario are atomic.

I wrote a program to verify my thought. Surprisingly, the readers did
occasionally read back mixed data. For example, in the first pwrite(),
the writer writes all 0x11, and in the 2nd pwrite(), it writes all
0x22, and in the 3rd write, it writes all 0x33... Occasionally, a
reader can read back data like "0x11, 0x11, .... 0x11, 0x22, 0x22....
0x22". The data appears to be from two consecutive pwrite() calls. I
checked the offset where the broken starts. The offset seems to be
sector-aligned (512-byte-aligned).

Why could such consistent issue happen? Did I miss anything in my
analysis and theory?

Thanks,
Leo