server mode and verify_dump

server mode and verify_dump

[Sitsofe Wheeler]

Hi,

I've noticed that when running fio in server mode it's not possible to
get a dump of using verify_dump when a verify mismatch occurs. Is it
possible to add a mode where the server process saves the mismatch
files to its current working directory for inspection later?

-- 
Sitsofe | http://sucs.org/~sits/

Re: server mode and verify_dump

[Jens Axboe]

On 08/03/2015 03:12 PM, Sitsofe Wheeler wrote:
> Hi,
>
> I've noticed that when running fio in server mode it's not possible to
> get a dump of using verify_dump when a verify mismatch occurs. Is it
> possible to add a mode where the server process saves the mismatch
> files to its current working directory for inspection later?

That sounds like an oversight. To save time, can you let me know how you 
are invoking fio and with what job file? Should not be that hard to 
fixup, it should just work.

-- 
Jens Axboe

Re: server mode and verify_dump

[Jens Axboe]

On 08/14/2015 09:47 AM, Jens Axboe wrote:
> On 08/03/2015 03:12 PM, Sitsofe Wheeler wrote:
>> Hi,
>>
>> I've noticed that when running fio in server mode it's not possible to
>> get a dump of using verify_dump when a verify mismatch occurs. Is it
>> possible to add a mode where the server process saves the mismatch
>> files to its current working directory for inspection later?
>
> That sounds like an oversight. To save time, can you let me know how you
> are invoking fio and with what job file? Should not be that hard to
> fixup, it should just work.

Just tested this, and the fio backend does dump the files if 
verify_dump=1 is set:

[...]
<lenny> write3: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, 
iodepth=1
<lenny> Starting 1 process
<lenny> crc16: verify failed at file file offset 2023424, length 4096
<lenny>        Expected CRC: 5b54
<lenny>        Received CRC: c1aa
<lenny>        received data dumped as file.2023424.received
<lenny>        expected data dumped as file.2023424.expected


-- 
Jens Axboe

Re: server mode and verify_dump

[Sitsofe Wheeler]

On 14 August 2015 at 16:58, Jens Axboe <axboe@kernel.dk> wrote:
> On 08/14/2015 09:47 AM, Jens Axboe wrote:
>>
>> On 08/03/2015 03:12 PM, Sitsofe Wheeler wrote:
>>>
>>> Hi,
>>>
>>> I've noticed that when running fio in server mode it's not possible to
>>> get a dump of using verify_dump when a verify mismatch occurs. Is it
>>> possible to add a mode where the server process saves the mismatch
>>> files to its current working directory for inspection later?
>>
>>
>> That sounds like an oversight. To save time, can you let me know how you
>> are invoking fio and with what job file? Should not be that hard to
>> fixup, it should just work.
>
>
> Just tested this, and the fio backend does dump the files if verify_dump=1
> is set:
>
> [...]
> <lenny> write3: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync,
> iodepth=1
> <lenny> Starting 1 process
> <lenny> crc16: verify failed at file file offset 2023424, length 4096
> <lenny>        Expected CRC: 5b54
> <lenny>        Received CRC: c1aa
> <lenny>        received data dumped as file.2023424.received
> <lenny>        expected data dumped as file.2023424.expected

You're absolutely right. I think I got fooled because of two things:
1. I was trying to use dd to stomp zeros while also using the sync
ioengine on a file in /dev/shm and just getting a file I/O error.
2. If fio encounters an error while trying to read the header (even if
the blocks are all of the same size) it doesn't dump anything at all
(even in non-server circumstances)...

-- 
Sitsofe | http://sucs.org/~sits/

Re: server mode and verify_dump

[Jens Axboe]

On 08/14/2015 01:11 PM, Sitsofe Wheeler wrote:
> On 14 August 2015 at 16:58, Jens Axboe <axboe@kernel.dk> wrote:
>> On 08/14/2015 09:47 AM, Jens Axboe wrote:
>>>
>>> On 08/03/2015 03:12 PM, Sitsofe Wheeler wrote:
>>>>
>>>> Hi,
>>>>
>>>> I've noticed that when running fio in server mode it's not possible to
>>>> get a dump of using verify_dump when a verify mismatch occurs. Is it
>>>> possible to add a mode where the server process saves the mismatch
>>>> files to its current working directory for inspection later?
>>>
>>>
>>> That sounds like an oversight. To save time, can you let me know how you
>>> are invoking fio and with what job file? Should not be that hard to
>>> fixup, it should just work.
>>
>>
>> Just tested this, and the fio backend does dump the files if verify_dump=1
>> is set:
>>
>> [...]
>> <lenny> write3: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync,
>> iodepth=1
>> <lenny> Starting 1 process
>> <lenny> crc16: verify failed at file file offset 2023424, length 4096
>> <lenny>        Expected CRC: 5b54
>> <lenny>        Received CRC: c1aa
>> <lenny>        received data dumped as file.2023424.received
>> <lenny>        expected data dumped as file.2023424.expected
>
> You're absolutely right. I think I got fooled because of two things:
> 1. I was trying to use dd to stomp zeros while also using the sync
> ioengine on a file in /dev/shm and just getting a file I/O error.
> 2. If fio encounters an error while trying to read the header (even if
> the blocks are all of the same size) it doesn't dump anything at all
> (even in non-server circumstances)...

Right, I suspected that might have been the case. If the header isn't 
valid, then fio doesn't dump any contents. One could argue that it would 
be cleaner if the backend send the files to the frontend to store, since 
that's where you are running fio from. But at least they are there, on 
the backend side.

-- 
Jens Axboe

Re: server mode and verify_dump

[Sitsofe Wheeler]

On 14 August 2015 at 20:17, Jens Axboe <axboe@kernel.dk> wrote:
> Right, I suspected that might have been the case. If the header isn't valid,
> then fio doesn't dump any contents. One could argue that it would be cleaner

What do you think it would take to dump a block's worth of data when
encountering a header error during a verify?

-- 
Sitsofe | http://sucs.org/~sits/

Re: server mode and verify_dump

[Jens Axboe]

On 08/14/2015 01:21 PM, Sitsofe Wheeler wrote:
> On 14 August 2015 at 20:17, Jens Axboe <axboe@kernel.dk> wrote:
>> Right, I suspected that might have been the case. If the header isn't valid,
>> then fio doesn't dump any contents. One could argue that it would be cleaner
>
> What do you think it would take to dump a block's worth of data when
> encountering a header error during a verify?

Without a valid header, we can't re-generate the data. That's why it 
currently doesn't dump the expected contents if the header verification 
fails.

-- 
Jens Axboe

Re: server mode and verify_dump

[Sitsofe Wheeler]

On 14 August 2015 at 20:22, Jens Axboe <axboe@kernel.dk> wrote:
> On 08/14/2015 01:21 PM, Sitsofe Wheeler wrote:
>>
>> On 14 August 2015 at 20:17, Jens Axboe <axboe@kernel.dk> wrote:
>>>
>>> Right, I suspected that might have been the case. If the header isn't
>>> valid,
>>> then fio doesn't dump any contents. One could argue that it would be
>>> cleaner
>>
>>
>> What do you think it would take to dump a block's worth of data when
>> encountering a header error during a verify?
>
>
> Without a valid header, we can't re-generate the data. That's why it
> currently doesn't dump the expected contents if the header verification
> fails.

I see. Even fixed patterns would be a pain unless they have an
entirely separate code path. How about just dumping whatever you
received even if it was just a header's worth?

-- 
Sitsofe | http://sucs.org/~sits/

Re: server mode and verify_dump

[Jens Axboe]

On 08/14/2015 01:41 PM, Sitsofe Wheeler wrote:
> On 14 August 2015 at 20:22, Jens Axboe <axboe@kernel.dk> wrote:
>> On 08/14/2015 01:21 PM, Sitsofe Wheeler wrote:
>>>
>>> On 14 August 2015 at 20:17, Jens Axboe <axboe@kernel.dk> wrote:
>>>>
>>>> Right, I suspected that might have been the case. If the header isn't
>>>> valid,
>>>> then fio doesn't dump any contents. One could argue that it would be
>>>> cleaner
>>>
>>>
>>> What do you think it would take to dump a block's worth of data when
>>> encountering a header error during a verify?
>>
>>
>> Without a valid header, we can't re-generate the data. That's why it
>> currently doesn't dump the expected contents if the header verification
>> fails.
>
> I see. Even fixed patterns would be a pain unless they have an
> entirely separate code path. How about just dumping whatever you
> received even if it was just a header's worth?

It's not guaranteed that we could do it. We could always just fake up a 
new header, but we don't know the seed and other important things. It 
would be trivial to dump the received block of data, but that seems a 
bit pointless when we don't have the expected data. Since the 
offset+size is listed in the output, getting the same from 'dd' or a 
similar tool would be easy enough.

But I'm willing to be convinced. Dumping the received data is a quick 
change.

-- 
Jens Axboe