"errror=Invalid or incomplete multibyte or wide character" & "verify: bad header numberio" errors

"errror=Invalid or incomplete multibyte or wide character" & "verify: bad header numberio" errors

[Saeed]

Has anyone seen these errors before?

Here's the job file:

bs=4k
ioengine=libaio
iodepth=32
size=1g
direct=1
runtime=60
filename=/dev/sdb
group_reporting
numjobs=10

# validation
do_verify=1
verify_fatal=1
verify_dump=1

verify_pattern=0x01020304

[mixed-a]
rw=readwrite
stonewall

[mixed-b]
rw=randrw
stonewall

[mixed-c]
rw=readwrite:4k
stonewall

[mixed-d]
rw=randrw:4k
stonewall

Errors:

Multiple writers may overwrite blocks that belong to other jobs. This 
can cause verification failures.
mixed-verify-random-with-holes-4k: (g=3): rw=randrw, 
bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.12
Starting 40 processes
fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete, 
error=Invalid or incomplete multibyte or wide character
fio: pid=5814, err=84/file:io_u.c:1968, func=io_u_queued_complete, 
error=Invalid or incomplete multibyte or wide character
fio: pid=5808, err=84/file:io_u.c:1968, func=io_u_queued_complete, 
error=Invalid or incomplete multibyte or wide character

verify: bad header numberio 2985, wanted 11 at file /dev/sdb offset 
1224704, length 4096
       hdr_fail data dumped as sdb.1224704.hdr_fail
verify: bad header numberio 2986, wanted 12 at file /dev/sdb offset 
1228800, length 4096
       hdr_fail data dumped as sdb.1228800.hdr_fail
verify: bad header numberio 23727, wanted 22903 at file /dev/sdb offset 
106496, length 4096
       hdr_fail data dumped as sdb.106496.hdr_fail
verify: bad header numberio 13399, wanted 1319 at file /dev/sdb offset 
311296, length 4096
       hdr_fail data dumped as sdb.311296.hdr_fail

Re: "errror=Invalid or incomplete multibyte or wide character" & "verify: bad header numberio" errors

[Sitsofe Wheeler]

On 22 July 2016 at 17:29, Saeed <ionictea@gmail.com> wrote:
> Has anyone seen these errors before?
[...]
> fio-2.12
> Starting 40 processes
> fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete,
> error=Invalid or incomplete multibyte or wide character

This frequently happens when a header is bad during verification. For
example if the initial verification header magic bytes are bad then
verify_header()  (from verify.c - see
https://github.com/axboe/fio/blob/fio-2.12/verify.c#L788 ) will end up
doing the following:

        return EILSEQ;

EILSEQ is defined as 84.
http://man7.org/linux/man-pages/man3/errno.3.html says this represents
"Illegal byte sequence". However deep in glibc this is mapped to
"Invalid or incomplete multibyte or wide character" when using
strerror (see https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/gnu/errlist.c;h=a9a5960503a97e78143a0c795e7255f418a22933;hb=HEAD#l929
). This seems to stem from its frequent usage with multibyte functions
(see http://www.lysator.liu.se/c/na1.html ).

So glibc has given EILSEQ a narrow error message ("Invalid or
incomplete multibyte or wide character") even though it originally had
a general meaning ("Illegal byte sequence") and fio is uses EILSEQ to
indicate to indicate it has found a corrupt header. I'm not sure
there's all that much that can be done about this...

-- 
Sitsofe | http://sucs.org/~sits/

Re: "errror=Invalid or incomplete multibyte or wide character" & "verify: bad header numberio" errors

[Sitsofe Wheeler]

On 22 July 2016 at 17:29, Saeed <ionictea@gmail.com> wrote:
> Has anyone seen these errors before?
>
> Here's the job file:
>
> bs=4k
> ioengine=libaio
> iodepth=32
> size=1g
> direct=1
> runtime=60
> filename=/dev/sdb
> group_reporting
> numjobs=10
>
> # validation
> do_verify=1
> verify_fatal=1
> verify_dump=1
>
> verify_pattern=0x01020304
>
> [mixed-a]
> rw=readwrite
> stonewall
>
> [mixed-b]
> rw=randrw
> stonewall
[...]
> fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete,
> error=Invalid or incomplete multibyte or wide character

I forgot to say that you might be getting these errors because you
limit the runtime of individual jobs to 60 seconds and that might not
be enough time for your mixed-a job to overwrite the entire 1GByte of
the file before it stopped due to running out of time. If later
verification jobs try to verify an area not written to, the header may
not be there or/malformed thus generating error messages.

-- 
Sitsofe | http://sucs.org/~sits/

On 22 July 2016 at 17:29, Saeed <ionictea@gmail.com> wrote:
> Has anyone seen these errors before?
>
> Here's the job file:
>
> bs=4k
> ioengine=libaio
> iodepth=32
> size=1g
> direct=1
> runtime=60
> filename=/dev/sdb
> group_reporting
> numjobs=10
>
> # validation
> do_verify=1
> verify_fatal=1
> verify_dump=1
>
> verify_pattern=0x01020304
>
> [mixed-a]
> rw=readwrite
> stonewall
>
> [mixed-b]
> rw=randrw
> stonewall
>
> [mixed-c]
> rw=readwrite:4k
> stonewall
>
> [mixed-d]
> rw=randrw:4k
> stonewall
>
> Errors:
>
> Multiple writers may overwrite blocks that belong to other jobs. This can
> cause verification failures.
> mixed-verify-random-with-holes-4k: (g=3): rw=randrw, bs=4K-4K/4K-4K/4K-4K,
> ioengine=libaio, iodepth=32
> ...
> fio-2.12
> Starting 40 processes
> fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete,
> error=Invalid or incomplete multibyte or wide character
> fio: pid=5814, err=84/file:io_u.c:1968, func=io_u_queued_complete,
> error=Invalid or incomplete multibyte or wide character
> fio: pid=5808, err=84/file:io_u.c:1968, func=io_u_queued_complete,
> error=Invalid or incomplete multibyte or wide character
>
> verify: bad header numberio 2985, wanted 11 at file /dev/sdb offset 1224704,
> length 4096
>       hdr_fail data dumped as sdb.1224704.hdr_fail
> verify: bad header numberio 2986, wanted 12 at file /dev/sdb offset 1228800,
> length 4096
>       hdr_fail data dumped as sdb.1228800.hdr_fail
> verify: bad header numberio 23727, wanted 22903 at file /dev/sdb offset
> 106496, length 4096
>       hdr_fail data dumped as sdb.106496.hdr_fail
> verify: bad header numberio 13399, wanted 1319 at file /dev/sdb offset
> 311296, length 4096
>       hdr_fail data dumped as sdb.311296.hdr_fail
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Sitsofe | http://sucs.org/~sits/

Re: "errror=Invalid or incomplete multibyte or wide character" & "verify: bad header numberio" errors

[Saeed]

I reduced the number of jobs down to 1 and now getting no errors.

A


https://bugs.chromium.org/p/chromium/issues/detail?id=337651


On 2016-07-22 19:06:08 +0000, Sitsofe Wheeler said:

> On 22 July 2016 at 17:29, Saeed 
> <ionictea@gmail.com> wrote:
>> Has anyone seen these errors before?
>> 
>> Here's the job file:
>> 
>> bs=4k
>> ioengine=libaio
>> iodepth=32
>> size=1g
>> direct=1
>> runtime=60
>> filename=/dev/sdb
>> group_reporting
>> numjobs=10
>> 
>> # validation
>> do_verify=1
>> verify_fatal=1
>> verify_dump=1
>> 
>> verify_pattern=0x01020304
>> 
>> [mixed-a]
>> rw=readwrite
>> stonewall
>> 
>> [mixed-b]
>> rw=randrw
>> stonewall
> [...]
>> fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete,
>> error=Invalid or incomplete multibyte or wide character
> 
> I forgot to say that you might be getting these errors because you
> limit the runtime of individual jobs to 60 seconds and that might not
> be enough time for your mixed-a job to overwrite the entire 1GByte of
> the file before it stopped due to running out of time. If later
> verification jobs try to verify an area not written to, the header may
> not be there or/malformed thus generating error messages.
> 
> -- 
> Sitsofe | http://sucs.org/~sits/
> 
> On 22 July 2016 at 17:29, Saeed 
> <ionictea@gmail.com> wrote:
>> Has anyone seen these errors before?
>> 
>> Here's the job file:
>> 
>> bs=4k
>> ioengine=libaio
>> iodepth=32
>> size=1g
>> direct=1
>> runtime=60
>> filename=/dev/sdb
>> group_reporting
>> numjobs=10
>> 
>> # validation
>> do_verify=1
>> verify_fatal=1
>> verify_dump=1
>> 
>> verify_pattern=0x01020304
>> 
>> [mixed-a]
>> rw=readwrite
>> stonewall
>> 
>> [mixed-b]
>> rw=randrw
>> stonewall
>> 
>> [mixed-c]
>> rw=readwrite:4k
>> stonewall
>> 
>> [mixed-d]
>> rw=randrw:4k
>> stonewall
>> 
>> Errors:
>> 
>> Multiple writers may overwrite blocks that belong to other jobs. This can
>> cause verification failures.
>> mixed-verify-random-with-holes-4k: (g=3): rw=randrw, bs=4K-4K/4K-4K/4K-4K,
>> ioengine=libaio, iodepth=32
>> ...
>> fio-2.12
>> Starting 40 processes
>> fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete,
>> error=Invalid or incomplete multibyte or wide character
>> fio: pid=5814, err=84/file:io_u.c:1968, func=io_u_queued_complete,
>> error=Invalid or incomplete multibyte or wide character
>> fio: pid=5808, err=84/file:io_u.c:1968, func=io_u_queued_complete,
>> error=Invalid or incomplete multibyte or wide character
>> 
>> verify: bad header numberio 2985, wanted 11 at file /dev/sdb offset 1224704,
>> length 4096
>> hdr_fail data dumped as sdb.1224704.hdr_fail
>> verify: bad header numberio 2986, wanted 12 at file /dev/sdb offset 1228800,
>> length 4096
>> hdr_fail data dumped as sdb.1228800.hdr_fail
>> verify: bad header numberio 23727, wanted 22903 at file /dev/sdb offset
>> 106496, length 4096
>> hdr_fail data dumped as sdb.106496.hdr_fail
>> verify: bad header numberio 13399, wanted 1319 at file /dev/sdb offset
>> 311296, length 4096
>> hdr_fail data dumped as sdb.311296.hdr_fail
>> 
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "errror=Invalid or incomplete multibyte or wide character" & "verify: bad header numberio" errors

[Jens Axboe]

On 07/24/2016 09:33 AM, Saeed wrote:
> I reduced the number of jobs down to 1 and now getting no errors.

That was your problem. If you look at your original fio output:

"Multiple writers may overwrite blocks that belong to other jobs. This 
can cause verification failures. "

fio warns you about that as well. And your actual verification errors 
show that to be the case as well:

"verify: bad header numberio 2985, wanted 11 at file /dev/sdb offset
1224704"

-- 
Jens Axboe