Truncated file on posix mount

Truncated file on posix mount

[sverre stoltenberg]

I have a ceph posix mount for openstack instances, and have found some 
of my kernels truncated. It looks rather random, and I have not found a 
way to reproduce it, but it will happen from time to time with normal use.

the 5e...aa checksum is the correct one.

root@t06-16:/var/lib/nova/instances# md5sum */kernel
3e0a48130ddb8bfe88239df05c073a0b  instance-00000001/kernel
5e9463bd77757a3889a1d9758b2b94aa  instance-0000000f/kernel
5e9463bd77757a3889a1d9758b2b94aa  instance-00000010/kernel
5e9463bd77757a3889a1d9758b2b94aa  instance-00000011/kernel
5e9463bd77757a3889a1d9758b2b94aa  instance-00000012/kernel
5e9463bd77757a3889a1d9758b2b94aa  instance-00000013/kernel
3e0a48130ddb8bfe88239df05c073a0b  instance-00000014/kernel

root@t06-16:/var/lib/nova/instances/instance-00000001# stat kernel
   File: `kernel'
   Size: 2097152         Blocks: 4096       IO Block: 4194304 regular file
Device: 0h/0d   Inode: 1099511627899  Links: 1
Access: (0664/-rw-rw-r--)  Uid: (  109/libvirt-qemu)   Gid: (  111/     kvm)
Access: 2012-06-13 11:56:56.462433802 +0200
Modify: 2012-06-04 11:47:17.730781000 +0200
Change: 2012-06-04 11:47:17.730781000 +0200
  Birth: -

root@t06-16:/var/lib/nova/instances/instance-00000010# stat kernel
   File: `kernel'
   Size: 2424448         Blocks: 4736       IO Block: 4194304 regular file
Device: 0h/0d   Inode: 1099511627989  Links: 1
Access: (0664/-rw-rw-r--)  Uid: (  109/libvirt-qemu)   Gid: (  111/     kvm)
Access: 2012-06-13 09:44:03.779892491 +0200
Modify: 2012-06-13 09:43:47.178125000 +0200
Change: 2012-06-13 09:44:00.020102878 +0200
  Birth: -

-- 
/S.

Re: Truncated file on posix mount

[Wido den Hollander]

Hi,

On 15-06-12 11:43, sverre stoltenberg wrote:
> I have a ceph posix mount for openstack instances, and have found some
> of my kernels truncated. It looks rather random, and I have not found a
> way to reproduce it, but it will happen from time to time with normal use.
>

Can you share the following information?

- Kernel version
- Ceph version
- "ceph -s" output

The MDS / POSIX filesystem hasn't been tested very well at the moment, 
in the next sprint (starting soon) the developers will start focusing on 
the POSIX filesystem.

Thanks,

Wido

> the 5e...aa checksum is the correct one.
>
> root@t06-16:/var/lib/nova/instances# md5sum */kernel
> 3e0a48130ddb8bfe88239df05c073a0b instance-00000001/kernel
> 5e9463bd77757a3889a1d9758b2b94aa instance-0000000f/kernel
> 5e9463bd77757a3889a1d9758b2b94aa instance-00000010/kernel
> 5e9463bd77757a3889a1d9758b2b94aa instance-00000011/kernel
> 5e9463bd77757a3889a1d9758b2b94aa instance-00000012/kernel
> 5e9463bd77757a3889a1d9758b2b94aa instance-00000013/kernel
> 3e0a48130ddb8bfe88239df05c073a0b instance-00000014/kernel
>
> root@t06-16:/var/lib/nova/instances/instance-00000001# stat kernel
> File: `kernel'
> Size: 2097152 Blocks: 4096 IO Block: 4194304 regular file
> Device: 0h/0d Inode: 1099511627899 Links: 1
> Access: (0664/-rw-rw-r--) Uid: ( 109/libvirt-qemu) Gid: ( 111/ kvm)
> Access: 2012-06-13 11:56:56.462433802 +0200
> Modify: 2012-06-04 11:47:17.730781000 +0200
> Change: 2012-06-04 11:47:17.730781000 +0200
> Birth: -
>
> root@t06-16:/var/lib/nova/instances/instance-00000010# stat kernel
> File: `kernel'
> Size: 2424448 Blocks: 4736 IO Block: 4194304 regular file
> Device: 0h/0d Inode: 1099511627989 Links: 1
> Access: (0664/-rw-rw-r--) Uid: ( 109/libvirt-qemu) Gid: ( 111/ kvm)
> Access: 2012-06-13 09:44:03.779892491 +0200
> Modify: 2012-06-13 09:43:47.178125000 +0200
> Change: 2012-06-13 09:44:00.020102878 +0200
> Birth: -
>

Re: Truncated file on posix mount

[sverre stoltenberg]

On 06/15/2012 01:32 PM, Wido den Hollander wrote:
> Hi,
>
> On 15-06-12 11:43, sverre stoltenberg wrote:
>> I have a ceph posix mount for openstack instances, and have found some
>> of my kernels truncated. It looks rather random, and I have not found a
>> way to reproduce it, but it will happen from time to time with normal
>> use.
>>
>
> Can you share the following information?
>
> - Kernel version
> - Ceph version
> - "ceph -s" output

The ceph filesystem is running on 3 debian wheezy boxes:

Linux t06-13 3.1.0-1-amd64 #1 SMP Tue Jan 10 05:01:58 UTC 2012 x86_64 
GNU/Linux

root@t06-13:~# ceph --version
ceph version 0.46 (commit:cb7f1c9c7520848b0899b26440ac34a8acea58d1)

root@t06-13:~# ceph -s
2012-06-15 14:36:59.978738    pg v696676: 800 pgs: 800 active+clean; 
65033 MB data, 158 GB used, 245 GB / 408 GB avail
2012-06-15 14:36:59.998849   mds e87: 1/1/1 up 
{0=t06-13-iscsi=up:active}, 2 up:standby
2012-06-15 14:36:59.998887   osd e1331: 4 osds: 3 up, 3 in
2012-06-15 14:37:00.010743   log 2012-06-11 10:55:04.785427 osd.2 
10.20.230.11:6801/31505 9228 : [WRN] old request 
osd_op(client.2822106.1:1146772 1000000007f.0000000f [write 
880640~4096,startsync 0~0] 0.c3ee5287 snapc 1=[] ordersnap) received at 
2012-06-11 10:54:34.667593 currently waiting for sub ops
2012-06-15 14:37:00.010870   mon e1: 3 mons at 
{0=10.20.230.13:6789/0,1=10.20.230.12:6789/0,2=10.20.230.11:6789/0}

The machine that is trying to mount the filesystem is ubuntu 12.04

Linux t06-16 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 
2012 x86_64 x86_64 x86_64 GNU/Linux

root@t06-16:/var/log# ceph --version
ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)


Did not see this when running everything on ceph 0.46, so I will try to 
upgrade the debian boxes.

> The MDS / POSIX filesystem hasn't been tested very well at the moment,
> in the next sprint (starting soon) the developers will start focusing on
> the POSIX filesystem.

That's good news. Thanks a lot!

-- 
/S.