[Qemu-devel] Please help corrupt filesystem image

[Qemu-devel] Please help corrupt filesystem image

[Alexander Marx]

Dear List!

I have a big problem and hope you can help me.
I built a new windows 2016 domain with virtual servers. 2 dc and 9 rds
hosts.
I was nearly finished with the setup and ready to migrate the users from
old to new domain.

Then i had to restart the physical servers. Unfortunately 1 dc and 2 rds
hosts could not be startet anymore.
Error says:

qemu-img: Could not open 'vm-150-disk-0.qcow2': Could not read qcow2
header: Input/output error

Even worst, a qemu-img check  also gave that error. and i was not able
to do anything i found in the internet to fix it.
always that error or the qemu-img could not determine the format.

Is there a chance to fix these images? What can i do?

I have no backup because the domain was not fully configured. But if i
have to rebuild the whole domain, i will need several weeks for it.
Any help is appreciated.

Thank you

Alex

Re: [Qemu-devel] Please help corrupt filesystem image

[Kevin Wolf]

Am 17.02.2019 um 05:13 hat Alexander Marx geschrieben:
> Dear List!
> 
> I have a big problem and hope you can help me.
> I built a new windows 2016 domain with virtual servers. 2 dc and 9 rds
> hosts.
> I was nearly finished with the setup and ready to migrate the users from
> old to new domain.
> 
> Then i had to restart the physical servers. Unfortunately 1 dc and 2 rds
> hosts could not be startet anymore.
> Error says:
> 
> qemu-img: Could not open 'vm-150-disk-0.qcow2': Could not read qcow2
> header: Input/output error

The very first thing the qcow2 driver in QEMU does is reading in the
image file header, and this is the step that produces the error message
you quoted. This is before even looking at any content of the file. So
it looks to me as if the image file was corrupted on the file system
level or even had bad sectors on the physical hard disk.

You can try running 'qemu-img check' under 'strace -f' and double check
which syscall returns an error before it errors out. I think it will be
the first pread64() call, returning EIO. If not, please post the output
you're getting.

Another thing you could try is whether the image file is still
accessible for tools like cp, dd or hexdump.

If these tests confirm that there is a more general problem with the
file, running fsck for your host file system that contains the image
might be a good idea.

> Even worst, a qemu-img check  also gave that error. and i was not able
> to do anything i found in the internet to fix it.
> always that error or the qemu-img could not determine the format.
> 
> Is there a chance to fix these images? What can i do?
> 
> I have no backup because the domain was not fully configured. But if i
> have to rebuild the whole domain, i will need several weeks for it.
> Any help is appreciated.

If only the header is inaccessible, reconstructing it may be possible.
Otherwise, probably not.

We can look into this if your results for the above show that you can
still access the image outside of QEMU, and what exactly is even going
wrong.

Kevin

Re: [Qemu-devel] Please help corrupt filesystem image

[Alexander Marx]

Thank you for replying.

The error was in the underlying filesystem. I had to setup new servers.

Alex


Am 18.02.19 um 10:51 schrieb Kevin Wolf:
> Am 17.02.2019 um 05:13 hat Alexander Marx geschrieben:
>> Dear List!
>>
>> I have a big problem and hope you can help me.
>> I built a new windows 2016 domain with virtual servers. 2 dc and 9 rds
>> hosts.
>> I was nearly finished with the setup and ready to migrate the users from
>> old to new domain.
>>
>> Then i had to restart the physical servers. Unfortunately 1 dc and 2 rds
>> hosts could not be startet anymore.
>> Error says:
>>
>> qemu-img: Could not open 'vm-150-disk-0.qcow2': Could not read qcow2
>> header: Input/output error
> The very first thing the qcow2 driver in QEMU does is reading in the
> image file header, and this is the step that produces the error message
> you quoted. This is before even looking at any content of the file. So
> it looks to me as if the image file was corrupted on the file system
> level or even had bad sectors on the physical hard disk.
>
> You can try running 'qemu-img check' under 'strace -f' and double check
> which syscall returns an error before it errors out. I think it will be
> the first pread64() call, returning EIO. If not, please post the output
> you're getting.
>
> Another thing you could try is whether the image file is still
> accessible for tools like cp, dd or hexdump.
>
> If these tests confirm that there is a more general problem with the
> file, running fsck for your host file system that contains the image
> might be a good idea.
>
>> Even worst, a qemu-img check  also gave that error. and i was not able
>> to do anything i found in the internet to fix it.
>> always that error or the qemu-img could not determine the format.
>>
>> Is there a chance to fix these images? What can i do?
>>
>> I have no backup because the domain was not fully configured. But if i
>> have to rebuild the whole domain, i will need several weeks for it.
>> Any help is appreciated.
> If only the header is inaccessible, reconstructing it may be possible.
> Otherwise, probably not.
>
> We can look into this if your results for the above show that you can
> still access the image outside of QEMU, and what exactly is even going
> wrong.
>
> Kevin

[Qemu-devel] Please help corrupt filesystem image

[Alexander Marx]

Dear List!

I have a big problem and hope you can help me.
I built a new windows 2016 domain with virtual servers. 2 dc and 9 rds
hosts.
I was nearly finished with the setup and ready to migrate the users from
old to new domain.

Then i had to restart the physical servers. Unfortunately 1 dc and 2 rds
hosts could not be startet anymore.
Error says:

qemu-img: Could not open 'vm-150-disk-0.qcow2': Could not read qcow2
header: Input/output error

Even worst, a qemu-img check  also gave that error. and i was not able
to do anything i found in the internet to fix it.
always that error or the qemu-img could not determine the format.

Is there a chance to fix these images? What can i do?

I have no backup because the domain was not fully configured. But if i
have to rebuild the whole domain, i will need several weeks for it.
Any help is appreciated.

Thank you

Alex

Re: [Qemu-devel] Please help corrupt filesystem image

[John Snow]

On 2/16/19 10:54 PM, Alexander Marx wrote:
> Dear List!
> 
> I have a big problem and hope you can help me.
> I built a new windows 2016 domain with virtual servers. 2 dc and 9 rds
> hosts.
> I was nearly finished with the setup and ready to migrate the users from
> old to new domain.
> 
> Then i had to restart the physical servers. Unfortunately 1 dc and 2 rds
> hosts could not be startet anymore.
> Error says:
> 
> qemu-img: Could not open 'vm-150-disk-0.qcow2': Could not read qcow2
> header: Input/output error
> 
> Even worst, a qemu-img check  also gave that error. and i was not able
> to do anything i found in the internet to fix it.
> always that error or the qemu-img could not determine the format.
> 
> Is there a chance to fix these images? What can i do?
> 
> I have no backup because the domain was not fully configured. But if i
> have to rebuild the whole domain, i will need several weeks for it.
> Any help is appreciated.
> 
> Thank you
> 
> Alex
> 

What versions of QEMU were you running?
What filesystem were these images stored on?
What happened at the time of the corruption, is there some detail that
might help explain the nature of the corruption? Did you shut down the
physical host while the VMs were still running?

There may not be a way to recover these images, but we should probably
make sure that your configuration isn't unsafe to prevent this kind of
problem in the future.

Re: [Qemu-devel] Please help corrupt filesystem image

[Alexander Marx]

Thank you for replying.
Well i am using latest PROXMOX in a cluster of 4 physical servers.
during the weekend i had to stop all hosts because electricians had to
work on the fuse box.

i shutted down all vm's then powered off all physical hosts. One of them
took very long.
this host had a raid5 of 27 TB on it (XFS) where some vm images were saved.

i waited long time but i knew that at weekends the raid does a patrol
read. So i switched that host hard off.

After rebooting, all hosts and nearly all vms came back online without
problems. only these two images could not be started.
As i told before, they were not critical as they were part of a new
domain which was not used in productuction.

Then i found out that the XFS Filesystem has problems. i stopped all
vm's again and did a xfs_check on that storage.
Some Errors were correted which causes 2 other vm's to die.

These vm's i could restore from backup, so no problem. The other 2
Win216 servers i had to delete and install new.
Thank you very much for your interest on my problem.
It is solved.

Have a nice week and please go on developing that great piece of software!


Am 18.02.19 um 21:54 schrieb John Snow:
>
> On 2/16/19 10:54 PM, Alexander Marx wrote:
>> Dear List!
>>
>> I have a big problem and hope you can help me.
>> I built a new windows 2016 domain with virtual servers. 2 dc and 9 rds
>> hosts.
>> I was nearly finished with the setup and ready to migrate the users from
>> old to new domain.
>>
>> Then i had to restart the physical servers. Unfortunately 1 dc and 2 rds
>> hosts could not be startet anymore.
>> Error says:
>>
>> qemu-img: Could not open 'vm-150-disk-0.qcow2': Could not read qcow2
>> header: Input/output error
>>
>> Even worst, a qemu-img check  also gave that error. and i was not able
>> to do anything i found in the internet to fix it.
>> always that error or the qemu-img could not determine the format.
>>
>> Is there a chance to fix these images? What can i do?
>>
>> I have no backup because the domain was not fully configured. But if i
>> have to rebuild the whole domain, i will need several weeks for it.
>> Any help is appreciated.
>>
>> Thank you
>>
>> Alex
>>
> What versions of QEMU were you running?
> What filesystem were these images stored on?
> What happened at the time of the corruption, is there some detail that
> might help explain the nature of the corruption? Did you shut down the
> physical host while the VMs were still running?
>
> There may not be a way to recover these images, but we should probably
> make sure that your configuration isn't unsafe to prevent this kind of
> problem in the future.