Poor VMWare disk performance on XFS partition

Poor VMWare disk performance on XFS partition

[Hendrik .]

I've been converting some of my drives from EXT3 to
XFS a while ago. Now I notice poor disk performance
when using XFS as underlying filesystem for a VMware
virtual drive. I did some experiments and it really
seems to be the XFS filesystem 'trashing' the speed of
a VMware Windows XP guest.

The first thing that I noticed is that when I shut
down the virtual machine it takes a very long time
after the machine seems to be shut down until the
VMware window becomes responsive again. In the mean
time there is heavy disk I/O. I found out that VMware
seems to write some kind of memory map on the host
hard disk which get heavily fragmented. The removal of
this file is probably very I/O intensive which causes
the delay. This is very annoying but not really a
problem as it only happens when a virtual machine is
shut down.

But there seems to be another problem when running the
guest operating system itself. I made two exact copies
of a Windows XP virtual machine on two hard disks of
the same type, size and brand. The first hard disk had
a XFS partition to host the virtual machine, the
seconds harddisk was formatted as EXT3. The XFS
partition has no fragmentation at all thus all files
only consisted of 1 extent. The EXT3 files were a bit
fragmented but this was only marginal (some larger
disk image files consisted of 17 extents where 16 was
optimal, reported by 'filefrag').

Then I ran the virtual machines one by one and started
a defragmentation program to cause a lot of I/O on the
guest operating system. Defragmentation of the XP host
running from the EXT3 partition took only 2m36 but the
exact same guest on the XFS partition took 10m5 to
complete. On the EXT3 partition hardly any noise was
heard from the drive heads as if the host operating
system was caching and delaying the I/O operations. On
the XFS host however a lot of noise was heard as if
the harddisk was trashing heavily.

The machine I ran the tests on had the following
specifications:
- Dual core AMD Athlon64 5200+
- 2 GB memory
- 2x 80 GB Samsung IDE harddisk
- Ubuntu Gutsy Gibbon (7.10) 64-bit
- Standard stock Ubuntu kernel
- VMware workstation 6.0.2 build-59824

The guest was a bare Windows XP SP-2 installation,
running O&O defrag 10.

Can anyone give me some information what might be
causing this massive slowdown? 

Regards,
Hendrik van den Boogaard


      ____________________________________________________________________________________
Looking for last minute shopping deals?  
Find them fast with Yahoo! Search.  http://tools.search.yahoo.com/newsearch/category.php?category=shopping

Re: Poor VMWare disk performance on XFS partition

[Eric Sandeen]

Hendrik . wrote:
> I've been converting some of my drives from EXT3 to
> XFS a while ago. Now I notice poor disk performance
> when using XFS as underlying filesystem for a VMware
> virtual drive. I did some experiments and it really
> seems to be the XFS filesystem 'trashing' the speed of
> a VMware Windows XP guest.
> 
> The first thing that I noticed is that when I shut
> down the virtual machine it takes a very long time
> after the machine seems to be shut down until the
> VMware window becomes responsive again. In the mean
> time there is heavy disk I/O. I found out that VMware
> seems to write some kind of memory map on the host
> hard disk which get heavily fragmented. 

What does xfs_bmap and/or filefrag say, is this file indeed very
fragmented?  And is it less so on ext3?  If the file is persistent then
preallocating it would probably help.

> The removal of
> this file is probably very I/O intensive which causes
> the delay. This is very annoying but not really a
> problem as it only happens when a virtual machine is
> shut down.
> 
> But there seems to be another problem when running the
> guest operating system itself. I made two exact copies
> of a Windows XP virtual machine on two hard disks of
> the same type, size and brand. The first hard disk had
> a XFS partition to host the virtual machine, the
> seconds harddisk was formatted as EXT3. The XFS
> partition has no fragmentation at all thus all files
> only consisted of 1 extent. The EXT3 files were a bit
> fragmented but this was only marginal (some larger
> disk image files consisted of 17 extents where 16 was
> optimal, reported by 'filefrag').

I've honestly never used vmware. How many disk image files per guest?
You said 2 copies of an XP VM, one on xfs and one on ext3, but then said
"EXT3 files" so I'm not sure what the big picture looks like here.

If a single guest uses multiple files, can you run xfs_bmap -v on them,
it may be that xfs is spreading them out between the AGs, thereby
putting them into different regions of the disk.

> Then I ran the virtual machines one by one and started
> a defragmentation program to cause a lot of I/O on the
> guest operating system. Defragmentation of the XP host
> running from the EXT3 partition took only 2m36 but the
> exact same guest on the XFS partition took 10m5 to
> complete. On the EXT3 partition hardly any noise was
> heard from the drive heads as if the host operating
> system was caching and delaying the I/O operations. On
> the XFS host however a lot of noise was heard as if
> the harddisk was trashing heavily.

Maybe try using seekwatcher to trace/graph IO of the vmware processes to
see what's going on?

-Eric

Re: Poor VMWare disk performance on XFS partition

[Jan Derfinak]

On Sun, 23 Mar 2008, Hendrik . wrote:

> I've been converting some of my drives from EXT3 to
> XFS a while ago. Now I notice poor disk performance
> when using XFS as underlying filesystem for a VMware
> virtual drive. I did some experiments and it really
> seems to be the XFS filesystem 'trashing' the speed of
> a VMware Windows XP guest.

Mount XFS partition with "nobarrier" option. I'm using also
logbufs=8,logbsize=256k for vmware.

jan

-- 

Re: Poor VMWare disk performance on XFS partition

[Andi Kleen]

Eric Sandeen <sandeen@sandeen.net> writes:
> 
> What does xfs_bmap and/or filefrag say, is this file indeed very
> fragmented?  

> And is it less so on ext3?  If the file is persistent then
> preallocating it would probably help.

Preallocating would prevent one of the main features of 
a sparse VM images: starting small and only growing as the
virtual machine needs more storage without having to
resize the virtual partitions.

I remember XFS had a mmap problem a long time ago (in 2.4) which
sounded similar (iirc it trickered with samba), but I thought
it was long fixed. The problem back then was that page
flushing on mmaps didn't get merged due to some unfortunate VM
interactions and then thousands of extents got created on flushing
a mmap. A lot of extents seems to make XFS slow.

-Andi

Re: Poor VMWare disk performance on XFS partition

[Russell Cattelan]

Jan Derfinak wrote:
> On Sun, 23 Mar 2008, Hendrik . wrote:
>
>   
>> I've been converting some of my drives from EXT3 to
>> XFS a while ago. Now I notice poor disk performance
>> when using XFS as underlying filesystem for a VMware
>> virtual drive. I did some experiments and it really
>> seems to be the XFS filesystem 'trashing' the speed of
>> a VMware Windows XP guest.
>>     
>
> Mount XFS partition with "nobarrier" option. I'm using also
> logbufs=8,logbsize=256k for vmware.
>
> jan
>
>   
I can verify that ... barriers are killers when running vmware guest 
disk/memory images.

The preallocation would also help out quite bit if you don't mind 
dedicating the
disk space vs the sparse file method, which allow for over subscribing 
the physical
space. Going through once in a while and shutting down the guests and 
defragmenting
it a good idea.

I would be interested to find out the results of seekwatcher on shutdown
but I have also seen the long shutdowns.

-Russell

Re: Poor VMWare disk performance on XFS partition

[Peter Grandi]

>>> On Mon, 24 Mar 2008 09:00:04 -0500, Russell Cattelan
>>> <cattelan@thebarn.com> said:

>> [ ... ] Mount XFS partition with "nobarrier" option. [ ... ]

> I can verify that ... barriers are killers when running vmware
> guest disk/memory images. [ ... ]

But of course running VM images with 'nobarrier' is quite brave:
because it removes *any* integrity guarantee to IO initiated
inside the virtual machine.

This is because the only safe behaviour for a virtual machine
software is to turn all VM storage operations into synchronous
ones (or else detect the use of barriers inside the virtual
machine).

Of course this is going to be catastrophic with XFS's delayed
allocations, which relies on large numbers of outstanding writes
to coalesce them into large segments.

> [ ... ] The preallocation would also help out quite bit if you
> don't mind dedicating the disk space vs the sparse file method,
> which allow for over subscribing the physical space. [ ... ]

But sparse files are a crazy idea for virtual machines, because
the software inside is built on the idea that its storage is
allocated in contiguous volumes, and relies on that for its own
optimizations.

Or else one ends up with a CP/CMS situation where the "CMS"
inside the virtual machine is fully aware that it is running
inside a virtual machine and passes "out-of-bandwidth" hints to
the virtual machine software.

Re: Poor VMWare disk performance on XFS partition

[Jan Derfinak]

On Mon, 24 Mar 2008, Peter Grandi wrote:

> >>> On Mon, 24 Mar 2008 09:00:04 -0500, Russell Cattelan
> >>> <cattelan@thebarn.com> said:
> 
> >> [ ... ] Mount XFS partition with "nobarrier" option. [ ... ]
> 
> > I can verify that ... barriers are killers when running vmware
> > guest disk/memory images. [ ... ]
> 
> But of course running VM images with 'nobarrier' is quite brave:
> because it removes *any* integrity guarantee to IO initiated
> inside the virtual machine.

But of course barrier is workaround for hardware problems like power outage.
So if you want to have integrity in VM without performance penalty you must
ensure integrity in HW and don't leave it to barrier option. VMWare
performace is compromised with barrier on. And if you compare ext3 with xfs
you must have on mind that ext3 does not use barrier by default.
 
jan

-- 

Re: Poor VMWare disk performance on XFS partition

[Josef 'Jeff' Sipek]

On Mon, Mar 24, 2008 at 03:24:46PM +0000, Peter Grandi wrote:
> >>> On Mon, 24 Mar 2008 09:00:04 -0500, Russell Cattelan
...
> > [ ... ] The preallocation would also help out quite bit if you
> > don't mind dedicating the disk space vs the sparse file method,
> > which allow for over subscribing the physical space. [ ... ]
> 
> But sparse files are a crazy idea for virtual machines, because
> the software inside is built on the idea that its storage is
> allocated in contiguous volumes, and relies on that for its own
> optimizations.

Yup, gets you LOTS of extents => takes forever to unlink.

> Or else one ends up with a CP/CMS situation where the "CMS"
> inside the virtual machine is fully aware that it is running
> inside a virtual machine and passes "out-of-bandwidth" hints to
> the virtual machine software.

IIRC, only the first version of CMS was capable of running on bare hardware.
It was rather silly (and still is) to do that, and so all sort of tweaks
were made to make it faster, but at the same time make it depend on CP (the
DIAG instruction, etc.).

There's nothing wrong with this aproach, it just happens that vmware does it
to a _very_ limited extent (using the vmware tools you install in the guest
- last I checked it was used to enhance the display/mouse/etc.).

Ok, this was somewhat offtopic.

Josef 'Jeff' Sipek.

-- 
Only two things are infinite, the universe and human stupidity, and I'm not
sure about the former.
		- Albert Einstein