Question about hyper-threading of domains

Question about hyper-threading of domains

[Jacob Gorm Hansen]

hi,

is it possible, and will it make sense, to allow an SMP-domain access to 
both SMT/Hyper threads of a CPU?

My performance measurements of the small address spaces implementation 
(SAS) seem to indicate that having the driver domain in a separate hyper 
thread is still a little faster than SAS for a non-SMP/SMT domU.  I 
guess that means that that SMT IPIs are about the same cost as ring 
switches.

So quite likely SAS is only interesting if:

1) You don't have SMT (luckily, I have a bunch of machines like that 
back home ;-)). Do the recent Opterons and Itaniums have SMT btw?

2) You want to run SMT in the domUs.

3) You give up driver isolation and manage to get dom0 running in ring0 
(to remove the cost of ring switches). Some comments in the Xen code 
indicate that this will be hard to do, so it may not be worth it.

I previously stated that using small address spaces means giving up 
driver isolation, but I think that with the right use of segments that 
should not be necessary, so there is no trade-off with regards to isolation.

My patch to Xen and XenLinux is fairly small, though a few things 
(mainly the PERDOMAIN mapping and thus use of segments in domains) are 
not handled correctly at this point. Also, I do not use segments to keep 
domUs out of dom0, which I will at some point.

Basically, I have added per-domain pgd lower and upper bounds, and I 
have added an extra pgd slot for the linear page tables of dom0. The 
linear_* macros now take an exec_domain* as argument, and I have a 
pgd-version check in __context-switch() that potentially updates the 
cached entries at the top of the domUs pgd, and does nothing (i.e. does 
not write cr3) otherwise.  If needed, I flush the TLB before 
pdg-updates,  and when unpinning a pgd I clear the cached area before 
put_page() and friends.

Most of the changes are of a nature that could be made applicable to 
other architectures as well. I think they could be activated at run-time 
with little or no overhead when not in use.

I would be happy to share my changes with anyone interested.

Jacob

Re: Question about hyper-threading of domains

[Lars Roland]

On 4/29/05, Jacob Gorm Hansen <jacobg@diku.dk> wrote:
> hi,
> 
> is it possible, and will it make sense, to allow an SMP-domain access to
> both SMT/Hyper threads of a CPU?
> 
> My performance measurements of the small address spaces implementation
> (SAS) seem to indicate that having the driver domain in a separate hyper
> thread is still a little faster than SAS for a non-SMP/SMT domU.  I
> guess that means that that SMT IPIs are about the same cost as ring
> switches.

Seans fair enough - Out of curiosity have you published these
measurements anywere ?

> 
> So quite likely SAS is only interesting if:
> 
> 1) You don't have SMT (luckily, I have a bunch of machines like that
> back home ;-)). Do the recent Opterons and Itaniums have SMT btw?

No neither the latest Opterons nor the Itaniums have explicit SMT.

> 
> 2) You want to run SMT in the domUs.
> 
> 3) You give up driver isolation and manage to get dom0 running in ring0
> (to remove the cost of ring switches). Some comments in the Xen code
> indicate that this will be hard to do, so it may not be worth it.
> 
> I previously stated that using small address spaces means giving up
> driver isolation, but I think that with the right use of segments that
> should not be necessary, so there is no trade-off with regards to isolation.

Could you explain a little further how excalty you plan to avoid this
- I have been digging into this myslef and It is not obvious to me how
this can be achived,

> 
> My patch to Xen and XenLinux is fairly small, though a few things
> (mainly the PERDOMAIN mapping and thus use of segments in domains) are
> not handled correctly at this point. Also, I do not use segments to keep
> domUs out of dom0, which I will at some point.
> 
> Basically, I have added per-domain pgd lower and upper bounds, and I
> have added an extra pgd slot for the linear page tables of dom0. The
> linear_* macros now take an exec_domain* as argument, and I have a
> pgd-version check in __context-switch() that potentially updates the
> cached entries at the top of the domUs pgd, and does nothing (i.e. does
> not write cr3) otherwise.  If needed, I flush the TLB before
> pdg-updates,  and when unpinning a pgd I clear the cached area before
> put_page() and friends.
> 
> Most of the changes are of a nature that could be made applicable to
> other architectures as well. I think they could be activated at run-time
> with little or no overhead when not in use.
> 
> I would be happy to share my changes with anyone interested.

Yes please - I have the time (and hardware) to check this.


Regards.

Lars Roland

Re: Question about hyper-threading of domains

[Keir Fraser]

On 29 Apr 2005, at 20:42, Jacob Gorm Hansen wrote:

> is it possible, and will it make sense, to allow an SMP-domain access 
> to both SMT/Hyper threads of a CPU?
>
> My performance measurements of the small address spaces implementation 
> (SAS) seem to indicate that having the driver domain in a separate 
> hyper thread is still a little faster than SAS for a non-SMP/SMT domU. 
>  I guess that means that that SMT IPIs are about the same cost as ring 
> switches.

HT is designed to look almost identical to SMP, so it is no problem to 
give both HTs to a multi-processor domain. This will make sense if the 
domain is running a workload that benefits from HT (e.g., probably 
certain types of multi-threaded application).

In most cases I expect it will give better overall performance to 
allocate the second HT solely to a driver domain.

  -- Keir

Re: Question about hyper-threading of domains

[Jacob Gorm Hansen]

Lars Roland wrote:

> Seans fair enough - Out of curiosity have you published these
> measurements anywere ?

No nothing yet, apart from the iperf numbers posted previously to this list.

>>I previously stated that using small address spaces means giving up
>>driver isolation, but I think that with the right use of segments that
>>should not be necessary, so there is no trade-off with regards to isolation.
> 
> 
> Could you explain a little further how excalty you plan to avoid this
> - I have been digging into this myslef and It is not obvious to me how
> this can be achived,

I have not implemented this, and perhaps I am not understanding the x86 
docs correctly, but it seems I can set up and expands-down segment to 
confine dom0 to its allowed range of addresses (0xf0000000 - 
0xfc000000). If that does not work, I can move dom0 to the range 0x0 - 
0x0c000000, and use the segment base to move it to 0xf0000000. The 
latter approach will require that I fix up copy_from_user and friends 
inside Xen to take this into account, but that seems doable.

> Yes please - I have the time (and hardware) to check this.

I will try to roll something up later today or tomorrow, before flying 
to Boston for NSDI.

Jacob

Re: Question about hyper-threading of domains

[Mark Williamson]

> > 1) You don't have SMT (luckily, I have a bunch of machines like that
> > back home ;-)). Do the recent Opterons and Itaniums have SMT btw?
>
> No neither the latest Opterons nor the Itaniums have explicit SMT.

I don't think AMD has much interest in SMT for the foreseeable future - 
they're jumping straight to multicore.  Intel doesn't bother with HT on 
anything but the Pentium 4 because its pipeline is so much longer than 
anything else...

And the thought of implementing an SMT Itanium really scares me :-) (although 
I don't doubt they do consider it an option)

Cheers,
Mark