lhype progress...

lhype progress...

[Rusty Russell]

Hi all,

	Some of you know I've been working on a trivial in-kernel hypervisor to
demonstrate paravirt_ops.  The good news is that it works!  (And not
just for me!)  The bad news is that I haven't written any documentation
and now I'm going to be offline (moving to a farm) for several weeks.

	Nonetheless, if you're the kind of person who wants to poke around with
hypervisors, no doubt you can figure it out.  Tony Breeds might even
write a HOWTO.  There's certainly a lot to be done (see FIXMEs in the
code for example).  Console device, block device and inter-domain
networking all work.  From here it should be easy 8)

	So, you can download the patch "lhype.patch" from the paravirt
mercurial patch queue.  It should apply by itself to a recent -mm tree.

	http://ozlabs.org/~rusty/paravirt

Thanks, and have fun playing!
Rusty.

Re: lhype progress...

[James Morris]

On Thu, 16 Nov 2006, Rusty Russell wrote:

> 	Nonetheless, if you're the kind of person who wants to poke around with
> hypervisors, no doubt you can figure it out.  Tony Breeds might even
> write a HOWTO.

Any patches I have queued for Rusty will be available here:
http://people.redhat.com/jmorris/lhype/patches/

(Currently just a couple of very minor things).



- James
-- 
James Morris
<jmorris@redhat.com>

[PATCH] lhype: Add TLS support

[James Morris]

This is queued for when Rusty returns, but may be generally useful now to 
people playing with lhype.

TLS support is required for some binaries (notably, Fedora & co), and also 
any binary which explicitly uses __thread variables (which all segfault 
without this patch).

It works well enough to bring FC6 up as far as it goes without an initrd, 
although there may be some further work required.  e.g.

o Need to ensure fixup_gdt() doesn't stomp on TLS segement descriptors.
o In some cases, a binary may fall back to calling modify_ldt(2) instead 
  of get_thread_area(2), and this needs to be handled.

---

Virtualize thread local storage for guests.  A new hypercall,
LHCALL_LOAD_TLS, is implemented, mapping to the load_tls()
paravirt op.  The virtualized GDT is now updated correctly
from the guest's per-thread TLS array.

Signed-off-by: James Morris <jmorris@namei.org>
---
 arch/i386/kernel/lhype.c   |    6 ++++++
 drivers/lhype/hypercalls.c |   22 +++++++++++++++++++++-
 include/asm-i386/lhype.h   |    1 +
 3 files changed, 28 insertions(+), 1 deletions(-)

diff --git a/arch/i386/kernel/lhype.c b/arch/i386/kernel/lhype.c
index d4376c5..35c4616 100644
--- a/arch/i386/kernel/lhype.c
+++ b/arch/i386/kernel/lhype.c
@@ -221,6 +221,11 @@ static fastcall void lhype_set_ldt(const
 	BUG_ON(entries);
 }
 
+static fastcall void lhype_load_tls(struct thread_struct *t, unsigned int cpu)
+{
+	hcall(LHCALL_LOAD_TLS, __pa(t), cpu, 0);
+}
+
 static fastcall void lhype_set_debugreg(int regno, unsigned long value)
 {
 	/* FIXME: Implement */
@@ -385,6 +390,7 @@ #endif
 	paravirt_ops.load_esp0 = lhype_load_esp0;
 	paravirt_ops.load_tr_desc = lhype_load_tr_desc;
 	paravirt_ops.set_ldt = lhype_set_ldt;
+	paravirt_ops.load_tls = lhype_load_tls;
 	paravirt_ops.set_debugreg = lhype_set_debugreg;
 	paravirt_ops.clts = lhype_clts;
 	paravirt_ops.read_cr0 = lhype_read_cr0;
diff --git a/drivers/lhype/hypercalls.c b/drivers/lhype/hypercalls.c
index bc8639c..f3ff693 100644
--- a/drivers/lhype/hypercalls.c
+++ b/drivers/lhype/hypercalls.c
@@ -178,13 +178,29 @@ static int guest_set_stack(struct lhype_
 	return 1;
 }
 
+static void guest_load_tls(struct lhype_dom_info *info,
+			   const struct thread_struct __user *t)
+{
+	int i;
+	struct desc_struct *gdt = info->domain->gdt_table;
+	
+	for (i = GDT_ENTRY_TLS_MIN; i <= GDT_ENTRY_TLS_MAX; i++) {
+		struct desc_struct d;
+
+		lhread(info, &d, (u32)&t->tls_array[i - GDT_ENTRY_TLS_MIN],
+		       sizeof(d));
+		gdt[i] = d;
+	}
+}
+
 void hypercall(struct lhype_dom_info *linfo, struct lhype_regs *regs)
 {
 	static unsigned long last_jifs;
 	/* These three happen regularly, so we limit them... */
 	if ((regs->eax != LHCALL_IRET
 	     && regs->eax != LHCALL_HALT
-	     && regs->eax != LHCALL_TIMER_READ)
+	     && regs->eax != LHCALL_TIMER_READ
+	     && regs->eax != LHCALL_LOAD_TLS)
 	    || printk_timed_ratelimit(&last_jifs, 10000)) {
 		printk(KERN_DEBUG "%i: %s\n", linfo->domid,
 		       regs->eax == LHCALL_LHYPE_PAGE ? "LHYPE_PAGE" :
@@ -208,6 +224,7 @@ void hypercall(struct lhype_dom_info *li
 		       regs->eax == LHCALL_SET_PTE ? "SET_PTE" :
 		       regs->eax == LHCALL_SET_SOME_PTE ? "SET_SOME_PTE" :
 		       regs->eax == LHCALL_SET_PUD ? "SET_PUD" :
+		       regs->eax == LHCALL_LOAD_TLS ? "LOAD_TLS" :
 		       "UNKNOWN");
 	}
 
@@ -325,6 +342,9 @@ void hypercall(struct lhype_dom_info *li
 	case LHCALL_SET_PUD:
 		guest_set_pud(linfo, regs->edx);
 		break;
+	case LHCALL_LOAD_TLS:
+		guest_load_tls(linfo, (struct thread_struct __user*)regs->edx);
+		break;
 	}
 	return;
 }
diff --git a/include/asm-i386/lhype.h b/include/asm-i386/lhype.h
index df86d97..af0c03b 100644
--- a/include/asm-i386/lhype.h
+++ b/include/asm-i386/lhype.h
@@ -26,6 +26,7 @@ #define LHCALL_SEND_DMA		18
 #define LHCALL_SET_PTE		19
 #define LHCALL_SET_SOME_PTE	20
 #define LHCALL_SET_PUD		21
+#define LHCALL_LOAD_TLS		22
 
 #define LHYPE_TRAP_ENTRY 0x81
 
-- 
1.4.2.1

Re: [PATCH] lhype: Add TLS support

[Rusty Russell]

On Mon, 2006-11-27 at 18:37 -0500, James Morris wrote:
> This is queued for when Rusty returns, but may be generally useful now to 
> people playing with lhype.
> 
> TLS support is required for some binaries (notably, Fedora & co), and also 
> any binary which explicitly uses __thread variables (which all segfault 
> without this patch).

Right...

	This unfortunately brings a security issue; GLIBC uses huge positive
offsets (4G minus a smidge) against these segments to access some
variables.  This means that the segments must cover the entire address
space, which would give them access to the hypervisor.

	There are two simple solutions to this.  The first is to trim the
segments, and use a modified libc (a-la Xen).  The second is to only
allow those TLS mappings in usermode: the U/S bit on the pagetables will
prevent usermode from accessing the hypervisor anyway.  This becomes a
little harder later, when we allow syscall traps directly into guest,
but I have another trick for that...

	Untested, but based on your patch.  Does it work?

Thanks!
Rusty.
diff -r d4fb2c676c1c arch/i386/kernel/lhype.c
--- a/arch/i386/kernel/lhype.c	Thu Nov 16 08:34:46 2006 +1100
+++ b/arch/i386/kernel/lhype.c	Wed Nov 29 16:23:43 2006 +1100
@@ -221,6 +221,11 @@ static fastcall void lhype_set_ldt(const
 	BUG_ON(entries);
 }
 
+static fastcall void lhype_load_tls(struct thread_struct *t, unsigned int cpu)
+{
+	hcall(LHCALL_LOAD_TLS, __pa(&t->tls_array), cpu, 0);
+}
+
 static fastcall void lhype_set_debugreg(int regno, unsigned long value)
 {
 	/* FIXME: Implement */
@@ -385,6 +390,7 @@ static __attribute_used__ __init void lh
 	paravirt_ops.load_esp0 = lhype_load_esp0;
 	paravirt_ops.load_tr_desc = lhype_load_tr_desc;
 	paravirt_ops.set_ldt = lhype_set_ldt;
+	paravirt_ops.load_tls = lhype_load_tls;
 	paravirt_ops.set_debugreg = lhype_set_debugreg;
 	paravirt_ops.clts = lhype_clts;
 	paravirt_ops.read_cr0 = lhype_read_cr0;
diff -r d4fb2c676c1c drivers/lhype/hypercalls.c
--- a/drivers/lhype/hypercalls.c	Thu Nov 16 08:34:46 2006 +1100
+++ b/drivers/lhype/hypercalls.c	Wed Nov 29 16:32:56 2006 +1100
@@ -178,13 +178,20 @@ static int guest_set_stack(struct lhype_
 	return 1;
 }
 
+static void guest_load_tls(struct lhype_dom_info *linfo,
+			   const struct desc_struct __user *tls_array)
+{
+	lhread(linfo, &linfo->tls, (u32)tls_array, sizeof(linfo->tls));
+}
+
 void hypercall(struct lhype_dom_info *linfo, struct lhype_regs *regs)
 {
 	static unsigned long last_jifs;
 	/* These three happen regularly, so we limit them... */
 	if ((regs->eax != LHCALL_IRET
 	     && regs->eax != LHCALL_HALT
-	     && regs->eax != LHCALL_TIMER_READ)
+	     && regs->eax != LHCALL_TIMER_READ
+	     && regs->eax != LHCALL_LOAD_TLS)
 	    || printk_timed_ratelimit(&last_jifs, 10000)) {
 		printk(KERN_DEBUG "%i: %s\n", linfo->domid,
 		       regs->eax == LHCALL_LHYPE_PAGE ? "LHYPE_PAGE" :
@@ -208,6 +215,7 @@ void hypercall(struct lhype_dom_info *li
 		       regs->eax == LHCALL_SET_PTE ? "SET_PTE" :
 		       regs->eax == LHCALL_SET_SOME_PTE ? "SET_SOME_PTE" :
 		       regs->eax == LHCALL_SET_PUD ? "SET_PUD" :
+		       regs->eax == LHCALL_LOAD_TLS ? "LOAD_TLS" :
 		       "UNKNOWN");
 	}
 
@@ -325,6 +333,9 @@ void hypercall(struct lhype_dom_info *li
 	case LHCALL_SET_PUD:
 		guest_set_pud(linfo, regs->edx);
 		break;
+	case LHCALL_LOAD_TLS:
+		guest_load_tls(linfo, (struct desc_struct __user*)regs->edx);
+		break;
 	}
 	return;
 }
diff -r d4fb2c676c1c drivers/lhype/init.c
--- a/drivers/lhype/init.c	Thu Nov 16 08:34:46 2006 +1100
+++ b/drivers/lhype/init.c	Wed Nov 29 16:32:14 2006 +1100
@@ -340,6 +340,17 @@ static int run_thread(struct lhype_dom_i
 		} else
 			cond_resched();
 
+		/* We can let them used the TLS 4G segments iff in
+		 * userspace: the U/S bit on the pagetables protects
+		 * hypervisor */
+		if ((linfo->domain->regs.cs & 3) == 3) {
+			memcpy(linfo->domain->gdt_table + GDT_ENTRY_TLS_MIN,
+			       linfo->tls, sizeof(linfo->tls));
+		} else {
+			memset(linfo->domain->gdt_table + GDT_ENTRY_TLS_MIN,
+			       0, sizeof(linfo->tls));
+		}
+
 		local_irq_disable();
 		/* FIXME: We should be able to be smart here. */
 		math_state_restore();
diff -r d4fb2c676c1c drivers/lhype/lhype.h
--- a/drivers/lhype/lhype.h	Thu Nov 16 08:34:46 2006 +1100
+++ b/drivers/lhype/lhype.h	Wed Nov 29 16:32:00 2006 +1100
@@ -108,6 +108,7 @@ struct lhype_dom_info
 	int halted;
 	u32 last_timer;
 	struct file *input;
+	struct desc_struct tls[GDT_ENTRY_TLS_ENTRIES];
 
 	u32 *pgdir;
 
diff -r d4fb2c676c1c include/asm-i386/lhype.h
--- a/include/asm-i386/lhype.h	Thu Nov 16 08:34:46 2006 +1100
+++ b/include/asm-i386/lhype.h	Wed Nov 29 16:18:38 2006 +1100
@@ -26,6 +26,7 @@
 #define LHCALL_SET_PTE		19
 #define LHCALL_SET_SOME_PTE	20
 #define LHCALL_SET_PUD		21
+#define LHCALL_LOAD_TLS		22
 
 #define LHYPE_TRAP_ENTRY 0x81
 

Re: [PATCH] lhype: Add TLS support

[Rusty Russell]

On Wed, 2006-11-29 at 16:37 +1100, Rusty Russell wrote:
> 	Untested, but based on your patch.  Does it work?

Actually, I suspect it won't.  Instead of memset'ing to 0, the TLS
segment descriptors should be set to a valid value, such as copying
USER_DS.

Cheers,
Rusty.

Re: [PATCH] lhype: Add TLS support

[James Morris]

On Thu, 30 Nov 2006, Rusty Russell wrote:

> On Wed, 2006-11-29 at 16:37 +1100, Rusty Russell wrote:
> > 	Untested, but based on your patch.  Does it work?


FYI, I've resynced with your latest fix (from the paravirt queue), and it 
seems to be working ok.

Some updated docs here:
http://virt.kernelnewbies.org/RustyvisorQuickstart



- James
-- 
James Morris
<jmorris@redhat.com>