nat does not work with hvm

nat does not work with hvm

[Erdem Bayer]

Hello all

I am using open source xen v 3.0.4. When network-nat and vif-nat is 
activated in the config file, modified guests boot ok but hvm guests 
does not have network connection. Here is the config files:

xend-config.sxp
-------------------------------------
#(network-script network-bridge)
#(vif-script vif-bridge)

(network-script network-nat)
(vif-script vif-nat)

-------------------------------------
I have commented the following lines from qemu-ifup script as I don't 
want to use bridge.
-------------------------------------
#ifconfig $1 0.0.0.0 up
#brctl addif $2 $1
-------------------------------------
I have also try commenting only the second line from the script. Still 
no luck. In that case an interface named tap is created also.

And the config file for the domain:
-------------------------------------
import os, re
arch = os.uname()[4]
if re.search('64', arch):
   arch_libdir = 'lib64'
else:
   arch_libdir = 'lib'
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
memory = 512
shadow_memory = 8
name = "winxp"
acpi=0
vif = [ 'ip=192.168.3.1, mac=aa:00:00:50:02:f0, type=ioemu, 
model=rtl8139' ]
disk = [ 'file:/mnt/sda5/xen/images/winxp1.img,hda,w',
'phy:/dev/cdrom,hdb:cdrom,r' ]
device_model = '/usr/' + arch_libdir + '/xen/bin/qemu-dm'
sdl=1
vnc=0
vncpasswd=''
stdvga=0
serial='pty'
-------------------------------------

When this domain is booted, it boots ok. Here is the network-nat status 
output.
============================================================
eth0      Link encap:Ethernet  HWaddr 00:17:08:37:F1:B7
         inet addr:10.0.0.5  Bcast:10.255.255.255  Mask:255.0.0.0
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:30 errors:0 dropped:0 overruns:0 frame:0
         TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:4530 (4.4 Kb)  TX bytes:2217 (2.1 Kb)
         Interrupt:20

lo        Link encap:Local Loopback
         inet addr:127.0.0.1  Mask:255.0.0.0
         UP LOOPBACK RUNNING  MTU:16436  Metric:1
         RX packets:6 errors:0 dropped:0 overruns:0 frame:0
         TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:328 (328.0 b)  TX bytes:328 (328.0 b)

vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
         inet addr:192.168.3.128  Bcast:0.0.0.0  Mask:255.255.255.255
         UP BROADCAST MULTICAST  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:32
         RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)


192.168.3.1 dev vif1.0  scope link  src 192.168.3.128
10.0.0.0/8 dev eth0  proto kernel  scope link  src 10.0.0.5
127.0.0.0/8 via 127.0.0.1 dev lo  scope link
default via 10.0.0.2 dev eth0

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use 
Iface
192.168.3.1     0.0.0.0         255.255.255.255 UH    0      0        0 
vif1.0
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 
eth0
127.0.0.0       127.0.0.1       255.0.0.0       UG    0      0        0 lo
0.0.0.0         10.0.0.2        0.0.0.0         UG    0      0        0 
eth0
============================================================

When this domain is booted, it cannot connect anywhere. It can not even 
ping 192.168.3.128 !!!!
And when the dom0 pings 192.168.3.1 Destination host unreachable error 
is get. The tcpdump in dom0 does not show any packets flowing.

I have tried the same config with an ubuntu dapper hvm guest and still 
no connections.
But when a modified guest is booted with the same network config, there 
is no problem. Everyone can ping each other and domU does have an 
internet connection. Here is the network-nat status output of the 
modified guest.

============================================================
eth0      Link encap:Ethernet  HWaddr 00:17:08:37:F1:B7
          inet addr:10.0.0.5  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1706 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1546 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1333224 (1.2 Mb)  TX bytes:159280 (155.5 Kb)
          Interrupt:20

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:9 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:500 (500.0 b)  TX bytes:500 (500.0 b)

vif10.0   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet addr:192.168.3.128  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32
          RX bytes:1242 (1.2 Kb)  TX bytes:3118 (3.0 Kb)


192.168.3.1 dev vif10.0  scope link  src 192.168.3.128
10.0.0.0/8 dev eth0  proto kernel  scope link  src 10.0.0.5
127.0.0.0/8 via 127.0.0.1 dev lo  scope link
default via 10.0.0.2 dev eth0

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use 
Iface
192.168.3.1     0.0.0.0         255.255.255.255 UH    0      0        0 
vif10.0
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth0
127.0.0.0       127.0.0.1       255.0.0.0       UG    0      0        0 lo
0.0.0.0         10.0.0.2        0.0.0.0         UG    0      0        0 eth0
============================================================


I have also tried dhcp instead of statically giving an ip address. Still 
the modified guest takes an ip address and have network, the hvm guest 
can not even reach the dhcp server in dom0.

Any ideas?
Erdem

Re: nat does not work with hvm

[Erdem Bayer]

But the network-nat script still runs in that case right? So what should 
be in the qemu-ifup script? Also should there also be a qemu-ifdown 
script to undo the changes?

I think in the qemu-ifup I should write a rule to switch the vifX.0 
created by the network-nat script and the tap interface created by the 
qemu-ifup script? Am I right about this?

Thanks for help

Erdem

Keir Fraser wrote On 14-04-2007 12:37:
> On 14/4/07 07:24, "Erdem Bayer" <ebayer@ttnet.net.tr> wrote:
>
>   
>> -------------------------------------
>> #ifconfig $1 0.0.0.0 up
>> #brctl addif $2 $1
>> -------------------------------------
>> I have also try commenting only the second line from the script. Still
>> no luck. In that case an interface named tap is created also.
>>     
>
> If you comment out those two lines and don't provide an alternative then the
> network backend interface will not even be brought up. It's then no surprise
> that no traffic can get to or from your HVM guest.
>
> If you want to route traffic rather than bridging it then you need to add
> commands to set up a route to your VM. And you almost certainly want to keep
> the first line that you commented out. That's the one that actually brings
> the interface online. If routing you'll want to put an actual IP address
> rather than 0.0.0.0.
>
> I don't think an IP address can currently be specified to that script, by
> the way. At least not directly.
>
>  -- Keir
>
>
>
>   

Re: nat does not work with hvm

[Keir Fraser]

On 14/4/07 07:24, "Erdem Bayer" <ebayer@ttnet.net.tr> wrote:

> -------------------------------------
> #ifconfig $1 0.0.0.0 up
> #brctl addif $2 $1
> -------------------------------------
> I have also try commenting only the second line from the script. Still
> no luck. In that case an interface named tap is created also.

If you comment out those two lines and don't provide an alternative then the
network backend interface will not even be brought up. It's then no surprise
that no traffic can get to or from your HVM guest.

If you want to route traffic rather than bridging it then you need to add
commands to set up a route to your VM. And you almost certainly want to keep
the first line that you commented out. That's the one that actually brings
the interface online. If routing you'll want to put an actual IP address
rather than 0.0.0.0.

I don't think an IP address can currently be specified to that script, by
the way. At least not directly.

 -- Keir

Re: nat does not work with hvm

[Keir Fraser]

On 14/4/07 07:56, "Erdem Bayer" <ebayer@ttnet.net.tr> wrote:

> But the network-nat script still runs in that case right?

Network-nat will run once, when xend is started. It's not run per-vif.

> So what should be in the qemu-ifup script?

Something a bit like what's in vif-nat, perhaps? That's the script that gets
run per-vif for non-hvm domains.

> Also should there also be a qemu-ifdown script to undo the changes?

Not needed. Everything should tear down automatically when the qemu process
dies.

 -- Keir

> I think in the qemu-ifup I should write a rule to switch the vifX.0
> created by the network-nat script and the tap interface created by the
> qemu-ifup script? Am I right about this?

Adding PTE Flag during Page Fault

[Joe Laws]

I altered the method, "__spurious_page_fault", in "xen/arch/x86/traps.c"
to change a page to present when it is marked as not present.  I am
attempting to do this using the "l1e_add_flags(l1e, _PAGE_PRESENT);" 
definition call.  I then set it to return EXCRET_fault_fixed, so the
trap should have the client retry what it was doing with the changes I
made.  Unfortunately after it returns from the page fault to the user,
it immediately faults again on the same error at the same memory
location.  This means my "l1e_add_flags(l1e, _PAGE_PRESENT)" is not
actually updating the correct location in memory or  if it is, the
changes get discarded immediately.  Does anyone who is familiar with the
page_fault trap code have any idea why my change to the PTE is not
sticking.  I tried doing a TLB flush for the memory address, but the
same thing happens.  My only other guess is that the l1e defined in the
function is just a copy of the actual table entry, so I need to look
back further where l1e was defined.

Thanks,
Joe

Re: Adding PTE Flag during Page Fault

[Joe Laws]

Tried changing the l1e by doing this:

l1e_write_atomic(&l1t[l1_table_offset(addr)],l1e);
unmap_domain_page(l1t);

Now it just causes Xen to crash and my computer to reboot. 
Unfortunately I can only see the output to the log by doing xm dmesg,
which gets cleared on every reboot.  Are there any efficient techniques
to get the latest xm dmesg output to file before the system crashes?

-Joe

Joe Laws wrote:
> I altered the method, "__spurious_page_fault", in "xen/arch/x86/traps.c"
> to change a page to present when it is marked as not present.  I am
> attempting to do this using the "l1e_add_flags(l1e, _PAGE_PRESENT);" 
> definition call.  I then set it to return EXCRET_fault_fixed, so the
> trap should have the client retry what it was doing with the changes I
> made.  Unfortunately after it returns from the page fault to the user,
> it immediately faults again on the same error at the same memory
> location.  This means my "l1e_add_flags(l1e, _PAGE_PRESENT)" is not
> actually updating the correct location in memory or  if it is, the
> changes get discarded immediately.  Does anyone who is familiar with the
> page_fault trap code have any idea why my change to the PTE is not
> sticking.  I tried doing a TLB flush for the memory address, but the
> same thing happens.  My only other guess is that the l1e defined in the
> function is just a copy of the actual table entry, so I need to look
> back further where l1e was defined.
>
> Thanks,
> Joe
>
>
>   

Re: Re: Adding PTE Flag during Page Fault

[Keir Fraser]

On 14/4/07 22:29, "Joe Laws" <jlaws@andrew.cmu.edu> wrote:

> Now it just causes Xen to crash and my computer to reboot.
> Unfortunately I can only see the output to the log by doing xm dmesg,
> which gets cleared on every reboot.  Are there any efficient techniques
> to get the latest xm dmesg output to file before the system crashes?

You can use kdump (basically chain boot into a special crash-dump Linux
kernel with block or net drivers which will scrape memory for things like
Xen heap and Xen logging output).

 -- Keir

Re: Adding PTE Flag during Page Fault

[Keir Fraser]

On 14/4/07 21:50, "Joe Laws" <jlaws@andrew.cmu.edu> wrote:

> My only other guess is that the l1e defined in the
> function is just a copy of the actual table entry, so I need to look
> back further where l1e was defined.

Well, l1e is a local variable in that function. *l1t points at the actual
page table entry. So you want something like:
 l1e_add_flags(l1e, _PAGE_PRESENT);
 l1e_write_atomic(l1t, l1e);

 -- Keir

Re: nat does not work with hvm

[Erdem Bayer]

Hi

I have been playing with the xen-unstable and observed that the vif-nat 
script is still running in the case of nat in qemu (although on the 
wrong interface as you have explained) Did I miss something or is it 
still running in the unstable vers. because it will have support on one 
of the next releases. Will nat be supported in qemu by xen in future 
versions or should I write my own qemu-ifup script? Also I was thinking 
if it is possible to have more than one qemu-ifup scripts and activate 
them from the config. Is this possible?

For the qemu-ifup script the steps I think is as follows:

1. Determine ip information.
2. Bring interface tap up with the ip
3. write iptables rules to route packages to the domU through tap 
interface ip

So the problem is passing the necessary values to qemu-ifup scripts. I 
wonder where qemu-ifup script is called. Is it possible to change its 
parameters?

Am I right? Do you have anything to add? If you can guide me through 
this I am sure I can come up with a reasonable script.

Thanks for help
Erdem

Keir Fraser wrote On 14-04-2007 13:19:
> On 14/4/07 08:12, "Erdem Bayer" <ebayer@ttnet.net.tr> wrote:
>
>   
>> Will the vif-nat script be run for the hvm domain? If not how does the
>> vifX.0 interface and route rules in my first config be created? I think
>> the the answer of this question is yes. Then what will the qemu-ifup
>> script do? What I wonder is what is the purpose of the qemu-ifup script
>> if the vif-nat script will be run also?
>>     
>
> Vif-nat isn't run in this case.
>
> Well actually it is run in 3.0.4 (but not in the very latest xen-unstable
> repository, so not in the forthcoming Xen 3.0.5). But it'll get run on the
> wrong interface. The vif2.0, or whatever, interfaces aren't used by qemu.
> qemu creates its own tap interface and the qemu-ifup script is executed to
> configure that tap interface.
>
>  -- Keir
>
>
>
>   

Re: nat does not work with hvm

[Keir Fraser]

On 17/4/07 12:32, "Erdem Bayer" <ebayer@ttnet.net.tr> wrote:

> So the problem is passing the necessary values to qemu-ifup scripts. I
> wonder where qemu-ifup script is called. Is it possible to change its
> parameters?
> 
> Am I right? Do you have anything to add? If you can guide me through
> this I am sure I can come up with a reasonable script.

I think you're along the right lines. The qemu-ifup script is run from
qemu-dm (source is in tools/ioemu).

 -- Keir

Re: Re: Adding PTE Flag during Page Fault

[Joe Laws]

Thanks for the suggestion Keir, unfortunately my kernel lacks the option
for kexec, despite the fact that all the source files are included. 
Anyway, I was able to get the information I need by passing certain args
to the xen kernel via grub:

kernel /xen.gz noreboot console=vga vga=keep xencons=tty

Using this output I managed to get my stuff working this morning also. 
I was surprised to learn that of the ~90K pages saved (for a 380MB
memory image) only ~1500 are needed for the system to start up and
continue running.  Granted every time something is done another 10-20
pages get loaded, the fact that less than 2% only need to be present to
start the VM again is crazy.  This means that only ~6MB of the memory
image needs to be present to resume. 

-Joe

Keir Fraser wrote:
>
> On 14/4/07 22:29, "Joe Laws" <jlaws@andrew.cmu.edu> wrote:
>
>   
>> Now it just causes Xen to crash and my computer to reboot.
>> Unfortunately I can only see the output to the log by doing xm dmesg,
>> which gets cleared on every reboot.  Are there any efficient techniques
>> to get the latest xm dmesg output to file before the system crashes?
>>     
>
> You can use kdump (basically chain boot into a special crash-dump Linux
> kernel with block or net drivers which will scrape memory for things like
> Xen heap and Xen logging output).
>
>  -- Keir
>
>
>
>   

Re: nat does not work with hvm

[Erdem Bayer]

Hi

Keir Fraser wrote On 14-04-2007 13:19:
>
> Vif-nat isn't run in this case.
>
> Well actually it is run in 3.0.4 (but not in the very latest xen-unstable
> repository, so not in the forthcoming Xen 3.0.5). But it'll get run on the
> wrong interface. The vif2.0, or whatever, interfaces aren't used by qemu.
> qemu creates its own tap interface and the qemu-ifup script is executed to
> configure that tap interface.
>
>  -- Keir
>
>   

In xen-unstable vif-nat scripts still runs with qemu-ifup script, when a 
hvm domain is booted. There are some problems with that. First of all, 
an un-needed vif interface is created and configured with the values 
from the hvm configuration file, and you can not write a qemu-ifup 
script that takes same info from config file. (for example you cannot 
configure the ip address of your tap interface because that ip is taken 
by vif interface, wrongly.)

Also I strongly believe that a qemu-ifdown script is needed. For example 
if you write some iptables rules in qemu-ifup, then these rules should 
be deleted from iptables when the domain goes down.

Please correct me if I am wrong. I think there should be only one 
network script per config, (ex: vif-nat, vif-bridge, etc) and that 
script should determine whether the domain is a hvm or a modified one 
and make necessary configuration accordingly. This way there is no need 
for seperate scripts for qemu and vif and when a qemu domain shuts down, 
it's settings can be de-configured properly. Or as an alternative there 
should be two script for each config (ex. vif-nat-qemu + vif-nat-other) 
and but only one of them should be executed.

So the question is: What is the plan about implementing other types of 
network scripts in qemu domains? Is this discussed or planned? If not I 
want to make the necessary changes. I figured that tools/ioemu/vl.c is 
responsible for creating a hvm domain and execute the correct network 
script. But I could not figure out what piece of code is responsible for 
calling the network script with necessary parameters when a modified 
guest boots. Also are these changes that I propose are acceptable (or 
reasonable at least)? I would appreciate if you share your knowledge and 
ideas about this matter.

Thanks
Erdem