Re: loop in copy_user_generic_string

["Zdenek Kabelac" <zdenek.kabelac@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3509 bytes --]

2008/3/12, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
> 2008/3/12, Andi Kleen <andi@firstfloor.org>:
>
> > > Argh - being stupid here - it looks like these 'working' kernels were
>  >  > not SMP actually.
>  >  > As long as the SMP is used - I still get the busy loop :(
>  >  > Now being clueless....
>  >
>  >
>  > Sorry don't have the cycles to look into your problem, but the
>  >  standard procedure for hard problems that can be reproduced
>  >  is to git bisect them down to the change set that introduced the
>  >  problem originally and then complain to whoever authored that.
>
>
> The problem is - I don't know about any working SMP kernel which would
>  survive this test - thought haven't got into a really big history -
>  tried something like 2.6.22 kernels - no luck - also many kernel seems
>  to be unbootable in SMP mode on my machine giving many oopses - in
>  fact just 2.6.24 series starts to be at least reliable in booting in
>  my Qemu setup without failing during disk mounting or in some other
>  place...
>
>  Will try to find probably some 2.6.18 kernel and will check what happens.
>
>  On the other hand - I've tried to replicate my bug on few other
>  machines with no luck actually - so it's something which might not be
>  easy to trace :(
>

Btw - just for testing purposes - I've taken current fedora rawhide kernel.
Started machine with this kernel and installed it into qemu guest as well.

And this is what I get when running in SMP mode:

BUG: soft lockup - CPU#1 stuck for 61s! [udevd:583]
CPU 1:
Modules linked in: floppy ata_piix ata_generic pata_acpi pcnet32 mii
libata scsi_m
od
Pid: 583, comm: udevd Not tainted 2.6.25-0.105.rc5.fc9 #1
RIP: 0010:[<ffffffff8113b907>]  [<ffffffff8113b907>] clear_page_c+0x7/0x10
RSP: 0000:ffff810015455b20  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff810015455be8 RCX: 0000000000000200
RDX: 00000000000006a0 RSI: ffff810015455a74 RDI: ffff810015001000
RBP: ffff810000000000 R08: 0000000015562000 R09: ffff810000000000
R10: 0000000000000292 R11: 0000000000000001 R12: 00003ffffffff000
R13: ffff810000009540 R14: ffff810015454000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff810017509320(0063) knlGS:00000000f7f1d720
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: ffff810015001000 CR3: 00000000159e9000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff810833ac>] ? get_page_from_freelist+0x51f/0x6b6
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810f0d4b>] ? show_stat+0x2a/0x4af
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810a621d>] ? __kmalloc+0x3e/0xf0
 [<ffffffff810c555f>] ? seq_read+0x143/0x2a2
 [<ffffffff810c5532>] ? seq_read+0x116/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810e9df5>] ? proc_reg_read+0x8a/0xa7
 [<ffffffff810ab489>] ? vfs_read+0xab/0x154
 [<ffffffff810ab5f6>] ? sys_read+0x47/0x70
 [<ffffffff81023f32>] ? ia32_sysret+0x0/0xa

(Full trace attached)
So I guess I'm kind of lucky that my own kernels actually boot in smp
mode properly.
Guest was started with 384MB - host has 2GB - around 1GB was free when started.
Kernel boots with nosmp flag.

Zdenek

[-- Attachment #2: qemu-debian.tty --]
[-- Type: application/octet-stream, Size: 14586 bytes --]

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.25-0.105.rc5.fc9 (mockbuild@) (gcc version 4.3.0 20080307 (Red Hat 4.3.0-2) (GCC) ) #1 SMP Mon Mar 10 20:59:23 EDT 2008
Command line: root=/dev/sda1 ro console=ttyS0 console=tty0 selinux=off
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 0000000017ff0000 (usable)
 BIOS-e820: 0000000017ff0000 - 0000000018000000 (ACPI data)
 BIOS-e820: 00000000fffbd000 - 0000000100000000 (reserved)
end_pfn_map = 1048576
DMI 2.4 present.
ACPI: RSDP 000FB9F0, 0014 (r0 QEMU  )
ACPI: RSDT 17FF0000, 002C (r1 QEMU   QEMURSDT        1 QEMU        1)
ACPI: FACP 17FF002C, 0074 (r1 QEMU   QEMUFACP        1 QEMU        1)
ACPI: DSDT 17FF0100, 1632 (r1   BXPC   BXDSDT        1 INTL 20061109)
ACPI: FACS 17FF00C0, 0040
ACPI: APIC 17FF1738, 00E0 (r1 QEMU   QEMUAPIC        1 QEMU        1)
No NUMA configuration found
Faking a node at 0000000000000000-0000000017ff0000
Bootmem setup node 0 0000000000000000-0000000017ff0000
  NODE_DATA [000000000000b000 - 0000000000012fff]
  bootmap [0000000000013000 -  0000000000015fff] pages 3
early res: 0 [0-fff] BIOS data page
early res: 1 [6000-7fff] SMP_TRAMPOLINE
early res: 2 [200000-bf5b57] TEXT DATA BSS
early res: 3 [17a7f000-17fdfb5e] RAMDISK
early res: 4 [9fc00-a0bff] EBDA
early res: 5 [8000-afff] PGTABLE
Zone PFN ranges:
  DMA             0 ->     4096
  DMA32        4096 ->  1048576
  Normal    1048576 ->  1048576
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0:        0 ->      159
    0:      256 ->    98288
ACPI: PM-Timer IO Port: 0xb008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 (Bootup-CPU)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] disabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] disabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] disabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] disabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0c] disabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0d] disabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0e] disabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0f] disabled)
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000e8000
PM: Registered nosave memory: 00000000000e8000 - 0000000000100000
Allocating PCI resources starting at 20000000 (gap: 18000000:e7fbd000)
SMP: Allowing 16 CPUs, 14 hotplug CPUs
PERCPU: Allocating 436896 bytes of per cpu data
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 93330
Policy zone: DMA32
Kernel command line: root=/dev/sda1 ro console=ttyS0 console=tty0 selinux=off
Initializing CPU#0
PID hash table entries: 2048 (order: 11, 16384 bytes)
TSC calibrated against PM_TIMER
Marking TSC unstable due to TSCs unsynchronized
time.c: Detected 2194.538 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
console [ttyS0] enabled
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES:    8
... MAX_LOCK_DEPTH:          48
... MAX_LOCKDEP_KEYS:        2048
... CLASSHASH_SIZE:           1024
... MAX_LOCKDEP_ENTRIES:     8192
... MAX_LOCKDEP_CHAINS:      16384
... CHAINHASH_SIZE:          8192
 memory used by lock dependency info: 1712 kB
 per task-struct memory footprint: 3456 bytes
Checking aperture...
Memory: 359744k/393152k available (2718k kernel code, 33020k reserved, 1460k data, 740k init)
SLUB: Genslabs=13, HWalign=64, Order=0-1, MinObjects=4, CPUs=16, Nodes=1
Calibrating delay using timer specific routine.. 4410.71 BogoMIPS (lpj=2205359)
Security Framework initialized
SELinux:  Disabled at boot.
Capability LSM initialized
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU 0/0 -> Node 0
checking if image is initramfs... it is
debug: unmapping init memory ffff810017a7f000..ffff810017fe0000
ACPI: Core revision 20070126
ACPI: Checking initramfs for custom DSDT
Using local APIC timer interrupts.
Detected 62.501 MHz APIC timer.
lockdep: fixing up alternatives.
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 4389.10 BogoMIPS (lpj=2194550)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU 1/1 -> Node 0
QEMU Virtual CPU version 0.9.1 stepping 03
Brought up 2 CPUs
khelper used greatest stack depth: 6264 bytes left
net_namespace: 1152 bytes
Time: 16:00:57  Date: 03/12/08
NET: Registered protocol family 16
No dock devices found.
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI
pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB
ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 7 devices
ACPI: ACPI bus type pnp unregistered
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
DMAR:parse DMAR table failure.
PCI-GART: No AMD northbridge found.
Time: acpi_pm clocksource has been installed.
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
audit: initializing netlink socket (disabled)
type=2000 audit(1205337655.109:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
pci 0000:00:01.0: PIIX3: Enabling Passive Release
pci 0000:00:01.0: Activating ISA DMA hang workarounds
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI: ACPI0007:00 is registered as cooling_device0
ACPI: ACPI0007:01 is registered as cooling_device1
khelper used greatest stack depth: 6240 bytes left
Non-volatile memory driver v1.2
Linux agpgart interface v0.103
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
ÿserial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16450
00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16450
isa bounce pool size: 16 pages
brd: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one day
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
Clocksource tsc unstable (delta = 4386122119919 ns)
usbcore: registered new interface driver usbhid
drivers/hid/usbhid/hid-core.c: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
registered taskstats version 1
  Magic number: 0:540:34
debug: unmapping init memory ffffffff814ca000..ffffffff81583000
Write protecting the kernel read-only data: 1144k
busybox used greatest stack depth: 5776 bytes left
busybox used greatest stack depth: 5456 bytes left
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2
SCSI subsystem initialized
pcnet32.c:v1.34-NAPI 14.Aug.2007 tsbogend@alpha.franken.de
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
ACPI: PCI Interrupt 0000:00:03.0[A] -> Link [LNKC] -> GSI 11 (level, high) -> IRQ 11
pcnet32: PCnet/PCI II 79C970A at 0xc020, 52:54:00:12:34:56 assigned IRQ 11.
eth0: registered as PCnet/PCI II 79C970A
pcnet32: 1 cards_found.
modprobe used greatest stack depth: 4832 bytes left
exe used greatest stack depth: 4752 bytes left
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14
ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15
ata1.00: ATA-7: QEMU HARDDISK, 0.9.1, max UDMA/100
ata1.00: 4194304 sectors, multi 16: LBA48 
ata1.01: ATA-7: QEMU HARDDISK, 0.9.1, max UDMA/100
ata1.01: 16777216 sectors, multi 16: LBA48 
ata1.00: configured for MWDMA2
ata1.01: configured for MWDMA2
FDC 0 is a S82078B
ata2.00: ATA-7: QEMU HARDDISK, 0.9.1, max UDMA/100
ata2.00: 1433600 sectors, multi 16: LBA48 
ata2.00: configured for MWDMA2
scsi 0:0:0:0: Direct-Access     ATA      QEMU HARDDISK    0.9. PQ: 0 ANSI: 5
scsi 0:0:1:0: Direct-Access     ATA      QEMU HARDDISK    0.9. PQ: 0 ANSI: 5
scsi 1:0:0:0: Direct-Access     ATA      QEMU HARDDISK    0.9. PQ: 0 ANSI: 5
modprobe used greatest stack depth: 3904 bytes left
BUG: soft lockup - CPU#1 stuck for 61s! [udevd:583]
CPU 1:
Modules linked in: floppy ata_piix ata_generic pata_acpi pcnet32 mii libata scsi_mod
Pid: 583, comm: udevd Not tainted 2.6.25-0.105.rc5.fc9 #1
RIP: 0010:[<ffffffff8113b907>]  [<ffffffff8113b907>] clear_page_c+0x7/0x10
RSP: 0000:ffff810015455b20  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff810015455be8 RCX: 0000000000000200
RDX: 00000000000006a0 RSI: ffff810015455a74 RDI: ffff810015001000
RBP: ffff810000000000 R08: 0000000015562000 R09: ffff810000000000
R10: 0000000000000292 R11: 0000000000000001 R12: 00003ffffffff000
R13: ffff810000009540 R14: ffff810015454000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff810017509320(0063) knlGS:00000000f7f1d720
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: ffff810015001000 CR3: 00000000159e9000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff810833ac>] ? get_page_from_freelist+0x51f/0x6b6
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810f0d4b>] ? show_stat+0x2a/0x4af
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810a621d>] ? __kmalloc+0x3e/0xf0
 [<ffffffff810c555f>] ? seq_read+0x143/0x2a2
 [<ffffffff810c5532>] ? seq_read+0x116/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810e9df5>] ? proc_reg_read+0x8a/0xa7
 [<ffffffff810ab489>] ? vfs_read+0xab/0x154
 [<ffffffff810ab5f6>] ? sys_read+0x47/0x70
 [<ffffffff81023f32>] ? ia32_sysret+0x0/0xa

BUG: soft lockup - CPU#1 stuck for 61s! [udevd:583]
CPU 1:
Modules linked in: floppy ata_piix ata_generic pata_acpi pcnet32 mii libata scsi_mod
Pid: 583, comm: udevd Not tainted 2.6.25-0.105.rc5.fc9 #1
RIP: 0010:[<ffffffff8113b907>]  [<ffffffff8113b907>] clear_page_c+0x7/0x10
RSP: 0000:ffff810015455b20  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff810015455be8 RCX: 0000000000000200
RDX: 00000000000006a0 RSI: ffff810015455a74 RDI: ffff810015001000
RBP: ffff810000000000 R08: 0000000015562000 R09: ffff810000000000
R10: 0000000000000292 R11: 0000000000000001 R12: 00003ffffffff000
R13: ffff810000009540 R14: ffff810015454000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff810017509320(0063) knlGS:00000000f7f1d720
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: ffff810015001000 CR3: 00000000159e9000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff810833ac>] ? get_page_from_freelist+0x51f/0x6b6
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810f0d4b>] ? show_stat+0x2a/0x4af
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810a621d>] ? __kmalloc+0x3e/0xf0
 [<ffffffff810c555f>] ? seq_read+0x143/0x2a2
 [<ffffffff810c5532>] ? seq_read+0x116/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810e9df5>] ? proc_reg_read+0x8a/0xa7
 [<ffffffff810ab489>] ? vfs_read+0xab/0x154
 [<ffffffff810ab5f6>] ? sys_read+0x47/0x70
 [<ffffffff81023f32>] ? ia32_sysret+0x0/0xa


[-- Attachment #3: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #4: Type: text/plain, Size: 158 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-devel