[uml-devel] kernel BUG: while fuzzying a 32 bit Linux user mode guest with trinity

[uml-devel] kernel BUG: while fuzzying a 32 bit Linux user mode guest with trinity

[Toralf Förster]

I could force a crash using latest kernel tree (v3.15-rc3-159-g6c6ca9c with applied fix3.patch for the mremap syscall) and latest trinity tree (1.1-1349-g18ebf71).

The backtrace of the core dump gives :

tfoerste@n22 ~/tmp $ gdb /home/tfoerste/devel/linux/linux --core=/mnt/ramdisk/core -batch -ex 'thread apply all bt'
[New LWP 23912]

warning: Could not load shared library symbols for linux-gate.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `/home/tfoerste/devel/linux/linux earlyprintk ubda=/home/tfoerste/virtual/uml/tr'.
Program terminated with signal 6, Aborted.
#0  0xb7741424 in __kernel_vsyscall ()

Thread 1 (LWP 23912):
#0  0xb7741424 in __kernel_vsyscall ()
#1  0x0848ac75 in kill ()
#2  0x08072a5d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08072d95 in os_dump_core () at arch/um/os-Linux/util.c:148
#4  0x0806257d in panic_exit (self=0x86c9618 <panic_exit_notifier>, unused1=0, unused2=0x8700960 <buf.17019>) at arch/um/kernel/um_arch.c:240
#5  0x0809a266 in notifier_call_chain (nl=0x0, val=0, v=0x8700960 <buf.17019>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a381 in __atomic_notifier_call_chain (nh=0x8700944 <panic_notifier_list>, val=0, v=0x8700960 <buf.17019>, nr_to_call=0, nr_calls=0x0) at kernel/notifier.c:182
#7  0x0809a3bf in atomic_notifier_call_chain (nh=0x0, val=0, v=0x0) at kernel/notifier.c:191
#8  0x084e742c in panic (fmt=0x0) at kernel/panic.c:130
#9  0x080cc265 in __delete_from_page_cache (page=0xa303520, shadow=0x0) at mm/filemap.c:202
#10 0x080cc32b in delete_from_page_cache (page=0xa303520) at mm/filemap.c:234
#11 0x080d7af7 in truncate_complete_page (page=<optimized out>, mapping=<optimized out>) at mm/truncate.c:145
#12 truncate_inode_page (mapping=0x4592c974, page=0xa303520) at mm/truncate.c:180
#13 0x080de69d in shmem_undo_range (inode=0x4592c8bc, lstart=26525858516, lend=3247232753107730432, unfalloc=false) at mm/shmem.c:429
#14 0x080df591 in shmem_truncate_range (inode=0x4592c8bc, lstart=0, lend=3247230382285783040) at mm/shmem.c:526
#15 0x080df6a8 in shmem_fallocate (file=0x0, mode=3, offset=0, len=1048576) at mm/shmem.c:1741
#16 0x081045da in do_fallocate (file=0x458bf300, mode=3, offset=0, len=1048576) at fs/open.c:298
#17 0x080e6b91 in madvise_remove (end=<optimized out>, start=<optimized out>, prev=<optimized out>, vma=<optimized out>) at mm/madvise.c:332
#18 madvise_vma (behavior=<optimized out>, end=<optimized out>, start=<optimized out>, prev=<optimized out>, vma=<optimized out>) at mm/madvise.c:384
#19 SYSC_madvise (behavior=<optimized out>, len_in=<optimized out>, start=<optimized out>) at mm/madvise.c:534
#20 SyS_madvise (start=1076387840, len_in=1048576, behavior=9) at mm/madvise.c:465
#21 0x08062b34 in handle_syscall (r=0x2d38e3e0) at arch/um/kernel/skas/syscall.c:35
#22 0x08074875 in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:193
#23 userspace (regs=0x2d38e3e0) at arch/um/os-Linux/skas/process.c:426
#24 0x0805f770 in fork_handler () at arch/um/kernel/process.c:149
#25 0x00000000 in ?? ()



The output of the UML guest is :


Kernel panic - not syncing: BUG!
CPU: 0 PID: 1988 Comm: trinity-c2 Not tainted 3.15.0-rc3-00159-g6c6ca9c-dirty #8
Stack:
 085a4f54 085a4f54 2d107bbc 00000004 086c8547 0a303520 0000003f 4592c974
 2d107bcc 084eafa5 00000000 00000000 2d107bf4 084e7410 085b08ec 08700960
 085a1ca5 2d107c00 00000000 0a303520 0000003f 4592c974 2d107c2c 080cc265
Call Trace:
 [<080cc265>] ? __delete_from_page_cache+0x215/0x270
 [<084eafa5>] dump_stack+0x26/0x28
 [<084e7410>] panic+0x7a/0x194
 [<080cc265>] __delete_from_page_cache+0x215/0x270
 [<080cc32b>] delete_from_page_cache+0x6b/0x90
 [<080d7af7>] truncate_inode_page+0x97/0xb0
 [<080de69d>] shmem_undo_range+0x1bd/0x620
 [<080df591>] shmem_truncate_range+0x31/0x60
 [<080df6a8>] shmem_fallocate+0xe8/0x360
 [<0849a605>] ? __gettimeofday+0x15/0x30
 [<08071dfe>] ? set_signals+0x1e/0x40
 [<081045da>] do_fallocate+0x14a/0x1d0
 [<080e6b91>] SyS_madvise+0x1d1/0x720
 [<080aef0d>] ? __getnstimeofday+0x3d/0x100
 [<0807fa68>] ? SyS_gettimeofday+0x38/0x80
 [<08062b34>] handle_syscall+0x64/0x80
 [<0849d621>] ? ptrace+0x31/0x80
 [<08079802>] ? get_fp_registers+0x22/0x40
 [<08074875>] userspace+0x475/0x5f0
 [<0849d621>] ? ptrace+0x31/0x80
 [<08079d66>] ? os_set_thread_area+0x26/0x40
 [<08078d30>] ? do_set_thread_area+0x20/0x50
 [<08078ea8>] ? arch_switch_tls+0xb8/0x100
 [<0805f770>] fork_handler+0x60/0x70
/home/tfoerste/workspace/bin/start_uml.sh: line 110: 23912 Aborted                 (core dumped) $LINUX earlyprintk ubda=$ROOTFS ubdb=$SWAP eth0=$NET mem=$MEM $TTY umid=uml_$NAME rootfstype=ext4 "$ARGS"
[3g        




There's no trinity log available, I lost it, sry.

FWIW the host system is a stable 32 bit Gentoo Linux with kernel 3.14.2.

-- 
Toralf


------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.  Get 
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] kernel BUG: while fuzzying a 32 bit Linux user mode guest with trinity

[Toralf Förster]

[-- Attachment #1: Type: text/plain, Size: 306 bytes --]

On 05/03/2014 06:04 PM, Toralf Förster wrote:
> There's no trinity log available, I lost it, sry.

For a similar trinity run with similar results for back trace and UML
guest output now with full logging of trinity enabled I attached the
last 2000 lines of the appropriate trinity log file.


-- 
Toralf


[-- Attachment #2: t --]
[-- Type: application/octet-stream, Size: 252267 bytes --]

[-- Attachment #3: Type: text/plain, Size: 388 bytes --]

------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.  Get 
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs

[-- Attachment #4: Type: text/plain, Size: 194 bytes --]

_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] kernel BUG: while fuzzying a 32 bit Linux user mode guest with trinity

[Richard Weinberger]

On Sat, May 3, 2014 at 6:04 PM, Toralf Förster <toralf.foerster@gmx.de> wrote:
> I could force a crash using latest kernel tree (v3.15-rc3-159-g6c6ca9c with applied fix3.patch for the mremap syscall) and latest trinity tree (1.1-1349-g18ebf71).
>
> The backtrace of the core dump gives :
>
> tfoerste@n22 ~/tmp $ gdb /home/tfoerste/devel/linux/linux --core=/mnt/ramdisk/core -batch -ex 'thread apply all bt'
> [New LWP 23912]
>
> warning: Could not load shared library symbols for linux-gate.so.1.
> Do you need "set solib-search-path" or "set sysroot"?
> Core was generated by `/home/tfoerste/devel/linux/linux earlyprintk ubda=/home/tfoerste/virtual/uml/tr'.
> Program terminated with signal 6, Aborted.
> #0  0xb7741424 in __kernel_vsyscall ()
>
> Thread 1 (LWP 23912):
> #0  0xb7741424 in __kernel_vsyscall ()
> #1  0x0848ac75 in kill ()
> #2  0x08072a5d in uml_abort () at arch/um/os-Linux/util.c:93
> #3  0x08072d95 in os_dump_core () at arch/um/os-Linux/util.c:148
> #4  0x0806257d in panic_exit (self=0x86c9618 <panic_exit_notifier>, unused1=0, unused2=0x8700960 <buf.17019>) at arch/um/kernel/um_arch.c:240
> #5  0x0809a266 in notifier_call_chain (nl=0x0, val=0, v=0x8700960 <buf.17019>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
> #6  0x0809a381 in __atomic_notifier_call_chain (nh=0x8700944 <panic_notifier_list>, val=0, v=0x8700960 <buf.17019>, nr_to_call=0, nr_calls=0x0) at kernel/notifier.c:182
> #7  0x0809a3bf in atomic_notifier_call_chain (nh=0x0, val=0, v=0x0) at kernel/notifier.c:191
> #8  0x084e742c in panic (fmt=0x0) at kernel/panic.c:130
> #9  0x080cc265 in __delete_from_page_cache (page=0xa303520, shadow=0x0) at mm/filemap.c:202
> #10 0x080cc32b in delete_from_page_cache (page=0xa303520) at mm/filemap.c:234
> #11 0x080d7af7 in truncate_complete_page (page=<optimized out>, mapping=<optimized out>) at mm/truncate.c:145
> #12 truncate_inode_page (mapping=0x4592c974, page=0xa303520) at mm/truncate.c:180
> #13 0x080de69d in shmem_undo_range (inode=0x4592c8bc, lstart=26525858516, lend=3247232753107730432, unfalloc=false) at mm/shmem.c:429
> #14 0x080df591 in shmem_truncate_range (inode=0x4592c8bc, lstart=0, lend=3247230382285783040) at mm/shmem.c:526
> #15 0x080df6a8 in shmem_fallocate (file=0x0, mode=3, offset=0, len=1048576) at mm/shmem.c:1741
> #16 0x081045da in do_fallocate (file=0x458bf300, mode=3, offset=0, len=1048576) at fs/open.c:298
> #17 0x080e6b91 in madvise_remove (end=<optimized out>, start=<optimized out>, prev=<optimized out>, vma=<optimized out>) at mm/madvise.c:332
> #18 madvise_vma (behavior=<optimized out>, end=<optimized out>, start=<optimized out>, prev=<optimized out>, vma=<optimized out>) at mm/madvise.c:384
> #19 SYSC_madvise (behavior=<optimized out>, len_in=<optimized out>, start=<optimized out>) at mm/madvise.c:534
> #20 SyS_madvise (start=1076387840, len_in=1048576, behavior=9) at mm/madvise.c:465
> #21 0x08062b34 in handle_syscall (r=0x2d38e3e0) at arch/um/kernel/skas/syscall.c:35
> #22 0x08074875 in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:193
> #23 userspace (regs=0x2d38e3e0) at arch/um/os-Linux/skas/process.c:426
> #24 0x0805f770 in fork_handler () at arch/um/kernel/process.c:149
> #25 0x00000000 in ?? ()
>
>
>
> The output of the UML guest is :
>
>
> Kernel panic - not syncing: BUG!
> CPU: 0 PID: 1988 Comm: trinity-c2 Not tainted 3.15.0-rc3-00159-g6c6ca9c-dirty #8
> Stack:
>  085a4f54 085a4f54 2d107bbc 00000004 086c8547 0a303520 0000003f 4592c974
>  2d107bcc 084eafa5 00000000 00000000 2d107bf4 084e7410 085b08ec 08700960
>  085a1ca5 2d107c00 00000000 0a303520 0000003f 4592c974 2d107c2c 080cc265
> Call Trace:
>  [<080cc265>] ? __delete_from_page_cache+0x215/0x270
>  [<084eafa5>] dump_stack+0x26/0x28
>  [<084e7410>] panic+0x7a/0x194
>  [<080cc265>] __delete_from_page_cache+0x215/0x270
>  [<080cc32b>] delete_from_page_cache+0x6b/0x90
>  [<080d7af7>] truncate_inode_page+0x97/0xb0
>  [<080de69d>] shmem_undo_range+0x1bd/0x620
>  [<080df591>] shmem_truncate_range+0x31/0x60
>  [<080df6a8>] shmem_fallocate+0xe8/0x360
>  [<0849a605>] ? __gettimeofday+0x15/0x30
>  [<08071dfe>] ? set_signals+0x1e/0x40
>  [<081045da>] do_fallocate+0x14a/0x1d0
>  [<080e6b91>] SyS_madvise+0x1d1/0x720
>  [<080aef0d>] ? __getnstimeofday+0x3d/0x100
>  [<0807fa68>] ? SyS_gettimeofday+0x38/0x80
>  [<08062b34>] handle_syscall+0x64/0x80
>  [<0849d621>] ? ptrace+0x31/0x80
>  [<08079802>] ? get_fp_registers+0x22/0x40
>  [<08074875>] userspace+0x475/0x5f0
>  [<0849d621>] ? ptrace+0x31/0x80
>  [<08079d66>] ? os_set_thread_area+0x26/0x40
>  [<08078d30>] ? do_set_thread_area+0x20/0x50
>  [<08078ea8>] ? arch_switch_tls+0xb8/0x100
>  [<0805f770>] fork_handler+0x60/0x70
> /home/tfoerste/workspace/bin/start_uml.sh: line 110: 23912 Aborted                 (core dumped) $LINUX earlyprintk ubda=$ROOTFS ubdb=$SWAP eth0=$NET mem=$MEM $TTY umid=uml_$NAME rootfstype=ext4 "$ARGS"
> [3g
>
>
>
>
> There's no trinity log available, I lost it, sry.
>
> FWIW the host system is a stable 32 bit Gentoo Linux with kernel 3.14.2.

As written two days ago, this seems to be a known issue:
https://lkml.org/lkml/2014/4/15/577

-- 
Thanks,
//richard

------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.  Get 
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] kernel BUG: while fuzzying a 32 bit Linux user mode guest with trinity

[Toralf Förster]

On 05/03/2014 09:15 PM, Richard Weinberger wrote:
> On Sat, May 3, 2014 at 6:04 PM, Toralf Förster <toralf.foerster@gmx.de> wrote:
>> I could force a crash using latest kernel tree (v3.15-rc3-159-g6c6ca9c with applied fix3.patch for the mremap syscall) and latest trinity tree (1.1-1349-g18ebf71).
...
>> #9  0x080cc265 in __delete_from_page_cache (page=0xa303520, shadow=0x0) at mm/filemap.c:202
...
> As written two days ago, this seems to be a known issue:
> https://lkml.org/lkml/2014/4/15/577

Just FWIW :
If I exclude the syscall "madvise" from the trinity fuzzer then this
issue can't be reproduced (till now). Allowing that syscall however
crashes the UML usually within less than 1/2 hour.


-- 
Toralf


------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] kernel BUG: while fuzzying a 32 bit Linux user mode guest with trinity

[Toralf Förster]

On 05/17/2014 05:24 PM, Toralf Förster wrote:
> On 05/03/2014 09:15 PM, Richard Weinberger wrote:
>> On Sat, May 3, 2014 at 6:04 PM, Toralf Förster <toralf.foerster@gmx.de> wrote:
>>> I could force a crash using latest kernel tree (v3.15-rc3-159-g6c6ca9c with applied fix3.patch for the mremap syscall) and latest trinity tree (1.1-1349-g18ebf71).
> ...
>>> #9  0x080cc265 in __delete_from_page_cache (page=0xa303520, shadow=0x0) at mm/filemap.c:202
> ...
>> As written two days ago, this seems to be a known issue:
>> https://lkml.org/lkml/2014/4/15/577
> 
> Just FWIW :
> If I exclude the syscall "madvise" from the trinity fuzzer then this
> issue can't be reproduced (till now). Allowing that syscall however
> crashes the UML usually within less than 1/2 hour.
> 
> 
Well, I was wrong, it just takes longer time, but here's an example for the issue using another syscall 

Kernel panic - not syncing: BUG!
CPU: 0 PID: 4400 Comm: trinity Not tainted 3.15.0-rc5-00077-g14186fe-dirty #17
Stack:
 085a4fd4 085a4fd4 48397c20 00000004 086c8547 0a5b8bc0 0000003f 48054244
 48397c30 084eb115 00000000 00000000 48397c58 084e7580 085b096c 08700960
 085a1d25 48397c64 00000000 0a5b8bc0 0000003f 48054244 48397c90 080cc2c5
Call Trace:
 [<080cc2c5>] ? __delete_from_page_cache+0x215/0x270
 [<084eb115>] dump_stack+0x26/0x28
 [<084e7580>] panic+0x7a/0x194
 [<080cc2c5>] __delete_from_page_cache+0x215/0x270
 [<080cc38b>] delete_from_page_cache+0x6b/0x90
 [<080d7a87>] truncate_inode_page+0x97/0xb0
 [<080de64d>] shmem_undo_range+0x1bd/0x620
 [<080df541>] shmem_truncate_range+0x31/0x60
 [<080dfb06>] shmem_evict_inode+0x86/0x150
 [<0811d87f>] evict+0xbf/0x170
 [<080fff98>] ? kmem_cache_free+0xe8/0x120
 [<080ec5a4>] ? remove_vma+0x44/0x50
 [<0811e2fd>] iput+0x14d/0x160
 [<0811ab08>] dentry_kill.isra.29+0x158/0x220
 [<0811ae8d>] dput+0xfd/0x120
 [<08107795>] __fput+0x175/0x190
 [<081075e0>] ? file_free_rcu+0x0/0x40
 [<081077eb>] ____fput+0xb/0x10
 [<08093b26>] task_work_run+0x76/0x90
 [<0805f95a>] interrupt_end+0x4a/0x80
 [<0807497b>] userspace+0x57b/0x5f0
 [<0849d7a1>] ? ptrace+0x31/0x80
 [<08079d66>] ? os_set_thread_area+0x26/0x40
 [<08078d30>] ? do_set_thread_area+0x20/0x50
 [<08078ea8>] ? arch_switch_tls+0xb8/0x100
 [<0805f770>] fork_handler+0x60/0x70
/home/tfoerste/workspace/bin/start_uml.sh: line 110:  8342 Aborted                 (core dumped) $LINUX earlyprintk ubda=$ROOTFS ubdb=$SWAP eth0=$NET mem=$MEM $TTY umid=uml_$NAME rootfstype=ext4 "$ARGS"




and the gdb back trace of the core file gives :

Thread 1 (LWP 8342):
#0  0xb7759424 in __kernel_vsyscall ()
#1  0x0848adf5 in kill ()
#2  0x08072a5d in uml_abort () at arch/um/os-Linux/util.c:93
#3  0x08072d95 in os_dump_core () at arch/um/os-Linux/util.c:148
#4  0x0806257d in panic_exit (self=0x86c9618 <panic_exit_notifier>, unused1=0, unused2=0x8700960 <buf.17021>) at arch/um/kernel/um_arch.c:240
#5  0x0809a2c6 in notifier_call_chain (nl=0x0, val=0, v=0x8700960 <buf.17021>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93
#6  0x0809a3e1 in __atomic_notifier_call_chain (nh=0x8700944 <panic_notifier_list>, val=0, v=0x8700960 <buf.17021>, nr_to_call=0, nr_calls=0x0) at kernel/notifier.c:182
#7  0x0809a41f in atomic_notifier_call_chain (nh=0x0, val=0, v=0x0) at kernel/notifier.c:191
#8  0x084e759c in panic (fmt=0x0) at kernel/panic.c:130
#9  0x080cc2c5 in __delete_from_page_cache (page=0xa5b8bc0, shadow=0x0) at mm/filemap.c:202
#10 0x080cc38b in delete_from_page_cache (page=0xa5b8bc0) at mm/filemap.c:234
#11 0x080d7a87 in truncate_complete_page (page=<optimized out>, mapping=<optimized out>) at mm/truncate.c:145
#12 truncate_inode_page (mapping=0x48054244, page=0xa5b8bc0) at mm/truncate.c:180
#13 0x080de64d in shmem_undo_range (inode=0x4805418c, lstart=26981530424, lend=5204328695673653632, unfalloc=false) at mm/shmem.c:429
#14 0x080df541 in shmem_truncate_range (inode=0x4805418c, lstart=0, lend=5204326324851703808) at mm/shmem.c:526
#15 0x080dfb06 in shmem_evict_inode (inode=0x4805418c) at mm/shmem.c:570
#16 0x0811d87f in evict (inode=0x4805418c) at fs/inode.c:550
#17 0x0811e2fd in iput_final (inode=<optimized out>) at fs/inode.c:1418
#18 iput (inode=0x4805418c) at fs/inode.c:1436
#19 0x0811ab08 in dentry_iput (dentry=<optimized out>) at fs/dcache.c:292
#20 dentry_kill (dentry=0x3789d4d0, unlock_on_failure=<optimized out>) at fs/dcache.c:507
#21 0x0811ae8d in dput (dentry=0x3789d4d0) at fs/dcache.c:582
#22 0x08107795 in __fput (file=0x48ac89c0) at fs/file_table.c:228
#23 0x081077eb in ____fput (work=0x48ac89c0) at fs/file_table.c:246
#24 0x08093b26 in task_work_run () at kernel/task_work.c:123
#25 0x0805f95a in tracehook_notify_resume (regs=<optimized out>) at include/linux/tracehook.h:196
#26 interrupt_end () at arch/um/kernel/process.c:98
#27 0x0807497b in userspace (regs=0x45f822e0) at arch/um/os-Linux/skas/process.c:459
#28 0x0805f770 in fork_handler () at arch/um/kernel/process.c:149
#29 0x00000000 in ?? ()

-- 
Toralf


------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel