* ARM64/KVM: Bad page state in process iperf @ 2015-12-15 3:46 Bhushan Bharat 2015-12-15 9:29 ` Christoffer Dall 2015-12-15 9:34 ` Marc Zyngier 0 siblings, 2 replies; 9+ messages in thread From: Bhushan Bharat @ 2015-12-15 3:46 UTC (permalink / raw) To: linux-arm-kernel Hi All, I am running "iperf" in KVM guest on ARM64 machine and observing below crash. ============================= $iperf -c 3.3.3.3 -P 4 -t 0 -i 5 -w 90k ------------------------------------------------------------ Client connecting to 3.3.3.3, TCP port 5001 TCP window size: 180 KByte (WARNING: requested 90.0 KByte) ------------------------------------------------------------ [ 3] local 3.3.3.1 port 51131 connected with 3.3.3.3 port 5001 [ 6] local 3.3.3.1 port 51134 connected with 3.3.3.3 port 5001 [ 5] local 3.3.3.1 port 51133 connected with 3.3.3.3 port 5001 [ 4] local 3.3.3.1 port 51132 connected with 3.3.3.3 port 5001 [ 53.088567] random: nonblocking pool is initialized [ ID] Interval Transfer Bandwidth [ 3] 0.0- 5.0 sec 638 MBytes 1.07 Gbits/sec [ 4] 35.0-40.0 sec 1.66 GBytes 2.85 Gbits/sec [ 5] 40.0-45.0 sec 1.11 GBytes 1.90 Gbits/sec [ 4] 40.0-45.0 sec 1.16 GBytes 1.99 Gbits/sec [ 98.895207] BUG: Bad page state in process iperf pfn:0a584 [ 98.896164] page:ffff780000296100 count:-1 mapcount:0 mapping: (null) index:0x0 [ 98.897436] flags: 0x0() [ 98.897885] page dumped because: nonzero _count [ 98.898640] Modules linked in: [ 98.899178] CPU: 0 PID: 1639 Comm: iperf Not tainted 4.1.8-00461-ge5431ad #141 [ 98.900302] Hardware name: linux,dummy-virt (DT) [ 98.901014] Call trace: [ 98.901406] [<ffff800000096cac>] dump_backtrace+0x0/0x12c [ 98.902522] [<ffff800000096de8>] show_stack+0x10/0x1c [ 98.903441] [<ffff800000678dc8>] dump_stack+0x8c/0xdc [ 98.904202] [<ffff800000145480>] bad_page+0xc4/0x114 [ 98.904945] [<ffff8000001487a4>] get_page_from_freelist+0x590/0x63c [ 98.905871] [<ffff80000014893c>] __alloc_pages_nodemask+0xec/0x794 [ 98.906791] [<ffff80000059fc80>] skb_page_frag_refill+0x70/0xa8 [ 98.907678] [<ffff80000059fcd8>] sk_page_frag_refill+0x20/0xd0 [ 98.908550] [<ffff8000005edc04>] tcp_sendmsg+0x1f8/0x9a8 [ 98.909368] [<ffff80000061419c>] inet_sendmsg+0x5c/0xd0 [ 98.910178] [<ffff80000059bb44>] sock_sendmsg+0x14/0x58 [ 98.911027] [<ffff80000059bbec>] sock_write_iter+0x64/0xbc [ 98.912119] [<ffff80000019b5b8>] __vfs_write+0xac/0x10c [ 98.913126] [<ffff80000019bcb8>] vfs_write+0x90/0x1a0 [ 98.913963] [<ffff80000019c53c>] SyS_write+0x40/0xa0 ----------- ^ permalink raw reply [flat|nested] 9+ messages in thread
* ARM64/KVM: Bad page state in process iperf 2015-12-15 3:46 ARM64/KVM: Bad page state in process iperf Bhushan Bharat @ 2015-12-15 9:29 ` Christoffer Dall 2015-12-15 9:33 ` Bhushan Bharat 2015-12-15 9:34 ` Marc Zyngier 1 sibling, 1 reply; 9+ messages in thread From: Christoffer Dall @ 2015-12-15 9:29 UTC (permalink / raw) To: linux-arm-kernel On Tue, Dec 15, 2015 at 03:46:03AM +0000, Bhushan Bharat wrote: > > Hi All, > > I am running "iperf" in KVM guest on ARM64 machine and observing below crash. Which host/guest kernel version is this? Which hardware? -Christoffer > > ============================= > $iperf -c 3.3.3.3 -P 4 -t 0 -i 5 -w 90k > ------------------------------------------------------------ > Client connecting to 3.3.3.3, TCP port 5001 > TCP window size: 180 KByte (WARNING: requested 90.0 KByte) > ------------------------------------------------------------ > [ 3] local 3.3.3.1 port 51131 connected with 3.3.3.3 port 5001 > [ 6] local 3.3.3.1 port 51134 connected with 3.3.3.3 port 5001 > [ 5] local 3.3.3.1 port 51133 connected with 3.3.3.3 port 5001 > [ 4] local 3.3.3.1 port 51132 connected with 3.3.3.3 port 5001 > [ 53.088567] random: nonblocking pool is initialized > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 5.0 sec 638 MBytes 1.07 Gbits/sec > [ 4] 35.0-40.0 sec 1.66 GBytes 2.85 Gbits/sec > [ 5] 40.0-45.0 sec 1.11 GBytes 1.90 Gbits/sec > [ 4] 40.0-45.0 sec 1.16 GBytes 1.99 Gbits/sec > [ 98.895207] BUG: Bad page state in process iperf pfn:0a584 > [ 98.896164] page:ffff780000296100 count:-1 mapcount:0 mapping: (null) index:0x0 > [ 98.897436] flags: 0x0() > [ 98.897885] page dumped because: nonzero _count > [ 98.898640] Modules linked in: > [ 98.899178] CPU: 0 PID: 1639 Comm: iperf Not tainted 4.1.8-00461-ge5431ad #141 > [ 98.900302] Hardware name: linux,dummy-virt (DT) > [ 98.901014] Call trace: > [ 98.901406] [<ffff800000096cac>] dump_backtrace+0x0/0x12c > [ 98.902522] [<ffff800000096de8>] show_stack+0x10/0x1c > [ 98.903441] [<ffff800000678dc8>] dump_stack+0x8c/0xdc > [ 98.904202] [<ffff800000145480>] bad_page+0xc4/0x114 > [ 98.904945] [<ffff8000001487a4>] get_page_from_freelist+0x590/0x63c > [ 98.905871] [<ffff80000014893c>] __alloc_pages_nodemask+0xec/0x794 > [ 98.906791] [<ffff80000059fc80>] skb_page_frag_refill+0x70/0xa8 > [ 98.907678] [<ffff80000059fcd8>] sk_page_frag_refill+0x20/0xd0 > [ 98.908550] [<ffff8000005edc04>] tcp_sendmsg+0x1f8/0x9a8 > [ 98.909368] [<ffff80000061419c>] inet_sendmsg+0x5c/0xd0 > [ 98.910178] [<ffff80000059bb44>] sock_sendmsg+0x14/0x58 > [ 98.911027] [<ffff80000059bbec>] sock_write_iter+0x64/0xbc > [ 98.912119] [<ffff80000019b5b8>] __vfs_write+0xac/0x10c > [ 98.913126] [<ffff80000019bcb8>] vfs_write+0x90/0x1a0 > [ 98.913963] [<ffff80000019c53c>] SyS_write+0x40/0xa0 > > ----------- > _______________________________________________ > kvmarm mailing list > kvmarm at lists.cs.columbia.edu > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 9+ messages in thread
* ARM64/KVM: Bad page state in process iperf 2015-12-15 9:29 ` Christoffer Dall @ 2015-12-15 9:33 ` Bhushan Bharat 0 siblings, 0 replies; 9+ messages in thread From: Bhushan Bharat @ 2015-12-15 9:33 UTC (permalink / raw) To: linux-arm-kernel > -----Original Message----- > From: Christoffer Dall [mailto:christoffer.dall at linaro.org] > Sent: Tuesday, December 15, 2015 2:59 PM > To: Bhushan Bharat-R65777 <Bharat.Bhushan@freescale.com> > Cc: kvmarm at lists.cs.columbia.edu; kvm at vger.kernel.org; linux-arm- > kernel at lists.infradead.org; linux-kernel at vger.kernel.org > Subject: Re: ARM64/KVM: Bad page state in process iperf > > On Tue, Dec 15, 2015 at 03:46:03AM +0000, Bhushan Bharat wrote: > > > > Hi All, > > > > I am running "iperf" in KVM guest on ARM64 machine and observing below > crash. > > Which host/guest kernel version is this? We are using Linux-v4.1 for both host (4K page size) and guest (64K page size). > > Which hardware? This is observed on Freescale LS2085 hardware based on A57 (8 CPUs). Thanks -Bharat > > -Christoffer > > > > > ============================= > > $iperf -c 3.3.3.3 -P 4 -t 0 -i 5 -w 90k > > ------------------------------------------------------------ > > Client connecting to 3.3.3.3, TCP port 5001 TCP window size: 180 > > KByte (WARNING: requested 90.0 KByte) > > ------------------------------------------------------------ > > [ 3] local 3.3.3.1 port 51131 connected with 3.3.3.3 port 5001 [ 6] > > local 3.3.3.1 port 51134 connected with 3.3.3.3 port 5001 [ 5] local > > 3.3.3.1 port 51133 connected with 3.3.3.3 port 5001 [ 4] local > > 3.3.3.1 port 51132 connected with 3.3.3.3 port 5001 > > [ 53.088567] random: nonblocking pool is initialized > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0- 5.0 sec 638 MBytes 1.07 Gbits/sec > > [ 4] 35.0-40.0 sec 1.66 GBytes 2.85 Gbits/sec [ 5] 40.0-45.0 sec > > 1.11 GBytes 1.90 Gbits/sec [ 4] 40.0-45.0 sec 1.16 GBytes 1.99 > > Gbits/sec > > [ 98.895207] BUG: Bad page state in process iperf pfn:0a584 > > [ 98.896164] page:ffff780000296100 count:-1 mapcount:0 mapping: > (null) index:0x0 > > [ 98.897436] flags: 0x0() > > [ 98.897885] page dumped because: nonzero _count > > [ 98.898640] Modules linked in: > > [ 98.899178] CPU: 0 PID: 1639 Comm: iperf Not tainted 4.1.8-00461- > ge5431ad #141 > > [ 98.900302] Hardware name: linux,dummy-virt (DT) > > [ 98.901014] Call trace: > > [ 98.901406] [<ffff800000096cac>] dump_backtrace+0x0/0x12c > > [ 98.902522] [<ffff800000096de8>] show_stack+0x10/0x1c > > [ 98.903441] [<ffff800000678dc8>] dump_stack+0x8c/0xdc > > [ 98.904202] [<ffff800000145480>] bad_page+0xc4/0x114 > > [ 98.904945] [<ffff8000001487a4>] get_page_from_freelist+0x590/0x63c > > [ 98.905871] [<ffff80000014893c>] __alloc_pages_nodemask+0xec/0x794 > > [ 98.906791] [<ffff80000059fc80>] skb_page_frag_refill+0x70/0xa8 > > [ 98.907678] [<ffff80000059fcd8>] sk_page_frag_refill+0x20/0xd0 > > [ 98.908550] [<ffff8000005edc04>] tcp_sendmsg+0x1f8/0x9a8 > > [ 98.909368] [<ffff80000061419c>] inet_sendmsg+0x5c/0xd0 > > [ 98.910178] [<ffff80000059bb44>] sock_sendmsg+0x14/0x58 > > [ 98.911027] [<ffff80000059bbec>] sock_write_iter+0x64/0xbc > > [ 98.912119] [<ffff80000019b5b8>] __vfs_write+0xac/0x10c > > [ 98.913126] [<ffff80000019bcb8>] vfs_write+0x90/0x1a0 > > [ 98.913963] [<ffff80000019c53c>] SyS_write+0x40/0xa0 > > > > ----------- > > _______________________________________________ > > kvmarm mailing list > > kvmarm at lists.cs.columbia.edu > > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 9+ messages in thread
* ARM64/KVM: Bad page state in process iperf 2015-12-15 3:46 ARM64/KVM: Bad page state in process iperf Bhushan Bharat 2015-12-15 9:29 ` Christoffer Dall @ 2015-12-15 9:34 ` Marc Zyngier 2015-12-15 9:53 ` Bhushan Bharat 1 sibling, 1 reply; 9+ messages in thread From: Marc Zyngier @ 2015-12-15 9:34 UTC (permalink / raw) To: linux-arm-kernel On 15/12/15 03:46, Bhushan Bharat wrote: > > Hi All, > > I am running "iperf" in KVM guest on ARM64 machine and observing below crash. > > ============================= > $iperf -c 3.3.3.3 -P 4 -t 0 -i 5 -w 90k > ------------------------------------------------------------ > Client connecting to 3.3.3.3, TCP port 5001 > TCP window size: 180 KByte (WARNING: requested 90.0 KByte) > ------------------------------------------------------------ > [ 3] local 3.3.3.1 port 51131 connected with 3.3.3.3 port 5001 > [ 6] local 3.3.3.1 port 51134 connected with 3.3.3.3 port 5001 > [ 5] local 3.3.3.1 port 51133 connected with 3.3.3.3 port 5001 > [ 4] local 3.3.3.1 port 51132 connected with 3.3.3.3 port 5001 > [ 53.088567] random: nonblocking pool is initialized > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 5.0 sec 638 MBytes 1.07 Gbits/sec > [ 4] 35.0-40.0 sec 1.66 GBytes 2.85 Gbits/sec > [ 5] 40.0-45.0 sec 1.11 GBytes 1.90 Gbits/sec > [ 4] 40.0-45.0 sec 1.16 GBytes 1.99 Gbits/sec > [ 98.895207] BUG: Bad page state in process iperf pfn:0a584 > [ 98.896164] page:ffff780000296100 count:-1 mapcount:0 mapping: (null) index:0x0 > [ 98.897436] flags: 0x0() > [ 98.897885] page dumped because: nonzero _count > [ 98.898640] Modules linked in: > [ 98.899178] CPU: 0 PID: 1639 Comm: iperf Not tainted 4.1.8-00461-ge5431ad #141 > [ 98.900302] Hardware name: linux,dummy-virt (DT) > [ 98.901014] Call trace: > [ 98.901406] [<ffff800000096cac>] dump_backtrace+0x0/0x12c > [ 98.902522] [<ffff800000096de8>] show_stack+0x10/0x1c > [ 98.903441] [<ffff800000678dc8>] dump_stack+0x8c/0xdc > [ 98.904202] [<ffff800000145480>] bad_page+0xc4/0x114 > [ 98.904945] [<ffff8000001487a4>] get_page_from_freelist+0x590/0x63c > [ 98.905871] [<ffff80000014893c>] __alloc_pages_nodemask+0xec/0x794 > [ 98.906791] [<ffff80000059fc80>] skb_page_frag_refill+0x70/0xa8 > [ 98.907678] [<ffff80000059fcd8>] sk_page_frag_refill+0x20/0xd0 > [ 98.908550] [<ffff8000005edc04>] tcp_sendmsg+0x1f8/0x9a8 > [ 98.909368] [<ffff80000061419c>] inet_sendmsg+0x5c/0xd0 > [ 98.910178] [<ffff80000059bb44>] sock_sendmsg+0x14/0x58 > [ 98.911027] [<ffff80000059bbec>] sock_write_iter+0x64/0xbc > [ 98.912119] [<ffff80000019b5b8>] __vfs_write+0xac/0x10c > [ 98.913126] [<ffff80000019bcb8>] vfs_write+0x90/0x1a0 > [ 98.913963] [<ffff80000019c53c>] SyS_write+0x40/0xa0 This looks quite bad, but I don't see anything here that links it to KVM (apart from being a guest). Do you have any indication that this is due to KVM misbehaving? I'd appreciate a few more details. Thanks, M. -- Jazz is not dead. It just smells funny... ^ permalink raw reply [flat|nested] 9+ messages in thread
* ARM64/KVM: Bad page state in process iperf 2015-12-15 9:34 ` Marc Zyngier @ 2015-12-15 9:53 ` Bhushan Bharat 2015-12-15 10:19 ` Marc Zyngier 0 siblings, 1 reply; 9+ messages in thread From: Bhushan Bharat @ 2015-12-15 9:53 UTC (permalink / raw) To: linux-arm-kernel Hi Mark, > -----Original Message----- > From: Marc Zyngier [mailto:marc.zyngier at arm.com] > Sent: Tuesday, December 15, 2015 3:05 PM > To: Bhushan Bharat-R65777 <Bharat.Bhushan@freescale.com>; > kvmarm at lists.cs.columbia.edu; kvm at vger.kernel.org; linux-arm- > kernel at lists.infradead.org; linux-kernel at vger.kernel.org > Subject: Re: ARM64/KVM: Bad page state in process iperf > > On 15/12/15 03:46, Bhushan Bharat wrote: > > > > Hi All, > > > > I am running "iperf" in KVM guest on ARM64 machine and observing below > crash. > > > > ============================= > > $iperf -c 3.3.3.3 -P 4 -t 0 -i 5 -w 90k > > ------------------------------------------------------------ > > Client connecting to 3.3.3.3, TCP port 5001 TCP window size: 180 > > KByte (WARNING: requested 90.0 KByte) > > ------------------------------------------------------------ > > [ 3] local 3.3.3.1 port 51131 connected with 3.3.3.3 port 5001 [ 6] > > local 3.3.3.1 port 51134 connected with 3.3.3.3 port 5001 [ 5] local > > 3.3.3.1 port 51133 connected with 3.3.3.3 port 5001 [ 4] local > > 3.3.3.1 port 51132 connected with 3.3.3.3 port 5001 > > [ 53.088567] random: nonblocking pool is initialized > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0- 5.0 sec 638 MBytes 1.07 Gbits/sec > > [ 4] 35.0-40.0 sec 1.66 GBytes 2.85 Gbits/sec [ 5] 40.0-45.0 sec > > 1.11 GBytes 1.90 Gbits/sec [ 4] 40.0-45.0 sec 1.16 GBytes 1.99 > > Gbits/sec > > [ 98.895207] BUG: Bad page state in process iperf pfn:0a584 > > [ 98.896164] page:ffff780000296100 count:-1 mapcount:0 mapping: > (null) index:0x0 > > [ 98.897436] flags: 0x0() > > [ 98.897885] page dumped because: nonzero _count > > [ 98.898640] Modules linked in: > > [ 98.899178] CPU: 0 PID: 1639 Comm: iperf Not tainted 4.1.8-00461- > ge5431ad #141 > > [ 98.900302] Hardware name: linux,dummy-virt (DT) > > [ 98.901014] Call trace: > > [ 98.901406] [<ffff800000096cac>] dump_backtrace+0x0/0x12c > > [ 98.902522] [<ffff800000096de8>] show_stack+0x10/0x1c > > [ 98.903441] [<ffff800000678dc8>] dump_stack+0x8c/0xdc > > [ 98.904202] [<ffff800000145480>] bad_page+0xc4/0x114 > > [ 98.904945] [<ffff8000001487a4>] get_page_from_freelist+0x590/0x63c > > [ 98.905871] [<ffff80000014893c>] __alloc_pages_nodemask+0xec/0x794 > > [ 98.906791] [<ffff80000059fc80>] skb_page_frag_refill+0x70/0xa8 > > [ 98.907678] [<ffff80000059fcd8>] sk_page_frag_refill+0x20/0xd0 > > [ 98.908550] [<ffff8000005edc04>] tcp_sendmsg+0x1f8/0x9a8 > > [ 98.909368] [<ffff80000061419c>] inet_sendmsg+0x5c/0xd0 > > [ 98.910178] [<ffff80000059bb44>] sock_sendmsg+0x14/0x58 > > [ 98.911027] [<ffff80000059bbec>] sock_write_iter+0x64/0xbc > > [ 98.912119] [<ffff80000019b5b8>] __vfs_write+0xac/0x10c > > [ 98.913126] [<ffff80000019bcb8>] vfs_write+0x90/0x1a0 > > [ 98.913963] [<ffff80000019c53c>] SyS_write+0x40/0xa0 > > This looks quite bad, but I don't see anything here that links it to KVM (apart > from being a guest). Do you have any indication that this is due to KVM > misbehaving? I never observed this issue in host Linux but observed this issue always in guest Linux. This issue does not comes immediately after I run "iperf" but after some time. > I'd appreciate a few more details. We have a networking hardware and we are directly assigning the h/w to guest. When using the same networking hardware in host it always works as expected (tried 100s of times). Also this issue is not observed when we have only one vCPU in guest but seen when we have SMP guest. Thanks -Bharat > > Thanks, > > M. > -- > Jazz is not dead. It just smells funny... ^ permalink raw reply [flat|nested] 9+ messages in thread
* ARM64/KVM: Bad page state in process iperf 2015-12-15 9:53 ` Bhushan Bharat @ 2015-12-15 10:19 ` Marc Zyngier 2015-12-15 10:57 ` Bhushan Bharat 0 siblings, 1 reply; 9+ messages in thread From: Marc Zyngier @ 2015-12-15 10:19 UTC (permalink / raw) To: linux-arm-kernel On 15/12/15 09:53, Bhushan Bharat wrote: > Hi Mark, > >> -----Original Message----- >> From: Marc Zyngier [mailto:marc.zyngier at arm.com] >> Sent: Tuesday, December 15, 2015 3:05 PM >> To: Bhushan Bharat-R65777 <Bharat.Bhushan@freescale.com>; >> kvmarm at lists.cs.columbia.edu; kvm at vger.kernel.org; linux-arm- >> kernel at lists.infradead.org; linux-kernel at vger.kernel.org >> Subject: Re: ARM64/KVM: Bad page state in process iperf >> >> On 15/12/15 03:46, Bhushan Bharat wrote: >>> >>> Hi All, >>> >>> I am running "iperf" in KVM guest on ARM64 machine and observing below >> crash. >>> >>> ============================= >>> $iperf -c 3.3.3.3 -P 4 -t 0 -i 5 -w 90k >>> ------------------------------------------------------------ >>> Client connecting to 3.3.3.3, TCP port 5001 TCP window size: 180 >>> KByte (WARNING: requested 90.0 KByte) >>> ------------------------------------------------------------ >>> [ 3] local 3.3.3.1 port 51131 connected with 3.3.3.3 port 5001 [ 6] >>> local 3.3.3.1 port 51134 connected with 3.3.3.3 port 5001 [ 5] local >>> 3.3.3.1 port 51133 connected with 3.3.3.3 port 5001 [ 4] local >>> 3.3.3.1 port 51132 connected with 3.3.3.3 port 5001 >>> [ 53.088567] random: nonblocking pool is initialized >>> [ ID] Interval Transfer Bandwidth >>> [ 3] 0.0- 5.0 sec 638 MBytes 1.07 Gbits/sec >>> [ 4] 35.0-40.0 sec 1.66 GBytes 2.85 Gbits/sec [ 5] 40.0-45.0 sec >>> 1.11 GBytes 1.90 Gbits/sec [ 4] 40.0-45.0 sec 1.16 GBytes 1.99 >>> Gbits/sec >>> [ 98.895207] BUG: Bad page state in process iperf pfn:0a584 >>> [ 98.896164] page:ffff780000296100 count:-1 mapcount:0 mapping: >> (null) index:0x0 >>> [ 98.897436] flags: 0x0() >>> [ 98.897885] page dumped because: nonzero _count >>> [ 98.898640] Modules linked in: >>> [ 98.899178] CPU: 0 PID: 1639 Comm: iperf Not tainted 4.1.8-00461- >> ge5431ad #141 >>> [ 98.900302] Hardware name: linux,dummy-virt (DT) >>> [ 98.901014] Call trace: >>> [ 98.901406] [<ffff800000096cac>] dump_backtrace+0x0/0x12c >>> [ 98.902522] [<ffff800000096de8>] show_stack+0x10/0x1c >>> [ 98.903441] [<ffff800000678dc8>] dump_stack+0x8c/0xdc >>> [ 98.904202] [<ffff800000145480>] bad_page+0xc4/0x114 >>> [ 98.904945] [<ffff8000001487a4>] get_page_from_freelist+0x590/0x63c >>> [ 98.905871] [<ffff80000014893c>] __alloc_pages_nodemask+0xec/0x794 >>> [ 98.906791] [<ffff80000059fc80>] skb_page_frag_refill+0x70/0xa8 >>> [ 98.907678] [<ffff80000059fcd8>] sk_page_frag_refill+0x20/0xd0 >>> [ 98.908550] [<ffff8000005edc04>] tcp_sendmsg+0x1f8/0x9a8 >>> [ 98.909368] [<ffff80000061419c>] inet_sendmsg+0x5c/0xd0 >>> [ 98.910178] [<ffff80000059bb44>] sock_sendmsg+0x14/0x58 >>> [ 98.911027] [<ffff80000059bbec>] sock_write_iter+0x64/0xbc >>> [ 98.912119] [<ffff80000019b5b8>] __vfs_write+0xac/0x10c >>> [ 98.913126] [<ffff80000019bcb8>] vfs_write+0x90/0x1a0 >>> [ 98.913963] [<ffff80000019c53c>] SyS_write+0x40/0xa0 >> >> This looks quite bad, but I don't see anything here that links it to KVM (apart >> from being a guest). Do you have any indication that this is due to KVM >> misbehaving? > > I never observed this issue in host Linux but observed this issue always in guest Linux. This issue does not comes immediately after I run "iperf" but after some time. > >> I'd appreciate a few more details. > > We have a networking hardware and we are directly assigning the h/w to guest. When using the same networking hardware in host it always works as expected (tried 100s of times). > Also this issue is not observed when we have only one vCPU in guest but seen when we have SMP guest. Can you reproduce the same issue without VFIO (using virtio, for example)? Is that platform VFIO? or PCI? Thanks, M. -- Jazz is not dead. It just smells funny... ^ permalink raw reply [flat|nested] 9+ messages in thread
* ARM64/KVM: Bad page state in process iperf 2015-12-15 10:19 ` Marc Zyngier @ 2015-12-15 10:57 ` Bhushan Bharat 2015-12-15 11:19 ` Marc Zyngier 0 siblings, 1 reply; 9+ messages in thread From: Bhushan Bharat @ 2015-12-15 10:57 UTC (permalink / raw) To: linux-arm-kernel > -----Original Message----- > From: Marc Zyngier [mailto:marc.zyngier at arm.com] > Sent: Tuesday, December 15, 2015 3:50 PM > To: Bhushan Bharat-R65777 <Bharat.Bhushan@freescale.com>; > kvmarm at lists.cs.columbia.edu; kvm at vger.kernel.org; linux-arm- > kernel at lists.infradead.org; linux-kernel at vger.kernel.org > Subject: Re: ARM64/KVM: Bad page state in process iperf > > On 15/12/15 09:53, Bhushan Bharat wrote: > > Hi Mark, > > > >> -----Original Message----- > >> From: Marc Zyngier [mailto:marc.zyngier at arm.com] > >> Sent: Tuesday, December 15, 2015 3:05 PM > >> To: Bhushan Bharat-R65777 <Bharat.Bhushan@freescale.com>; > >> kvmarm at lists.cs.columbia.edu; kvm at vger.kernel.org; linux-arm- > >> kernel at lists.infradead.org; linux-kernel at vger.kernel.org > >> Subject: Re: ARM64/KVM: Bad page state in process iperf > >> > >> On 15/12/15 03:46, Bhushan Bharat wrote: > >>> > >>> Hi All, > >>> > >>> I am running "iperf" in KVM guest on ARM64 machine and observing > >>> below > >> crash. > >>> > >>> ============================= > >>> $iperf -c 3.3.3.3 -P 4 -t 0 -i 5 -w 90k > >>> ------------------------------------------------------------ > >>> Client connecting to 3.3.3.3, TCP port 5001 TCP window size: 180 > >>> KByte (WARNING: requested 90.0 KByte) > >>> ------------------------------------------------------------ > >>> [ 3] local 3.3.3.1 port 51131 connected with 3.3.3.3 port 5001 [ > >>> 6] local 3.3.3.1 port 51134 connected with 3.3.3.3 port 5001 [ 5] > >>> local > >>> 3.3.3.1 port 51133 connected with 3.3.3.3 port 5001 [ 4] local > >>> 3.3.3.1 port 51132 connected with 3.3.3.3 port 5001 > >>> [ 53.088567] random: nonblocking pool is initialized > >>> [ ID] Interval Transfer Bandwidth > >>> [ 3] 0.0- 5.0 sec 638 MBytes 1.07 Gbits/sec > >>> [ 4] 35.0-40.0 sec 1.66 GBytes 2.85 Gbits/sec [ 5] 40.0-45.0 sec > >>> 1.11 GBytes 1.90 Gbits/sec [ 4] 40.0-45.0 sec 1.16 GBytes 1.99 > >>> Gbits/sec > >>> [ 98.895207] BUG: Bad page state in process iperf pfn:0a584 > >>> [ 98.896164] page:ffff780000296100 count:-1 mapcount:0 mapping: > >> (null) index:0x0 > >>> [ 98.897436] flags: 0x0() > >>> [ 98.897885] page dumped because: nonzero _count > >>> [ 98.898640] Modules linked in: > >>> [ 98.899178] CPU: 0 PID: 1639 Comm: iperf Not tainted 4.1.8-00461- > >> ge5431ad #141 > >>> [ 98.900302] Hardware name: linux,dummy-virt (DT) > >>> [ 98.901014] Call trace: > >>> [ 98.901406] [<ffff800000096cac>] dump_backtrace+0x0/0x12c > >>> [ 98.902522] [<ffff800000096de8>] show_stack+0x10/0x1c > >>> [ 98.903441] [<ffff800000678dc8>] dump_stack+0x8c/0xdc > >>> [ 98.904202] [<ffff800000145480>] bad_page+0xc4/0x114 > >>> [ 98.904945] [<ffff8000001487a4>] > get_page_from_freelist+0x590/0x63c > >>> [ 98.905871] [<ffff80000014893c>] > __alloc_pages_nodemask+0xec/0x794 > >>> [ 98.906791] [<ffff80000059fc80>] skb_page_frag_refill+0x70/0xa8 > >>> [ 98.907678] [<ffff80000059fcd8>] sk_page_frag_refill+0x20/0xd0 > >>> [ 98.908550] [<ffff8000005edc04>] tcp_sendmsg+0x1f8/0x9a8 > >>> [ 98.909368] [<ffff80000061419c>] inet_sendmsg+0x5c/0xd0 > >>> [ 98.910178] [<ffff80000059bb44>] sock_sendmsg+0x14/0x58 > >>> [ 98.911027] [<ffff80000059bbec>] sock_write_iter+0x64/0xbc > >>> [ 98.912119] [<ffff80000019b5b8>] __vfs_write+0xac/0x10c > >>> [ 98.913126] [<ffff80000019bcb8>] vfs_write+0x90/0x1a0 > >>> [ 98.913963] [<ffff80000019c53c>] SyS_write+0x40/0xa0 > >> > >> This looks quite bad, but I don't see anything here that links it to > >> KVM (apart from being a guest). Do you have any indication that this > >> is due to KVM misbehaving? > > > > I never observed this issue in host Linux but observed this issue always in > guest Linux. This issue does not comes immediately after I run "iperf" but > after some time. > > > >> I'd appreciate a few more details. > > > > We have a networking hardware and we are directly assigning the h/w to > guest. When using the same networking hardware in host it always works as > expected (tried 100s of times). > > Also this issue is not observed when we have only one vCPU in guest but > seen when we have SMP guest. > > Can you reproduce the same issue without VFIO (using virtio, for example)? With virtio I have not observed this issue. > Is that platform VFIO? or PCI? It is not vfio-pci and vfio-platform. It is vfio-fls-mc (some Freescale new hardware), similar to the lines of vfio-platform uses same set of VFIO APIs used by vfio-pci/platform. Do you think this can be some h/w specific issue. Thanks -Bharat > > Thanks, > > M. > -- > Jazz is not dead. It just smells funny... ^ permalink raw reply [flat|nested] 9+ messages in thread
* ARM64/KVM: Bad page state in process iperf 2015-12-15 10:57 ` Bhushan Bharat @ 2015-12-15 11:19 ` Marc Zyngier 2015-12-15 11:26 ` Bhushan Bharat 0 siblings, 1 reply; 9+ messages in thread From: Marc Zyngier @ 2015-12-15 11:19 UTC (permalink / raw) To: linux-arm-kernel On 15/12/15 10:57, Bhushan Bharat wrote: > > >> -----Original Message----- >> From: Marc Zyngier [mailto:marc.zyngier at arm.com] >> Sent: Tuesday, December 15, 2015 3:50 PM >> To: Bhushan Bharat-R65777 <Bharat.Bhushan@freescale.com>; >> kvmarm at lists.cs.columbia.edu; kvm at vger.kernel.org; linux-arm- >> kernel at lists.infradead.org; linux-kernel at vger.kernel.org >> Subject: Re: ARM64/KVM: Bad page state in process iperf >> >> On 15/12/15 09:53, Bhushan Bharat wrote: >>> Hi Mark, >>> >>>> -----Original Message----- >>>> From: Marc Zyngier [mailto:marc.zyngier at arm.com] >>>> Sent: Tuesday, December 15, 2015 3:05 PM >>>> To: Bhushan Bharat-R65777 <Bharat.Bhushan@freescale.com>; >>>> kvmarm at lists.cs.columbia.edu; kvm at vger.kernel.org; linux-arm- >>>> kernel at lists.infradead.org; linux-kernel at vger.kernel.org >>>> Subject: Re: ARM64/KVM: Bad page state in process iperf >>>> >>>> On 15/12/15 03:46, Bhushan Bharat wrote: >>>>> >>>>> Hi All, >>>>> >>>>> I am running "iperf" in KVM guest on ARM64 machine and observing >>>>> below >>>> crash. >>>>> >>>>> ============================= >>>>> $iperf -c 3.3.3.3 -P 4 -t 0 -i 5 -w 90k >>>>> ------------------------------------------------------------ >>>>> Client connecting to 3.3.3.3, TCP port 5001 TCP window size: 180 >>>>> KByte (WARNING: requested 90.0 KByte) >>>>> ------------------------------------------------------------ >>>>> [ 3] local 3.3.3.1 port 51131 connected with 3.3.3.3 port 5001 [ >>>>> 6] local 3.3.3.1 port 51134 connected with 3.3.3.3 port 5001 [ 5] >>>>> local >>>>> 3.3.3.1 port 51133 connected with 3.3.3.3 port 5001 [ 4] local >>>>> 3.3.3.1 port 51132 connected with 3.3.3.3 port 5001 >>>>> [ 53.088567] random: nonblocking pool is initialized >>>>> [ ID] Interval Transfer Bandwidth >>>>> [ 3] 0.0- 5.0 sec 638 MBytes 1.07 Gbits/sec >>>>> [ 4] 35.0-40.0 sec 1.66 GBytes 2.85 Gbits/sec [ 5] 40.0-45.0 sec >>>>> 1.11 GBytes 1.90 Gbits/sec [ 4] 40.0-45.0 sec 1.16 GBytes 1.99 >>>>> Gbits/sec >>>>> [ 98.895207] BUG: Bad page state in process iperf pfn:0a584 >>>>> [ 98.896164] page:ffff780000296100 count:-1 mapcount:0 mapping: >>>> (null) index:0x0 >>>>> [ 98.897436] flags: 0x0() >>>>> [ 98.897885] page dumped because: nonzero _count >>>>> [ 98.898640] Modules linked in: >>>>> [ 98.899178] CPU: 0 PID: 1639 Comm: iperf Not tainted 4.1.8-00461- >>>> ge5431ad #141 >>>>> [ 98.900302] Hardware name: linux,dummy-virt (DT) >>>>> [ 98.901014] Call trace: >>>>> [ 98.901406] [<ffff800000096cac>] dump_backtrace+0x0/0x12c >>>>> [ 98.902522] [<ffff800000096de8>] show_stack+0x10/0x1c >>>>> [ 98.903441] [<ffff800000678dc8>] dump_stack+0x8c/0xdc >>>>> [ 98.904202] [<ffff800000145480>] bad_page+0xc4/0x114 >>>>> [ 98.904945] [<ffff8000001487a4>] >> get_page_from_freelist+0x590/0x63c >>>>> [ 98.905871] [<ffff80000014893c>] >> __alloc_pages_nodemask+0xec/0x794 >>>>> [ 98.906791] [<ffff80000059fc80>] skb_page_frag_refill+0x70/0xa8 >>>>> [ 98.907678] [<ffff80000059fcd8>] sk_page_frag_refill+0x20/0xd0 >>>>> [ 98.908550] [<ffff8000005edc04>] tcp_sendmsg+0x1f8/0x9a8 >>>>> [ 98.909368] [<ffff80000061419c>] inet_sendmsg+0x5c/0xd0 >>>>> [ 98.910178] [<ffff80000059bb44>] sock_sendmsg+0x14/0x58 >>>>> [ 98.911027] [<ffff80000059bbec>] sock_write_iter+0x64/0xbc >>>>> [ 98.912119] [<ffff80000019b5b8>] __vfs_write+0xac/0x10c >>>>> [ 98.913126] [<ffff80000019bcb8>] vfs_write+0x90/0x1a0 >>>>> [ 98.913963] [<ffff80000019c53c>] SyS_write+0x40/0xa0 >>>> >>>> This looks quite bad, but I don't see anything here that links it to >>>> KVM (apart from being a guest). Do you have any indication that this >>>> is due to KVM misbehaving? >>> >>> I never observed this issue in host Linux but observed this issue always in >> guest Linux. This issue does not comes immediately after I run "iperf" but >> after some time. >>> >>>> I'd appreciate a few more details. >>> >>> We have a networking hardware and we are directly assigning the h/w to >> guest. When using the same networking hardware in host it always works as >> expected (tried 100s of times). >>> Also this issue is not observed when we have only one vCPU in guest but >> seen when we have SMP guest. >> >> Can you reproduce the same issue without VFIO (using virtio, for example)? > > With virtio I have not observed this issue. > >> Is that platform VFIO? or PCI? > > It is not vfio-pci and vfio-platform. It is vfio-fls-mc (some > Freescale new hardware), similar to the lines of vfio-platform uses > same set of VFIO APIs used by vfio-pci/platform. Do you think this > can be some h/w specific issue. I have no idea, but by the look of it, something could be doing DMA on top of your guest page tables, which is not really expected. I suggest you carefully look at: 1) the DMA addresses that are passed to your device 2) the page tables that are programmed into the SMMU 3) the resulting translation Hopefully this will give you a clue about what is generating this. Thanks, M. -- Jazz is not dead. It just smells funny... ^ permalink raw reply [flat|nested] 9+ messages in thread
* ARM64/KVM: Bad page state in process iperf 2015-12-15 11:19 ` Marc Zyngier @ 2015-12-15 11:26 ` Bhushan Bharat 0 siblings, 0 replies; 9+ messages in thread From: Bhushan Bharat @ 2015-12-15 11:26 UTC (permalink / raw) To: linux-arm-kernel > -----Original Message----- > From: Marc Zyngier [mailto:marc.zyngier at arm.com] > Sent: Tuesday, December 15, 2015 4:49 PM > To: Bhushan Bharat-R65777 <Bharat.Bhushan@freescale.com>; > kvmarm at lists.cs.columbia.edu; kvm at vger.kernel.org; linux-arm- > kernel at lists.infradead.org; linux-kernel at vger.kernel.org > Subject: Re: ARM64/KVM: Bad page state in process iperf > > On 15/12/15 10:57, Bhushan Bharat wrote: > > > > > >> -----Original Message----- > >> From: Marc Zyngier [mailto:marc.zyngier at arm.com] > >> Sent: Tuesday, December 15, 2015 3:50 PM > >> To: Bhushan Bharat-R65777 <Bharat.Bhushan@freescale.com>; > >> kvmarm at lists.cs.columbia.edu; kvm at vger.kernel.org; linux-arm- > >> kernel at lists.infradead.org; linux-kernel at vger.kernel.org > >> Subject: Re: ARM64/KVM: Bad page state in process iperf > >> > >> On 15/12/15 09:53, Bhushan Bharat wrote: > >>> Hi Mark, > >>> > >>>> -----Original Message----- > >>>> From: Marc Zyngier [mailto:marc.zyngier at arm.com] > >>>> Sent: Tuesday, December 15, 2015 3:05 PM > >>>> To: Bhushan Bharat-R65777 <Bharat.Bhushan@freescale.com>; > >>>> kvmarm at lists.cs.columbia.edu; kvm at vger.kernel.org; linux-arm- > >>>> kernel at lists.infradead.org; linux-kernel at vger.kernel.org > >>>> Subject: Re: ARM64/KVM: Bad page state in process iperf > >>>> > >>>> On 15/12/15 03:46, Bhushan Bharat wrote: > >>>>> > >>>>> Hi All, > >>>>> > >>>>> I am running "iperf" in KVM guest on ARM64 machine and observing > >>>>> below > >>>> crash. > >>>>> > >>>>> ============================= > >>>>> $iperf -c 3.3.3.3 -P 4 -t 0 -i 5 -w 90k > >>>>> ------------------------------------------------------------ > >>>>> Client connecting to 3.3.3.3, TCP port 5001 TCP window size: 180 > >>>>> KByte (WARNING: requested 90.0 KByte) > >>>>> ------------------------------------------------------------ > >>>>> [ 3] local 3.3.3.1 port 51131 connected with 3.3.3.3 port 5001 [ > >>>>> 6] local 3.3.3.1 port 51134 connected with 3.3.3.3 port 5001 [ 5] > >>>>> local > >>>>> 3.3.3.1 port 51133 connected with 3.3.3.3 port 5001 [ 4] local > >>>>> 3.3.3.1 port 51132 connected with 3.3.3.3 port 5001 > >>>>> [ 53.088567] random: nonblocking pool is initialized > >>>>> [ ID] Interval Transfer Bandwidth > >>>>> [ 3] 0.0- 5.0 sec 638 MBytes 1.07 Gbits/sec > >>>>> [ 4] 35.0-40.0 sec 1.66 GBytes 2.85 Gbits/sec [ 5] 40.0-45.0 > >>>>> sec > >>>>> 1.11 GBytes 1.90 Gbits/sec [ 4] 40.0-45.0 sec 1.16 GBytes 1.99 > >>>>> Gbits/sec > >>>>> [ 98.895207] BUG: Bad page state in process iperf pfn:0a584 > >>>>> [ 98.896164] page:ffff780000296100 count:-1 mapcount:0 mapping: > >>>> (null) index:0x0 > >>>>> [ 98.897436] flags: 0x0() > >>>>> [ 98.897885] page dumped because: nonzero _count > >>>>> [ 98.898640] Modules linked in: > >>>>> [ 98.899178] CPU: 0 PID: 1639 Comm: iperf Not tainted 4.1.8-00461- > >>>> ge5431ad #141 > >>>>> [ 98.900302] Hardware name: linux,dummy-virt (DT) > >>>>> [ 98.901014] Call trace: > >>>>> [ 98.901406] [<ffff800000096cac>] dump_backtrace+0x0/0x12c > >>>>> [ 98.902522] [<ffff800000096de8>] show_stack+0x10/0x1c > >>>>> [ 98.903441] [<ffff800000678dc8>] dump_stack+0x8c/0xdc > >>>>> [ 98.904202] [<ffff800000145480>] bad_page+0xc4/0x114 > >>>>> [ 98.904945] [<ffff8000001487a4>] > >> get_page_from_freelist+0x590/0x63c > >>>>> [ 98.905871] [<ffff80000014893c>] > >> __alloc_pages_nodemask+0xec/0x794 > >>>>> [ 98.906791] [<ffff80000059fc80>] skb_page_frag_refill+0x70/0xa8 > >>>>> [ 98.907678] [<ffff80000059fcd8>] sk_page_frag_refill+0x20/0xd0 > >>>>> [ 98.908550] [<ffff8000005edc04>] tcp_sendmsg+0x1f8/0x9a8 > >>>>> [ 98.909368] [<ffff80000061419c>] inet_sendmsg+0x5c/0xd0 > >>>>> [ 98.910178] [<ffff80000059bb44>] sock_sendmsg+0x14/0x58 > >>>>> [ 98.911027] [<ffff80000059bbec>] sock_write_iter+0x64/0xbc > >>>>> [ 98.912119] [<ffff80000019b5b8>] __vfs_write+0xac/0x10c > >>>>> [ 98.913126] [<ffff80000019bcb8>] vfs_write+0x90/0x1a0 > >>>>> [ 98.913963] [<ffff80000019c53c>] SyS_write+0x40/0xa0 > >>>> > >>>> This looks quite bad, but I don't see anything here that links it > >>>> to KVM (apart from being a guest). Do you have any indication that > >>>> this is due to KVM misbehaving? > >>> > >>> I never observed this issue in host Linux but observed this issue > >>> always in > >> guest Linux. This issue does not comes immediately after I run > >> "iperf" but after some time. > >>> > >>>> I'd appreciate a few more details. > >>> > >>> We have a networking hardware and we are directly assigning the h/w > >>> to > >> guest. When using the same networking hardware in host it always > >> works as expected (tried 100s of times). > >>> Also this issue is not observed when we have only one vCPU in guest > >>> but > >> seen when we have SMP guest. > >> > >> Can you reproduce the same issue without VFIO (using virtio, for > example)? > > > > With virtio I have not observed this issue. > > > >> Is that platform VFIO? or PCI? > > > > It is not vfio-pci and vfio-platform. It is vfio-fls-mc (some > > Freescale new hardware), similar to the lines of vfio-platform uses > > same set of VFIO APIs used by vfio-pci/platform. Do you think this can > > be some h/w specific issue. > > I have no idea, but by the look of it, something could be doing DMA on top of > your guest page tables, which is not really expected. I suggest you carefully > look at: > > 1) the DMA addresses that are passed to your device > 2) the page tables that are programmed into the SMMU > 3) the resulting translation Thanks Mark, this is good info. I will continue debugging keeping these points in my mind. -Bharat > > Hopefully this will give you a clue about what is generating this. > > Thanks, > > M. > -- > Jazz is not dead. It just smells funny... ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2015-12-15 11:26 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-12-15 3:46 ARM64/KVM: Bad page state in process iperf Bhushan Bharat 2015-12-15 9:29 ` Christoffer Dall 2015-12-15 9:33 ` Bhushan Bharat 2015-12-15 9:34 ` Marc Zyngier 2015-12-15 9:53 ` Bhushan Bharat 2015-12-15 10:19 ` Marc Zyngier 2015-12-15 10:57 ` Bhushan Bharat 2015-12-15 11:19 ` Marc Zyngier 2015-12-15 11:26 ` Bhushan Bharat
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).