* segmentation fault on power system
@ 2013-07-23 10:13 Han Pingtian
2013-07-23 10:23 ` Erwan Velu
0 siblings, 1 reply; 11+ messages in thread
From: Han Pingtian @ 2013-07-23 10:13 UTC (permalink / raw)
To: fio
Hey there,
When trying to run fio on one of our power system, segmentation fault
occured and I got two core file. This is the backtraces:
Core was generated by `./fio/fio fio-jobs/randomw.fio '.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000010062aac in init_disk_util (td=0xfff8ad00000) at diskutil.c:481
481 if (!td->o.do_disk_util ||
(gdb) bt
#0 0x0000000010062aac in init_disk_util (td=0xfff8ad00000) at diskutil.c:481
#1 0x0000000010050f30 in run_threads () at backend.c:1691
#2 0x000000001005179c in fio_backend () at backend.c:1911
#3 0x0000000010066bc4 in main (argc=<value optimized out>, argv=0xfffd33602d8, envp=<value optimized out>)
at fio.c:50
(gdb) p td->io_ops
$1 = (struct ioengine_ops *) 0x0
(gdb) p td
$2 = (struct thread_data *) 0xfff8ad00000
(gdb) p threads->io_ops
$3 = (struct ioengine_ops *) 0x100349126c0
(gdb)
Core was generated by `./fio/fio fio-jobs/randomw.fio '.
Program terminated with signal 4, Illegal instruction.
#0 mfspr (envp=<value optimized out>) at arch/arch-ppc.h:55
55 asm volatile("mfspr %0,%1": "=r" (val) : "K" (reg));
(gdb) bt
#0 mfspr (envp=<value optimized out>) at arch/arch-ppc.h:55
#1 get_cpu_clock (envp=<value optimized out>) at arch/arch-ppc.h:71
#2 atb_child (envp=<value optimized out>) at arch/arch-ppc.h:88
#3 atb_clocktest (envp=<value optimized out>) at arch/arch-ppc.h:98
#4 arch_init (envp=<value optimized out>) at arch/arch-ppc.h:116
#5 initialize_fio (envp=<value optimized out>) at libfio.c:248
#6 0x0000000010066b44 in main (argc=<value optimized out>, argv=<value optimized out>,
envp=<value optimized out>) at fio.c:33
(gdb)
This is the job file:
[sequential-read]
rw=read
ioengine=libaio
direct=0
iodepth=4
size=${SIZE}
numjobs=${NUMJOBS}
; -- end job file --
Please have a look. Thanks in advance!
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: segmentation fault on power system
2013-07-23 10:13 Han Pingtian
@ 2013-07-23 10:23 ` Erwan Velu
2013-07-23 15:15 ` Jens Axboe
2013-07-24 1:40 ` Han Pingtian
0 siblings, 2 replies; 11+ messages in thread
From: Erwan Velu @ 2013-07-23 10:23 UTC (permalink / raw)
To: Han Pingtian; +Cc: fio
On 23/07/2013 12:13, Han Pingtian wrote:
> Hey there,
>
> When trying to run fio on one of our power system, segmentation fault
>
Can you give us the kind of cpu you are using ? (/proc/cpuinfo)
I'm not used with PPC but maybe your processor doesn't support ATBU call
on mfspr.
Erwan,
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: segmentation fault on power system
2013-07-23 10:23 ` Erwan Velu
@ 2013-07-23 15:15 ` Jens Axboe
2013-07-24 2:10 ` Han Pingtian
2013-07-24 1:40 ` Han Pingtian
1 sibling, 1 reply; 11+ messages in thread
From: Jens Axboe @ 2013-07-23 15:15 UTC (permalink / raw)
To: Erwan Velu; +Cc: Han Pingtian, fio
On 07/23/2013 04:23 AM, Erwan Velu wrote:
> On 23/07/2013 12:13, Han Pingtian wrote:
>> Hey there,
>>
>> When trying to run fio on one of our power system, segmentation fault
>>
> Can you give us the kind of cpu you are using ? (/proc/cpuinfo)
> I'm not used with PPC but maybe your processor doesn't support ATBU call
> on mfspr.
That is definitely the problem, so CPU info would help. In the mean
time, you can use clocksource=clock_gettime to get rid of the illegal
instruction.
--
Jens Axboe
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: segmentation fault on power system
2013-07-23 10:23 ` Erwan Velu
2013-07-23 15:15 ` Jens Axboe
@ 2013-07-24 1:40 ` Han Pingtian
1 sibling, 0 replies; 11+ messages in thread
From: Han Pingtian @ 2013-07-24 1:40 UTC (permalink / raw)
To: Erwan Velu; +Cc: fio
On Tue, Jul 23, 2013 at 12:23:19PM +0200, Erwan Velu wrote:
> On 23/07/2013 12:13, Han Pingtian wrote:
> >Hey there,
> >
> >When trying to run fio on one of our power system, segmentation fault
> >
> Can you give us the kind of cpu you are using ? (/proc/cpuinfo)
> I'm not used with PPC but maybe your processor doesn't support ATBU
> call on mfspr.
>
> Erwan,
This is the contents of /proc/cpuinfo:
processor : 0
cpu : POWER7 (architected), altivec supported
clock : 3550.000000MHz
revision : 2.3 (pvr 003f 0203)
processor : 1
cpu : POWER7 (architected), altivec supported
clock : 3550.000000MHz
revision : 2.3 (pvr 003f 0203)
processor : 2
cpu : POWER7 (architected), altivec supported
clock : 3550.000000MHz
revision : 2.3 (pvr 003f 0203)
processor : 3
cpu : POWER7 (architected), altivec supported
clock : 3550.000000MHz
revision : 2.3 (pvr 003f 0203)
timebase : 512000000
platform : pSeries
model : IBM,8231-E2C
machine : CHRP IBM,8231-E2C
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: segmentation fault on power system
2013-07-23 15:15 ` Jens Axboe
@ 2013-07-24 2:10 ` Han Pingtian
2013-07-24 6:21 ` Han Pingtian
0 siblings, 1 reply; 11+ messages in thread
From: Han Pingtian @ 2013-07-24 2:10 UTC (permalink / raw)
To: Jens Axboe; +Cc: Erwan Velu, fio
On Tue, Jul 23, 2013 at 09:15:40AM -0600, Jens Axboe wrote:
> On 07/23/2013 04:23 AM, Erwan Velu wrote:
> > On 23/07/2013 12:13, Han Pingtian wrote:
> >> Hey there,
> >>
> >> When trying to run fio on one of our power system, segmentation fault
> >>
> > Can you give us the kind of cpu you are using ? (/proc/cpuinfo)
> > I'm not used with PPC but maybe your processor doesn't support ATBU call
> > on mfspr.
>
> That is definitely the problem, so CPU info would help. In the mean
> time, you can use clocksource=clock_gettime to get rid of the illegal
> instruction.
This is the contents of /proc/cpuinfo:
processor : 0
cpu : POWER7 (architected), altivec supported
clock : 3550.000000MHz
revision : 2.3 (pvr 003f 0203)
processor : 1
cpu : POWER7 (architected), altivec supported
clock : 3550.000000MHz
revision : 2.3 (pvr 003f 0203)
processor : 2
cpu : POWER7 (architected), altivec supported
clock : 3550.000000MHz
revision : 2.3 (pvr 003f 0203)
processor : 3
cpu : POWER7 (architected), altivec supported
clock : 3550.000000MHz
revision : 2.3 (pvr 003f 0203)
timebase : 512000000
platform : pSeries
model : IBM,8231-E2C
machine : CHRP IBM,8231-E2C
But 'clocksource=clock_gettime' doesn't fix the fault. I can also get
the same two cores.
If I changed the code like this:
================================================================================
diff --git a/arch/arch-ppc.h b/arch/arch-ppc.h
index 65e6b74..30c315c 100644
--- a/arch/arch-ppc.h
+++ b/arch/arch-ppc.h
@@ -67,15 +67,15 @@ static inline unsigned long long get_cpu_clock(void)
unsigned long long ret;
do {
- if (arch_flags & ARCH_FLAG_1) {
- tbu0 = mfspr(SPRN_ATBU);
- tbl = mfspr(SPRN_ATBL);
- tbu1 = mfspr(SPRN_ATBU);
- } else {
+ //if (arch_flags & ARCH_FLAG_1) {
+ // tbu0 = mfspr(SPRN_ATBU);
+ // tbl = mfspr(SPRN_ATBL);
+ // tbu1 = mfspr(SPRN_ATBU);
+ //} else {
tbu0 = mfspr(SPRN_TBRU);
tbl = mfspr(SPRN_TBRL);
tbu1 = mfspr(SPRN_TBRU);
- }
+ //}
} while (tbu0 != tbu1);
ret = (((unsigned long long)tbu0) << 32) | tbl;
================================================================================
then only one core dumpped which has this backtrace:
================================================================================
Core was generated by `./fio/fio --debug=parse fio-jobs/randomw.fio '.
Program terminated with signal 11, Segmentation fault.
#0 0x000000001006288c in init_disk_util (td=0xfff9a730000) at diskutil.c:481
481 if (!td->o.do_disk_util ||
(gdb) bt
#0 0x000000001006288c in init_disk_util (td=0xfff9a730000) at diskutil.c:481
#1 0x0000000010050d30 in run_threads () at backend.c:1691
#2 0x000000001005159c in fio_backend () at backend.c:1911
#3 0x00000000100669a4 in main (argc=<value optimized out>, argv=0xfffd20c9ec8, envp=<value optimized out>)
at fio.c:50
(gdb) p td
$1 = (struct thread_data *) 0xfff9a730000
(gdb) p threads
$2 = (struct thread_data *) 0xfff9a730000
(gdb) p td->io_ops
$3 = (struct ioengine_ops *) 0x0
(gdb) p threads->io_ops
$4 = (struct ioengine_ops *) 0x1000e262720
(gdb)
================================================================================
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: segmentation fault on power system
2013-07-24 2:10 ` Han Pingtian
@ 2013-07-24 6:21 ` Han Pingtian
2013-07-25 14:34 ` Jens Axboe
0 siblings, 1 reply; 11+ messages in thread
From: Han Pingtian @ 2013-07-24 6:21 UTC (permalink / raw)
To: Jens Axboe; +Cc: Erwan Velu, fio
I think I have found what's wrong. After 'make distclean' then 'make',
the problem of segfault has gone. There were some object files
compiled on another machine. The recompilation fixed it. But looks like
the ATBU call dosen't work on power7 system.
On Wed, Jul 24, 2013 at 10:10:19AM +0800, Han Pingtian wrote:
> On Tue, Jul 23, 2013 at 09:15:40AM -0600, Jens Axboe wrote:
> > On 07/23/2013 04:23 AM, Erwan Velu wrote:
> > > On 23/07/2013 12:13, Han Pingtian wrote:
> > >> Hey there,
> > >>
> > >> When trying to run fio on one of our power system, segmentation fault
> > >>
> > > Can you give us the kind of cpu you are using ? (/proc/cpuinfo)
> > > I'm not used with PPC but maybe your processor doesn't support ATBU call
> > > on mfspr.
> >
> > That is definitely the problem, so CPU info would help. In the mean
> > time, you can use clocksource=clock_gettime to get rid of the illegal
> > instruction.
>
> This is the contents of /proc/cpuinfo:
>
> processor : 0
> cpu : POWER7 (architected), altivec supported
> clock : 3550.000000MHz
> revision : 2.3 (pvr 003f 0203)
>
> processor : 1
> cpu : POWER7 (architected), altivec supported
> clock : 3550.000000MHz
> revision : 2.3 (pvr 003f 0203)
>
> processor : 2
> cpu : POWER7 (architected), altivec supported
> clock : 3550.000000MHz
> revision : 2.3 (pvr 003f 0203)
>
> processor : 3
> cpu : POWER7 (architected), altivec supported
> clock : 3550.000000MHz
> revision : 2.3 (pvr 003f 0203)
>
> timebase : 512000000
> platform : pSeries
> model : IBM,8231-E2C
> machine : CHRP IBM,8231-E2C
>
> But 'clocksource=clock_gettime' doesn't fix the fault. I can also get
> the same two cores.
>
> If I changed the code like this:
>
> ================================================================================
> diff --git a/arch/arch-ppc.h b/arch/arch-ppc.h
> index 65e6b74..30c315c 100644
> --- a/arch/arch-ppc.h
> +++ b/arch/arch-ppc.h
> @@ -67,15 +67,15 @@ static inline unsigned long long get_cpu_clock(void)
> unsigned long long ret;
>
> do {
> - if (arch_flags & ARCH_FLAG_1) {
> - tbu0 = mfspr(SPRN_ATBU);
> - tbl = mfspr(SPRN_ATBL);
> - tbu1 = mfspr(SPRN_ATBU);
> - } else {
> + //if (arch_flags & ARCH_FLAG_1) {
> + // tbu0 = mfspr(SPRN_ATBU);
> + // tbl = mfspr(SPRN_ATBL);
> + // tbu1 = mfspr(SPRN_ATBU);
> + //} else {
> tbu0 = mfspr(SPRN_TBRU);
> tbl = mfspr(SPRN_TBRL);
> tbu1 = mfspr(SPRN_TBRU);
> - }
> + //}
> } while (tbu0 != tbu1);
>
> ret = (((unsigned long long)tbu0) << 32) | tbl;
> ================================================================================
>
> then only one core dumpped which has this backtrace:
>
> ================================================================================
> Core was generated by `./fio/fio --debug=parse fio-jobs/randomw.fio '.
> Program terminated with signal 11, Segmentation fault.
> #0 0x000000001006288c in init_disk_util (td=0xfff9a730000) at diskutil.c:481
> 481 if (!td->o.do_disk_util ||
> (gdb) bt
> #0 0x000000001006288c in init_disk_util (td=0xfff9a730000) at diskutil.c:481
> #1 0x0000000010050d30 in run_threads () at backend.c:1691
> #2 0x000000001005159c in fio_backend () at backend.c:1911
> #3 0x00000000100669a4 in main (argc=<value optimized out>, argv=0xfffd20c9ec8, envp=<value optimized out>)
> at fio.c:50
> (gdb) p td
> $1 = (struct thread_data *) 0xfff9a730000
> (gdb) p threads
> $2 = (struct thread_data *) 0xfff9a730000
> (gdb) p td->io_ops
> $3 = (struct ioengine_ops *) 0x0
> (gdb) p threads->io_ops
> $4 = (struct ioengine_ops *) 0x1000e262720
> (gdb)
> ================================================================================
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: segmentation fault on power system
2013-07-24 6:21 ` Han Pingtian
@ 2013-07-25 14:34 ` Jens Axboe
2013-08-09 8:39 ` Han Pingtian
0 siblings, 1 reply; 11+ messages in thread
From: Jens Axboe @ 2013-07-25 14:34 UTC (permalink / raw)
To: Han Pingtian; +Cc: Erwan Velu, fio
On 07/24/2013 12:21 AM, Han Pingtian wrote:
> I think I have found what's wrong. After 'make distclean' then 'make',
> the problem of segfault has gone. There were some object files
> compiled on another machine. The recompilation fixed it. But looks like
> the ATBU call dosen't work on power7 system.
So the recompilation fixed the segfault, but not the illegal instruction
due to ATBU?
--
Jens Axboe
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: segmentation fault on power system
2013-07-25 14:34 ` Jens Axboe
@ 2013-08-09 8:39 ` Han Pingtian
2013-08-09 15:02 ` Jens Axboe
0 siblings, 1 reply; 11+ messages in thread
From: Han Pingtian @ 2013-08-09 8:39 UTC (permalink / raw)
To: Jens Axboe; +Cc: Erwan Velu, fio
On Thu, Jul 25, 2013 at 08:34:41AM -0600, Jens Axboe wrote:
> On 07/24/2013 12:21 AM, Han Pingtian wrote:
> > I think I have found what's wrong. After 'make distclean' then 'make',
> > the problem of segfault has gone. There were some object files
> > compiled on another machine. The recompilation fixed it. But looks like
> > the ATBU call dosen't work on power7 system.
>
> So the recompilation fixed the segfault, but not the illegal instruction
> due to ATBU?
Yes. If we set unlimit core dump, fio will generate a core with
backtrace like this one:
#0 mfspr (envp=<value optimized out>) at arch/arch-ppc.h:55
55 asm volatile("mfspr %0,%1": "=r" (val) : "K" (reg));
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6.ppc64 libaio-0.3.107-10.el6.ppc64
(gdb) bt
#0 mfspr (envp=<value optimized out>) at arch/arch-ppc.h:55
#1 get_cpu_clock (envp=<value optimized out>) at arch/arch-ppc.h:71
#2 atb_child (envp=<value optimized out>) at arch/arch-ppc.h:88
#3 atb_clocktest (envp=<value optimized out>) at arch/arch-ppc.h:98
#4 arch_init (envp=<value optimized out>) at arch/arch-ppc.h:116
#5 initialize_fio (envp=<value optimized out>) at libfio.c:248
#6 0x0000000010067294 in main (argc=<value optimized out>, argv=<value optimized out>,
envp=<value optimized out>) at fio.c:33
Apologize for late to reply.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: segmentation fault on power system
2013-08-09 8:39 ` Han Pingtian
@ 2013-08-09 15:02 ` Jens Axboe
0 siblings, 0 replies; 11+ messages in thread
From: Jens Axboe @ 2013-08-09 15:02 UTC (permalink / raw)
To: Han Pingtian; +Cc: Erwan Velu, fio
On 08/09/2013 02:39 AM, Han Pingtian wrote:
> On Thu, Jul 25, 2013 at 08:34:41AM -0600, Jens Axboe wrote:
>> On 07/24/2013 12:21 AM, Han Pingtian wrote:
>>> I think I have found what's wrong. After 'make distclean' then 'make',
>>> the problem of segfault has gone. There were some object files
>>> compiled on another machine. The recompilation fixed it. But looks like
>>> the ATBU call dosen't work on power7 system.
>>
>> So the recompilation fixed the segfault, but not the illegal instruction
>> due to ATBU?
>
> Yes. If we set unlimit core dump, fio will generate a core with
> backtrace like this one:
>
> #0 mfspr (envp=<value optimized out>) at arch/arch-ppc.h:55
> 55 asm volatile("mfspr %0,%1": "=r" (val) : "K" (reg));
> Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6.ppc64 libaio-0.3.107-10.el6.ppc64
> (gdb) bt
> #0 mfspr (envp=<value optimized out>) at arch/arch-ppc.h:55
> #1 get_cpu_clock (envp=<value optimized out>) at arch/arch-ppc.h:71
> #2 atb_child (envp=<value optimized out>) at arch/arch-ppc.h:88
> #3 atb_clocktest (envp=<value optimized out>) at arch/arch-ppc.h:98
> #4 arch_init (envp=<value optimized out>) at arch/arch-ppc.h:116
> #5 initialize_fio (envp=<value optimized out>) at libfio.c:248
> #6 0x0000000010067294 in main (argc=<value optimized out>, argv=<value optimized out>,
> envp=<value optimized out>) at fio.c:33
>
>
> Apologize for late to reply.
Alright, I'll have PPC default to the normal clock until we can improve
and detect this properly.
--
Jens Axboe
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: segmentation fault on power system
@ 2013-10-10 6:30 Saritha Vinod
2013-10-11 16:28 ` Jens Axboe
0 siblings, 1 reply; 11+ messages in thread
From: Saritha Vinod @ 2013-10-10 6:30 UTC (permalink / raw)
To: fio
Hi Jens,
>> Yes. If we set unlimit core dump, fio will generate a core with
>> backtrace like this one:
>>
>> #0 mfspr (envp=<value optimized out>) at arch/arch-ppc.h:55
>> 55 asm volatile("mfspr %0,%1": "=r" (val) : "K" (reg));
>> Missing separate debuginfos, use: debuginfo-install
glibc-2.12-1.107.el6.ppc64 libaio-0.3.107-10.el6.ppc64
>> (gdb) bt
>> #0 mfspr (envp=<value optimized out>) at arch/arch-ppc.h:55
>> #1 get_cpu_clock (envp=<value optimized out>) at arch/arch-ppc.h:71
>> #2 atb_child (envp=<value optimized out>) at arch/arch-ppc.h:88
>> #3 atb_clocktest (envp=<value optimized out>) at arch/arch-ppc.h:98
>> #4 arch_init (envp=<value optimized out>) at arch/arch-ppc.h:116
>> #5 initialize_fio (envp=<value optimized out>) at libfio.c:248
>> #6 0x0000000010067294 in main (argc=<value optimized out>, argv=<value
optimized out>,
>> envp=<value optimized out>) at fio.c:33
>>
>>
>> Apologize for late to reply.
> Alright, I'll have PPC default to the normal clock until we can improve
> and detect this properly.
Is there a new rpm for fio available with this fix?
With fio version fio-2.1.2, I am still getting the core dump.
Thanks,
Saritha
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: segmentation fault on power system
2013-10-10 6:30 segmentation fault on power system Saritha Vinod
@ 2013-10-11 16:28 ` Jens Axboe
0 siblings, 0 replies; 11+ messages in thread
From: Jens Axboe @ 2013-10-11 16:28 UTC (permalink / raw)
To: Saritha Vinod, fio
On 10/10/2013 12:30 AM, Saritha Vinod wrote:
>
> Hi Jens,
>
>>> Yes. If we set unlimit core dump, fio will generate a core with
>>> backtrace like this one:
>>>
>>> #0 mfspr (envp=<value optimized out>) at arch/arch-ppc.h:55
>>> 55 asm volatile("mfspr %0,%1": "=r" (val) : "K" (reg));
>>> Missing separate debuginfos, use: debuginfo-install
> glibc-2.12-1.107.el6.ppc64 libaio-0.3.107-10.el6.ppc64
>>> (gdb) bt
>>> #0 mfspr (envp=<value optimized out>) at arch/arch-ppc.h:55
>>> #1 get_cpu_clock (envp=<value optimized out>) at arch/arch-ppc.h:71
>>> #2 atb_child (envp=<value optimized out>) at arch/arch-ppc.h:88
>>> #3 atb_clocktest (envp=<value optimized out>) at arch/arch-ppc.h:98
>>> #4 arch_init (envp=<value optimized out>) at arch/arch-ppc.h:116
>>> #5 initialize_fio (envp=<value optimized out>) at libfio.c:248
>>> #6 0x0000000010067294 in main (argc=<value optimized out>, argv=<value
> optimized out>,
>>> envp=<value optimized out>) at fio.c:33
>>>
>>>
>>> Apologize for late to reply.
>
>> Alright, I'll have PPC default to the normal clock until we can improve
>> and detect this properly.
>
> Is there a new rpm for fio available with this fix?
> With fio version fio-2.1.2, I am still getting the core dump.
Use latest git of fio and it should work.
--
Jens Axboe
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2013-10-11 16:29 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-10-10 6:30 segmentation fault on power system Saritha Vinod
2013-10-11 16:28 ` Jens Axboe
-- strict thread matches above, loose matches on Subject: below --
2013-07-23 10:13 Han Pingtian
2013-07-23 10:23 ` Erwan Velu
2013-07-23 15:15 ` Jens Axboe
2013-07-24 2:10 ` Han Pingtian
2013-07-24 6:21 ` Han Pingtian
2013-07-25 14:34 ` Jens Axboe
2013-08-09 8:39 ` Han Pingtian
2013-08-09 15:02 ` Jens Axboe
2013-07-24 1:40 ` Han Pingtian
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox