[LTP] LTP: memcg_stress_test hanging whole box

[LTP] LTP: memcg_stress_test hanging whole box

[Nikola Ciprich]

[-- Attachment #1.1: Type: text/plain, Size: 1159 bytes --]

Hello,

I hope I won't be bothering with dumb question, but I wasn't able to find any solution to my problem...

I'm playing with LTP, and it works nice for me (few tests fail, but with obvious reasons, which I will fix,
so this is Ok), but I'm having problem with memcg_stress test. It always almost immediately gets host machine to
heavy swapping, and never finishes... when I had 2GB of swap, machine usually got into totally unusable state,
when I lowered swap to 512M, I'm at least able to kill test after some time...
I guess this is not expected behaviour, but my question is, what should I check to get this working properly?
I'm getting same results for latest x86_64 2.6.32.x and 3.0.x. Testing machine has 4 cores and 4GB of RAM.

Could somebody help me with that please?

with best regards

nikola ciprich

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #1.2: Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 368 bytes --]

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure 
contains a definitive record of customers, application performance, 
security threats, fraudulent activity, and more. Splunk takes this 
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d

[-- Attachment #3: Type: text/plain, Size: 155 bytes --]

_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] LTP: memcg_stress_test hanging whole box

[Shubham Goyal]

On Monday 28 November 2011 01:42 AM, Nikola Ciprich wrote:
> I hope I won't be bothering with dumb question, but I wasn't able to find any solution to my problem...
>
> I'm playing with LTP, and it works nice for me (few tests fail, but with obvious reasons, which I will fix,
> so this is Ok), but I'm having problem with memcg_stress test. It always almost immediately gets host machine to
> heavy swapping, and never finishes... when I had 2GB of swap, machine usually got into totally unusable state,
> when I lowered swap to 512M, I'm at least able to kill test after some time...
> I guess this is not expected behaviour, but my question is, what should I check to get this working properly?
> I'm getting same results for latest x86_64 2.6.32.x and 3.0.x. Testing machine has 4 cores and 4GB of RAM.
>
> Could somebody help me with that please?
>

Hi Nikola,

I have also seen some hangs with memcg_stress test on some of my x 
machines. But this is not consistent for me as the same set of tests 
works fine on one machine and hangs on other. Yes this is not expected 
behavior and ideally tests should complete. Can you please give a try 
after changing swap space to twice the size of RAM? Also can you paste 
here the the exact process which is hanging for you i.e. paste the 
output of 'ps -ef | grep ltp' during the system hang state.

Thanks,
Shubham


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure 
contains a definitive record of customers, application performance, 
security threats, fraudulent activity, and more. Splunk takes this 
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] LTP: memcg_stress_test hanging whole box

[Nikola Ciprich]

[-- Attachment #1.1: Type: text/plain, Size: 1445 bytes --]

> I have also seen some hangs with memcg_stress test on some of my x  
> machines. But this is not consistent for me as the same set of tests  
> works fine on one machine and hangs on other. Yes this is not expected  
> behavior and ideally tests should complete. Can you please give a try  
> after changing swap space to twice the size of RAM? Also can you paste  
> here the the exact process which is hanging for you i.e. paste the  
> output of 'ps -ef | grep ltp' during the system hang state.

Hello Shubham,

I tried setting swap to 8GB, but the result is the same... the box gets into state
that I can't login to it even after few hours, so hard reset is needed..
When I have top started, I see a lot of memcg_process_stress processes.
What is strange, when I start the test with swap disabled, memcg_process_stress processes
just seem to eat all physical memory and then just sleep. OOM killer doesn't kill anything,
but the box doesn't hang..
weird..
I'm wondering whether to report this to kernel list?
or do You have any idea on where to look?
cheers
nik

>
> Thanks,
> Shubham
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #1.2: Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 368 bytes --]

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure 
contains a definitive record of customers, application performance, 
security threats, fraudulent activity, and more. Splunk takes this 
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d

[-- Attachment #3: Type: text/plain, Size: 155 bytes --]

_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] LTP: memcg_stress_test hanging whole box

[Shubham Goyal]

On Wednesday 30 November 2011 03:33 AM, Nikola Ciprich wrote:
> Hello Shubham,
>
> I tried setting swap to 8GB, but the result is the same... the box gets into state
> that I can't login to it even after few hours, so hard reset is needed..
> When I have top started, I see a lot of memcg_process_stress processes.
> What is strange, when I start the test with swap disabled, memcg_process_stress processes
> just seem to eat all physical memory and then just sleep. OOM killer doesn't kill anything,
> but the box doesn't hang..
> weird..
> I'm wondering whether to report this to kernel list?
> or do You have any idea on where to look?
> cheers
> nik

Hi Nikola,

What I recently noticed was the controller test hang problem is coming 
with old versions of LTP. I tried with latest release of LTP on the same 
affected machine and I did not observed any test case or system hang. I 
hope you are using latest LTP downloaded from 'ltp.sourceforge.net'?

There might be a issue with oom that it is not able to kill the memory 
intensive process but you need to gather more information about the 
hanging process first. You might want to run only the hanging test i.e. 
memcg_stress and not complete LTP to narrow down the issue. Just check 
whether you are able to reproduce the issue when only memcg_stress test 
is running There are chances that test case itself has some issues.

Thanks,
Shubham



------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure 
contains a definitive record of customers, application performance, 
security threats, fraudulent activity, and more. Splunk takes this 
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] LTP: memcg_stress_test hanging whole box

[Nikola Ciprich]

[-- Attachment #1.1: Type: text/plain, Size: 1330 bytes --]

> Hi Nikola,
Hello Shubham,

>
> What I recently noticed was the controller test hang problem is coming  
> with old versions of LTP. I tried with latest release of LTP on the same  
> affected machine and I did not observed any test case or system hang. I  
> hope you are using latest LTP downloaded from 'ltp.sourceforge.net'?
yes, I'm using 20110915.

>
> There might be a issue with oom that it is not able to kill the memory  
> intensive process but you need to gather more information about the  
> hanging process first. You might want to run only the hanging test i.e.  
> memcg_stress and not complete LTP to narrow down the issue. Just check  
> whether you are able to reproduce the issue when only memcg_stress test  
> is running There are chances that test case itself has some issues.
yup, it's possible to reproduce just with this one test..
so do You want me to collect OOM traces and report here? or directly to
LKML?

thanks
n.


>
> Thanks,
> Shubham
>
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #1.2: Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 368 bytes --]

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure 
contains a definitive record of customers, application performance, 
security threats, fraudulent activity, and more. Splunk takes this 
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d

[-- Attachment #3: Type: text/plain, Size: 155 bytes --]

_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] LTP: memcg_stress_test hanging whole box

[Shubham Goyal]

Hi Nikola,

> yup, it's possible to reproduce just with this one test..
> so do You want me to collect OOM traces and report here? or directly to
> LKML?

I suggest you report it here as well as to LKML. I will try to have a 
look from test case perspective. Please clearly mention the test which 
is causing the issue, architecture/hardware and kernel details on which 
you are getting this issue.

Thanks,
Shubham


------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of 
discussion for anyone considering optimizing the pricing and packaging model 
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list