[Xenomai] unmapping and remapping /dev/rtheap

[Xenomai] unmapping and remapping /dev/rtheap

[Adam Jacobvitz]

Hi, I need to take a checkpoint using criu of an application compiled using
xenomai's posix skin. The problem is that criu doesn't know what to do with
the /dev/rtheap mapping. Is there a way to either manually unmap
/dev/rtheap from the process and then remap it when restored, or is there a
way to compile my xenomai application without using rtheap?

Thanks!

Adam

Re: [Xenomai] unmapping and remapping /dev/rtheap

[Jan Kiszka]

On 2013-09-04 12:04, Adam Jacobvitz wrote:
> Hi, I need to take a checkpoint using criu of an application compiled using
> xenomai's posix skin. The problem is that criu doesn't know what to do with
> the /dev/rtheap mapping. Is there a way to either manually unmap
> /dev/rtheap from the process and then remap it when restored, or is there a
> way to compile my xenomai application without using rtheap?

Err, how does your CRIU version deal with all the Xenomai-internal
state? I guess you didn't patch the Xenomai core and skins to
export/import it, did you?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] unmapping and remapping /dev/rtheap

[Adam Jacobvitz]

Yeah...I suppose that's the fundamental issue. I'm brand new to xenomai and
I'm not really familiar to what state xenomai maintains for an application.

Do you know what state I would need to checkpoint and if its even possible
to checkpoint it?

Thanks,

Adam
On Sep 4, 2013 6:34 AM, "Jan Kiszka" <jan.kiszka@siemens.com> wrote:

> On 2013-09-04 12:04, Adam Jacobvitz wrote:
> > Hi, I need to take a checkpoint using criu of an application compiled
> using
> > xenomai's posix skin. The problem is that criu doesn't know what to do
> with
> > the /dev/rtheap mapping. Is there a way to either manually unmap
> > /dev/rtheap from the process and then remap it when restored, or is
> there a
> > way to compile my xenomai application without using rtheap?
>
> Err, how does your CRIU version deal with all the Xenomai-internal
> state? I guess you didn't patch the Xenomai core and skins to
> export/import it, did you?
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux
>

Re: [Xenomai] unmapping and remapping /dev/rtheap

[Jan Kiszka]

On 2013-09-04 12:39, Adam Jacobvitz wrote:
> Yeah...I suppose that's the fundamental issue. I'm brand new to xenomai and
> I'm not really familiar to what state xenomai maintains for an application.
> 
> Do you know what state I would need to checkpoint and if its even possible
> to checkpoint it?

There is a lot, starting with core thread objects, sync objects etc.,
then there are skin-specific extensions of those and also objects that
are shared between processes. But even if you export all this, Xenomai
wasn't designed with this use case in mind. So you may find many tricky
corner cases around checkpoint/restart - just like in Linux...

What overall use case are you aiming at with CRIU for your RT
process(es)? What states would your processes be in when
saving/restoring? And what Xenomai version do you consider for this, 2.6
or upcoming 3.0?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] unmapping and remapping /dev/rtheap

[Adam J]

I'm a researcher who's trying to use a linux system as a testbed for a 
new program sampling methodology. What I want to be able to do is take a 
checkpoint at a given point in a program and then run that checkpoint 
for a set amount of instructions repeatedly with deterministic run time. 
I am new to running programs deterministically on Linux.

Currently I am running my experiments on a vanilla 3.10.10 linux system. 
I have all user-space processes disabled except for the shell I'm using 
to launch jobs. I am running my process under test with SCHED_FIFO 99 
priority on a CPUSet with a single CPU and I have NUMA emulation setup 
to segregate a portion of memory for that CPU from the rest of the 
system. DVFS is disabled, and if there is an involuntary context switch 
the run is said to have failed. I'm also flushing the pagecache using 
drop_caches before restoring the program to get the system state as 
repeatable as possible.

My test programs are controlled using performance counters via PAPI and 
signaling. For example, if I want to run a program for 100M instructions 
I write 100,000,000 to a specific file and then send a signal to the 
program telling it to start. The program reads the file, runs for 100M 
instructions, produces the unhalted cycle count, and then pauses. I want 
this unhalted cycle count to be the same every time.

I have my experiments set up to be able to repeatedly warm-up and run a 
region of a program. For example, I have program A which I want to 
measure run time for 100M instructions starting at Instruction 300M in 
the program. To setup the experiment take a checkpoint at Instruction 
100M. When running the experiment I restore the checkpoint at 
Instruction 100M and then run the program for 200M instructions to 
warm-up the micro-architectural state. Then I run the program for 100M 
instructions and measure the run time.

The issue I'm having right now is that I'm getting non-deterministic run 
times when running samples and I'm not sure of the cause. The programs I 
am sampling are from SPEC2006 which are not real-time applications. I am 
also running the SPEC applications on a non-real time linux system. One 
hypothesis I have is because the system isn't real time there are system 
level effects causing the program region to run for a different amount 
of time for each execution. To test this hypothesis I wanted to use a 
real-time linux system such as Xenomai to see if running my program 
under Xenomai will eliminate the variability. The programs I'm using are 
still not designed for real-time linux, but since I'm only looking at a 
small region of the program I'm hoping compiling the program unmodified 
using Xenomai would be sufficient.

Another hypothesis we have that the issue is due to different memory 
mappings being used after every restore resulting in the program 
occupying different ways in the cache. I am not sure how to determine if 
this is the cause of the run time variability though.

To test my first hypothesis I am trying to take a checkpoint of my 
application using CRIU so I can compare the run time variability of the 
Xenomai compiled application to that of the standard compilation of the 
application. If you think though this is not a valid hypothesis, I am 
open to other ideas.

Thank you,

Adam



On 9/4/2013 6:53 AM, Jan Kiszka wrote:
> On 2013-09-04 12:39, Adam Jacobvitz wrote:
>> Yeah...I suppose that's the fundamental issue. I'm brand new to xenomai and
>> I'm not really familiar to what state xenomai maintains for an application.
>>
>> Do you know what state I would need to checkpoint and if its even possible
>> to checkpoint it?
> There is a lot, starting with core thread objects, sync objects etc.,
> then there are skin-specific extensions of those and also objects that
> are shared between processes. But even if you export all this, Xenomai
> wasn't designed with this use case in mind. So you may find many tricky
> corner cases around checkpoint/restart - just like in Linux...
>
> What overall use case are you aiming at with CRIU for your RT
> process(es)? What states would your processes be in when
> saving/restoring? And what Xenomai version do you consider for this, 2.6
> or upcoming 3.0?
>
> Jan
>

Re: [Xenomai] unmapping and remapping /dev/rtheap

[Gilles Chanteperdrix]

Adam J wrote:
> I'm a researcher who's trying to use a linux system as a testbed for a
> new program sampling methodology. What I want to be able to do is take a
> checkpoint at a given point in a program and then run that checkpoint
> for a set amount of instructions repeatedly with deterministic run time.
> I am new to running programs deterministically on Linux.
>
> Currently I am running my experiments on a vanilla 3.10.10 linux system.
> I have all user-space processes disabled except for the shell I'm using
> to launch jobs. I am running my process under test with SCHED_FIFO 99

Maybe I am stating the obvious, but have you tuned
/proc/sys/kernel/sched_rt_sched_rt_period_us
and
/proc/sys/kernel/sched_rt_runtime_us

-- 
                                              Gilles

Re: [Xenomai] unmapping and remapping /dev/rtheap

[Adam J]

Oops, yeah I forgot to mention that. Before running the experiments I do:

sysctl -w kernel.sched_rt_runtime_us=-1
sysctl -w kernel.sched_rt_period_us=1000000000

Adam

On 9/4/2013 8:03 AM, Gilles Chanteperdrix wrote:
> Adam J wrote:
>> I'm a researcher who's trying to use a linux system as a testbed for a
>> new program sampling methodology. What I want to be able to do is take a
>> checkpoint at a given point in a program and then run that checkpoint
>> for a set amount of instructions repeatedly with deterministic run time.
>> I am new to running programs deterministically on Linux.
>>
>> Currently I am running my experiments on a vanilla 3.10.10 linux system.
>> I have all user-space processes disabled except for the shell I'm using
>> to launch jobs. I am running my process under test with SCHED_FIFO 99
> Maybe I am stating the obvious, but have you tuned
> /proc/sys/kernel/sched_rt_sched_rt_period_us
> and
> /proc/sys/kernel/sched_rt_runtime_us
>