Write Error: No space left on device

Write Error: No space left on device

[Mukesh G]

Hi,
  I am using a vanilla build of the Linux kernel (2.6.26) where I have
enabled cpusets and cgroups. I was able to successfully create a "test"
cgroup. When I try to add any process to the tasks under "test" cgroup, I
get the error "Write Error: No space left on device".

Any help is appreciated...

Thanks in advance

Mukesh

Re: Write Error: No space left on device

[Paul Jackson]

Mukesh wrote:
> "Write Error: No space left on device".

Search for

     ENOSPC

on the (brand new) cpuset(7) man page, such as at:

     ftp://oss.sgi.com/projects/cpusets/download/cpuset.7

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj-sJ/iWh9BUns@public.gmane.org> 1.940.382.4214

Re: Write Error: No space left on device

[Mukesh G]

Hi Paul,
     How do I determine the no of memory nodes in shared systems (E.g: x86
based systems)? How do you define a memory node in the context of the shared
systems?

I figured that I need to increase the memory nodes but unable to do so...

Thanks

Mukesh


On Fri, Sep 5, 2008 at 6:30 PM, Paul Jackson <pj-sJ/iWh9BUns@public.gmane.org> wrote:

> Mukesh wrote:
> > "Write Error: No space left on device".
>
> Search for
>
>     ENOSPC
>
> on the (brand new) cpuset(7) man page, such as at:
>
>     ftp://oss.sgi.com/projects/cpusets/download/cpuset.7
>
> --
>                  I won't rest till it's the best ...
>                  Programmer, Linux Scalability
>                  Paul Jackson <pj-sJ/iWh9BUns@public.gmane.org> 1.940.382.4214
>

Re: [Devel] Re: Write Error: No space left on device

[Paul Menage]

Hi Mukesh,

You should either:

- mount with just the cgroup subsystems that you care about (e.g.
"-ocpu,memory")

- set the "cpuset.cpus" and "cpuset.mems" files in the child cgroup to
be equal to those of the parent (root) cgroup.

Paul

On Mon, Sep 15, 2008 at 4:55 AM, Mukesh G <mukgbv-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi Paul,
>     How do I determine the no of memory nodes in shared systems (E.g: x86
> based systems)? How do you define a memory node in the context of the shared
> systems?
>
> I figured that I need to increase the memory nodes but unable to do so...
>
> Thanks
>
> Mukesh
>
>
> On Fri, Sep 5, 2008 at 6:30 PM, Paul Jackson <pj-sJ/iWh9BUns@public.gmane.org> wrote:
>
>> Mukesh wrote:
>> > "Write Error: No space left on device".
>>
>> Search for
>>
>>     ENOSPC
>>
>> on the (brand new) cpuset(7) man page, such as at:
>>
>>     ftp://oss.sgi.com/projects/cpusets/download/cpuset.7
>>
>> --
>>                  I won't rest till it's the best ...
>>                  Programmer, Linux Scalability
>>                  Paul Jackson <pj-sJ/iWh9BUns@public.gmane.org> 1.940.382.4214
>>
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linux-foundation.org/mailman/listinfo/containers
>
> _______________________________________________
> Devel mailing list
> Devel-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org
> https://openvz.org/mailman/listinfo/devel
>

Re: [Devel] Re: Write Error: No space left on device

[Mukesh G]

Hi
 Thanks for the help... I got cpu, memory cgroups at least mounted... I got
a basic task (my shell) assigned to cpu cgroup, memory cgroup

mount -t cgroup -o cpu none /container/cpu
mount -t cgroup -o memory none /container/memory
mount -t cgroup -o cpuset none /container/cpuset

Each of the sub-directories have files that can be modified for a designated
purpose....

The questions that I am now trying to resolve

a)Definition of cpu.shares?
   the default value is 1024. What is the max? What is the relation b/w cpu
cores, freq of the cores etc...? What is 1 cpu share worth?
b) What is the relation between cpuset.cpu* values and cpu cgroup?
c) What is the relation between cpuset.mem* values and memory cgroup?

Thanks in advance

Mukesh

On Tue, Sep 16, 2008 at 5:56 AM, Paul Menage <menage-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:

> Hi Mukesh,
>
> You should either:
>
> - mount with just the cgroup subsystems that you care about (e.g.
> "-ocpu,memory")
>
> - set the "cpuset.cpus" and "cpuset.mems" files in the child cgroup to
> be equal to those of the parent (root) cgroup.
>
> Paul
>
> On Mon, Sep 15, 2008 at 4:55 AM, Mukesh G <mukgbv-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > Hi Paul,
> >     How do I determine the no of memory nodes in shared systems (E.g: x86
> > based systems)? How do you define a memory node in the context of the
> shared
> > systems?
> >
> > I figured that I need to increase the memory nodes but unable to do so...
> >
> > Thanks
> >
> > Mukesh
> >
> >
> > On Fri, Sep 5, 2008 at 6:30 PM, Paul Jackson <pj-sJ/iWh9BUns@public.gmane.org> wrote:
> >
> >> Mukesh wrote:
> >> > "Write Error: No space left on device".
> >>
> >> Search for
> >>
> >>     ENOSPC
> >>
> >> on the (brand new) cpuset(7) man page, such as at:
> >>
> >>     ftp://oss.sgi.com/projects/cpusets/download/cpuset.7
> >>
> >> --
> >>                  I won't rest till it's the best ...
> >>                  Programmer, Linux Scalability
> >>                  Paul Jackson <pj-sJ/iWh9BUns@public.gmane.org> 1.940.382.4214
> >>
> > _______________________________________________
> > Containers mailing list
> > Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> > https://lists.linux-foundation.org/mailman/listinfo/containers
> >
> > _______________________________________________
> > Devel mailing list
> > Devel-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org
> > https://openvz.org/mailman/listinfo/devel
> >
>

Re: [Devel] Re: Write Error: No space left on device

[Paul Menage]

On Wed, Sep 17, 2008 at 5:57 AM, Mukesh G <mukgbv-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> a)Definition of cpu.shares?
>    the default value is 1024. What is the max?

See Documentation/scheduler/sched-design-CFS.txt

> b) What is the relation between cpuset.cpu* values and cpu cgroup?

None.

> c) What is the relation between cpuset.mem* values and memory cgroup?

None.

Paul

Re: [Devel] Re: Write Error: No space left on device

[Mukesh G]

Hi,

I need your help in clarifying some doubts...

I have setup a container for cpu. Within the container with a cpu.shares of
1024 (default), which is made up of 2 cpu containers (C2048; cpu.shares
2048) and (C1024: cpu.shares 1024).

Enclosing a cpubusy script that keeps the cpu busy performing some dumb
processing. 2 instances of this script is executed on a dual core Intel x86
systems (PIDS: 21804, 21805)

#!/usr/bin/perl

$goal = 2181818;

while (TRUE) {
        for ($i=0; $i<=$goal; $i++) {
                $x = 0.000001;
                $y = sin($x);
                $y = $y + 0.00001;

        }

        next;
        $y = $y + 0.01;
}

By default, I have observed the running process becomes part of the tasks of
the main container system. The cpu's are 100% utilized, which is correct as
both the processes are in the same container (i.e. the parent cpu
container).

I now add a running cpubusy script process (PID: 21804) to the C2048
container.

Behavior 1: I noticed that the cpu utilization is still 100%, even though I
was expecting a change in the cpu utilization as there is some enforced
isolation. Am I reading this right?

I now add the second cpubusy script process (PID: 21805) to the C1024
container

Behavior 2: I noticed that the cpu utilization still at 100%. This time I
was expecting a change in cpu utilization. Not sure where I am going wrong?

The cpubusy script is inherently single threaded so created 2 additional
cpubusy scripts to simulate more processes (PID: 21872, 21873). Now, I see
the utilization change.

PID 21804: 66%
PID 21805: 50%
PID 21872: 50%
PID 21873: 33%

Behavior 3: I was expecting PIDs 21872 and 21873 as they belong to the
parent cpu container to have higher cpu utilization. The results are
surpursing as shown above.

Now, I add the process with PID: 21872 to container system C1024 and process
with PID: 21873

PID 21804: 66%
PID 21873: 66%
PID 21805: 33%
PID 21873: 33%

Behavior 4: This is the expected behavior.

Thanks

Mukesh

On Thu, Sep 18, 2008 at 6:10 AM, Paul Menage <menage-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:

> On Wed, Sep 17, 2008 at 5:57 AM, Mukesh G <mukgbv-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > a)Definition of cpu.shares?
> >    the default value is 1024. What is the max?
>
> See Documentation/scheduler/sched-design-CFS.txt
>
> > b) What is the relation between cpuset.cpu* values and cpu cgroup?
>
> None.
>
> > c) What is the relation between cpuset.mem* values and memory cgroup?
>
> None.
>
> Paul
>

Re: [Devel] Re: Write Error: No space left on device

[Dhaval Giani]

On Tue, Oct 28, 2008 at 12:58:16PM +0530, Mukesh G wrote:
> Hi,
> 
> I need your help in clarifying some doubts...
> 
> I have setup a container for cpu. Within the container with a cpu.shares of
> 1024 (default), which is made up of 2 cpu containers (C2048; cpu.shares
> 2048) and (C1024: cpu.shares 1024).
> 
> Enclosing a cpubusy script that keeps the cpu busy performing some dumb
> processing. 2 instances of this script is executed on a dual core Intel x86
> systems (PIDS: 21804, 21805)
> 
> #!/usr/bin/perl
> 
> $goal = 2181818;
> 
> while (TRUE) {
>         for ($i=0; $i<=$goal; $i++) {
>                 $x = 0.000001;
>                 $y = sin($x);
>                 $y = $y + 0.00001;
> 
>         }
> 
>         next;
>         $y = $y + 0.01;
> }
> 
> By default, I have observed the running process becomes part of the tasks of
> the main container system. The cpu's are 100% utilized, which is correct as
> both the processes are in the same container (i.e. the parent cpu
> container).
> 
> I now add a running cpubusy script process (PID: 21804) to the C2048
> container.
> 
> Behavior 1: I noticed that the cpu utilization is still 100%, even though I
> was expecting a change in the cpu utilization as there is some enforced
> isolation. Am I reading this right?
> 

Hi Mukesh,

Not really. You would not see any change in CPU utlization. The group
scheduler is a propotional scheduler, that is it divides the CPU
bandwidth as per the propotion of the cpu shares. Now, you have only 1
process running, so it will get all the bandwidth available.

> I now add the second cpubusy script process (PID: 21805) to the C1024
> container
> 
> Behavior 2: I noticed that the cpu utilization still at 100%. This time I
> was expecting a change in cpu utilization. Not sure where I am going wrong?
>

Right, I notice you are on a dual core machine. That means both the
processes will be on different CPUs, and so you get to see the 100%
utilization for both the processes. (since a process can utilize more
than one CPU )
 
> The cpubusy script is inherently single threaded so created 2 additional
> cpubusy scripts to simulate more processes (PID: 21872, 21873). Now, I see
> the utilization change.
> 
> PID 21804: 66%
> PID 21805: 50%
> PID 21872: 50%
> PID 21873: 33%
> 
> Behavior 3: I was expecting PIDs 21872 and 21873 as they belong to the
> parent cpu container to have higher cpu utilization. The results are
> surpursing as shown above.
> 

Not really. What happens is that the CPU bandwidth available to a
control group is divided between the tasks and task groups as per their
weight. Assuming that you were running just nice 0 tasks, I would expect
C2048:C1024:PID21872:PID21873 to share CPU bandwidth in the ratio
2:1:1:1. Since there are 4 tasks, they would be split 2 on each core,
with the division of bandwidth being as per their weights. (So ideally
it should have been 80%, 40%, 40%, 40%, but is what you see).

> Now, I add the process with PID: 21872 to container system C1024 and process
> with PID: 21873
> 
> PID 21804: 66%
> PID 21873: 66%
> PID 21805: 33%
> PID 21873: 33%
> 
> Behavior 4: This is the expected behavior.
> 

Yes, absolutely true :).

Hope that helped

Thanks,
-- 
regards,
Dhaval

Re: [Devel] Re: Write Error: No space left on device

[Mukesh G]

Hi Dhaval,
    Thanks for clarifying some doubts. Further to your mail and my
understanding, I can make some conclusions

1) Even though there is a hierarchical order for containers, the shares are
not allocated in terms of that order i.e. If the parent container has a
certain no of cpu.shares and more child containers. The parent and child
containers have the same priority.

2) For only 2 process running on a dual core system and If I have one
process assigned to C2048 container and another process assigned to C1024
container, I would expect the scheduler to allocate in the order of 2:1.
Behavior 2 indicates that allocation did not happen.

3) I ran the experiment having 2 instances of multi-threaded version of java
workload on the same system and noticed that 2:1 scenario did not repeat.
Hence, a multi threaded code does not benefit from this kind of a scheduler.

4) If you have more processes than the no of cores for the cpu shares to
work.

Thanks

Mukesh

On Tue, Oct 28, 2008 at 2:08 PM, Dhaval Giani <dhaval-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>wrote:

> On Tue, Oct 28, 2008 at 12:58:16PM +0530, Mukesh G wrote:
> > Hi,
> >
> > I need your help in clarifying some doubts...
> >
> > I have setup a container for cpu. Within the container with a cpu.shares
> of
> > 1024 (default), which is made up of 2 cpu containers (C2048; cpu.shares
> > 2048) and (C1024: cpu.shares 1024).
> >
> > Enclosing a cpubusy script that keeps the cpu busy performing some dumb
> > processing. 2 instances of this script is executed on a dual core Intel
> x86
> > systems (PIDS: 21804, 21805)
> >
> > #!/usr/bin/perl
> >
> > $goal = 2181818;
> >
> > while (TRUE) {
> >         for ($i=0; $i<=$goal; $i++) {
> >                 $x = 0.000001;
> >                 $y = sin($x);
> >                 $y = $y + 0.00001;
> >
> >         }
> >
> >         next;
> >         $y = $y + 0.01;
> > }
> >
> > By default, I have observed the running process becomes part of the tasks
> of
> > the main container system. The cpu's are 100% utilized, which is correct
> as
> > both the processes are in the same container (i.e. the parent cpu
> > container).
> >
> > I now add a running cpubusy script process (PID: 21804) to the C2048
> > container.
> >
> > Behavior 1: I noticed that the cpu utilization is still 100%, even though
> I
> > was expecting a change in the cpu utilization as there is some enforced
> > isolation. Am I reading this right?
> >
>
> Hi Mukesh,
>
> Not really. You would not see any change in CPU utlization. The group
> scheduler is a propotional scheduler, that is it divides the CPU
> bandwidth as per the propotion of the cpu shares. Now, you have only 1
> process running, so it will get all the bandwidth available.
>
> > I now add the second cpubusy script process (PID: 21805) to the C1024
> > container
> >
> > Behavior 2: I noticed that the cpu utilization still at 100%. This time I
> > was expecting a change in cpu utilization. Not sure where I am going
> wrong?
> >
>
> Right, I notice you are on a dual core machine. That means both the
> processes will be on different CPUs, and so you get to see the 100%
> utilization for both the processes. (since a process can utilize more
> than one CPU )
>
> > The cpubusy script is inherently single threaded so created 2 additional
> > cpubusy scripts to simulate more processes (PID: 21872, 21873). Now, I
> see
> > the utilization change.
> >
> > PID 21804: 66%
> > PID 21805: 50%
> > PID 21872: 50%
> > PID 21873: 33%
> >
> > Behavior 3: I was expecting PIDs 21872 and 21873 as they belong to the
> > parent cpu container to have higher cpu utilization. The results are
> > surpursing as shown above.
> >
>
> Not really. What happens is that the CPU bandwidth available to a
> control group is divided between the tasks and task groups as per their
> weight. Assuming that you were running just nice 0 tasks, I would expect
> C2048:C1024:PID21872:PID21873 to share CPU bandwidth in the ratio
> 2:1:1:1. Since there are 4 tasks, they would be split 2 on each core,
> with the division of bandwidth being as per their weights. (So ideally
> it should have been 80%, 40%, 40%, 40%, but is what you see).
>
> > Now, I add the process with PID: 21872 to container system C1024 and
> process
> > with PID: 21873
> >
> > PID 21804: 66%
> > PID 21873: 66%
> > PID 21805: 33%
> > PID 21873: 33%
> >
> > Behavior 4: This is the expected behavior.
> >
>
> Yes, absolutely true :).
>
> Hope that helped
>
> Thanks,
> --
> regards,
> Dhaval
>

Re: [Devel] Re: Write Error: No space left on device

[Dhaval Giani]

On Tue, Oct 28, 2008 at 03:37:52PM +0530, Mukesh G wrote:
> Hi Dhaval,
>     Thanks for clarifying some doubts. Further to your mail and my
> understanding, I can make some conclusions
> 
> 1) Even though there is a hierarchical order for containers, the shares are
> not allocated in terms of that order i.e. If the parent container has a
> certain no of cpu.shares and more child containers. The parent and child
> containers have the same priority.
> 

No, the hierarchy is always maintained for the group scheduler. The parent
group's shares are used to see how long/when it will get scheduled with
respect to other tasks/groups at that level. As an example, if you have
a hierarchy like this

/
|-A
| |-A1
| |-A2
|
|-B
|-t1
|-t2

Let A, B have shares 1024 and t1 and t2 be tasks at nice 0. With this
setting, A, B, t1 and t2 will get equal CPU bandwidth. The tasks and
groups in A will split bandwidth available to A in a similar fashion. So
the cpu shares is not some resource which is split but it is a value which
will decide in what proportion the CPU bandwidth will be shared. So, if
you have no tasks in /, then setting the shares of A and B to 10 is the
same as setting the shares of A and B to 1000 (not exactly in terms of
load balancing, but enough for the purpose of this discussion).

> 2) For only 2 process running on a dual core system and If I have one
> process assigned to C2048 container and another process assigned to C1024
> container, I would expect the scheduler to allocate in the order of 2:1.
> Behavior 2 indicates that allocation did not happen.
> 

Well, its not possible to run one thread on more than one core at the
sametime :-). So we do not see the 2:1 usage happen.

> 3) I ran the experiment having 2 instances of multi-threaded version of java
> workload on the same system and noticed that 2:1 scenario did not repeat.
> Hence, a multi threaded code does not benefit from this kind of a scheduler.
> 

I am not sure if I have understood your question. But I think the answer
to the previous question should help.

> 4) If you have more processes than the no of cores for the cpu shares to
> work.
> 

Well, the physical limit on how much CPU a thread can hog is 100%, so
yes, if you have lesser threads than number of CPUs, you don't expect
shares to  work as expected, and each thread will get 100% CPU time.

thanks,
-- 
regards,
Dhaval

Re: [Devel] Re: Write Error: No space left on device

[Mukesh G]

Hi Dhaval,
     Back to the drawing board for me...

I have expressed my example in a diagram to help me understand better.

Parent Container / (cpu.shares = 1024; Will get complete CPU bandwidth)
|
|
----------------------------------------------------------------------------
|                                |    |
Container C2048                          C1024     |
    ----------------        ---------------- ---------------
    |               |      |               | |       |
Processes   A1              A2             B1              B2         T1
     T2


A1:A2:B1:B2:T1:T2 = 2:2:1:1:1:1 (Is my understanding right? If so, then my
question on hierachial priority still stands especially for parent container
processes, Is this by design?)

A1:B1 or A1:T1 = 2:1

In a single threaded case,it makes sense that we cannot have more than 100%
CPu utilization. Hence if T1 can take up 100 but A1 can go upto 100 but not
beyond.

In a multi-threaded case, a process can take potentially than more 100% cpu
utilization (In my dual core case, both the cores) then in that case, I
would expect the above ordering to apply with A1 getting more cpu time than
B1 or T1.

Thanks

Mukesh


On Tue, Oct 28, 2008 at 5:42 PM, Dhaval Giani <dhaval-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>wrote:

> On Tue, Oct 28, 2008 at 03:37:52PM +0530, Mukesh G wrote:
> > Hi Dhaval,
> >     Thanks for clarifying some doubts. Further to your mail and my
> > understanding, I can make some conclusions
> >
> > 1) Even though there is a hierarchical order for containers, the shares
> are
> > not allocated in terms of that order i.e. If the parent container has a
> > certain no of cpu.shares and more child containers. The parent and child
> > containers have the same priority.
> >
>
> No, the hierarchy is always maintained for the group scheduler. The parent
> group's shares are used to see how long/when it will get scheduled with
> respect to other tasks/groups at that level. As an example, if you have
> a hierarchy like this
>
> /
> |-A
> | |-A1
> | |-A2
> |
> |-B
> |-t1
> |-t2
>
> Let A, B have shares 1024 and t1 and t2 be tasks at nice 0. With this
> setting, A, B, t1 and t2 will get equal CPU bandwidth. The tasks and
> groups in A will split bandwidth available to A in a similar fashion. So
> the cpu shares is not some resource which is split but it is a value which
> will decide in what proportion the CPU bandwidth will be shared. So, if
> you have no tasks in /, then setting the shares of A and B to 10 is the
> same as setting the shares of A and B to 1000 (not exactly in terms of
> load balancing, but enough for the purpose of this discussion).
>
> > 2) For only 2 process running on a dual core system and If I have one
> > process assigned to C2048 container and another process assigned to C1024
> > container, I would expect the scheduler to allocate in the order of 2:1.
> > Behavior 2 indicates that allocation did not happen.
> >
>
> Well, its not possible to run one thread on more than one core at the
> sametime :-). So we do not see the 2:1 usage happen.
>
> > 3) I ran the experiment having 2 instances of multi-threaded version of
> java
> > workload on the same system and noticed that 2:1 scenario did not repeat.
> > Hence, a multi threaded code does not benefit from this kind of a
> scheduler.
> >
>
> I am not sure if I have understood your question. But I think the answer
> to the previous question should help.
>
> > 4) If you have more processes than the no of cores for the cpu shares to
> > work.
> >
>
> Well, the physical limit on how much CPU a thread can hog is 100%, so
> yes, if you have lesser threads than number of CPUs, you don't expect
> shares to  work as expected, and each thread will get 100% CPU time.
>
> thanks,
> --
> regards,
> Dhaval
>