* teuthology task waiting for machines (> 8h)
@ 2014-06-28 10:27 Loic Dachary
2014-06-28 15:47 ` Yuri Weinstein
2014-06-30 14:10 ` Zack Cerza
0 siblings, 2 replies; 6+ messages in thread
From: Loic Dachary @ 2014-06-28 10:27 UTC (permalink / raw)
To: Zack Cerza; +Cc: Ceph Development
[-- Attachment #1: Type: text/plain, Size: 928 bytes --]
Hi Zack,
http://pulpito.ceph.com/loic-2014-06-27_18:45:37-upgrade:firefly-x:stress-split-wip-8475-testing-basic-plana/329515/
seems to indicate that the tasks cannot obtain the machines it needs:
2014-06-27T17:55:19.072 INFO:teuthology.task.internal:Locking machines...
2014-06-27T17:55:19.110 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 5)...
2014-06-27T17:55:29.175 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 5)...
...
2014-06-28T03:22:13.745 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 0)...
2014-06-28T03:22:23.787 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 0)...
Is it something expected (for instance when tasks with a higher priorty take precedence) ? If it is then all that's needed is patience right ?
Cheers
--
Loïc Dachary, Artisan Logiciel Libre
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: teuthology task waiting for machines (> 8h)
2014-06-28 10:27 teuthology task waiting for machines (> 8h) Loic Dachary
@ 2014-06-28 15:47 ` Yuri Weinstein
2014-06-28 16:51 ` Loic Dachary
2014-06-30 14:11 ` Zack Cerza
2014-06-30 14:10 ` Zack Cerza
1 sibling, 2 replies; 6+ messages in thread
From: Yuri Weinstein @ 2014-06-28 15:47 UTC (permalink / raw)
To: Loic Dachary; +Cc: Zack Cerza, Ceph Development
Technically yes.
If queue is busy - patience is needed.
Assuming that there are no runs in the queue which are hung. Zack is
diligently looking and fixing to prevent hung tests. If we see runs
older then say one day, we kill them (altho 'teuthology-kill' is not
working for me today :( )
Another option to speed up run - use PRIO (for priority) when
scheduling it and/or use not plana machines as they are in high
demand.
Thx
YuriW
On Sat, Jun 28, 2014 at 3:27 AM, Loic Dachary <loic@dachary.org> wrote:
> Hi Zack,
>
> http://pulpito.ceph.com/loic-2014-06-27_18:45:37-upgrade:firefly-x:stress-split-wip-8475-testing-basic-plana/329515/
>
> seems to indicate that the tasks cannot obtain the machines it needs:
>
> 2014-06-27T17:55:19.072 INFO:teuthology.task.internal:Locking machines...
> 2014-06-27T17:55:19.110 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 5)...
> 2014-06-27T17:55:29.175 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 5)...
> ...
> 2014-06-28T03:22:13.745 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 0)...
> 2014-06-28T03:22:23.787 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 0)...
>
> Is it something expected (for instance when tasks with a higher priorty take precedence) ? If it is then all that's needed is patience right ?
>
> Cheers
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: teuthology task waiting for machines (> 8h)
2014-06-28 15:47 ` Yuri Weinstein
@ 2014-06-28 16:51 ` Loic Dachary
2014-06-30 14:11 ` Zack Cerza
1 sibling, 0 replies; 6+ messages in thread
From: Loic Dachary @ 2014-06-28 16:51 UTC (permalink / raw)
To: Yuri Weinstein; +Cc: Ceph Development
[-- Attachment #1: Type: text/plain, Size: 1860 bytes --]
On 28/06/2014 17:47, Yuri Weinstein wrote:
> Technically yes.
>
> If queue is busy - patience is needed.
>
> Assuming that there are no runs in the queue which are hung. Zack is
> diligently looking and fixing to prevent hung tests. If we see runs
> older then say one day, we kill them (altho 'teuthology-kill' is not
> working for me today :( )
Hi Yuri,
This is reassuring :-) Patience is easy during the week-ends.
> Another option to speed up run - use PRIO (for priority) when
> scheduling it and/or use not plana machines as they are in high
> demand.
Unless there is a specific reason to run rados / upgrade suites on plana machines, which other types of machine do you recommend ?
Cheers
>
> Thx
> YuriW
>
> On Sat, Jun 28, 2014 at 3:27 AM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Zack,
>>
>> http://pulpito.ceph.com/loic-2014-06-27_18:45:37-upgrade:firefly-x:stress-split-wip-8475-testing-basic-plana/329515/
>>
>> seems to indicate that the tasks cannot obtain the machines it needs:
>>
>> 2014-06-27T17:55:19.072 INFO:teuthology.task.internal:Locking machines...
>> 2014-06-27T17:55:19.110 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 5)...
>> 2014-06-27T17:55:29.175 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 5)...
>> ...
>> 2014-06-28T03:22:13.745 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 0)...
>> 2014-06-28T03:22:23.787 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 0)...
>>
>> Is it something expected (for instance when tasks with a higher priorty take precedence) ? If it is then all that's needed is patience right ?
>>
>> Cheers
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
>>
--
Loïc Dachary, Artisan Logiciel Libre
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: teuthology task waiting for machines (> 8h)
2014-06-28 15:47 ` Yuri Weinstein
2014-06-28 16:51 ` Loic Dachary
@ 2014-06-30 14:11 ` Zack Cerza
1 sibling, 0 replies; 6+ messages in thread
From: Zack Cerza @ 2014-06-30 14:11 UTC (permalink / raw)
To: Yuri Weinstein; +Cc: Loic Dachary, Ceph Development
On Sat, Jun 28, 2014 at 9:47 AM, Yuri Weinstein
<yuri.weinstein@inktank.com> wrote:
> altho 'teuthology-kill' is not
> working for me today :(
Uhoh, why not?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: teuthology task waiting for machines (> 8h)
2014-06-28 10:27 teuthology task waiting for machines (> 8h) Loic Dachary
2014-06-28 15:47 ` Yuri Weinstein
@ 2014-06-30 14:10 ` Zack Cerza
2014-06-30 15:10 ` Loic Dachary
1 sibling, 1 reply; 6+ messages in thread
From: Zack Cerza @ 2014-06-30 14:10 UTC (permalink / raw)
To: Loic Dachary; +Cc: Ceph Development
Hi Loic,
At this point I don't really have a way to look back in time to see
what was going on, but in the future when jobs are blocked waiting for
machines for unreasonable periods it's useful to know what is holding
them:
teuthology-lock --brief -a --machine-type plana | sort -k +4
Thanks,
Zack
On Sat, Jun 28, 2014 at 4:27 AM, Loic Dachary <loic@dachary.org> wrote:
> Hi Zack,
>
> http://pulpito.ceph.com/loic-2014-06-27_18:45:37-upgrade:firefly-x:stress-split-wip-8475-testing-basic-plana/329515/
>
> seems to indicate that the tasks cannot obtain the machines it needs:
>
> 2014-06-27T17:55:19.072 INFO:teuthology.task.internal:Locking machines...
> 2014-06-27T17:55:19.110 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 5)...
> 2014-06-27T17:55:29.175 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 5)...
> ...
> 2014-06-28T03:22:13.745 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 0)...
> 2014-06-28T03:22:23.787 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 0)...
>
> Is it something expected (for instance when tasks with a higher priorty take precedence) ? If it is then all that's needed is patience right ?
>
> Cheers
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: teuthology task waiting for machines (> 8h)
2014-06-30 14:10 ` Zack Cerza
@ 2014-06-30 15:10 ` Loic Dachary
0 siblings, 0 replies; 6+ messages in thread
From: Loic Dachary @ 2014-06-30 15:10 UTC (permalink / raw)
To: Zack Cerza; +Cc: Ceph Development
[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]
Hi Zack,
Thanks for the tip, I'll try it next time :-)
Cheers
On 30/06/2014 16:10, Zack Cerza wrote:
> Hi Loic,
>
> At this point I don't really have a way to look back in time to see
> what was going on, but in the future when jobs are blocked waiting for
> machines for unreasonable periods it's useful to know what is holding
> them:
>
> teuthology-lock --brief -a --machine-type plana | sort -k +4
>
> Thanks,
> Zack
>
> On Sat, Jun 28, 2014 at 4:27 AM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Zack,
>>
>> http://pulpito.ceph.com/loic-2014-06-27_18:45:37-upgrade:firefly-x:stress-split-wip-8475-testing-basic-plana/329515/
>>
>> seems to indicate that the tasks cannot obtain the machines it needs:
>>
>> 2014-06-27T17:55:19.072 INFO:teuthology.task.internal:Locking machines...
>> 2014-06-27T17:55:19.110 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 5)...
>> 2014-06-27T17:55:29.175 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 5)...
>> ...
>> 2014-06-28T03:22:13.745 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 0)...
>> 2014-06-28T03:22:23.787 INFO:teuthology.task.internal:waiting for more machines to be free (need 3 see 0)...
>>
>> Is it something expected (for instance when tasks with a higher priorty take precedence) ? If it is then all that's needed is patience right ?
>>
>> Cheers
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
>>
--
Loïc Dachary, Artisan Logiciel Libre
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-06-30 15:10 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-06-28 10:27 teuthology task waiting for machines (> 8h) Loic Dachary
2014-06-28 15:47 ` Yuri Weinstein
2014-06-28 16:51 ` Loic Dachary
2014-06-30 14:11 ` Zack Cerza
2014-06-30 14:10 ` Zack Cerza
2014-06-30 15:10 ` Loic Dachary
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.