From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Elder Subject: Re: rbd map command hangs for 15 minutes during system start up Date: Tue, 18 Dec 2012 10:09:03 -0600 Message-ID: <50D0951F.6050106@inktank.com> References: <50B7C788.6040404@inktank.com> <50B93FCA.2060801@inktank.com> <50BC2DE6.6050307@inktank.com> <50C65AE5.2050704@inktank.com> <50C774FD.9030107@inktank.com> <50C78D09.5050403@inktank.com> <50CB57E2.10703@inktank.com> <50CB69E5.7080001@inktank.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: Received: from mail-ie0-f177.google.com ([209.85.223.177]:45971 "EHLO mail-ie0-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755081Ab2LRQJD (ORCPT ); Tue, 18 Dec 2012 11:09:03 -0500 Received: by mail-ie0-f177.google.com with SMTP id k13so1132158iea.8 for ; Tue, 18 Dec 2012 08:09:02 -0800 (PST) In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Nick Bartos Cc: Sage Weil , Gregory Farnum , Josh Durgin , Mandell Degerness , "ceph-devel@vger.kernel.org" On 12/17/2012 11:12 AM, Nick Bartos wrote: > Here's a log with the rbd debugging enabled: > > https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log > > On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: >> On 12/14/2012 10:53 AM, Nick Bartos wrote: >>> Yes I was only enabling debugging for libceph. I'm adding debugging >>> for rbd as well. I'll do a repro later today when a test cluster >>> opens up. >> >> Excellent, thank you. -Alex I looked through these debugging messages. Looking only at the rbd debugging, what I see seems to indicate that rbd is idle at the point the "hang" seems to start. This suggests that the hang is not due to rbd itself, but rather whatever it is that might be responsible for using the rbd image once it has been mapped. Is that possible? I don't know what process you have that is mapping the rbd image, and what is supposed to be the next thing it does. (I realize this may not make a lot of sense, given a patch in rdb seems to have caused the hang to begin occurring.) Also note that the debugging information available (i.e., the lines in the code that can output debugging information) may well be incomplete. So if you don't find anything it may be necessary to provide you with another update which might include more debugging. Anyway, could you provide a little more context about what is going on sort of *around* rbd when activity seems to stop? Thanks a lot. -Alex