From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60891) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UrWe9-0006Li-4J for qemu-devel@nongnu.org; Tue, 25 Jun 2013 13:03:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UrWe2-0007vJ-J3 for qemu-devel@nongnu.org; Tue, 25 Jun 2013 13:03:13 -0400 Received: from e7.ny.us.ibm.com ([32.97.182.137]:39820) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UrWe2-0007tt-Fb for qemu-devel@nongnu.org; Tue, 25 Jun 2013 13:03:06 -0400 Received: from /spool/local by e7.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 25 Jun 2013 13:03:05 -0400 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by d01dlp03.pok.ibm.com (Postfix) with ESMTP id 4FE95C9005C for ; Tue, 25 Jun 2013 13:03:03 -0400 (EDT) Received: from d03av05.boulder.ibm.com (d03av05.boulder.ibm.com [9.17.195.85]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r5PH32tI282954 for ; Tue, 25 Jun 2013 13:03:03 -0400 Received: from d03av05.boulder.ibm.com (loopback [127.0.0.1]) by d03av05.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r5PH2goK009307 for ; Tue, 25 Jun 2013 11:02:43 -0600 Message-ID: <51C9CD20.9010409@linux.vnet.ibm.com> Date: Tue, 25 Jun 2013 13:02:24 -0400 From: "Michael R. Hines" MIME-Version: 1.0 References: <1372125485-11795-1-git-send-email-mrhines@linux.vnet.ibm.com> <1372125485-11795-15-git-send-email-mrhines@linux.vnet.ibm.com> <8761x21pvx.fsf@elfo.elfo> <51C96D39.2020603@redhat.com> <51C99EC6.2050008@linux.vnet.ibm.com> <877ghiz34t.fsf@elfo.elfo> In-Reply-To: <877ghiz34t.fsf@elfo.elfo> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v11 14/15] rdma: introduce MIG_STATE_NONE and change MIG_STATE_SETUP state transition List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: quintela@redhat.com Cc: aliguori@us.ibm.com, qemu-devel@nongnu.org, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, Paolo Bonzini , chegu_vinod@hp.com, knoel@redhat.com On 06/25/2013 10:17 AM, Juan Quintela wrote: > "Michael R. Hines" wrote: >> On 06/25/2013 06:13 AM, Paolo Bonzini wrote: >>> Il 25/06/2013 11:49, Juan Quintela ha scritto: >>>> mrhines@linux.vnet.ibm.com wrote: >>>>> From: "Michael R. Hines" >>>>> >>>>> As described in the previous patch, until now, the MIG_STATE_SETUP >>>>> state was not really a 'formal' state. It has been used as a 'zero' state >>>>> (what we're calling 'NONE' here) and QEMU has been unconditionally transitioning >>>>> into this state when the QMP migration command was called. Instead we want to >>>>> introduce MIG_STATE_NONE, which is our starting state in the state machine, and >>>>> then immediately transition into the MIG_STATE_SETUP state when the QMP migrate >>>>> command is issued. >>>>> >>>>> In order to do this, we must delay the transition into MIG_STATE_ACTIVE until >>>>> later in the migration_thread(). This is done to be able to timestamp the amount of >>>>> time spent in the SETUP state for proper accounting to the user during >>>>> an RDMA migration. >>>>> >>>>> Furthermore, the management software, until now, has never been aware of the >>>>> existence of the SETUP state whatsoever. This must change, because, timing of this >>>>> state implies that the state actually exists. >>>>> >>>>> These two patches cannot be separated because the 'query_migrate' QMP >>>>> switch statement needs to know how to handle this new state transition. >>>>> >>>>> Tested-by: Michael R. Hines >>>>> Signed-off-by: Michael R. Hines >>>>> @@ -316,6 +321,7 @@ static void migrate_fd_cancel(MigrationState *s) >>>>> { >>>>> DPRINTF("cancelling migration\n"); >>>>> + migrate_set_state(s, MIG_STATE_SETUP, MIG_STATE_CANCELLED); >>>>> migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_CANCELLED); >>>>> } >>>> This chunk is wrong. >>>> >>>> we can call qme_migrate_cancel() at any point, and it is going to be >>>> called normally from MIG_STATE_ACTIVE. >>>> >>>> migrate_set_satet(s, s->state, MIG_STATE_CANCELLED) >>>> >>>> should do the trick. Or something like that, what do you think? >>> I don't like the three-arguments migrate_set_state, but I don't have any >>> better idea. >>> >>> With Juan's modification, it is fine (but not reviewed-by me :)). While >>> you resend, the first 13 patches of v10 can be merged (pull request). >>> You can then rebase the last three on top. >> So, I should send a v11 before Juan applies and generates the pull request? >> >> Did I understand correctly? >> >>> Michael, did you look at the "debugging/getting the protocol ready" mode >>> where all pages are unpinned? >>> >>> Paolo >>> >> >> We did not come up with a design on how such a mode would be >> exposed to the user. Implementing this in migration-rdma.c is >> just a few lines of code, but without a clear use case in this patch >> series, I've not chosen expose it to the user yet without some kind >> of agreement with you guys. >> >> Recommendations? > Dunno. I have tried to use the thing: > > I am using -incoming x-rdma:deus:4444 > > (qemu) char device redirected to /dev/pts/2 (label serial0) > librdmacm: Warning: couldn't read ABI version. > librdmacm: Warning: assuming: 4 > librdmacm: Fatal: unable to get RDMA device list > RDMA ERROR: could not create rdma event channel > -incoming x-rdma:deus:4444: RDMA ERROR: could not create rdma event channel > This means that your RDMA environment / IB configuration (outside of QEMU) is not configured properly. There are several "sanity checks" you can do to make sure it is working: Run: $ ibv_devices <= You should see a list of configured IB devices $ ibv_devinfo <= You should see "PORT_ACTIVE" in the output somewhere for each IB device $ ifconfig ib0 <= if using standard infiniband $ ifconfig eth0 <= if using RDMA over Converged ethernet Furthermore, there are several latency/throughput utilities that come along with the OpenFabrics software stack you can run to make sure your two hosts are communicating with each other: There's these commands: $ ucmatose [args...] $ ibv_rc_pingpong [args ..] $ rdma_server/rdma_client [args...] > I also used "0" instead of deus (the machine name). > > The only thing that I did appart from intsalling librdmacm-devel was: > > > # cat /etc/udev/rules.d/80-infinibad.rules > KERNEL="rdma_cm", NAME="infiniband/%k", MODE="0666" > > As per README. Something else I need to do to use the tcp > implementations? > > Later, Juan. > > > > >