From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hirofumi Tsujimura Subject: measuring time of a domain creation with several device mode Date: Fri, 13 Apr 2007 22:07:04 +0900 Message-ID: <461F8078.9030709@jp.fujitsu.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-2022-JP Content-Transfer-Encoding: 7bit Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Hi all. About five month ago, there was a topic that a domain with many disks could not create because of a time-out of timer. I measured time of domain creation with VBD(file) and VBD(LVM partition) and block tap. A example of the measurement result is the following. A domain creation spends O(n^2) time. -----------------------------+------+-------+-------+--------- disk | 1 | 10 | 50 | 100 -------+---------------------+------+-------+-------+--------- time |VBD(file) | 1.35 | 10.37 | 66.23 | 205.19 [sec.] |VBD(LVM partition) | 1.18 | 7.52 | 53.65 | 175.52 |block tap | 1.17 | 4.43 | 19.30 | 35.43 -------+---------------------+------+-------+-------+--------- I think that a lock contention wastes the time because a jumbo lock is used on tools/examples/xen-backend.agent script. Is it possible to avoid using the lock? The measurement environment is as follows. - Architecture : x86_32 with VTx - Xen change set: xen-unstable changeset 14535 - CPU : intel core duo processor 1.8GHz - Memory : 1G Byte Details of the investigation are described as follows. On the day, I measured creation time up to 87 of disks with VBD(file). A example of the measurement result is the following. table1: use VBD(file) -----------------+------------------ number of disk | time [sec.] -----------------+------------------ 1 | 1.27 10 | 11.43 50 | 129.19 87 | 491.41 -----------------+------------------ On the next day, I measured the rest from 87 of disks. A example of the measurement result is the following. table2: use VBD(file) -----------------+------------------ number of disk | time [sec.] -----------------+------------------ 87 | 158.01 90 | 481.14 100 | 607.35 -----------------+------------------ I wondered why the time at 87 of disks was different by the table1 and the table2. And so, I watched any process with "top" command when creating a domain. Then many "xen-backend.agent" processes were executed. The number of processes was same as the number of devices defined by a config file. Firstly, I investigated a tools/examples/xen-backend.agent. ---------------- tools/examples/xen-backend.agent -------------------- #! /bin/bash PATH=/etc/xen/scripts:$PATH . /etc/xen/scripts/locking.sh claim_lock xenbus_hotplug_global case "$XENBUS_TYPE" in tap) /etc/xen/scripts/blktap "$ACTION" ;; vbd) /etc/xen/scripts/block "$ACTION" ... esac release_lock xenbus_hotplug_global ---------------------------------------------------------------------- The xen-backend.agent process tries to get a lock named "xenbus_hotplug_global". For example, if a domain A with 100 of disks is created and a domain B with 100 of disks is destroyed, 200 of the processes tries to get the lock. I thought that the creation time is related to the lock contention. By way of experiment, I created a domain after confirming that xen-backend.agent process do not exist, then I measured it. A example of the measurement result is the following. table3: use VBD(file) -----------------+------------------ number of disk | time [sec.] -----------------+------------------ 1 | 1.35 10 | 10.37 50 | 66.23 100 | 205.19 -----------------+------------------ The time is not so fast. Secondly, I investigated a tools/examples/block that is executed by the xen-backend.agent process. The block process searches a usable loop device from all loop devices to mount a file. At the same time, it confirms that the file is not used on other loop devices. I thought that the creation time is related to the searching and the checking. By way of experiment, I measured the time with VBD(LVM partition) and block tap.The LVM parition was created on a USB HDD. A example of the measurement result is the following. table4: use VBD(LVM partition) -----------------+------------------ number of disk | time [sec.] -----------------+------------------ 1 | 1.18 10 | 7.52 50 | 53.65 100 | 175.52 -----------------+------------------ table5: use block tap -----------------+------------------ number of disk | time [sec.] -----------------+------------------ 1 | 1.17 10 | 4.43 50 | 19.30 100 | 35.43 -----------------+------------------ Thanks. -- Hirofumi Tsujimura