From mboxrd@z Thu Jan 1 00:00:00 1970 From: John McCullough Subject: Re: blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13 Date: Wed, 14 Apr 2010 09:30:15 -0700 Message-ID: <4BC5ED97.9090608@cs.ucsd.edu> References: <4BC50A5A.1020501@cs.ucsd.edu> <1271216280.25413.1782.camel@agari.van.xensource.com> <4BC55531.7080005@cs.ucsd.edu> <1271233446.2449.59.camel@ramone.somacoma.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1271233446.2449.59.camel@ramone.somacoma.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Daniel Stodden Cc: Ian Pratt , "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org On 04/14/2010 01:24 AM, Daniel Stodden wrote: > On Wed, 2010-04-14 at 01:40 -0400, John McCullough wrote: > >> Daniel, >> >> That did the trick and got us up to 256, Thanks! >> >> Out of curiosity, what's standing in the way of more devices? >> > I must admit I never tried. Lack of maybe a couple sparse tables here > and there? > > >> We tried raising the MAX_*_DEVICES constants in these files to 512, but >> didn't have any luck: >> linux-2.6-pvops.git/drivers/xen/blktap/blktap.h >> tools/blktap2/include/blktaplib.h >> tools/blktap/lib/blktaplib.h >> >> (The error is now "vbd open failed: -6") >> > That would be an ENXIO, probably while trying to open the ring (can you > verify that with an strace -f?) > Looks like it is ENXIO: ... [pid 4154] open("/dev/xen/blktap-2/blktap256", O_RDWR [pid 4153] close(4) = 0 [pid 4153] fcntl(3, F_GETFL) = 0 (flags O_RDONLY) [pid 4153] brk(0) = 0xa1e000 [pid 4153] brk(0xa3f000) = 0xa3f000 [pid 4153] fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 [pid 4153] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6844b40000 [pid 4153] lseek(3, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) [pid 4153] read(3, [pid 4154] <... open resumed> ) = -1 ENXIO (No such device or address) [pid 4154] gettimeofday({1271262219, 398530}, NULL) = 0 [pid 4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 115, MSG_NOSIGNAL, NULL, 0) = 115 [pid 4154] close(3) = 0 [pid 4154] gettimeofday({1271262219, 398836}, NULL) = 0 [pid 4154] sendto(0, "<30>Apr 14 09:23:39 tapdisk2[415"..., 109, MSG_NOSIGNAL, NULL, 0) = 109 [pid 4154] gettimeofday({1271262219, 399161}, NULL) = 0 [pid 4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 86, MSG_NOSIGNAL, NULL, 0) = 86 [pid 4154] write(1, "-6: vbd open failed: -6\n", 24) = 24 ... (err, ioctl to control, close and exit) Thanks, John > Should indeed work for up to 2^20, or MAX_BLKTAP_DEVICEs. > > I don't think this fails in the ring code, we return ENODEV there (which > is a bug) and the kernel won't correct that code. > > Daniel > > >> I noticed an artificial limit of 26*26 in the tapdev naming scheme, but >> I didn't look very thoroughly. >> >> Thanks again, >> John >> >> >> Daniel Stodden wrote: >> >>> Hi. >>> >>> Please echo $((N * (32 * 11 + 50) + SOME_HEADROOM)) >>> to /proc/sys/fs/aio-max-nr. Or set it up in sysctl.conf. >>> >>> Where N is the number of devices you desire. >>> >>> As for the apparently missing big fat complaint you should have seen pop >>> in syslog, I'll keep it in mind for the next update. :} >>> >>> Cheers, >>> Daniel >>> >>> On Tue, 2010-04-13 at 20:20 -0400, John McCullough wrote: >>> >>> >>>> I have been working with a colleague to get a large number of small VMs >>>> running on a single system. We were hoping for at least 100, but we >>>> seem to be topping out around 81. Each VM has a disk image and a swap >>>> image. It seemed like we were hitting a blktap limit, so we tried >>>> bumping up the MAX macros in tool/blktap2 and the linux driver, with no >>>> change. (Though we haven't hit the theoretical 256 blktap devices yet). >>>> >>>> (Initially we were only able to get 64 VMs until we bumped >>>> CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs). >>>> >>>> To isolate the problem, I tried creating a large number of blktap >>>> devices in the dom0 with no guests running and I ran into the same >>>> ceiling (162 total devices). Commands to reproduce the problem follow: >>>> >>>> echo 9> /sys/class/blktap2/verbosity >>>> >>>> for x in `seq 0 163`; do >>>> if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1 >>>> seek=1M 2> /dev/null); then >>>> echo "Qemu fail on $x"; exit 1 >>>> fi >>>> if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then >>>> echo "blktap fail on $x"; exit 1 >>>> fi >>>> done >>>> >>>> The result: >>>> ... >>>> /dev/xen/blktap-2/tapdev159 >>>> /dev/xen/blktap-2/tapdev160 >>>> /dev/xen/blktap-2/tapdev161 >>>> /dev/xen/blktap-2/tapdev162 >>>> unrecognized child response >>>> blktap fail on 163 >>>> >>>> Dmesg output associated with 163: >>>> [ 1288.839978] blktap_sysfs_create: adding attributes for dev >>>> ffff88019e4d1e00 >>>> [ 1288.840947] blktap_sysfs_destroy >>>> >>>> (Output for the prior devices includes processing a request, and a >>>> blktap_device_finish_request) >>>> >>>> No related xm dmesg output. >>>> >>>> $ hg tip >>>> changeset: 21091:f28f1ee587c8 >>>> tag: tip >>>> user: Keir Fraser >>>> date: Wed Apr 07 12:38:28 2010 +0100 >>>> summary: Added signature for changeset 484179b2be5d >>>> >>>> $ uname -a >>>> Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 >>>> x86_64 GNU/Linux >>>> >>>> Has anyone had contrary experience? Does anyone know where the 162 max >>>> is coming from? >>>> >>>> Thanks, >>>> John >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >>>> >>>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >>> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >