Re: [PATCH 02/24] scripts/archive-source: speed up downloading subprojects

[Paolo Bonzini <pbonzini@redhat.com>]

On 9/22/25 10:08, Daniel P. Berrangé wrote:
> On Mon, Sep 22, 2025 at 10:59:11AM +0400, Marc-André Lureau wrote:
>> Hi
>>
>> On Fri, Sep 19, 2025 at 5:50 PM Daniel P. Berrangé <berrange@redhat.com>
>> wrote:
>>
>>> On Fri, Sep 19, 2025 at 05:32:56PM +0400, marcandre.lureau@redhat.com
>>> wrote:
>>>> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>>>>
>>>> Running meson on each subproject is quite slow.
>>>
>>> Why is this faster ?  Does it make meson do parallel downloads
>>> if you list then all at once ?
>>>
>>
>> It doesn't look like it does it in paralllel, yet it is x times faster...
>>
>> time sh -c 'for s in keycodemapdb libvfio-user berkeley-softfloat-3
>> berkeley-testfloat-3 anyhow-1-rs arbitrary-int-1-rs bilge-0.2-rs
>> bilge-impl-0.2-rs either-1-rs foreign-0.3-rs itertools-0.11-rs libc-0.2-rs
>> proc-macro2-1-rs proc-macro-error-1-rs proc-macro-error-attr-1-rs
>> quote-1-rs syn-2-rs unicode-ident-1-rs; do
>>    meson subprojects download $s >/dev/null
>> done'
>> sh -c   5.27s user 0.33s system 99% cpu 5.613 total
>>
>> time sh -c 'meson subprojects download keycodemapdb libvfio-user
>> berkeley-softfloat-3 berkeley-testfloat-3 anyhow-1-rs arbitrary-int-1-rs
>> bilge-0.2-rs bilge-impl-0.2-rs either-1-rs foreign-0.3-rs itertools-0.11-rs
>> libc-0.2-rs proc-macro2-1-rs proc-macro-error-1-rs
>> proc-macro-error-attr-1-rs quote-1-rs syn-2-rs unicode-ident-1-rs'
>> sh -c   0.32s user 0.01s system 99% cpu 0.339 total
> 
> Oh I bet that 0.32secs is the overhead of python importing all modules
> that meson uses. Running python 18 times, gives the 5 second overhead.
Oh, it actually is parallel!

     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
     executor = ThreadPoolExecutor(options.num_processes)
     ...
     for wrap in wraps:
         dirname = Path(source_dir, subproject_dir, 
wrap.directory).as_posix()
         runner = Runner(logger, r, wrap, dirname, options)
         task = loop.run_in_executor(executor, runner.run)
         tasks.append(task)
         task_names.append(wrap.name)
     results = loop.run_until_complete(asyncio.gather(*tasks))

and then Runner.run -> Runner.run_method -> Runner.download