sphinx-build is really slow, any way to improve that?

sphinx-build is really slow, any way to improve that?

[Claudio Fontana]

Hi all,

when I build qemu, there is a lot of time spent at the end of the build where one cpu goes 100% on sphinx-build.

Is there some way to parallelize that? It seems it is the current bottleneck for rebuilds for me..

Thanks,

Claudio

Re: sphinx-build is really slow, any way to improve that?

[Peter Maydell]

On Mon, 5 Sept 2022 at 20:51, Claudio Fontana <cfontana@suse.de> wrote:
> when I build qemu, there is a lot of time spent at the end of the build where one cpu goes 100% on sphinx-build.
>
> Is there some way to parallelize that? It seems it is the current bottleneck for rebuilds for me..

It's a big fat python program, so I suspect not, but
maybe I'm wrong.

You can always configure --disable-docs if you don't care
about the docs and want to make builds faster.

thanks
-- PMM

Re: sphinx-build is really slow, any way to improve that?

[Daniel P. Berrangé]

On Mon, Sep 05, 2022 at 10:21:55PM +0100, Peter Maydell wrote:
> On Mon, 5 Sept 2022 at 20:51, Claudio Fontana <cfontana@suse.de> wrote:
> > when I build qemu, there is a lot of time spent at the end of the build where one cpu goes 100% on sphinx-build.
> >
> > Is there some way to parallelize that? It seems it is the current bottleneck for rebuilds for me..
> 
> It's a big fat python program, so I suspect not, but
> maybe I'm wrong.

It annoys me too and I've had a look at what it is doing in the past and
failed to find an obvious way to improve it. I fear this could be an
inherant limitation of the way we use sphinx to build the docs as a
complete manual, as compared to say treating each docs source file as
a distinct standalone web page.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: sphinx-build is really slow, any way to improve that?

[Markus Armbruster]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Mon, 5 Sept 2022 at 20:51, Claudio Fontana <cfontana@suse.de> wrote:
>> when I build qemu, there is a lot of time spent at the end of the build where one cpu goes 100% on sphinx-build.
>>
>> Is there some way to parallelize that? It seems it is the current bottleneck for rebuilds for me..
>
> It's a big fat python program, so I suspect not, but
> maybe I'm wrong.
>
> You can always configure --disable-docs if you don't care
> about the docs and want to make builds faster.

I care about the docs, but the impact on turnaround time is so bad I
stopped building the docs in the build trees I use for development, and
instead keep a separate tree that has docs enabled.  Tends to delay
diagnosis of doc markup errors, but that's the lesser evil for me.

We used to have a similar problem with generated C: touch the QAPI
schema, rebuild everything and its dog.  That was because everything and
its dog depended on the generated QAPI header.  C projects 101: putting
everything in a single header slows down rebuilds.

We solved this by splitting up the generated header, and updating
generated headers only when they actually change.

I have no idea whether Sphinx could do a similarly incremental rebuild.

Re: sphinx-build is really slow, any way to improve that?

[Peter Maydell]

On Tue, 6 Sept 2022 at 08:55, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Mon, Sep 05, 2022 at 10:21:55PM +0100, Peter Maydell wrote:
> > On Mon, 5 Sept 2022 at 20:51, Claudio Fontana <cfontana@suse.de> wrote:
> > > when I build qemu, there is a lot of time spent at the end of the build where one cpu goes 100% on sphinx-build.
> > >
> > > Is there some way to parallelize that? It seems it is the current bottleneck for rebuilds for me..
> >
> > It's a big fat python program, so I suspect not, but
> > maybe I'm wrong.
>
> It annoys me too and I've had a look at what it is doing in the past and
> failed to find an obvious way to improve it. I fear this could be an
> inherant limitation of the way we use sphinx to build the docs as a
> complete manual, as compared to say treating each docs source file as
> a distinct standalone web page.

IIRC sphinx really really wants to process the whole document tree
in one go. You can see this in the way that for example the
HTML build process creates HTML files for the top-level rst
files that are supposed to be only for the manpage -- it will
suck in and process everything, not just the files reachable
via whatever top level file you point it at.

-- PMM

Re: sphinx-build is really slow, any way to improve that?

[Daniel P. Berrangé]

On Tue, Sep 06, 2022 at 02:41:13PM +0100, Peter Maydell wrote:
> On Tue, 6 Sept 2022 at 08:55, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Mon, Sep 05, 2022 at 10:21:55PM +0100, Peter Maydell wrote:
> > > On Mon, 5 Sept 2022 at 20:51, Claudio Fontana <cfontana@suse.de> wrote:
> > > > when I build qemu, there is a lot of time spent at the end of the build where one cpu goes 100% on sphinx-build.
> > > >
> > > > Is there some way to parallelize that? It seems it is the current bottleneck for rebuilds for me..
> > >
> > > It's a big fat python program, so I suspect not, but
> > > maybe I'm wrong.
> >
> > It annoys me too and I've had a look at what it is doing in the past and
> > failed to find an obvious way to improve it. I fear this could be an
> > inherant limitation of the way we use sphinx to build the docs as a
> > complete manual, as compared to say treating each docs source file as
> > a distinct standalone web page.
> 
> IIRC sphinx really really wants to process the whole document tree
> in one go. You can see this in the way that for example the
> HTML build process creates HTML files for the top-level rst
> files that are supposed to be only for the manpage -- it will
> suck in and process everything, not just the files reachable
> via whatever top level file you point it at.

Yeah, thats why I think we're limited by what sphinx upstream can do
for us. They need to be able to parallelize stuff in their loading
and generation code.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: sphinx-build is really slow, any way to improve that?

[Markus Armbruster]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Tue, 6 Sept 2022 at 08:55, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>
>> On Mon, Sep 05, 2022 at 10:21:55PM +0100, Peter Maydell wrote:
>> > On Mon, 5 Sept 2022 at 20:51, Claudio Fontana <cfontana@suse.de> wrote:
>> > > when I build qemu, there is a lot of time spent at the end of the build where one cpu goes 100% on sphinx-build.
>> > >
>> > > Is there some way to parallelize that? It seems it is the current bottleneck for rebuilds for me..
>> >
>> > It's a big fat python program, so I suspect not, but
>> > maybe I'm wrong.
>>
>> It annoys me too and I've had a look at what it is doing in the past and
>> failed to find an obvious way to improve it. I fear this could be an
>> inherant limitation of the way we use sphinx to build the docs as a
>> complete manual, as compared to say treating each docs source file as
>> a distinct standalone web page.
>
> IIRC sphinx really really wants to process the whole document tree
> in one go. You can see this in the way that for example the
> HTML build process creates HTML files for the top-level rst
> files that are supposed to be only for the manpage -- it will
> suck in and process everything, not just the files reachable
> via whatever top level file you point it at.

How do other projects deal with the resulting slowness?

Could we avoid rerunning sphinx-build when documentation hasn't changed?