[Qemu-devel] [Bug] Docs build fails at interop.rst

[Qemu-devel] [Bug] Docs build fails at interop.rst

[Aarushi Mehta]

https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw
running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31
(Rawhide)

uname - a
Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431
allows for the build to occur

Regards
Aarushi Mehta

Re: [Qemu-devel] [Bug] Docs build fails at interop.rst

[John Snow]

On 5/20/19 7:30 AM, Aarushi Mehta wrote:
> https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw
> running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31
> (Rawhide)
> 
> uname - a
> Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32
> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> 
> Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431
> allows for the build to occur
> 
> Regards
> Aarushi Mehta
> 
> 

Ah, dang. The blocks aren't strictly conforming json, but the version I
tested this under didn't seem to care. Your version is much newer. (I
was using 1.7 as provided by Fedora 29.)

For now, try reverting 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7 instead,
which should at least turn off the "warnings as errors" option, but I
don't think that reverting -n will turn off this warning.

I'll try to get ahold of this newer version and see if I can't fix it
more appropriately.

--js

[Qemu-devel] QMP Example Formatting in ReST, Sphinx, and Pygments (was: Re: [Bug] Docs build fails at interop.rst)

[John Snow]

On 5/20/19 12:37 PM, John Snow wrote:
> 
> 
> On 5/20/19 7:30 AM, Aarushi Mehta wrote:
>> https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw
>> running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31
>> (Rawhide)
>>
>> uname - a
>> Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32
>> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>>
>> Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431
>> allows for the build to occur
>>
>> Regards
>> Aarushi Mehta
>>
>>
> 
> Ah, dang. The blocks aren't strictly conforming json, but the version I
> tested this under didn't seem to care. Your version is much newer. (I
> was using 1.7 as provided by Fedora 29.)
> 
> For now, try reverting 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7 instead,
> which should at least turn off the "warnings as errors" option, but I
> don't think that reverting -n will turn off this warning.
> 
> I'll try to get ahold of this newer version and see if I can't fix it
> more appropriately.
> 
> --js
> 

...Sigh, okay.

So, I am still not actually sure what changed from pygments 2.2 and
sphinx 1.7 to pygments 2.4 and sphinx 2.0.1, but it appears as if Sphinx
by default always tries to do add a filter to the pygments lexer that
raises an error on highlighting failure, instead of the default behavior
which is to just highlight those errors in the output. There is no
option to Sphinx that I am aware of to retain this lexing behavior.
(Effectively, it's strict or nothing.)

This approach, apparently, is broken in Sphinx 1.7/Pygments 2.2, so the
build works with our malformed json.

There are a few options:

1. Update conf.py to ignore these warnings (and all future lexing
errors), and settle for the fact that there will be no QMP highlighting
wherever we use the directionality indicators ('->', '<-').

2. Update bitmaps.rst to remove the directionality indicators.

3. Update bitmaps.rst to format the QMP blocks as raw text instead of JSON.

4. Update bitmaps.rst to remove the "json" specification from the code
block. This will cause sphinx to "guess" the formatting, and the
pygments guesser will decide it's Python3.

This will parse well enough, but will mis-highlight 'true' and 'false'
which are not python keywords. This approach may break in the future if
the Python3 lexer is upgraded to be stricter (because '->' and '<-' are
still invalid), and leaves us at the mercy of both the guesser and the
lexer.

I'm not actually sure what I dislike the least; I think I dislike #1 the
most. #4 gets us most of what we want but is perhaps porcelain.

I suspect if we attempt to move more of our documentation to ReST and
Sphinx that we will need to answer for ourselves how we intend to
document QMP code flow examples.

--js

Re: [Qemu-devel] QMP Example Formatting in ReST, Sphinx, and Pygments (was: Re: [Bug] Docs build fails at interop.rst)

[Eduardo Habkost]

On Mon, May 20, 2019 at 05:25:28PM -0400, John Snow wrote:
> 
> 
> On 5/20/19 12:37 PM, John Snow wrote:
> > 
> > 
> > On 5/20/19 7:30 AM, Aarushi Mehta wrote:
> >> https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw
> >> running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31
> >> (Rawhide)
> >>
> >> uname - a
> >> Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32
> >> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> >>
> >> Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431
> >> allows for the build to occur
> >>
> >> Regards
> >> Aarushi Mehta
> >>
> >>
> > 
> > Ah, dang. The blocks aren't strictly conforming json, but the version I
> > tested this under didn't seem to care. Your version is much newer. (I
> > was using 1.7 as provided by Fedora 29.)
> > 
> > For now, try reverting 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7 instead,
> > which should at least turn off the "warnings as errors" option, but I
> > don't think that reverting -n will turn off this warning.
> > 
> > I'll try to get ahold of this newer version and see if I can't fix it
> > more appropriately.
> > 
> > --js
> > 
> 
> ...Sigh, okay.
> 
> So, I am still not actually sure what changed from pygments 2.2 and
> sphinx 1.7 to pygments 2.4 and sphinx 2.0.1, but it appears as if Sphinx
> by default always tries to do add a filter to the pygments lexer that
> raises an error on highlighting failure, instead of the default behavior
> which is to just highlight those errors in the output. There is no
> option to Sphinx that I am aware of to retain this lexing behavior.
> (Effectively, it's strict or nothing.)
> 
> This approach, apparently, is broken in Sphinx 1.7/Pygments 2.2, so the
> build works with our malformed json.
> 
> There are a few options:
> 
> 1. Update conf.py to ignore these warnings (and all future lexing
> errors), and settle for the fact that there will be no QMP highlighting
> wherever we use the directionality indicators ('->', '<-').
> 
> 2. Update bitmaps.rst to remove the directionality indicators.
> 
> 3. Update bitmaps.rst to format the QMP blocks as raw text instead of JSON.
> 
> 4. Update bitmaps.rst to remove the "json" specification from the code
> block. This will cause sphinx to "guess" the formatting, and the
> pygments guesser will decide it's Python3.
> 
> This will parse well enough, but will mis-highlight 'true' and 'false'
> which are not python keywords. This approach may break in the future if
> the Python3 lexer is upgraded to be stricter (because '->' and '<-' are
> still invalid), and leaves us at the mercy of both the guesser and the
> lexer.
> 
> I'm not actually sure what I dislike the least; I think I dislike #1 the
> most. #4 gets us most of what we want but is perhaps porcelain.
> 
> I suspect if we attempt to move more of our documentation to ReST and
> Sphinx that we will need to answer for ourselves how we intend to
> document QMP code flow examples.

Writing a custom lexer that handles "<-" and "->" was simple (see below).

Now, is it possible to convince Sphinx to register and use a custom lexer?

$ cat > /tmp/lexer.py <<EOF
from pygments.lexer import RegexLexer, DelegatingLexer
from pygments.lexers.data import JsonLexer
import re
from pygments.token import *

class QMPExampleMarkersLexer(RegexLexer):
    tokens = {
        'root': [
            (r' *-> *', Generic.Prompt),
            (r' *<- *', Generic.Output),
        ]
    }

class QMPExampleLexer(DelegatingLexer):
    def __init__(self, **options):
        super(QMPExampleLexer, self).__init__(JsonLexer, QMPExampleMarkersLexer, Error, **options)
EOF
$ pygmentize -l /tmp/lexer.py:QMPExampleLexer -x -f html <<EOF
    -> {
         "execute": "drive-backup",
         "arguments": {
           "device": "drive0",
           "bitmap": "bitmap0",
           "target": "drive0.inc0.qcow2",
           "format": "qcow2",
           "sync": "incremental",
           "mode": "existing"
         }
       }

    <- { "return": {} }
EOF
<div class="highlight"><pre><span></span><span class="gp">    -&gt; </span><span class="p">{</span>
         <span class="nt">&quot;execute&quot;</span><span class="p">:</span> <span class="s2">&quot;drive-backup&quot;</span><span class="p">,</span>
         <span class="nt">&quot;arguments&quot;</span><span class="p">:</span> <span class="p">{</span>
           <span class="nt">&quot;device&quot;</span><span class="p">:</span> <span class="s2">&quot;drive0&quot;</span><span class="p">,</span>
           <span class="nt">&quot;bitmap&quot;</span><span class="p">:</span> <span class="s2">&quot;bitmap0&quot;</span><span class="p">,</span>
           <span class="nt">&quot;target&quot;</span><span class="p">:</span> <span class="s2">&quot;drive0.inc0.qcow2&quot;</span><span class="p">,</span>
           <span class="nt">&quot;format&quot;</span><span class="p">:</span> <span class="s2">&quot;qcow2&quot;</span><span class="p">,</span>
           <span class="nt">&quot;sync&quot;</span><span class="p">:</span> <span class="s2">&quot;incremental&quot;</span><span class="p">,</span>
           <span class="nt">&quot;mode&quot;</span><span class="p">:</span> <span class="s2">&quot;existing&quot;</span>
         <span class="p">}</span>
       <span class="p">}</span>

<span class="go">    &lt;- </span><span class="p">{</span> <span class="nt">&quot;return&quot;</span><span class="p">:</span> <span class="p">{}</span> <span class="p">}</span>
</pre></div>
$ 


-- 
Eduardo

Re: [Qemu-devel] QMP Example Formatting in ReST, Sphinx, and Pygments (was: Re: [Bug] Docs build fails at interop.rst)

[John Snow]

On 5/20/19 7:04 PM, Eduardo Habkost wrote:
> On Mon, May 20, 2019 at 05:25:28PM -0400, John Snow wrote:
>>
>>
>> On 5/20/19 12:37 PM, John Snow wrote:
>>>
>>>
>>> On 5/20/19 7:30 AM, Aarushi Mehta wrote:
>>>> https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw
>>>> running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31
>>>> (Rawhide)
>>>>
>>>> uname - a
>>>> Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32
>>>> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>> Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431
>>>> allows for the build to occur
>>>>
>>>> Regards
>>>> Aarushi Mehta
>>>>
>>>>
>>>
>>> Ah, dang. The blocks aren't strictly conforming json, but the version I
>>> tested this under didn't seem to care. Your version is much newer. (I
>>> was using 1.7 as provided by Fedora 29.)
>>>
>>> For now, try reverting 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7 instead,
>>> which should at least turn off the "warnings as errors" option, but I
>>> don't think that reverting -n will turn off this warning.
>>>
>>> I'll try to get ahold of this newer version and see if I can't fix it
>>> more appropriately.
>>>
>>> --js
>>>
>>
>> ...Sigh, okay.
>>
>> So, I am still not actually sure what changed from pygments 2.2 and
>> sphinx 1.7 to pygments 2.4 and sphinx 2.0.1, but it appears as if Sphinx
>> by default always tries to do add a filter to the pygments lexer that
>> raises an error on highlighting failure, instead of the default behavior
>> which is to just highlight those errors in the output. There is no
>> option to Sphinx that I am aware of to retain this lexing behavior.
>> (Effectively, it's strict or nothing.)
>>
>> This approach, apparently, is broken in Sphinx 1.7/Pygments 2.2, so the
>> build works with our malformed json.
>>
>> There are a few options:
>>
>> 1. Update conf.py to ignore these warnings (and all future lexing
>> errors), and settle for the fact that there will be no QMP highlighting
>> wherever we use the directionality indicators ('->', '<-').
>>
>> 2. Update bitmaps.rst to remove the directionality indicators.
>>
>> 3. Update bitmaps.rst to format the QMP blocks as raw text instead of JSON.
>>
>> 4. Update bitmaps.rst to remove the "json" specification from the code
>> block. This will cause sphinx to "guess" the formatting, and the
>> pygments guesser will decide it's Python3.
>>
>> This will parse well enough, but will mis-highlight 'true' and 'false'
>> which are not python keywords. This approach may break in the future if
>> the Python3 lexer is upgraded to be stricter (because '->' and '<-' are
>> still invalid), and leaves us at the mercy of both the guesser and the
>> lexer.
>>
>> I'm not actually sure what I dislike the least; I think I dislike #1 the
>> most. #4 gets us most of what we want but is perhaps porcelain.
>>
>> I suspect if we attempt to move more of our documentation to ReST and
>> Sphinx that we will need to answer for ourselves how we intend to
>> document QMP code flow examples.
> 
> Writing a custom lexer that handles "<-" and "->" was simple (see below).
> 
> Now, is it possible to convince Sphinx to register and use a custom lexer?
> 

Spoilers, yes, and I've sent a patch to list. Thanks for your help!