Re: Kernel docs: muddying the waters a bit

[Hans Verkuil <hverkuil@xs4all.nl>]

Hi Markus,

On 04/08/16 17:12, Markus Heiser wrote:
> Hi kernel-doc authors,
> 
> motivated by this MT, I implemented a toolchain to migrate the kernel’s 
> DocBook XML documentation to reST markup. 
> 
> It converts 99% of the docs well ... to gain an impression how 
> kernel-docs could benefit from, visit my sphkerneldoc project page
> on github:
> 
>   http://return42.github.io/sphkerneldoc/
> 
> The sources available at:
> 
>   https://github.com/return42/sphkerneldoc
> 
> The work is underway, suggestions are welcome!

I have to admit that this looks pretty good :-)

My main remark based on my quick scan through the docs is that anything in
typewriter font seems to be shown as red text with a rectangle around it.
That's quite jarring for me and I think it should be just shown as normal
text, just using a non-proportional font, just like in the original.

I also noticed that the 'title' of tables ends with a '¶' character for
some reason.

See e.g. the struct v4l2_audioout table in
http://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/vidioc-g-audioout.html

Regards,

	Hans

> 
> .. have a nice weekend ..
> 
> --M--
> 
> 
> Am 13.03.2016 um 16:33 schrieb Markus Heiser <markus.heiser@darmarIT.de>:
> 
>>
>> Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab <mchehab@osg.samsung.com>:
>>
>>> Em Thu, 10 Mar 2016 12:25:58 +0200
>>> Jani Nikula <jani.nikula@intel.com> escreveu:
>>>
>>>> TL;DR? Skip to the last paragraph.
>>>>
>>>> On Wed, 09 Mar 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
>>>>> I guess the conversion to asciidoc format is now in good shape,
>>>>> at least to demonstrate that it is possible to use this format for the
>>>>> media docbook. Still, there are lots of broken references.  
>>>>
>>>> Getting references right with asciidoc is a big problem in the
>>>> kernel-doc side. As I wrote before, the proofs of concept only worked
>>>> because everything was processed as one big file (via includes). The
>>>> Asciidoctor inter-document references won't help, because we won't know
>>>> the target document name while processing kernel-doc.
>>>
>>> I was able to produce chunked htmls here with:
>>>
>>> 	asciidoctor -b docbook45 media_api.adoc
>>> 	xmlto -o html-dir html media_api.xml
>>>
>>> The results are at:
>>> 	https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/
>>>
>>> But yeah, all references seem to be broken there. It could be due to some
>>> conversion issue (I didn't actually tried to check what's wrong there),
>>> but I think that there's something not ok with docbook45
>>> output for multi-part documents (on both AsciiDoc and Asciidoctor).
>>>
>>>> Sphinx is massively better at handling cross references for
>>>> kernel-doc. We can use domains (C language) and roles (e.g. functions,
>>>> types, etc.) for the references, which provide kind of
>>>> namespaces. Sphinx warns for referencing non-existing targets, but
>>>> doesn't generate broken links in the result like Asciidoctor does.
>>>>
>>>> For example, in the documentation for a function that has struct foo as
>>>> parameter or return type, a cross reference to struct foo is added
>>>> automagically, but only if documentation for struct foo actually
>>>> exists. In Asciidoctor, we would have to blindly generate the references
>>>> ourselves, and try to resolve broken links ourselves by somehow
>>>> post-processing the result.
>>>>
>>>>> Yet, from my side, if we're willing to get rid of DocBook, then
>>>>> Asciidoctor seems to be the *only* alternative so far to parse the
>>>>> complex media documents.  
>>>>
>>>> I think you mean, "get rid of DocBook as source format", not altogether?
>>>> I'm yet to be convinved we could rely on Asciidoctor's native formats.
>>>
>>> What I mean is that, right now, I see only two alternatives for the
>>> media uAPI documentation:
>>> 	1) keep using DocBook;
>>> 	2) AsciiDoc/Asciidoctor.
>>>
>>> Sphinx doesn't have what's needed to support the complexity of the
>>> media books, specially since cell span seems to be possible only
>>> by using asciiArt formats. Writing a big table using asciiArt is
>>> something that is a *real pain*. Also, as tested, if the table is
>>> too big, it fails to parse such asciiArt tables. So, while Sphinx
>>> doesn't have a decent way to describe tables, we can't use it.
>>
>>
>> Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc,
>> (mostly) you have more then one choice .. e.g. import csv tables .. 
>> but this should be discussed by example ...
>>
>>
>>> If it starts implementing it, then we can check if the other
>>> features used by the media documentation are also supported.
>>> Probably, multi-part books would be another pain with Sphinx.
>>> We have actually 4 books inside a common body. A few chapters
>>> (like book licensing, bibliography, error codes) are shared
>>> by all 4 documents.
>>>
>>> But, so far, I can't see any way to port media books without
>>> lots of lot of work to develop new features at the Sphinx code.
>>
>>
>> may I can help you ...
>>
>>
>>>> The toolchain gets faster, easier to debug and simplified a lot with
>>>> DocBook out of the equation completely. Sphinx itself is stable, widely
>>>> available, and well documented. IMO there's sufficient native output
>>>> format support. There are plenty of really nice extensions
>>>> available. There's a possibility of doing kernel-doc as an extension in
>>>> the future (either by calling current kernel-doc from the extension or
>>>> by rewriting it).
>>>
>>> Well, if we go to Sphinx for kernel-doc, that means that we'll need
>>> 2 different tools for the documentation:
>>> 	- Sphinx for kernel-doc
>>> 	- either DocBook or Asciidoctor/AsciiDoc for media.
>>>
>>> IMHO, this is the worse scenario, as we'll keep depending on
>>> DocBook plus requiring Sphinx, but it is up to Jon to decide.
>>>
>>
>> The migration of kernel-doc is a long term project, not a
>> one shot job. The scope of documents to migrate is not limited
>> to the files with DocBook markup in, most documents have not
>> a real markup.
>>
>> Please take a look at my thoughts and efforts about migration.
>>
>> * https://sphkerneldoc.readthedocs.org
>>
>> * https://github.com/return42/sphkerneldoc.git
>>
>> sphkerneldoc.git is a small project started this weekend, within
>> this project I show you, how migration could be done and
>> we can discuss concerns like "tables and cell-spans" by example. 
>>
>> Believe me, most concerns discussed in this thread are a leak of
>> knowledge. I'am working with sphinx-doc since 7 years, switched
>> over from DocBook (escaped from a 8 years lasting XML hell).
>> DocBook and sphinx-doc are complete different, so sphinx-doc
>> might feels odd in the first time, but if you have switched 
>> like me, you will never go back again.
>>
>>>> Dan keeps bringing up the active community in Asciidoctor, and how
>>>> they're fixing things up as we speak... which is great, but Sphinx is
>>>> here now, packaged and shipping in distros ready to use. It seems that
>>>> of the two, an Asciidoctor based toolchain is currently more in need of
>>>> hacking and extending to meet our needs. Which brings us to the
>>>> implementation language, Python vs. Ruby.
>>>>
>>>> I won't make the mistake of comparing the relative merits of the
>>>> languages, but I'll boldly claim the set of kernel developers who know
>>>> Python is likely larger than the set of kernel developers who know Ruby
>>>> [citation needed]. AFAICT there are no Ruby tools in the kernel tree,
>>>> but there is a bunch of Python. My own very limited and subjective
>>>> experience with other tools around the kernel is that Python is much
>>>> more popular than Ruby. So my claim here is that we're in a better
>>>> position to hack on Sphinx extensions ourselves than Asciidoctor.
>>>
>>> Sorry, but I don't buy it. Python is, IMHO, a mess: each new version
>>> is incompatible with the previous one, and requires the source to
>>> change, in order to use a newer version than the one used to write
>>> the code. So, when talking about Python, we're actually talking about
>>> several different dialects that don't talk well to each other.
>>
>> Sorry, you are complete wrong ... I'am 15 years python programmer,
>> shipped out huge projects with my customers ... we never have seen
>> these problems ... sorry ...
>>
>>
>>> I don't know about Ruby. So far, I don't have anything against (or in
>>> favor) of it. I bet most Kernel developers would actually prefer a
>>> toolchain in C. If such tool doesn't exist, anything else seems
>>> equally the same ;)
>>
>> Why we are talking about script languages? What needed is a 
>> authoring system, which is as near as possible to the developers,
>> which are the authors.
>>
>> Sphinx-Doc is a standard authoring-tool versioned, maintained 
>> and extended by thousands of developers ...
>>
>>
>>>> My conclusion is that Sphinx covers the vast majority of the needs of
>>>> our documentation producers and consumers, in an amazing way, out of the
>>>> box, better than Asciidoctor.
>>>>
>>>> Which brings us to the minority and the parts where Sphinx falls short,
>>>> media documentation in particular. It's complex documentation, with very
>>>> specific requirements on the output, especially that many things remain
>>>> exactly as they are now. It also feels like the target is more to have
>>>> standalone media documentation, and not so much to be aligned with and
>>>> be part of the rest of the kernel documentation.
>>>>
>>>> I want to question the need to have all kernel documentation use tools
>>>> that meet the strict requirements of the outlier, when there's a better
>>>> alternative for the vast majority of the documentation. Especially when
>>>> Asciidoctor isn't a ready solution for media documentation either.
>>>>
>>>> In summary, my proposal is to go with Sphinx, leave media docs as
>>>> DocBook for now, and see if and how they can be converted to
>>>> Sphinx/reStructuredText later on when we have everything else in
>>>> place. It's not the perfect outcome, but IMHO it's the best overall
>>>> choice.
>>>
>>> Well, this could be done. We don't have any good reason to move
>>> the media docs out of DocBook.
>>
>> Sorry but again wrong: you lost many of the authors which are 
>> frustrated by a XML markup and you lost many developers to improve
>> the toolchain, frustrated by a complicated DocBook-XML XSLT
>> toolchain with SGML markup from the middle of the last epoch.
>>
>>> On the contrary, this means an extra
>>> work. The only advantage is that it is a way simpler to write
>>> documentation with a markup language, but converting from the PoC
>>> to its integration at the Kernel tree still require lots of work,
>>> specially due to the cross-refs "magic" scripts that we have under
>>> Documentation/DocBook/media/Makefile.
>>
>> Yes, you are right, migration is a process not a one shot 
>> job, as I mentioned before. You are a great programmer, your 
>> documentation is also great, this invest should be preserved.
>> So lets take a try. It would be a honor for me to show 
>> you all theses steps by example on my repository (see above).
>>
>>> As I said, the only big drawback is to keep depending on two
>>> different tools for kernel-doc and for media documentation.
>>
>> -- Markus --
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-media" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>