V4L docs and docbook

V4L docs and docbook

[Jonathan Corbet]

Hey, Mauro,

There's been a conversation going on that I keep meaning to bring you
into.  In short, there's a fair amount of interest in improving our
formatted kernel documentation, and, in particular, making it easier to
write; I'd like to be sure that work doesn't leave media behind.

Work pushed by Daniel Vetter and company has been aiming toward the
ability to use a lightweight markup language in the in-source kerneldoc
comments.  Initially Markdown was targeted; the most likely choice now
looks like ReStructuredText, though no decision has been made.  I've been
pushing for moving all of our formatted documentation to whatever markup
we use, leaving DocBook behind.  There are, I think, a lot of good
reasons to want to do that, including consistency between the template
files and the in-source comments, ease of authoring, and a less unwieldy
toolchain.

Various proof-of-concept patches have gone around showing that this idea
seems to be feasible.  The latest discussion is at:

	http://thread.gmane.org/gmane.linux.documentation/35773

The media community has a lot of investment in DocBook documentation.
Converting to another markup form is relatively easy, and it's something
I would be willing to help with when the time comes.  But it occurred to
me that I should probably ask what you all think of this.

There is no flag day here; there's no reason to rip out the current
DocBook-based build infrastructure as long as somebody's using it.  But
it would be nice to get rid of it eventually and work toward the creation
of a more integrated set of kernel documentation.

So...is this an idea that fills you with horror, or does it maybe have
some appeal?  Do you have any questions?

One other question I had for you was: which of the allegedly supported
output formats are important to you?

Thanks,

jon

Re: V4L docs and docbook

[Mauro Carvalho Chehab]

Hi Jon,

Em Wed, 17 Feb 2016 14:52:54 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> Hey, Mauro,
> 
> There's been a conversation going on that I keep meaning to bring you
> into.  In short, there's a fair amount of interest in improving our
> formatted kernel documentation, and, in particular, making it easier to
> write; I'd like to be sure that work doesn't leave media behind.
> 
> Work pushed by Daniel Vetter and company has been aiming toward the
> ability to use a lightweight markup language in the in-source kerneldoc
> comments.  Initially Markdown was targeted; the most likely choice now
> looks like ReStructuredText, though no decision has been made.  I've been
> pushing for moving all of our formatted documentation to whatever markup
> we use, leaving DocBook behind.  There are, I think, a lot of good
> reasons to want to do that, including consistency between the template
> files and the in-source comments, ease of authoring, and a less unwieldy
> toolchain.
> 
> Various proof-of-concept patches have gone around showing that this idea
> seems to be feasible.  The latest discussion is at:
> 
> 	http://thread.gmane.org/gmane.linux.documentation/35773
> 
> The media community has a lot of investment in DocBook documentation.
> Converting to another markup form is relatively easy, and it's something
> I would be willing to help with when the time comes.  But it occurred to
> me that I should probably ask what you all think of this.
> 
> There is no flag day here; there's no reason to rip out the current
> DocBook-based build infrastructure as long as somebody's using it.  But
> it would be nice to get rid of it eventually and work toward the creation
> of a more integrated set of kernel documentation.
> 
> So...is this an idea that fills you with horror, or does it maybe have
> some appeal?  Do you have any questions?

As you can see at:
	https://linuxtv.org/docs.php

We have 2 types of documentation for the Kernel part of the subsystem,
Both using DocBook:
- The uAPI documentation:
	https://linuxtv.org/downloads/v4l-dvb-apis
- The kAPI documentation:
	https://linuxtv.org/downloads/v4l-dvb-internals/device-drivers/mediadev.html

The kAPI uses kernel-doc. I believe it should be easy to convert and/or
add markup tags there to improve it. Actually, this is one of the things
we currently miss. So, anything to improve it will be very welcomed.

The uAPI is a different story. What we have there is a join of two
different documents:
- The V4L2 documentation, written directly in DocBook
- The DVB API documentation, written originally in LaTex and later
  migrated to DocBook.

Such documentation uses extensive usage of the DocBook features,
so, I think it won't be trivial to convert it.

In addition, we have some scripts embedded at the DocBook/media
Makefile that create cross-references between the public API
headers and the DocBook, warning us if some new ioctl, define,
enum, ... was created without the corresponding documentation.

While not perfect, those scripts help a lot for us to be sure that
no uAPI changes would reach upstream without the corresponding
documentation. They work by parsing the uapi/*.h files we use and
adding references to (almost) every data type there. As those
headers are included at the documentation as appendices, DocBook
generation with xmllint will check if all references exist and will
produce an alert if something is missing. I even run xmllint manually
to make it pedantic, using the script below:

	make cleanmediadocs
	make DOCBOOKS=media_api.xml htmldocs 2>&1 | grep -v "element.*: validity error : ID .* already defined"
	echo
	echo "Do some pedantic checks and generate DocBook/media/media_api.html without chunks"
	echo
	xmllint --noent --postvalid "$PWD/Documentation/DocBook/media_api.xml" >/tmp/x.xml 2>/dev/null
	xmllint --noent --postvalid --noout /tmp/x.xml
	xmlto html-nochunks -m ./Documentation/DocBook/stylesheet.xsl -o Documentation/DocBook/media Documentation/DocBook/media_api.

I run the above script every time a patch touches on one of the public
API headers, as part of my reviewing process.

I believe that porting it to whatever other documentation system we
decide would be painful and would require a lot of effort.

Also, as we're touching the documentation on almost all Kernel versions, 
such porting effort should happen quickly, as otherwise it would either
prevent us from adding new features at the subsystem, or we would need to
keep 2 copies of the documentation, and the ones porting it would have to
port later the new additions.

So, I guess we'll likely need to postpone converting the uAPI document
until we can find someone with time and knowledge to do it quicky.

> One other question I had for you was: which of the allegedly supported
> output formats are important to you?

The most important format is html, on both on multiple docs, like the one
hosted at linuxtv.org, or as a single big html file (with is nice to
quickly find something inside it).

We used to have pdf generation with LaTex and DocBook, but we lost with
the conversions, as several tables got truncated after some change.
So, we currently don't use pdf. This is something that I miss, personally,
as pdfs are easy to find things inside, specially if the documentation tool
add the proper indexes. So, having the capability of generating a nice pdf
without mangling with the tables would be a nice feature.

I guess man pages for the kABI would also be interesting, although I
don't think anyone on media is using it currently (but our kABI
documentation is young). I feel that man pages could be more interesting
for newbies as times goes by. So, it would be another nice to have.

Thanks,
Mauro

Re: V4L docs and docbook

[Russel Winder]

[-- Attachment #1: Type: text/plain, Size: 1952 bytes --]

On Wed, 2016-02-17 at 21:51 -0200, Mauro Carvalho Chehab wrote:
> […]
> 
> We have 2 types of documentation for the Kernel part of the
> subsystem,
> Both using DocBook:
> - The uAPI documentation:
> 	https://linuxtv.org/downloads/v4l-dvb-apis
> - The kAPI documentation:
> 	https://linuxtv.org/downloads/v4l-dvb-internals/device-drivers/
> mediadev.html
[…]

I may not be introducing new data here but…

Whilst ReStructuredText and Markdown are fairly popular text markup
languages, they are not related to the DocBook/XML toolchain.

Many people, especially authors of books etc. are not really willing to
write in DocBook/XML even though it is the re-purposable representation
of choice for most of the major publishers. This led to ASCIIDoc.

ASCIIDoc is a plain text markup language in the same way
ReStructuredText and Markdown are, but it's intention was always to be
a lightweight front end to DocBook/XML so as to allow authors to write
in a nice markup language but work with the DocBook/XML toolchain.

ASCIIDoc has gained quite a strong following. So much so that it now
has a life of its own separate from the DocBook/XML tool chain. There
is ASCIIDoctor which generates PDF, HTML,… from the source without
using DocBook/XML, yet the source can quite happily go through a
DocBook/XML toolchain as well.

Many of the open source projects I am involved with are now using
ASCIIDoctor as the documentation form. This has increased the number of
non-main-contributor contributions via pull requests. It is so much
easier to work with ASCIIDoc(tor) source than DocBook/XML source. 
 
-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: V4L docs and docbook

[Mauro Carvalho Chehab]

[-- Attachment #1: Type: text/plain, Size: 1884 bytes --]

Em Thu, 18 Feb 2016 08:17:00 +0000
Russel Winder <russel@winder.org.uk> escreveu:

> On Wed, 2016-02-17 at 21:51 -0200, Mauro Carvalho Chehab wrote:
> > […]
> > 
> > We have 2 types of documentation for the Kernel part of the
> > subsystem,
> > Both using DocBook:
> > - The uAPI documentation:
> > 	https://linuxtv.org/downloads/v4l-dvb-apis
> > - The kAPI documentation:
> > 	https://linuxtv.org/downloads/v4l-dvb-internals/device-drivers/
> > mediadev.html  
> […]
> 
> I may not be introducing new data here but…
> 
> Whilst ReStructuredText and Markdown are fairly popular text markup
> languages, they are not related to the DocBook/XML toolchain.
> 
> Many people, especially authors of books etc. are not really willing to
> write in DocBook/XML even though it is the re-purposable representation
> of choice for most of the major publishers. This led to ASCIIDoc.
> 
> ASCIIDoc is a plain text markup language in the same way
> ReStructuredText and Markdown are, but it's intention was always to be
> a lightweight front end to DocBook/XML so as to allow authors to write
> in a nice markup language but work with the DocBook/XML toolchain.
> 
> ASCIIDoc has gained quite a strong following. So much so that it now
> has a life of its own separate from the DocBook/XML tool chain. There
> is ASCIIDoctor which generates PDF, HTML,… from the source without
> using DocBook/XML, yet the source can quite happily go through a
> DocBook/XML toolchain as well.
> 
> Many of the open source projects I am involved with are now using
> ASCIIDoctor as the documentation form. This has increased the number of
> non-main-contributor contributions via pull requests. It is so much
> easier to work with ASCIIDoc(tor) source than DocBook/XML source. 

Are there any tools that would convert from DocBook to ASCIIDoc?

Thanks,
Mauro

[-- Attachment #2: Assinatura digital OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: V4L docs and docbook

[Mauro Carvalho Chehab]

[-- Attachment #1: Type: text/plain, Size: 9216 bytes --]

Em Thu, 18 Feb 2016 06:31:14 -0200
Mauro Carvalho Chehab <mchehab@infradead.org> escreveu:

> Em Thu, 18 Feb 2016 08:17:00 +0000
> Russel Winder <russel@winder.org.uk> escreveu:
> 
> > On Wed, 2016-02-17 at 21:51 -0200, Mauro Carvalho Chehab wrote:  
> > > […]
> > > 
> > > We have 2 types of documentation for the Kernel part of the
> > > subsystem,
> > > Both using DocBook:
> > > - The uAPI documentation:
> > > 	https://linuxtv.org/downloads/v4l-dvb-apis
> > > - The kAPI documentation:
> > > 	https://linuxtv.org/downloads/v4l-dvb-internals/device-drivers/
> > > mediadev.html    
> > […]
> > 
> > I may not be introducing new data here but…
> > 
> > Whilst ReStructuredText and Markdown are fairly popular text markup
> > languages, they are not related to the DocBook/XML toolchain.
> > 
> > Many people, especially authors of books etc. are not really willing to
> > write in DocBook/XML even though it is the re-purposable representation
> > of choice for most of the major publishers. This led to ASCIIDoc.
> > 
> > ASCIIDoc is a plain text markup language in the same way
> > ReStructuredText and Markdown are, but it's intention was always to be
> > a lightweight front end to DocBook/XML so as to allow authors to write
> > in a nice markup language but work with the DocBook/XML toolchain.
> > 
> > ASCIIDoc has gained quite a strong following. So much so that it now
> > has a life of its own separate from the DocBook/XML tool chain. There
> > is ASCIIDoctor which generates PDF, HTML,… from the source without
> > using DocBook/XML, yet the source can quite happily go through a
> > DocBook/XML toolchain as well.
> > 
> > Many of the open source projects I am involved with are now using
> > ASCIIDoctor as the documentation form. This has increased the number of
> > non-main-contributor contributions via pull requests. It is so much
> > easier to work with ASCIIDoc(tor) source than DocBook/XML source.   
> 
> Are there any tools that would convert from DocBook to ASCIIDoc?

Answering myself:

I found one tool at:
	https://github.com/oreillymedia/docbook2asciidoc

That seemed to work. I ran it with:
	$ make DOCBOOKS=media_api.xml htmldocs 2>&1 | grep -v "element.*: validity error : ID .* already defined"
	$ xmllint --noent --postvalid "$PWD/Documentation/DocBook/media_api.xml" >/tmp/x.xml 2>/dev/null
	$ java -jar saxon9he.jar -s /tmp/x.xml -o book.asciidoc d2a_docbook.xsl chunk-output=true

And it produced a series of documents, that I stored at:
	https://mchehab.fedorapeople.org/media-kabi-docs-test/

I noticed, however, that it kept some things using DocBook xml. Perhaps
because some things cannot be translated to markup (see appa.asciidoc)?

Also, converting them to HTML produced me some errors, but perhaps because
I don't know what I'm doing ;)

What I did was:
	for i in book-docinfo.xml *.asciidoc; do asciidoc $i; done

errors enclosed.

Yet, it seems there are some hope on using asciidoc for the kAPI
documentation.

Thanks,
Mauro

---

asciidoc: ERROR: book.asciidoc: line 9: only book doctypes can contain level 0 sections
asciidoc: ERROR: ch01.asciidoc: line 588: only book doctypes can contain level 0 sections
asciidoc: ERROR: ch02.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch03.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch03.asciidoc: line 370: illegal style name: capture example
asciidoc: ERROR: ch04.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch04.asciidoc: line 719: illegal style name: to do
asciidoc: ERROR: ch05.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch06.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch07.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch07.asciidoc: line 907: illegal style name: Solution?
asciidoc: ERROR: ch07.asciidoc: line 2208: illegal style name: to do - OSS/ALSA
asciidoc: ERROR: appa.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appa.asciidoc: line 7: missing style: [blockdef-pass]: appendix
asciidoc: ERROR: appb.asciidoc: line 2: section title not permitted in delimited block
asciidoc: WARNING: appb.asciidoc: line 7: missing style: [paradef-default]: appendix
asciidoc: ERROR: appc.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appc.asciidoc: line 7: missing style: [blockdef-pass]: appendix
asciidoc: ERROR: appd.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appd.asciidoc: line 7: missing style: [blockdef-pass]: appendix
asciidoc: ERROR: appe.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appe.asciidoc: line 5: missing style: [paradef-default]: appendix
asciidoc: ERROR: book.asciidoc: line 142: section title not permitted in delimited block
asciidoc: ERROR: book.asciidoc: line 148: section title not permitted in delimited block
asciidoc: ERROR: ch08.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch09.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch10.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch11.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch12.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch13.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: ch14.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: appf.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appf.asciidoc: line 7: missing style: [blockdef-pass]: appendix
asciidoc: ERROR: appg.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appg.asciidoc: line 7: missing style: [blockdef-pass]: appendix
asciidoc: ERROR: apph.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: apph.asciidoc: line 7: missing style: [blockdef-pass]: appendix
asciidoc: ERROR: appi.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appi.asciidoc: line 7: missing style: [blockdef-pass]: appendix
asciidoc: ERROR: appj.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appj.asciidoc: line 7: missing style: [blockdef-pass]: appendix
asciidoc: ERROR: appk.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appk.asciidoc: line 7: missing style: [blockdef-pass]: appendix
asciidoc: ERROR: book.asciidoc: line 193: section title not permitted in delimited block
asciidoc: ERROR: ch15.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: book.asciidoc: line 201: section title not permitted in delimited block
asciidoc: ERROR: ch16.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: appl.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appl.asciidoc: line 7: missing style: [blockdef-pass]: appendix
asciidoc: ERROR: ch17.asciidoc: line 2: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 3: section title not permitted in delimited block
asciidoc: WARNING: appm.asciidoc: line 6: missing style: [paradef-literal]: appendix
asciidoc: ERROR: appm.asciidoc: line 17: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 50: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 144: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 166: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 217: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 434: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 471: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 492: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 514: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 531: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 545: section title not permitted in delimited block
asciidoc: ERROR: appm.asciidoc: line 569: section title not permitted in delimited block
asciidoc: ERROR: ch04.asciidoc: line 2845: [blockdef-example] missing closing delimiter
asciidoc: ERROR: ch01.asciidoc: line 588: only book doctypes can contain level 0 sections
asciidoc: ERROR: ch01.asciidoc: line 4224: [blockdef-example] missing closing delimiter
asciidoc: ERROR: ch03.asciidoc: line 370: illegal style name: capture example
asciidoc: ERROR: ch04.asciidoc: line 719: illegal style name: to do
asciidoc: ERROR: ch07.asciidoc: line 907: illegal style name: Solution?
asciidoc: ERROR: ch07.asciidoc: line 2208: illegal style name: to do - OSS/ALSA



[-- Attachment #2: Assinatura digital OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: V4L docs and docbook

[Mauro Carvalho Chehab]

Em Thu, 18 Feb 2016 07:10:14 -0200
Mauro Carvalho Chehab <mchehab@infradead.org> escreveu:

> Em Thu, 18 Feb 2016 06:31:14 -0200
> Mauro Carvalho Chehab <mchehab@infradead.org> escreveu:
> 
> > Em Thu, 18 Feb 2016 08:17:00 +0000
> > Russel Winder <russel@winder.org.uk> escreveu:
> >   
> > > On Wed, 2016-02-17 at 21:51 -0200, Mauro Carvalho Chehab wrote:    
> > > > […]
> > > > 
> > > > We have 2 types of documentation for the Kernel part of the
> > > > subsystem,
> > > > Both using DocBook:
> > > > - The uAPI documentation:
> > > > 	https://linuxtv.org/downloads/v4l-dvb-apis
> > > > - The kAPI documentation:
> > > > 	https://linuxtv.org/downloads/v4l-dvb-internals/device-drivers/
> > > > mediadev.html      
> > > […]
> > > 
> > > I may not be introducing new data here but…
> > > 
> > > Whilst ReStructuredText and Markdown are fairly popular text markup
> > > languages, they are not related to the DocBook/XML toolchain.
> > > 
> > > Many people, especially authors of books etc. are not really willing to
> > > write in DocBook/XML even though it is the re-purposable representation
> > > of choice for most of the major publishers. This led to ASCIIDoc.
> > > 
> > > ASCIIDoc is a plain text markup language in the same way
> > > ReStructuredText and Markdown are, but it's intention was always to be
> > > a lightweight front end to DocBook/XML so as to allow authors to write
> > > in a nice markup language but work with the DocBook/XML toolchain.
> > > 
> > > ASCIIDoc has gained quite a strong following. So much so that it now
> > > has a life of its own separate from the DocBook/XML tool chain. There
> > > is ASCIIDoctor which generates PDF, HTML,… from the source without
> > > using DocBook/XML, yet the source can quite happily go through a
> > > DocBook/XML toolchain as well.
> > > 
> > > Many of the open source projects I am involved with are now using
> > > ASCIIDoctor as the documentation form. This has increased the number of
> > > non-main-contributor contributions via pull requests. It is so much
> > > easier to work with ASCIIDoc(tor) source than DocBook/XML source.     
> > 
> > Are there any tools that would convert from DocBook to ASCIIDoc?  
> 
> Answering myself:
> 
> I found one tool at:
> 	https://github.com/oreillymedia/docbook2asciidoc
> 
> That seemed to work. I ran it with:
> 	$ make DOCBOOKS=media_api.xml htmldocs 2>&1 | grep -v "element.*: validity error : ID .* already defined"
> 	$ xmllint --noent --postvalid "$PWD/Documentation/DocBook/media_api.xml" >/tmp/x.xml 2>/dev/null
> 	$ java -jar saxon9he.jar -s /tmp/x.xml -o book.asciidoc d2a_docbook.xsl chunk-output=true
> 
> And it produced a series of documents, that I stored at:
> 	https://mchehab.fedorapeople.org/media-kabi-docs-test/
> 
> I noticed, however, that it kept some things using DocBook xml. Perhaps
> because some things cannot be translated to markup (see appa.asciidoc)?
> 
> Also, converting them to HTML produced me some errors, but perhaps because
> I don't know what I'm doing ;)
> 
> What I did was:
> 	for i in book-docinfo.xml *.asciidoc; do asciidoc $i; done
> 
> errors enclosed.

Stupid me: it should be just:
	asciidoc book.asciidoc

Still, there are lots of broken things there, and lots of errors when
building it:
	https://mchehab.fedorapeople.org/media-kabi-docs-test/book.html

Ok, I would expect the need to handling some things manually, but
it worries that it broke the tables. For example, see
"Table 1. Control IDs" at https://mchehab.fedorapeople.org/media-kabi-docs-test/book.html

It was mapped as a 3 cols table, but this is how it should be,
instead:
	https://linuxtv.org/downloads/v4l-dvb-apis/control.html
	
As this table has actually 5 cols, because some controls have a list
of multiple values (see V4L2_CID_COLORFX for example).
-- 
Thanks,
Mauro

Re: V4L docs and docbook

[Russel Winder]

[-- Attachment #1: Type: text/plain, Size: 1678 bytes --]

On Thu, 2016-02-18 at 07:28 -0200, Mauro Carvalho Chehab wrote:
> Em Thu, 18 Feb 2016 07:10:14 -0200
> […]
> 
> Stupid me: it should be just:
> 	asciidoc book.asciidoc
> 
> Still, there are lots of broken things there, and lots of errors when
> building it:
> 	https://mchehab.fedorapeople.org/media-kabi-docs-test/book.html
> 
> Ok, I would expect the need to handling some things manually, but
> it worries that it broke the tables. For example, see
> "Table 1. Control IDs" at https://mchehab.fedorapeople.org/media-kabi
> -docs-test/book.html
> 
> It was mapped as a 3 cols table, but this is how it should be,
> instead:
> 	https://linuxtv.org/downloads/v4l-dvb-apis/control.html
> 	
> As this table has actually 5 cols, because some controls have a list
> of multiple values (see V4L2_CID_COLORFX for example).

I will not be able to do anything to help with any of this today, but I
can try and take a look tomorrow.

Certainly an automated translation will not do all the complicated
bits, that will need manual intervention. I may well be able to assist
on this. I think the transform from DocBook/XML is worth a bit of up
front effort now, so as to make everything so much easier for all
concerned in the future. Obviously I do not have data relating to this
project, just experience and anecdotal evidence from others.

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: V4L docs and docbook

[Mauro Carvalho Chehab]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

Em Fri, 19 Feb 2016 08:28:13 +0000
Russel Winder <russel@winder.org.uk> escreveu:

> On Thu, 2016-02-18 at 07:28 -0200, Mauro Carvalho Chehab wrote:
> > Em Thu, 18 Feb 2016 07:10:14 -0200
> > […]
> > 
> > Stupid me: it should be just:
> > 	asciidoc book.asciidoc
> > 
> > Still, there are lots of broken things there, and lots of errors when
> > building it:
> > 	https://mchehab.fedorapeople.org/media-kabi-docs-test/book.html
> > 
> > Ok, I would expect the need to handling some things manually, but
> > it worries that it broke the tables. For example, see
> > "Table 1. Control IDs" at https://mchehab.fedorapeople.org/media-kabi
> > -docs-test/book.html
> > 
> > It was mapped as a 3 cols table, but this is how it should be,
> > instead:
> > 	https://linuxtv.org/downloads/v4l-dvb-apis/control.html
> > 	
> > As this table has actually 5 cols, because some controls have a list
> > of multiple values (see V4L2_CID_COLORFX for example).  
> 
> I will not be able to do anything to help with any of this today, but I
> can try and take a look tomorrow.
> 
> Certainly an automated translation will not do all the complicated
> bits, that will need manual intervention.

True, although I would feel more comfortable if we can use a tool for
those nested tables, as there are a lot of them at the V4L2 part of
the media kAPI doc. It would probably worth to either write a script
or to patch an existing tool like pandoc to make it work with
nested tables.

> I may well be able to assist
> on this. I think the transform from DocBook/XML is worth a bit of up
> front effort now, so as to make everything so much easier for all
> concerned in the future. Obviously I do not have data relating to this
> project, just experience and anecdotal evidence from others.


Thanks!

-- 
Thanks,
Mauro

[-- Attachment #2: Assinatura digital OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: V4L docs and docbook

[Jani Nikula]

On Thu, 18 Feb 2016, Mauro Carvalho Chehab <mchehab@infradead.org> wrote:
> Are there any tools that would convert from DocBook to ASCIIDoc?

I used pandoc when I tested the asciidoc pipeline. Something along the
lines of this for filtering docbook in stdin to asciidoc in stdout:

pandoc --atx-headers -f docbook -t asciidoc


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: V4L docs and docbook

[Mauro Carvalho Chehab]

Em Thu, 18 Feb 2016 11:37:41 +0200
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Thu, 18 Feb 2016, Mauro Carvalho Chehab <mchehab@infradead.org> wrote:
> > Are there any tools that would convert from DocBook to ASCIIDoc?  
> 
> I used pandoc when I tested the asciidoc pipeline. Something along the
> lines of this for filtering docbook in stdin to asciidoc in stdout:
> 
> pandoc --atx-headers -f docbook -t asciidoc
>

Results are at:
	https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/

pandoc/asciidoc also broke the tables.


Thanks,
Mauro

Re: V4L docs and docbook

[Jani Nikula]

On Thu, 18 Feb 2016, Russel Winder <russel@winder.org.uk> wrote:
> On Wed, 2016-02-17 at 21:51 -0200, Mauro Carvalho Chehab wrote:
>> […]
>> 
>> We have 2 types of documentation for the Kernel part of the
>> subsystem,
>> Both using DocBook:
>> - The uAPI documentation:
>> 	https://linuxtv.org/downloads/v4l-dvb-apis
>> - The kAPI documentation:
>> 	https://linuxtv.org/downloads/v4l-dvb-internals/device-drivers/
>> mediadev.html
> […]
>
> I may not be introducing new data here but…
>
> Whilst ReStructuredText and Markdown are fairly popular text markup
> languages, they are not related to the DocBook/XML toolchain.
>
> Many people, especially authors of books etc. are not really willing to
> write in DocBook/XML even though it is the re-purposable representation
> of choice for most of the major publishers. This led to ASCIIDoc.
>
> ASCIIDoc is a plain text markup language in the same way
> ReStructuredText and Markdown are, but it's intention was always to be
> a lightweight front end to DocBook/XML so as to allow authors to write
> in a nice markup language but work with the DocBook/XML toolchain.

We have been looking at asciidoc too, so much so that there are draft
patches by Jon and me to build some of the documentation from asciidoc
source. Both are in the thread starting at [1]. But we really need to
consider reStructuredText too [2].

I think this thread is about figuring out how much we really depend on
DocBook. There is a lot of pain in dealing with DocBook, especially as a
source format, but also as an intermediate format. If we can produce the
end results (html, pdf, man, ...) from the source markup directly, it's
a win.

> ASCIIDoc has gained quite a strong following. So much so that it now
> has a life of its own separate from the DocBook/XML tool chain. There
> is ASCIIDoctor which generates PDF, HTML,… from the source without
> using DocBook/XML, yet the source can quite happily go through a
> DocBook/XML toolchain as well.

See some of the other threads. Asciidoc seems to be in maintenance
mode. Asciidoctor depends on a ruby environment which is not met with
enthusiasm. The HTML output can only be chunked (split to several pages)
via the DocBook output. Sphinx is active, doesn't add a lot of new
dependencies, and seems to be able to natively output all the end
results people have so far said they care about.

> Many of the open source projects I am involved with are now using
> ASCIIDoctor as the documentation form. This has increased the number of
> non-main-contributor contributions via pull requests. It is so much
> easier to work with ASCIIDoc(tor) source than DocBook/XML source. 

I'm hopeful this holds for any of the lightweight markups.


BR,
Jani.


[1] http://lkml.kernel.org/r/1453764522-29030-1-git-send-email-corbet@lwn.net
[2] http://lkml.kernel.org/r/20160213145317.247c63c7@lwn.net

-- 
Jani Nikula, Intel Open Source Technology Center

Re: V4L docs and docbook

[Hans Verkuil]

Hi Jon,

On 02/17/2016 10:52 PM, Jonathan Corbet wrote:
> Hey, Mauro,
> 
> There's been a conversation going on that I keep meaning to bring you
> into.  In short, there's a fair amount of interest in improving our
> formatted kernel documentation, and, in particular, making it easier to
> write; I'd like to be sure that work doesn't leave media behind.
> 
> Work pushed by Daniel Vetter and company has been aiming toward the
> ability to use a lightweight markup language in the in-source kerneldoc
> comments.  Initially Markdown was targeted; the most likely choice now
> looks like ReStructuredText, though no decision has been made.  I've been
> pushing for moving all of our formatted documentation to whatever markup
> we use, leaving DocBook behind.  There are, I think, a lot of good
> reasons to want to do that, including consistency between the template
> files and the in-source comments, ease of authoring, and a less unwieldy
> toolchain.

I looked at ReStructuredText and it looks like it will be a pain to convert
the media DocBook code to that, and the main reason is the poor table support.
The syntax for that looks very painful and the media DocBook is full of tables.

BTW, my daily build scripts also rebuilds the media spec and it is available
here: https://hverkuil.home.xs4all.nl/spec/media.html

Also missing in ReStructuredText seems to be support for formulas (see for
example the Colorspaces section in the spec), although to be fair standard
DocBook doesn't do a great job at that either.

Now, I hate DocBook so going to something easier would certainly be nice,
but I think it is going to be a difficult task.

Someone would have to prove that going to another formatting tool will
produce good results for our documentation. We can certainly give a few
representative sections of our doc to someone to convert, and if that
looks OK, then the full conversion can be done.

We have (and still are) put a lot of effort into our documentation and
we would like to keep the same level of quality.

Regards,

	Hans

> 
> Various proof-of-concept patches have gone around showing that this idea
> seems to be feasible.  The latest discussion is at:
> 
> 	http://thread.gmane.org/gmane.linux.documentation/35773
> 
> The media community has a lot of investment in DocBook documentation.
> Converting to another markup form is relatively easy, and it's something
> I would be willing to help with when the time comes.  But it occurred to
> me that I should probably ask what you all think of this.
> 
> There is no flag day here; there's no reason to rip out the current
> DocBook-based build infrastructure as long as somebody's using it.  But
> it would be nice to get rid of it eventually and work toward the creation
> of a more integrated set of kernel documentation.
> 
> So...is this an idea that fills you with horror, or does it maybe have
> some appeal?  Do you have any questions?
> 
> One other question I had for you was: which of the allegedly supported
> output formats are important to you?
> 
> Thanks,
> 
> jon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: V4L docs and docbook

[Jani Nikula]

On Thu, 18 Feb 2016, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> I looked at ReStructuredText and it looks like it will be a pain to convert
> the media DocBook code to that, and the main reason is the poor table support.
> The syntax for that looks very painful and the media DocBook is full of tables.

The table support seems to be one point in favor of asciidoc over
reStructuredText [citation needed].

> BTW, my daily build scripts also rebuilds the media spec and it is available
> here: https://hverkuil.home.xs4all.nl/spec/media.html
>
> Also missing in ReStructuredText seems to be support for formulas (see for
> example the Colorspaces section in the spec), although to be fair standard
> DocBook doesn't do a great job at that either.

This may be true for vanilla rst as supported by Python docutils, but
the Sphinx tool we're considering does support a lot of things through
extensions. The builtin extensions include support for rendering math
via PNG or javascript [1]. There's also support for embedded graphviz
[2] which may be of interest.

> Now, I hate DocBook so going to something easier would certainly be nice,
> but I think it is going to be a difficult task.
>
> Someone would have to prove that going to another formatting tool will
> produce good results for our documentation. We can certainly give a few
> representative sections of our doc to someone to convert, and if that
> looks OK, then the full conversion can be done.

It would be great to have you actively on board doing this yourself,
seeking the solutions, as you're the ones doing your documentation in
the end.

Speaking only for myself, I'd rather prove we can produce beautiful
documentation from lightweight markup for ourselves, and let others make
their own conclusions about switching over or sticking with DocBook.

> We have (and still are) put a lot of effort into our documentation and
> we would like to keep the same level of quality.

We are doing this because we (at least in the graphics community) also
put a lot of effort into documentation, and we would like to make it
*better*!

I believe switching to some lightweight markup will be helpful in
attracting more contributions to documentation.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: V4L docs and docbook

[Hans Verkuil]

On 02/18/16 11:19, Jani Nikula wrote:
> On Thu, 18 Feb 2016, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>> I looked at ReStructuredText and it looks like it will be a pain to convert
>> the media DocBook code to that, and the main reason is the poor table support.
>> The syntax for that looks very painful and the media DocBook is full of tables.
> 
> The table support seems to be one point in favor of asciidoc over
> reStructuredText [citation needed].
> 
>> BTW, my daily build scripts also rebuilds the media spec and it is available
>> here: https://hverkuil.home.xs4all.nl/spec/media.html
>>
>> Also missing in ReStructuredText seems to be support for formulas (see for
>> example the Colorspaces section in the spec), although to be fair standard
>> DocBook doesn't do a great job at that either.
> 
> This may be true for vanilla rst as supported by Python docutils, but
> the Sphinx tool we're considering does support a lot of things through
> extensions. The builtin extensions include support for rendering math
> via PNG or javascript [1]. There's also support for embedded graphviz
> [2] which may be of interest.
> 
>> Now, I hate DocBook so going to something easier would certainly be nice,
>> but I think it is going to be a difficult task.
>>
>> Someone would have to prove that going to another formatting tool will
>> produce good results for our documentation. We can certainly give a few
>> representative sections of our doc to someone to convert, and if that
>> looks OK, then the full conversion can be done.
> 
> It would be great to have you actively on board doing this yourself,
> seeking the solutions, as you're the ones doing your documentation in
> the end.
> 
> Speaking only for myself, I'd rather prove we can produce beautiful
> documentation from lightweight markup for ourselves, and let others make
> their own conclusions about switching over or sticking with DocBook.
> 
>> We have (and still are) put a lot of effort into our documentation and
>> we would like to keep the same level of quality.
> 
> We are doing this because we (at least in the graphics community) also
> put a lot of effort into documentation, and we would like to make it
> *better*!
> 
> I believe switching to some lightweight markup will be helpful in
> attracting more contributions to documentation.

Just to be clear: I really don't like DocBook at all, so something better and
easier would be very much appreciated.

But good table handling is a prerequisite for us since we rely heavily on that.

Regards,

	Hans

Re: V4L docs and docbook

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 5771 bytes --]

Hans Verkuil <hverkuil@xs4all.nl> writes:

> But good table handling is a prerequisite for us since we rely heavily on that.

I converted an asciidoc document that included some tables to sphinx via
docbook using pandoc; that seemed to generate workable results for me,
but my needs are pretty simple.

Asciidoc has more sophisticated table support, providing the ability to
align text within cells and paint different kinds of borders. Sphinx
provides for spanning rows and columns, and multi-line auto-wrapped cell
contents. The rst docs say there's an emacs mode that can help paint the
source format, but I haven't tried that yet.

In any case, here's some tables from the document I converted:

asciidoc via docbook:

http://altusmetrum.org/AltOS/doc/altusmetrum.html#_altus_metrum_hardware_specifications

sphinx:

http://keithp.com/~keithp/altusmetrum-sphinx/hardwarespecs.html#altus-metrum-hardware-specifications

While completely unconfigurable, rst tables do at least benefit from an
easier to read input syntax; asciidoc tables are about as readable in
source form as troff or latex...

asciidoc:

	.Altus Metrum Flight Computer Electronics
	[options="header"]
	|================================
	|Device | Barometer | Z-axis accel | GPS | 3D sensors | Storage | RF Output | Battery

	|TeleMetrum v1.0
	|MP3H6115 10km (33k')
	|MMA2202 50g
	|SkyTraq
	|-
	|1MB
	|10mW
	|3.7V

	|TeleMetrum v1.1
	|MP3H6115 10km (33k')
	|MMA2202 50g
	|SkyTraq
	|-
	|2MB
	|10mW
	|3.7V

	|TeleMetrum v1.2
	|MP3H6115 10km (33k')
	|ADXL78 70g
	|SkyTraq
	|-
	|2MB
	|10mW
	|3.7V

	|TeleMetrum v2.0
	|MS5607 30km (100k')
	|MMA6555 102g
	|uBlox Max-7Q
	|-
	|8MB
	|40mW
	|3.7V

	|TeleMini v1.0
	|MP3H6115 10km (33k')
	|-
	|-
	|-
	|5kB
	|10mW
	|3.7V

	|EasyMini v1.0
	|MS5607 30km (100k')
	|-
	|-
	|-
	|1MB
	|-
	|3.7-12V

	|TeleMega v1.0
	|MS5607 30km (100k')
	|MMA6555 102g
	|uBlox Max-7Q
	|MPU6000 HMC5883
	|8MB
	|40mW
	|3.7V

	|EasyMega v1.0
	|MS5607 30km (100k')
	|MMA6555 102g
	|-
	|MPU6000 HMC5883
	|8MB
	|-
	|3.7V

	|==============================

rst:

	+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
	| Device    | Barometer | Z-axis    | GPS       | 3D        | Storage   | RF Output | Battery   |
	|           |           | accel     |           | sensors   |           |           |           |
	+===========+===========+===========+===========+===========+===========+===========+===========+
	| TeleMetru | MP3H6115  | MMA2202   | SkyTraq   | -         | 1MB       | 10mW      | 3.7V      |
	| m         | 10km      | 50g       |           |           |           |           |           |
	| v1.0      | (33k')    |           |           |           |           |           |           |
	+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
	| TeleMetru | MP3H6115  | MMA2202   | SkyTraq   | -         | 2MB       | 10mW      | 3.7V      |
	| m         | 10km      | 50g       |           |           |           |           |           |
	| v1.1      | (33k')    |           |           |           |           |           |           |
	+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
	| TeleMetru | MP3H6115  | ADXL78    | SkyTraq   | -         | 2MB       | 10mW      | 3.7V      |
	| m         | 10km      | 70g       |           |           |           |           |           |
	| v1.2      | (33k')    |           |           |           |           |           |           |
	+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
	| TeleMetru | MS5607    | MMA6555   | uBlox     | -         | 8MB       | 40mW      | 3.7V      |
	| m         | 30km      | 102g      | Max-7Q    |           |           |           |           |
	| v2.0      | (100k')   |           |           |           |           |           |           |
	+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
	| TeleMini  | MP3H6115  | -         | -         | -         | 5kB       | 10mW      | 3.7V      |
	| v1.0      | 10km      |           |           |           |           |           |           |
	|           | (33k')    |           |           |           |           |           |           |
	+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
	| EasyMini  | MS5607    | -         | -         | -         | 1MB       | -         | 3.7-12V   |
	| v1.0      | 30km      |           |           |           |           |           |           |
	|           | (100k')   |           |           |           |           |           |           |
	+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
	| TeleMega  | MS5607    | MMA6555   | uBlox     | MPU6000   | 8MB       | 40mW      | 3.7V      |
	| v1.0      | 30km      | 102g      | Max-7Q    | HMC5883   |           |           |           |
	|           | (100k')   |           |           |           |           |           |           |
	+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
	| EasyMega  | MS5607    | MMA6555   | -         | MPU6000   | 8MB       | -         | 3.7V      |
	| v1.0      | 30km      | 102g      |           | HMC5883   |           |           |           |
	|           | (100k')   |           |           |           |           |           |           |
	+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+


-- 
-keith

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 810 bytes --]