Re: Possible changes to the project docs

["Mark Van De Vyver" <mark@taqtiqa.com>]

[-- Attachment #1: Type: text/plain, Size: 11158 bytes --]

On Thu, May 28, 2020 at 12:54 AM, Nicolas Dechesne wrote:

> 
> hi Mark,
> 
> thanks a lot for joining the discussion! This is very helpful and much
> appreciated.
> 
> On Thu, May 28, 2020 at 7:49 AM Mark Van De Vyver < mark@taqtiqa.com >
> wrote:
> 
>> Hmm, it seems the interwebs ate my prior message.   TLDR;
> 
> 
> I believe you replied privately to me, your message is not lost and is in
> good hands , well mine ;)
> 
> 
>> Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and Antora
>> 
>> * A switch from DocBook is a net benefit regardless of what you switch
>> too.
>> * Making docs more accessible to dev and user contributors is the key.
>> * I regard ascidoc/md/rst as near enough that they don't swing the
>> decision - in terms of being approachable to devs and users.
> 
> 
>> * A potential discriminator is l10n/i18n - on the face of it they all seem
>> to be imperfect (wip) and adequate.
> 
> 
> We don't have plans for that now, even if it might be a good thing for the
> project, I think we struggle enough with one language today.
> 
> 

Agreed.
Unless there is a group dedicated to a translation, it would only make sense to restrict the translation to some automated service - google translate at worse or some thing like Transifex. Transifex offer a freelicense to "Transifex for free to Open Source projects ( http://docs.transifex.com/faq/#what-is-considered-an-open-source-project ) that have no funding, revenue, and/or commercialization model".  Not clear if we qualify for that.
The question is whether the cost of a 'rough' translation is worth the benefit of new users who are technically profficient but whose English is not at a level to grok the docs fully.
Thinking here of the proverbial 14yo who has just got board/device xyz and stumbles upon yoctoproject.org.

> 
> 
> 
>> * Is there any other strategic benefit to be obtained?
> 
> The main goal is to ensure documentation is 'easier' to read and write,
> encouraging more contributors, while keeping the reputation of the quality
> of the project's documentation.
> 
> 
>> I think so.  Looking at https://antora.org/ this stands out:
>> *The ability to retrieve and aggregate all the content from different git
>> repositories.*
> 
> 
> I think there are 2 sides to the problem:
> 1. the format of the documentation: DocBook (XML) vs reST vs Asciidoc. I
> think it's well accepted that Markdown is 'not enough' for large tech
> documentation/book
> 
> 

MD is ruled out, to my mind, by the lack of multi-repo support;
However it is worth noting the scope of the syntax/functionality should be sufficient - is see here ( https://cloudflow.io/docs/current ) for a larger project and here ( https://jlenv.github.io/jlenv ) for my Julia port of rbenv (devs here might be familiar with the Python port of rbenv).
Hence, I would argue that, which ever of asciidoc or rst is used, we encourage users employ to a subset of the syntax to keep the docs simple and maintainable.
AFAICT we're not dealing Julia or Linux graphics code levels of complexity.  Happy to be corrected.

> 
> 2. the tools available to process the documentation (sphinx, asciidoctor,
> antora.. ). Antora is a toolset on top of Asciidoc.
> 
> The main reasons why we've started to explore/experiment with reST/Sphinx
> (as opposed to Asciidoc) are:
> 1. this is what Python uses, and our project is intimately linked to
> Python. Sphinx can be extended with plugins in Python, and we have very
> capable Python developers around here.
> 
> 

Developer productivity is key.
I'd argue constraining to tool X's functionality.
Again, I'm happy to be wrong, yet AFAICT there shouldn't be a level of complexity in this projects documentation requirements that exceeds these tools.

> 
> 2. we've closely studied what happened in the Linux kernel community, and
> their thought process when they moved away from Docbook a couple of years
> ago, which is nicely summarized in the following articles:
> https://lwn.net/Articles/671496/
> https://lwn.net/Articles/692704/
> https://lwn.net/Articles/692705/
> Even though the initial plan was to move from Docbook to Asciidoc, they
> ended up using reST/Sphinx in the end. Given the proximity of the YP and
> kernel communities, we should pay close attention to what they've done.
> 
> 

Agreed - I read that transition slightly differently: The kernel doc tooling was imperfect but generally tenable.  The Linux graphics code base complexity made the docs untenable, and forced an alternative to be sought. Once the graphics code base needs were satisfied it obviously made sense to standardize.  Just as Julia has use cases of that compexity.
Are we really dealing with that level of difficulty (as disctinct from very elborate/long chains of scripts/routines - each conceptually relatively straightword in isloation).

> 
> 
> That said, no decision has been made, Sphinx is just the first proof of
> concept which have been made. So now is definitely the right time to
> discuss all options!
> 
> 

Great. So I spent some time digging further (See Transifex note above)...

> 
> 
>> This provides the following headroom:
>> * Move documentation into the repo that holds the related code.  Bringing
>> documentation closer to devs who are more likely to write some preliminary
>> documentation and correct documentation errors/ambiguities.
> 
> I agree that the project documentation would benefit from some inline
> documentation. Not for everything, but maybe for classes, variables,
> functions, it would be easier to maintain if the documentation was
> 'closer' to the doc. The kernel documentation uses a mix of inline doc vs
> separate doc already (with sphinx).
> 

Agreed. Pretty much every tool can handle some level of rendering inline docs that should be adequate.
What Antora allows is doc sharing between repos, and AFAICT, Sphinx doesn't provide that headroom.

> 
> 
> 
> 
>> * Allow individual repo's/tools to evolve at their own pace - a release
>> that repo/tool X would like to make 'yesterday' can be made and everyone
>> can be confident the docs are still consistent (assumes that tool/repo-Y
>> does not make any statements about tool/repo-X in the Y-docs)
>> * Allow the project to treat a repo as a 1st class citizen even if it is
>> not under the openembedded.org ( http://openembedded.org ) or yoctoproject.org
>> ( http://yoctoproject.org )
> 
> I don't think there is such a concept of a 1st class citizen. The
> Openembedded ecosystem is made of a very large amount of layers, and only
> a minority of layers are hosted on oe.org ( http://oe.org ) or yp.org (
> http://yp.org ). I believe that all layers are treated equally, as soon as
> they are listed/added to the OE layer index maybe.
> 
> 

Yet such layers cannot present integrated documentation.
Lets take a concrete existing example:
http://git.yoctoproject.org/cgit/cgit.cgi/meta-amd/tree/?h=dunfell

Ideally there would be a way for that project to present their (currently) secondary documentation (assume the eight (8) md files are ported to which ever syntax or rst or asciidoc) integrated with the yocto/oe parts of their documentation.  Even if this is only for some group internal document portal or readthedocs or some such portal.
In fact on the dunfell branch the current DOCUMENTATION.md suggests a Antora " playbook ( https://docs.antora.org/antora/2.2/playbook/ ) " (untested):

site:
 title: Layer XYZ Documentation

content:
 sources:
 - url: .
   branches: dunfell
   start_path: documentation/introduction - url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/overview-manual - url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/ brief-yoctoprojectqs ( http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/brief-yoctoprojectqs?h=dunfell )
 - url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/ toaster-manual ( http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/brief-yoctoprojectqs?h=dunfell )
 - url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/ sdk-manual ( http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/brief-yoctoprojectqs?h=dunfell )
output: clean: true dir: public 
Take a second concrete example:
http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi

The proverbial 12-14yo with a board now has three sets of documentation to contend with (following the README.md):
1. https://meta-raspberrypi.readthedocs.io
2. Yocto/OE
3. raspberrypi.org

> 
> 
> 
>> * Corollary: Allow any repo outside of openembedded.org (
>> http://openembedded.org ) or yoctoproject.org ( http://yoctoproject.org ) to
>> incorporate whichever of the tools/repos they need (git cloning & forking
>> model allows/encourages that) and present documentation that is integrated
>> and targeted (Antora could - eventually - allow/encourage that).
> 
> 
> The YP documentation 'project' is about publishing documentation for what
> makes up the Yocto Project (OE-core, bitbake, ..).
> 

Agreed. I suppose I'm arguing the project specific elements of the documentation (i.e not reusable by/of interest to a repo/layer) could (ideally) be limited to page header, footer and brand image detail - everything else could be reusable by layers as they think best fits their userbase/audience.

> 
> The current Docbook structure is probably too complex (learning curve) to
> be adopted in other layers. So anything we could improve to make it
> simpler for other layers to benefit from doc infrastructure would be a
> nice bonus.
> 
> 
>> * Give the Yocto-project flexibility in how to organize itself going
>> forward - documentation is not an impediment.
>> 
>> Of course there is a learning curve and the docs need to be refactored
>> back to their home tool/repo.
>> To my mind there is a categorical distinction that can be made between the
>> DocBook alternatives.
> 
> 
> Can you provide more information about what Antora vs
> Asciidoc/asciidoctor?
> 
> 

Personally I've never encountered the need use asciidoc. Until now - I'll likely refactor the github.com/jlenv org documents so that each repo's documentation is pulled into a whole under jlenv.github.io/.
My understanding Antora builds on asciidoctor.js.
I'd argue for whichever tool is used: Project and layer docs should limit themselves to that tools built-in functionality - patching a doc tool detracts dev cycles.
I'd argue to limit documentation to what is effectively the markdown level of functionality - but layer/repo maintainers are best placed to make this call.

Kind regards
Mark

> 
> 
> 
>> The questions is whether any of the above conjectures are of value or
>> attractive?
>> 
>> HTH?
>> --
>> Kind regards
>> Mark
>> 
>> Mark Van de Vyver, B.Bus(Hons), PhD(Dist)
>> 
> 
> 

--
Kind regards
Mark

Mark Van de Vyver, B.Bus(Hons), PhD(Dist)

[-- Attachment #2: Type: text/html, Size: 17733 bytes --]