[Cocci] Addition of source code documentation generation to the build system?

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

Hello,

I try to generate some documentation from the source files of your current software.

elfring at Sonne:~/Projekte/Coccinelle/lokal> MY_DIR=../Probe/Doku/ &&
MY_NAMES=${MY_DIR}file_name_list11.txt && find . \( -name '*.ml' -o -name
'*.mli' \) -a ! -name 'myocamlbuild.ml' -type f -fprint0 $MY_NAMES &&
ocamldoc.opt -d ${MY_DIR} -I commons -I commons/ocamlextra -I
commons/ocollection -I globals -I engine -I parsing_c -I parsing_cocci -I ctl -I
ocaml -I ocamlsexp -I python -I popl09 -I extra -I /usr/local/share/menhir -I
bundles/pcre -I bundles/pycaml -dot -o ${MY_DIR}my_graph.gv -html $(sed 's/\x0/
/g' $MY_NAMES)


I get a couple of error messages because my approach is incomplete so far. I
stumble on technical challenges here.
Would you like to suggest an improved way for the construction of include
directories and file names which should be passed to the command?
Are there any chances to adapt it to the chosen build configuration?

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[Julia Lawall]

On Sun, 3 Mar 2013, SF Markus Elfring wrote:

> Hello,
>
> I try to generate some documentation from the source files of your current software.
>
> elfring at Sonne:~/Projekte/Coccinelle/lokal> MY_DIR=../Probe/Doku/ &&
> MY_NAMES=${MY_DIR}file_name_list11.txt && find . \( -name '*.ml' -o -name
> '*.mli' \) -a ! -name 'myocamlbuild.ml' -type f -fprint0 $MY_NAMES &&
> ocamldoc.opt -d ${MY_DIR} -I commons -I commons/ocamlextra -I
> commons/ocollection -I globals -I engine -I parsing_c -I parsing_cocci -I ctl -I
> ocaml -I ocamlsexp -I python -I popl09 -I extra -I /usr/local/share/menhir -I
> bundles/pcre -I bundles/pycaml -dot -o ${MY_DIR}my_graph.gv -html $(sed 's/\x0/
> /g' $MY_NAMES)
>
>
> I get a couple of error messages because my approach is incomplete so far. I
> stumble on technical challenges here.
> Would you like to suggest an improved way for the construction of include
> directories and file names which should be passed to the command?
> Are there any chances to adapt it to the chosen build configuration?

Maybe you could use find with the -exec argument, if it is acceptable to
work on only one file at a time?  I wouldn't know how to make it sensitive
to the configuration.  I guess one could write some code to parse the
configuration file, but that doesn't seem very appealing.  Perhaps you
could also just look for the ML files that have been compiled, if you do
make clean first.

julia

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

> Maybe you could use find with the -exec argument, if it is acceptable to
> work on only one file at a time?

I do not see that this suggestion works here because I understand the tool
"ocamldoc" in the way that it wants to get all relevant parameters (including
the source file names) passed at once.


> I wouldn't know how to make it sensitive to the configuration.

My attempt with the tool "find" alone leads to unwanted results. I would like to
avoid to maintain file name list manually.
It seems that your repository contains 321 OCaml source files from which
documentation information can eventually be extracted.

Your tool "spatch" reuses also some other software components. I see it as a
configuration challenge to exclude source files from these dependencies for the
suggested documentation process.

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[Julia Lawall]

On Mon, 4 Mar 2013, SF Markus Elfring wrote:

> > Maybe you could use find with the -exec argument, if it is acceptable to
> > work on only one file at a time?
>
> I do not see that this suggestion works here because I understand the tool
> "ocamldoc" in the way that it wants to get all relevant parameters (including
> the source file names) passed at once.

Actually, I am not sure what is the point for you to pick up all of the ml
files.  The things listed in the mli files are the only things that you
should use.  Unless there is an ML file that has no mli file, which is
unfortunately occasionally the case.  Perhaps the approach would be to
look for the compiled files, then see if the mli variant exists, and if
not see if the ml variant exists.

On the other hand, I'm not sure what the purpose of all of this is.  If it
is only to be able to write ocaml code that processes the C abstract
syntax tree, then it should be enough to consider the files in the
parser_c directory.  These are however the ones where there is a risk of a
missing mli file.

julia

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

> On the other hand, I'm not sure what the purpose of all of this is.

I hope to achieve a better understanding for your software by the presentation
style from the tool "ocamldoc". (It supports also to generate Graphviz files.)


> If it is only to be able to write ocaml code that processes the C abstract
> syntax tree, then it should be enough to consider the files in the
> parser_c directory.  These are however the ones where there is a risk of a
> missing mli file.

I guess that it is still a configuration challenge to set some variables in a
way so that all dependencies like needed modules can be resolved for the
documentation.

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[Julia Lawall]

On Mon, 4 Mar 2013, SF Markus Elfring wrote:

> > On the other hand, I'm not sure what the purpose of all of this is.
>
> I hope to achieve a better understanding for your software by the presentation
> style from the tool "ocamldoc". (It supports also to generate Graphviz files.)
>
>
> > If it is only to be able to write ocaml code that processes the C abstract
> > syntax tree, then it should be enough to consider the files in the
> > parser_c directory.  These are however the ones where there is a risk of a
> > missing mli file.
>
> I guess that it is still a configuration challenge to set some variables in a
> way so that all dependencies like needed modules can be resolved for the
> documentation.

It seems like a separate issue.  With the proper -I arguments, which you
can get from the makefile of the parser_c directory, ocamldoc should be
able to figure things out for itself, assuming make depend has been run
first.

julia

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

> The things listed in the mli files are the only things that you should use.
> Unless there is an ML file that has no mli file, which is unfortunately
> occasionally the case.

Would you like to add any interface descriptions for documentation improvements?

Which approach do you recommend to assign file names and include parameters to
different configuration variables for your source files (excluding external
software components)?

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[Julia Lawall]

On Tue, 5 Mar 2013, SF Markus Elfring wrote:

> > The things listed in the mli files are the only things that you should use.
> > Unless there is an ML file that has no mli file, which is unfortunately
> > occasionally the case.
>
> Would you like to add any interface descriptions for documentation
> improvements?

No, we don't have the resources for that.

> Which approach do you recommend to assign file names and include
> parameters to different configuration variables for your source files
> (excluding external software components)?

No idea what this means.

julia

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

> No, we don't have the resources for that.

It is a pity. - I assume that some users would be interested in more interface
descriptions. Will other software developers eventually help with documentation
improvements?


>> Which approach do you recommend to assign file names and include
>> parameters to different configuration variables for your source files
>> (excluding external software components)?
> 
> No idea what this means.

The file search gave unwanted results when I attempted it with the complete
project hierarchy. I imagine that usable file name lists can be automatically
generated if all your own source files (excluding bundled components) would be
stored in specific subdirectories. Several parts can be concatenated to achieve
the desired parameters for the call of the tool "ocamldoc".

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[Julia Lawall]

> The file search gave unwanted results when I attempted it with the complete
> project hierarchy. I imagine that usable file name lists can be automatically
> generated if all your own source files (excluding bundled components) would be
> stored in specific subdirectories. Several parts can be concatenated to achieve
> the desired parameters for the call of the tool "ocamldoc".

To the best of my understanding, our files are in separate directories
from the files for external libraries.  But if you need everything to get
any documentation, then I guess you would need the external libraries as
well.

I'm not sure why ocamldoc can't just use the -I paths to find the things
it needs.  That is you should be able to give just one file, and it should
be able to find the information for any dependencies.  But i haven't tried
it either.

julia

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

> To the best of my understanding, our files are in separate directories
> from the files for external libraries.

Your top directory contains also a few source files which might be interesting
here. I assume that the file "myocamlbuild.ml" belongs to the configuration
option "--enable-ocamlbuild" for example.
Are there any more conditions to consider?



> But if you need everything to get any documentation, then I guess you would need
> the external libraries as well.

I guess that I will need a technique to determine include paths for the
referenced modules consistently.


> I'm not sure why ocamldoc can't just use the -I paths to find the things
> it needs.  That is you should be able to give just one file, and it should
> be able to find the information for any dependencies.

Do you expect that more "intelligence" or "convenience" should be provided by
this tool?

I imagine that the desired (include) parameters need to be constructed by
adjustments for the configuration script.

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[Julia Lawall]

> > But if you need everything to get any documentation, then I guess you would need
> > the external libraries as well.
>
> I guess that I will need a technique to determine include paths for the
> referenced modules consistently.

I think you could grep for them in the makefile.

> > I'm not sure why ocamldoc can't just use the -I paths to find the things
> > it needs.  That is you should be able to give just one file, and it should
> > be able to find the information for any dependencies.
>
> Do you expect that more "intelligence" or "convenience" should be provided by
> this tool?

I don't know anything about this tool, but I would expect that it would
follow the same strategy as the rest of the ocaml tools for respolving
dependencies.

julia

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

>> I guess that I will need a technique to determine include paths for the
>> referenced modules consistently.
> 
> I think you could grep for them in the makefile.

This approach will not work because the "Makefile" will be generated by the
configuration script (build specification).


> I don't know anything about this tool, but I would expect that it would
> follow the same strategy as the rest of the ocaml tools for respolving
> dependencies.

I would also appreciate if the program "ocamldoc" could eventually benefit more
from the OCaml infrastructure to make the approach a bit more convenient and
safe.   ;-)
I find that the challenge remains to construct relevant parameters by usual
software development (with M4 programming language) here.

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[Julia Lawall]

On Tue, 5 Mar 2013, SF Markus Elfring wrote:

> >> I guess that I will need a technique to determine include paths for the
> >> referenced modules consistently.
> >
> > I think you could grep for them in the makefile.
>
> This approach will not work because the "Makefile" will be generated by the
> configuration script (build specification).

It would seem reasonable to run configure first.

But anyway, I don't think that the configureation script generates the
makefiles.  The make files are static, and they include other files that
are generated by the configuration file.  But the include information is
in the static part.

> > I don't know anything about this tool, but I would expect that it would
> > follow the same strategy as the rest of the ocaml tools for respolving
> > dependencies.
>
> I would also appreciate if the program "ocamldoc" could eventually benefit more
> from the OCaml infrastructure to make the approach a bit more convenient and
> safe.   ;-)
> I find that the challenge remains to construct relevant parameters by usual
> software development (with M4 programming language) here.

I think you should take it up with the ocaml developers.  Surely others
would appreciate this functionality, if it does not already exist.

julia

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

> But anyway, I don't think that the configureation script generates the
> makefiles.  The make files are static, and they include other files that
> are generated by the configuration file.  But the include information is
> in the static part.

I suggest to reconsider the understanding of Autotools functionality from your
current build specification "configure.ac". Some data need to be conditionally
forwarded through the involved processing stages until it will reach a
documentation generation command.


> I think you should take it up with the ocaml developers.  Surely others
> would appreciate this functionality, if it does not already exist.

Have you got any good contacts to them (e. g. from your university) to get
further helpful ideas eventually?

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[Arie Middelkoop]

I'm sorry but configure generates only constants containing paths,
which specify - depending on the system configuration and preferences
- where binaries/sources of various packages are located. It does not
generate invocations of the ocaml compiler nor of ocamldoc.

Like with all other tools, in order to use ocamldoc you'll need to
make sure that it gets the appropriate inputs, such as compiled
interface files. You could use our makefiles as a starting point to
help you in this process. However, it will involve manual labour. It
is neither clear to me that what you want is actually possible. When I
tried it once, ocamldoc failed to parse some sources, so you cannot
expect it to handle evertything.

As I wrote before, ocamlbuild has some hidden magic for documentation
generation. So if you can get coccinelle to build with it, you might
get documentation out as well without having to do a lot of work. Note
that we do not officially support ocamlbuild so if it does not work
for you, you can forget this last paragraph.

Arie

2013/3/5 SF Markus Elfring <elfring@users.sourceforge.net>:
>> But anyway, I don't think that the configureation script generates the
>> makefiles.  The make files are static, and they include other files that
>> are generated by the configuration file.  But the include information is
>> in the static part.
>
> I suggest to reconsider the understanding of Autotools functionality from your
> current build specification "configure.ac". Some data need to be conditionally
> forwarded through the involved processing stages until it will reach a
> documentation generation command.
>
>
>> I think you should take it up with the ocaml developers.  Surely others
>> would appreciate this functionality, if it does not already exist.
>
> Have you got any good contacts to them (e. g. from your university) to get
> further helpful ideas eventually?
>
> Regards,
> Markus

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

> I'm sorry but configure generates only constants containing paths,
> which specify - depending on the system configuration and preferences
> - where binaries/sources of various packages are located.

The Autotools infrastructure ensures that these macros will be applied by
Makefiles at the appropriate places.


> It does not generate invocations of the ocaml compiler nor of ocamldoc.

I agree because these tools do not belong to software which are supported by the
autoconf build system by default. I guess that this can be added by adjustments
of ".in" (or ".am" eventually) files easily.
http://www.gnu.org/software/automake/manual/html_node/automake-Invocation.html


> Like with all other tools, in order to use ocamldoc you'll need to
> make sure that it gets the appropriate inputs, such as compiled
> interface files.

I find it a technical challenge at the moment to collect the needed parameters
in a consistent way.


> However, it will involve manual labour.

I hope to avoid that.


> When I tried it once, ocamldoc failed to parse some sources, so you
> cannot expect it to handle evertything.

Thanks for your background information.


> As I wrote before, ocamlbuild has some hidden magic for documentation
> generation. So if you can get coccinelle to build with it, you might
> get documentation out as well without having to do a lot of work. Note
> that we do not officially support ocamlbuild so if it does not work
> for you, you can forget this last paragraph.

I hope that this aspect will change in the future.

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

> It is neither clear to me that what you want is actually possible.

How do you think about a result like
"http://xen.org/files/XenCloud/ocamldoc/codedoc.html"?
(I hope that a helpful display can also be achieved without a custom generator.)
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/configure.ac;h=5c3f5e8b60717c83256814bde98de82dab0d2bc9;hb=748e38f4719ea739f39b9529664e2221d6f8a394

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/configure.ac;h=5c3f5e8b60717c83256814bde98de82dab0d2bc9;hb=748e38f4719ea739f39b9529664e2221d6f8a394

I'm sorry. - I picked up a link from the wrong source code repository.
An other link is appropriate if you are eventually interested to look at an
alternative documentation approach for an application programming interface.
https://github.com/xen-org/xen-api/tree/master/ocaml/doc

Regards,
Markus

[Cocci] Addition of source code documentation generation to the build system?

[SF Markus Elfring]

> I think you should take it up with the ocaml developers.

I hope that useful advices will appear in an other discussion forum.
https://sympa.inria.fr/sympa/arc/caml-list/2013-03/msg00013.html

Regards,
Markus