Unify documentation system: LaTeX to plDoc

[wouterbeek]

17 libraries are currently documented both in LaTeX and in plDoc. They reside in swipl-devel/man/lib. They document modules that reside in swipl-devel/library.

[Anniepoo]

We do have too many formats and workflows for documentation.
swinb files, LaTeX files, plDoc, lpn's autoconverter, the website compiler chain, http://swish.swi-prolog.org, documentation embedded in that, http://pengines.swi-prolog.org/
Pack documentation on the pack's pages, or, often, only on random sites somewhere on the internet.

It contributes to 'I know there's this cool thing, if only I could find it'

[wouterbeek]

@Anniepoo Yes I agree: this issue is part of a larger goal to unify (SWI) Prolog wisdom under a more unified umbrella. My idea is to segment this larger goal into smaller issues that are easy to pick up by community members.

About the formats that you enumerate:

• LaTeX is only used for the manual (which is fine IMO), but also for the above mentioned 17 modules. The latter is not good since this currently duplicates plDoc documentation in the module file. This is not too difficult to solve over time.
• plDoc is a mix of Markdown, Doxygen and some Wiki format. It has good support for hooks into the Prolog system and is very versatile. Most documentation should move to this format. ATM plDoc is (irony) not very well documented (lots of formatting idiosyncrasies) and there is a fairly long list of bugs. Fixing this will be quite some work, but the benefit is clear.
• What is 'Ipn autoconverter'?
• What additional formats are used in the Web site?
• Jan has made recent updates to SWISH that allow HTML blocks to be inserted. It is also possible to hide most of the toolbars and console panes, resulting in a Web page with interactive Prolog snippets. I envision this to be a viable format for interactive Web tutorials, but this has to be tested out with a real tutorial first.
• Pengines is not a documentation format but the infra on which SWISH (among other things) runs, or am I missing something here?
• Pack documentation should be written in plDoc.
• If there is useful content on the Web somewhere then we should link to it or, if the above mentioned SWISH format proves viable, we should invite the author to make an adaptation in HTML/SWISH.

If the above would be applies we would end up with the vast majority of documentation residing in two formats: plDoc for code file and HTML/SWISH for (interactive) tutorials.

[Anniepoo]

The lpn autoconverter was written by Jan to convert the original LaTeX files for the book "Learn Prolog Now" to the SWISH-embedded form available on

http://lpn.swi-prolog.org/lpnpage.php?pageid=online

The students at the OS Academy did some work on improving the tool, but it still makes some mistakes.

[Anniepoo]

The manual pages on the website come from the LaTeX manual.
Wiki-ized pages come from plweb-www

Some pages are present as either .txt files or as .html files
.txt files are included as the content area after converting from markdown
infrequently used, but available, is .frg files, which are an incomplete piece of html to be inserted verbatim in the content area.

[Anniepoo]

"Jan has made recent updates to SWISH that allow HTML blocks to be inserted. It is also possible to hide most of the toolbars and console panes, resulting in a Web page with interactive Prolog snippets. I envision this to be a viable format for interactive Web tutorials, but this has to be tested out with a real tutorial first."
Agreed - when it reaches a place of some finality, I'd like to see the 'real world' tutorials merge into this.
(Oh, and htat's one more format. The "Real World" tutorials start life as asciidoc, and are further munged by a little prolog script. And some of them are still in obsolete formats (mostly hand coded HTML).

[Anniepoo]

Pengines is not a documentation format but the infra on which SWISH (among other things) runs, or am I missing something here?

Some of the documentation for the pengines system is only available at http://pengines.swi-prolog.org

While most of this site is obsolete, it's the only place the javascript client is documented.
It's this documentation I was referring to.

While thinking about larger issues re documentation, consider that we do have bits of code in other languages - javascript for pengines js client, and soon we'll have JavaPengines and others.

[Anniepoo]

Pack documentation should be written in plDoc.
The biggest issue with autodoc systems is that they discourage writing larger scale documentation.

So, while yes, module and predicate level documentation should be in plDoc, what about Michael Hendrix's excellent tutorial for Julian?

http://mndrix.github.io/julian/index.html

[Anniepoo]

8cD you know, I think we're making a piece of semantic web here.

[wouterbeek]

@Anniepoo Thanks for adding some formats I didn't even know about to the list :-) I still think that Prolog modules can largely be documented in plDoc and tutorials/manuals can largely be written in SWISH.

I did not yet think about the need to document other languages like, e.g., JavaScript. Do you know of any existing solutions to documenting multi-language projects? Quasi-quoting languages may have this?

To my knowledge there is no SW solution for annotating documentation. It may be possible to come up with something:

• pack has modules
• module has predicates,declarations,operators
• predicate has name,arity,arguments,clauses,determinism,description
• predicate is private,public
• predicate is-related-to predicate
• clause has head, body
• argument has instantiation,description

I’m probably forgetting things here but this should already cover a fair about of the strings that plDoc parses when it extracts documentation from a Prolog module. IOW it is not all that strange to (1) come up with a SW vocabulary for documentation and (2) apply this to SWI first because of its superb SW support.

[triska]

Wouter, this is an excellent goal; the implications though are very far reaching, and many things must be worked on before PlDoc or other (currently) largely ad-hoc markup languages become a suitable target for the documentation of all these libraries.

For example, only yesterday, Jan himself suggested moving the CLP(FD) documentation away from PlDoc due to major shortcomings of the available formatting instructions. I will probably move the documentation to LaTeX or a different format.

Richard O'Keefe raised a few excellent (and, as far as I can tell, unfortunately largely ignored) points about PlDoc in a prior discussion. Some of the issues I have so far encountered with PlDoc are:

• small issues like: no equivalent to HTML's   or LaTeX's non-breaking space ~ (#18).
• larger issues like: no official support for linking to sections (#17)
• major and fundamental issues such as the ones raised by Richard O'Keefe in the above thread.

...
I have not been complaining about *reading* this stuff.
I have been saying that
 - it is somewhere between difficult and impossible for me
   to write a *program* that reads it
 - it is excessively difficult for me to *write* this stuff
   because I am never quite sure what is legal and what is
   not and how to get various effects. 
...

In my view, these are all issues that should be filed and discussed in the pldoc repository, because these shortcomings block the conversion to PlDoc.

[JanWielemaker]

For example, only yesterday, Jan himself suggested moving the CLP(FD) documentation away from PlDoc due to major shortcomings of the available formatting instructions. I will probably move the documentation to LaTeX or a different format.

I only suggested to move extensive descriptions of a library out of the source file. Clp(fd) has now
really extensive multi-section documentation in the module header. I think that stresses PlDoc too much. It is intended to provide reference documentation on a library. library(clpfd) is a giant beast, counting 7148 lines of source, defining 42 public predicates and 16 public operators and has a 644 lines documentation header. The documentation of such an extensive library requires more structure than an introduction text and a description list of predicate definitions.

Using markdown files that include lists as below, you can group the predicates as you need them.

  - [[name1/arity1]]
  - [[name2/arity2]]

You can keep the description around the list in markdown or, if you prefer, in LaTeX or any mixture of them. See man/lib and in particular the new tabling library that uses this. Like clp(fd), tabling is a too complex topic to describe comfortably using wiki-style comments in the source code.

[triska]

OK, I will try this or move the documentation to LaTeX altogether, since I need features that are not available in Markdown.

[triska]

As recommended, I've now moved the CLP(FD) documentation to LaTeX since I need more flexibility and features than PlDoc (at least currently) provides to document the whole library coherently and efficiently.

Pertaining to this discussion, here are my experiences so far:

1. It became clear to me that I really need two kinds of documentation:
   1. Reference documentation, describing each individual predicate in sufficient detail to let users know what it does and point them to more information if necessary.
   2. Comprehensive documentation that is meant to be read consecutively like a book chapter, describing larger concepts and including the description of individual predicates while assuming some concepts are already known at particular points throughout the text. Please see the recent clpfd.doc for an example of this and compare it with the PlDoc output.
2. As already suggested by Jan, LaTeX is ideal for (ii). It really provides everything I need and I can reorder the predicate descriptions in a way that is both logical and efficient, because I can group predicates together and factor out common descriptions for each of them. In my view, everything short of an actual LaTeX equivalent will eventually fall short in some way for this use case, so I regard LaTeX as a necessity for the documentation process.
3. Conversely, I have found many examples where I would like to make the documentation of individual predicates more extensive when it is displayed in isolation (i.e., via a direct link), such as for example the description of (#=)/2: In a book-style chapter, a certain order is imposed and the last two sentences could be factored out over all arithmetic CLP(FD) constraints. This is what I have also done in the LaTeX version (please see). However, when I point users to (#=)/2 in isolation, then I would still like these sentences to be shown. For this use case, I am quite happy with PlDoc: It can be written in the source file for each predicate in isolation and is very well suited to point users to a somewhat detailed description of an individual predicate.
4. At this point, a question: What happens if I use both LaTeX and PlDoc documentation within the SWI codebase? Ideally, to cover (3), I would like the PlDoc description to be shown to users when I post links to individual predicates, and the LaTeX source (.doc) to be used for generating the entire HTML page of the library as well as for the PDF version of the manual. Is that currently the case? If not, what is the precise relation between these documentation options, and would you consider to handle them this way?
5. Assuming (4) works out as I hope, a followup question: Is there a good way to link from PlDoc to the HTML/LaTeX version of the library documentation and vice versa?

[JanWielemaker]

`Mixing' is achieved using Markdown files. You write the book like story that documents one or more
predicates from the library in one or more markdown files in man/lib as

## About bla bla {#bla}

bla bla

  - [[p/1]]
  - [[q/2]]

bla bla bla

  - [[hello//2]]

Etc. You add a rule to the Makefile to make .tex and you use \input from the main LaTeX file. References to predicates will work automatically. References to sections are probably possible using the <#name> trick as we've seen before. predicateoptions.doc and predopts.txt are an example. Several packages use this trick too.

[triska]

I have now done this: The CLP(FD) predicate overview is now produced from clpfdpreds.txt, using Markdown notation.

Two observations:

1. Links to sections work in Markdown, using [description](<#label>). That's great! Thank you!
2. Please consider the entry for [[element/3]]: I unexpectedly get two entries for this predicate in the resulting HTML, apparently because element/3 is also defined in other libraries. Is there a way to state "Please give me the element/3 documentation from only library(clpfd)?" I tried [[(clpfd:element)/3]], but it did not work.