Musings on Online/Improved Documentation

Okay, I'll start a thread about this, partly to get a discussion and partly so it's documented somewhere.

I've been looking at how to update the documentation and have a "write once, use many" model. Ideally, we can write the documentation in a markup language, and then it would become available on-line (on this site or elsewhere) and get bundled into the distribution (for the web interface help) with the same files.

Thoughts so far, in no particular order:

Hosting

* readthedocs.org works well, but is limited in language support (markdown and reStructuredText only).
* It integrates directly into github, though, so pushing to a documentation repository will automatically update the on-line help.
* However, readthedocs is using an old version of mkdocs, so its markdown translation isn't 100% perfect.
* We have a web site here, and could easily host it ourselves.

> readthedocs is simple, but if we can emulate the 'pull' from github then I'd suggest doing it in house.

* Using readthedocs gives automatic theme support; doing it in-house gives us CSS control for a common look-and-feel
* However, that obviously needs someone to do the CSS...!

Sample of the current documentation ported to markdown and auto-converted to readthedocs (automatic github pull and mkdocs conversion):

http://s12.postimg.org/lns724j65/Screenshot_240515_17_20_33.png

Language

* Markdown is the simplest and most widely-supported (default for github, stackexchange and here on this forum, in various flavours)
* Asciidoc is more powerful but less-well supported (e.g. no automatic import into readthedocs)
* reStructuredText is more powerful again, but syntax is harder to learn so is less likely to accept 'quick fix PRs' to the repository.
* reStructuredText is also overkill for simple HTML - we're not doing formula layouts and TeX/LaTeX conversions

> reStructuredText is only really useful if you're using readthedocs to host. Otherwise, it's not worth the hassle.

> markdown or some close cousin is the way to go. Asciidoc is probably the bet compromise rather than use an extended markdown (github flavoured, markdown here, etc.)

Conversion Tools

* readthedocs uses mkdocs to import markdown
* mkdocs has limitations on ToC levels, so we can't have a web menu such as:
Configuration
---General
---Access Entries
---DVB Inputs
------TV Adapters
------Networks

------Muxes

... etc.

* Nor can you 'nest' or 'include' sub-files in mkdocs

> That means a long list in the toc. You can start to see it in the image above.

* My plan was to use mkdocs build to then create html files (e.g. config_muxes.md becomes config_muxes.html) and then drop these straight into tvh as replacements for the current docs.
* However, mkdocs build doesn't work like that - it creates sub-folders by name, each with index.html (e.g. configu_muxes.md becomes config_muxes/index.html)

> That means post-processing to rename them back to what tvh currently expects

> Or it means changing the current URLs in tvh to look for the index.html files (not hard)

* Worse, mkdocs generates an entire page, complete with ToC. That's a show-stopper for me, as you just cannot have it embedded in a popup window like that.

> so mkdocs is a non-starter unless it can be tuned further.

* pandoc looks more promising. That will at least take the individual .md files and convert them to simple .html - not dissimilar to what we have now

> This is workflow, but relatively trivial to automate

Conclusions

There are some clear ease-of-use benefits to using readthedocs, and similar author ease in using markdown.

One option, then, is to use both of those, but simply use pandoc as part of a new release to convert then-current individual pages and bundle them as we do now.

I'm leaning towards:

* markdown for the raw files
* pandoc to translate to html only what we bundle into the web help
* House the raw files on github, probably as part of the main tvh repository
* Host the pages on readthedocs because updates are immediate from github

> that means we can take a static snapshot of the document as part of a release branch while continuing to update on master

I'll pause there and scribble more as and when. My main conclusion is that it's never as easy as it should be, but maybe I'm making it too hard.

If anyone wants to look:

https://github.com/ProfYaffle/tvheadend-documentation

https://tvheadend.readthedocs.org/en/latest/

Content

Obviously, we then also need to improve the content and bring it up to date. Volunteers welcome as we decide what format to use!

Other thoughts

Q: How much CSS work is needed in the pandoc output to get it to render correctly?
Q: What impact, if any, do the recent .gz changes to the webUI cause?

Q: Align image directories and similar between the readthedocs input and the webUI, so all references remain valid in the pandoc conversion

EDIT

Raw HTML output from pandoc renders fine:

http://s4.postimg.org/r5d1e8aod/raw_html.png

... but gets mangled by the CSS used by tvh. I hope this is just a case of stuff not being defined, as to change definitions in tvh's CSS would obviously affect everything. There are possibilities to add html tags into markdown, but then you're writing html again, which was the point in the first place...

http://s14.postimg.org/nthhmssqp/tvh_html.png

EDIT

Yes, it's the extjs (?) CSS - hacking this into the beginning of the file and it renders correctly:


<style type="text/css">
strong {font-weight: bold;}
em {font-style: italic;}
</style>

pandoc supports templates, so I wonder...

EDIT

Put this lot into a template, e.g. custom.css:


  <style type="text/css">
    strong {font-weight: bold;}
    em {font-style: italic;}
    ol li {
      list-style:decimal;}
    ul li {
      list-style:disc;}
    ul ul li {
      list-style:circle;}
    ul, ol {
      padding-left: 2em;}
  </style>

and then include it in the pandoc translation:

pandoc -t html5 -H custom.css input.md -o output.html

EDIT

And fix the line spacing between paragraphs. Pretty good vs the old look and feel now:


p {
  margin-top: 0em;
  margin-bottom: 1.5em;
}

Note: this feels awful - especially the paragraph spacing. Even inserting the CSS to defeat Sencha's formatting is very odd, but switching it off at a panel level seems to do terrible things to the overall interface. More homework required.

EDIT

The mkdocs conversion at readthedocs doesn't half bugger up file locations. What works fine locally gets mangled in translation as additional ../ directive are inserted into the paths.

Options:

1. Put everything into the docs level that it seems to expect. Will need some way of identifying the webui files (e.g. w_ prefix)

2. Duplicate the images, so that they're there locally for testing and then somewhere else for readthe docs.

Looks like it was a bug that was fixed in mkdocs some time ago, but hasn't made it to readthedocs...

EDIT

readthedocs is broken - too many issues with importing, syntax highlighting, nested documents, markup translation. Broken to the point where I'd give up and self-host...

However, swapping messages with d0ugal on IRC, he plans on upgrading it very soon (next week or so) now mkdocs is stable and has no known bugs. So I'll wait a week before aborting that route, and concentrate on the local rendering on the assumption that readthedocs will either catch up or I'll abandon it later.

EDIT

Latest mkdocs (0.13.0-dev) support multi-level tocs (whoop) and a search function. Install by:

pip install git+https://github.com/mkdocs/mkdocs.git

Also, any .css file in docs is picked up and used - so my custom.css is screwing up the default theme. Rename it so it isn't picked up, something innocuous like custom-header.txt, and then pass this to pandoc.

EDIT

Table formatting doesn't survive into pandoc/extjs. However, I just noticed that changing the CSS globally (as above) changes the layout of the entire web page - including the default tabs, which isn't what we want at all. So I'll need to think again there - there must be some way to tag things properly and use default CSS...

EDIT

Looks like hts-doc-text is the class for the popup contents. So we can tackle that in the CSS to restore em, ul, ol, etc.

Specifying that in the inserted CSS means we don't break the overall interface - 'raay. Downside is that pandoc has stripped out that class, so we need to add it in again - either as part of the header/footer or as explicit tags in the markdown.


<div class="hts-doc-text">
</div>

Inserting that into the pandoc-generated files means tvheadend's CSS can take care of it all (subject to whatever is defined for this class already and/or if it's used anywhere else, such as the EPG popups).

EDIT

Nope, simply tagging the markdown screws up the mkdocs theme (because it's now the wrong class when mkdocs converts it... d'oh). I'm feeling that a quick sed to replace <body></body> tags in the pandoc output might be ugly but practical - I'm converting the files for use elsewhere, after all.

Also, I can't see that that class is used anywhere else, which is good news. That means it's mine, all mine I tell you (mwahahaha)!

EDIT

Okay, much work, and the format and workflow is all there. I just need to improve/automate it a bit more now.

See: https://profyaffle.github.io/ for a live preview.

Repository remains on https://github.com/ProfYaffle/tvheadend-documentation

Readthedocs should catch up shortly...

EDIT

Much activity today, and I think this thread comes to a close.

Documentation is now live on http://docs.tvheadend.org (actually, a CNAME redirect to a live version of https://github.com/tvheadend/tvheadend.github.io).

Source documentation has been migrated to https://github.com/tvheadend/tvheadend-documentation.

Contributions welcome...