reST primer

We at SearXNG are using reStructuredText (aka reST) markup for all kind of documentation. With the builders from the Sphinx project a HTML output is generated and deployed at docs.searxng.org. For build prerequisites read Build docs.

The source files of SearXNG’s documentation are located at git://docs. Sphinx assumes source files to be encoded in UTF-8 by default. Run make docs.live to build HTML while editing.

Sphinx and reST have their place in the python ecosystem. Over that reST is used in popular projects, e.g the Linux kernel documentation [kernel doc].

reST is a plaintext markup language, its markup is mostly intuitive and you will not need to learn much to produce well formed articles with. I use the word mostly: like everything in live, reST has its advantages and disadvantages, some markups feel a bit grumpy (especially if you are used to other plaintext markups).

Soft skills

Before going any deeper into the markup let’s face on some soft skills a trained author brings with, to reach a well feedback from readers:

  • Documentation is dedicated to an audience and answers questions from the audience point of view.

  • Don’t detail things which are general knowledge from the audience point of view.

  • Limit the subject, use cross links for any further reading.

To be more concrete what a point of view means. In the (git://docs) folder we have three sections (and the blog folder), each dedicate to a different group of audience.

User’s POV: git://docs/user

A typical user knows about search engines and might have heard about meta crawlers and privacy.

Admin’s POV: git://docs/admin

A typical Admin knows about setting up services on a linux system, but he does not know all the pros and cons of a SearXNG setup.

Developer’s POV: git://docs/dev

Depending on the readability of code, a typical developer is able to read and understand source code. Describe what a item aims to do (e.g. a function). If the chronological order matters, describe it. Name the out-of-limits conditions and all the side effects a external developer will not know.

Basic inline markup

Basic inline markup is done with asterisks and backquotes. If asterisks or backquotes appear in running text and could be confused with inline markup delimiters, they have to be escaped with a backslash (\*pointer).

Table 15 basic inline markup

description

rendered

markup

one asterisk for emphasis

italics

*italics*

two asterisks for strong emphasis

boldface

**boldface**

backquotes for code samples and literals

foo()

``foo()``

quote asterisks or backquotes

*foo is a pointer

\*foo is a pointer

Basic article structure

The basic structure of an article makes use of heading adornments to markup chapter, sections and subsections.

reST template

reST template for an simple article:

.. _doc refname:

==============
Document title
==============

Lorem ipsum dolor sit amet, consectetur adipisici elit ..  Further read
:ref:`chapter refname`.

.. _chapter refname:

Chapter
=======

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquid ex ea commodi consequat ...

.. _section refname:

Section
-------

lorem ..

.. _subsection refname:

Subsection
~~~~~~~~~~

lorem ..

Headings

  1. title - with overline for document title:

==============
Document title
==============

  1. chapter - with anchor named anchor name:

    .. _anchor name:

Chapter
=======

  2. section

    Section
-------

  3. subsection

    Subsection
~~~~~~~~~~

Literal blocks

The simplest form of literal-blocks is a indented block introduced by two colons (::). For highlighting use highlight or code-block directive. To include literals from external files use literalinclude or kernel-include directive (latter one expands environment variables in the path name).

::

::

  Literal block

Lorem ipsum dolor::

  Literal block

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy
eirmod tempor invidunt ut labore ::

  Literal block

Literal block

Literal block

Lorem ipsum dolor:

Literal block

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore

Literal block

code-block

The code-block directive is a variant of the code directive with additional options. To learn more about code literals visit Showing code examples.

The URL ``/stats`` handle is shown in :ref:`stats-handle`

.. code-block:: Python
   :caption: python code block
   :name: stats-handle

   @app.route('/stats', methods=['GET'])
   def stats():
       """Render engine statistics page."""
       stats = get_engines_stats()
       return render(
           'stats.html'
           , stats = stats )

Code block

The URL /stats handle is shown in python code block

Listing 1 python code block
@app.route('/stats', methods=['GET'])
def stats():
    """Render engine statistics page."""
    stats = get_engines_stats()
    return render(
        'stats.html'
        , stats = stats )

Unicode substitution

The unicode directive converts Unicode character codes (numerical values) to characters. This directive can only be used within a substitution definition.

.. |copy| unicode:: 0xA9 .. copyright sign
.. |(TM)| unicode:: U+2122

Trademark |(TM)| and copyright |copy| glyphs.

Unicode

Trademark ™ and copyright © glyphs.

Roles

A custom interpreted text role (ref) is an inline piece of explicit markup. It signifies that that the enclosed text should be interpreted in a specific way.

The general markup is one of:

:rolename:`ref-name`
:rolename:`ref text <ref-name>`
Table 17 smart refs with sphinx.ext.extlinks and intersphinx

role

rendered example

markup

guilabel

Cancel

:guilabel:`&Cancel`

kbd

C-x C-f

:kbd:`C-x C-f`

menuselection

Open ‣ File

:menuselection:`Open --> File`

download

this file

:download:`this file <reST.rst>`

math

:math:`a^2 + b^2 = c^2`

ref

Simple SVG image.

:ref:`svg image example`

command

ls -la

:command:`ls -la`

emphasis

italic

:emphasis:`italic`

strong

bold

:strong:`bold`

literal

foo()

:literal:`foo()`

subscript

H2O

H\ :sub:`2`\ O

superscript

E = mc2

E = mc\ :sup:`2`

title-reference

Time

:title:`Time`

Figures & Images

SearXNG’s sphinx setup includes: Scalable figure and image handling. Scalable here means; scalable in sense of the build process. Normally in absence of a converter tool, the build process will break. From the authors POV it’s annoying to care about the build process when handling with images, especially since he has no access to the build process. With Scalable figure and image handling the build process continues and scales output quality in dependence of installed image processors.

If you want to add an image, you should use the kernel-figure (inheritance of figure) and kernel-image (inheritance of image) directives. E.g. to insert a figure with a scalable image format use SVG (Simple SVG image.):

.. _svg image example:

.. kernel-figure:: svg_image.svg
   :alt: SVG image example

   Simple SVG image

 To refer the figure, a caption block is needed: :ref:`svg image example`.
SVG image example

Fig. 4 Simple SVG image.

To refer the figure, a caption block is needed: Simple SVG image..

DOT files (aka Graphviz)

With kernel-figure & kernel-image reST support for DOT formatted files is given.

A simple example is shown in DOT’s hello world example:

.. _dot file example:

.. kernel-figure:: hello.dot
   :alt: hello world

   DOT's hello world example

hello.dot

hello world

Fig. 5 DOT’s hello world example

kernel-render DOT

Embed render markups (or languages) like Graphviz’s DOT is provided by the kernel-render directive. A simple example of embedded DOT is shown in figure Embedded DOT (Graphviz) code:

.. _dot render example:

.. kernel-render:: DOT
   :alt: digraph
   :caption: Embedded  DOT (Graphviz) code

   digraph foo {
     "bar" -> "baz";
   }

Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot
render example`.

Please note build tools. If Graphviz is installed, you will see an vector image. If not, the raw markup is inserted as literal-block.

kernel-render DOT

digraph

Fig. 6 Embedded DOT (Graphviz) code

Attribute caption is needed, if you want to refer the figure: Embedded DOT (Graphviz) code.

kernel-render SVG

A simple example of embedded SVG is shown in figure Embedded SVG markup:

.. _svg render example:

.. kernel-render:: SVG
   :caption: Embedded **SVG** markup
   :alt: so-nw-arrow

<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" version="1.1"
     baseProfile="full" width="70px" height="40px"
     viewBox="0 0 700 400"
     >
  <line x1="180" y1="370"
        x2="500" y2="50"
        stroke="black" stroke-width="15px"
        />
  <polygon points="585 0 525 25 585 50"
           transform="rotate(135 525 25)"
           />
</svg>

kernel-render SVG

so-nw-arrow

Fig. 7 Embedded SVG markup

List markups

Bullet list

List markup (ref) is simple:

- This is a bulleted list.

  1. Nested lists are possible, but be aware that they must be separated from
     the parent list items by blank line
  2. Second item of nested list

- It has two items, the second
  item uses two lines.

#. This is a numbered list.
#. It has two items too.

bullet list

  • This is a bulleted list.

    1. Nested lists are possible, but be aware that they must be separated from the parent list items by blank line

    2. Second item of nested list

  • It has two items, the second item uses two lines.

  1. This is a numbered list.

  2. It has two items too.

Horizontal list

The .. hlist:: transforms a bullet list into a more compact list.

.. hlist::

   - first list item
   - second list item
   - third list item
   ...

hlist

  • first list item

  • second list item

  • third list item

  • next list item

  • next list item xxxx

  • next list item yyyy

  • next list item zzzz

Definition list

Each definition list (ref) item contains a term, optional classifiers and a definition. A term is a simple one-line word or phrase. Optional classifiers may follow the term on the same line, each after an inline ‘ : ‘ (space, colon, space). A definition is a block indented relative to the term, and may contain multiple paragraphs and other body elements. There may be no blank line between a term line and a definition block (this distinguishes definition lists from block quotes). Blank lines are required before the first and after the last definition list item, but are optional in-between.

Definition lists are created as follows:

term 1 (up to a line of text)
    Definition 1.

See the typo : this line is not a term!

  And this is not term's definition.  **There is a blank line** in between
  the line above and this paragraph.  That's why this paragraph is taken as
  **block quote** (:duref:`ref <block-quotes>`) and not as term's definition!

term 2
    Definition 2, paragraph 1.

    Definition 2, paragraph 2.

term 3 : classifier
    Definition 3.

term 4 : classifier one : classifier two
    Definition 4.

definition list

term 1 (up to a line of text)

Definition 1.

See the typo : this line is not a term!

And this is not term’s definition. There is a blank line in between the line above and this paragraph. That’s why this paragraph is taken as block quote (ref) and not as term’s definition!

term 2

Definition 2, paragraph 1.

Definition 2, paragraph 2.

term 3classifier

Definition 3.

term 4 : classifier one : classifier two

Quoted paragraphs

Quoted paragraphs (ref) are created by just indenting them more than the surrounding paragraphs. Line blocks (ref) are a way of preserving line breaks:

normal paragraph ...
lorem ipsum.

   Quoted paragraph ...
   lorem ipsum.

| These lines are
| broken exactly like in
| the source file.

Quoted paragraph and line block

normal paragraph … lorem ipsum.

Quoted paragraph … lorem ipsum.

These lines are
broken exactly like in
the source file.

Field Lists

Field lists are used as part of an extension syntax, such as options for directives, or database-like records meant for further processing. Field lists are mappings from field names to field bodies. They marked up like this:

:fieldname: Field content
:foo:       first paragraph in field foo

            second paragraph in field foo

:bar:       Field content

Field List

fieldname:

Field content

foo:

first paragraph in field foo

second paragraph in field foo

bar:

Field content

They are commonly used in Python documentation:

def my_function(my_arg, my_other_arg):
    """A function just for me.

    :param my_arg: The first of my arguments.
    :param my_other_arg: The second of my arguments.

    :returns: A message (just for me, of course).
    """

Further list blocks

  • field lists (ref, with caveats noted in Field Lists)

  • option lists (ref)

  • quoted literal blocks (ref)

  • doctest blocks (ref)

Admonitions

Generic admonition

The generic admonition needs a title:

.. admonition:: generic admonition title

   lorem ipsum ..

generic admonition title

lorem ipsum ..

Specific admonitions

Specific admonitions: hint, note, tip attention, caution, danger, error, , important, and warning .

.. hint::

   lorem ipsum ..

.. note::

   lorem ipsum ..

.. warning::

   lorem ipsum ..

Hint

lorem ipsum ..

Note

lorem ipsum ..

Tip

lorem ipsum ..

Attention

lorem ipsum ..

Caution

lorem ipsum ..

Danger

lorem ipsum ..

Important

lorem ipsum ..

Error

lorem ipsum ..

Warning

lorem ipsum ..

Tables

ASCII-art tables like Simple tables and Grid tables might be comfortable for readers of the text-files, but they have huge disadvantages in the creation and modifying. First, they are hard to edit. Think about adding a row or a column to a ASCII-art table or adding a paragraph in a cell, it is a nightmare on big tables.

Second the diff of modifying ASCII-art tables is not meaningful, e.g. widening a cell generates a diff in which also changes are included, which are only ascribable to the ASCII-art. Anyway, if you prefer ASCII-art for any reason, here are some helpers:

Simple tables

Simple tables allow colspan but not rowspan. If your table need some metadata (e.g. a title) you need to add the .. table:: directive (ref) in front and place the table in its body:

.. table:: foo gate truth table
   :widths: grid
   :align: left

   ====== ====== ======
       Inputs    Output
   ------------- ------
   A      B      A or B
   ====== ====== ======
   False
   --------------------
   True
   --------------------
   True   False  True
          (foo)
   ------ ------ ------
   False  True
          (foo)
   ====== =============

Simple ASCII table

Table 18 foo gate truth table

Inputs

Output

A

B

A or B

False

True

True

False (foo)

True

False

True (foo)

Grid tables

Grid tables allow colspan colspan and rowspan:

.. table:: grid table example
   :widths: 1 1 5

   +------------+------------+-----------+
   | Header 1   | Header 2   | Header 3  |
   +============+============+===========+
   | body row 1 | column 2   | column 3  |
   +------------+------------+-----------+
   | body row 2 | Cells may span columns.|
   +------------+------------+-----------+
   | body row 3 | Cells may  | - Cells   |
   +------------+ span rows. | - contain |
   | body row 4 |            | - blocks. |
   +------------+------------+-----------+

ASCII grid table

Table 19 grid table example

Header 1

Header 2

Header 3

body row 1

column 2

column 3

body row 2

Cells may span columns.

body row 3

Cells may span rows.

  • Cells

  • contain

  • blocks.

body row 4

flat-table

The flat-table is a further developed variant of the list tables. It is a double-stage list similar to the list-table with some additional features:

column-span: cspan

with the role cspan a cell can be extended through additional columns

row-span: rspan

with the role rspan a cell can be extended through additional rows

auto-span:

spans rightmost cell of a table row over the missing cells on the right side of that table-row. With Option :fill-cells: this behavior can changed from auto span to auto fill, which automatically inserts (empty) cells instead of spanning the last cell.

options:
header-rows:

[int] count of header rows

stub-columns:

[int] count of stub columns

widths:

[[int] [int] … ] widths of columns

fill-cells:

instead of auto-span missing cells, insert missing cells

roles:
cspan:

[int] additional columns (morecols)

rspan:

[int] additional rows (morerows)

The example below shows how to use this markup. The first level of the staged list is the table-row. In the table-row there is only one markup allowed, the list of the cells in this table-row. Exception are comments ( .. ) and targets (e.g. a ref to row 2 of table’s body).

.. flat-table:: ``flat-table`` example
   :header-rows: 2
   :stub-columns: 1
   :widths: 1 1 1 1 2

   * - :rspan:`1` head / stub
     - :cspan:`3` head 1.1-4

   * - head 2.1
     - head 2.2
     - head 2.3
     - head 2.4

   * .. row body 1 / this is a comment

     - row 1
     - :rspan:`2` cell 1-3.1
     - cell 1.2
     - cell 1.3
     - cell 1.4

   * .. Comments and targets are allowed on *table-row* stage.
     .. _`row body 2`:

     - row 2
     - cell 2.2
     - :rspan:`1` :cspan:`1`
       cell 2.3 with a span over

       * col 3-4 &
       * row 2-3

   * - row 3
     - cell 3.2

   * - row 4
     - cell 4.1
     - cell 4.2
     - cell 4.3
     - cell 4.4

   * - row 5
     - cell 5.1 with automatic span to right end

   * - row 6
     - cell 6.1
     - ..

List table

Table 20 flat-table example

head / stub

head 1.1-4

head 2.1

head 2.2

head 2.3

head 2.4

row 1

cell 1-3.1

cell 1.2

cell 1.3

cell 1.4

row 2

cell 2.2

cell 2.3 with a span over

  • col 3-4 &

  • row 2-3

row 3

cell 3.2

row 4

cell 4.1

cell 4.2

cell 4.3

cell 4.4

row 5

cell 5.1 with automatic span to right end

row 6

cell 6.1

CSV table

CSV table might be the choice if you want to include CSV-data from a outstanding (build) process into your documentation.

.. csv-table:: CSV table example
   :header: .. , Column 1, Column 2
   :widths: 2 5 5
   :stub-columns: 1
   :file: csv_table.txt

Content of file csv_table.txt:

stub col row 1, column, "loremLorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy
eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam
voluptua."
stub col row 1, "At vero eos et accusam et justo duo dolores et ea rebum. Stet clita
kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.", column
stub col row 1, column, column

CSV table

Table 21 CSV table example

Column 1

Column 2

stub col row 1

column

loremLorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua.

stub col row 1

At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

column

stub col row 1

column

column

Templating

Templating is suitable for documentation which is created generic at the build time. The sphinx-jinja extension evaluates jinja templates in the Python environment (make install) (with SearXNG modules installed). We use this e.g. to build chapter: Configured Engines. Below the jinja directive from the git://docs/admin/engines.rst is shown:

==================
Configured Engines
==================

.. sidebar:: Further reading ..

   - :ref:`settings categories_as_tabs`
   - :ref:`engines-dev`
   - :ref:`settings engine`
   - :ref:`general engine configuration`

.. jinja:: searx

   SearXNG supports {{engines | length}} search engines of which
   {{enabled_engine_count}} are enabled by default.

   Engines can be assigned to multiple :ref:`categories <engine categories>`.
   The UI displays the tabs that are configured in :ref:`categories_as_tabs
   <settings categories_as_tabs>`.  In addition to these UI categories (also
   called *tabs*), engines can be queried by their name or the categories they
   belong to, by using a :ref:`\!bing syntax <search-syntax>`.

.. contents::
   :depth: 2
   :local:
   :backlinks: entry

.. jinja:: searx

   {% for category, engines in categories_as_tabs.items() %}

   tab ``!{{category.replace(' ', '_')}}``
   ---------------------------------------

   {% for group, group_bang, engines in engines | group_engines_in_tab %}

   {% if loop.length > 1 %}
   {% if group_bang %}group ``{{group_bang}}``{% else %}{{group}}{% endif %}
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   {% endif %}

   .. flat-table::
      :header-rows: 2
      :stub-columns: 1
      :widths: 10 1 10 1 1 1 1 1 1 1

      * - :cspan:`5` Engines configured by default (in :ref:`settings.yml <engine settings>`)
        - :cspan:`3` :ref:`Supported features <engine file>`

      * - Name
        - !bang
        - Module
        - Disabled
        - Timeout
        - Weight
        - Paging
        - Locale
        - Safe search
        - Time range

      {% for mod in engines %}

      * - `{{mod.name}} <{{mod.about and mod.about.website}}>`_
          {%- if mod.about and  mod.about.language %}
          ({{mod.about.language | upper}})
          {%- endif %}
        - ``!{{mod.shortcut}}``
        - {%- if 'searx.engines.' + mod.__name__ in documented_modules %}
          :py:mod:`~searx.engines.{{mod.__name__}}`
          {%- else %}
          :origin:`{{mod.__name__}} <searx/engines/{{mod.__name__}}.py>`
          {%- endif %}
        - {{(mod.disabled and "y") or ""}}
        - {{mod.timeout}}
        - {{mod.weight or 1 }}
        {% if mod.engine_type == 'online' %}
        - {{(mod.paging and "y") or ""}}
        - {{(mod.language_support and "y") or ""}}
        - {{(mod.safesearch and "y") or ""}}
        - {{(mod.time_range_support and "y") or ""}}
        {% else %}
        - :cspan:`3` not applicable ({{mod.engine_type}})
        {% endif %}

     {% endfor %}
     {% endfor %}
     {% endfor %}

The context for the template is selected in the line .. jinja:: searx. In sphinx’s build configuration (git://docs/conf.py) the searx context contains the engines and plugins.

import searx.search
import searx.engines
import searx.plugins
searx.search.initialize()
jinja_contexts = {
   'searx': {
      'engines': searx.engines.engines,
      'plugins': searx.plugins.plugins
   },
}

Tabbed views

With sphinx-tabs extension we have tabbed views. To provide installation instructions with one tab per distribution we use the group-tabs directive, others are basic-tabs and code-tabs. Below a group-tab example from Build docs is shown:

.. tabs::

   .. group-tab:: Ubuntu / debian

      .. code-block:: sh

         $ sudo apt install shellcheck

   .. group-tab:: Arch Linux

      .. code-block:: sh

         $ sudo pacman -S shellcheck

   .. group-tab::  Fedora / RHEL

      .. code-block:: sh

         $ sudo dnf install ShellCheck

Math equations

The input language for mathematics is LaTeX markup using the CTAN: amsmath package.

To embed LaTeX markup in reST documents, use role :math: for inline and directive .. math:: for block markup.

In :math:numref:`schroedinger general` the time-dependent Schrödinger equation
is shown.

.. math::
   :label: schroedinger general

    \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle =
          \hat{H} |\,\psi (t) \rangle.

LaTeX math equation

In (1) the time-dependent Schrödinger equation is shown.

The next example shows the difference of \tfrac (textstyle) and \dfrac (displaystyle) used in a inline markup or another fraction.

``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}`
``\dfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}`

Line spacing

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. … \tfrac inline example

At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. … \tfrac inline example

At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.