Author: Joachim Gabler

# Source code documentation using adoc

An essential characteristic of high quality source code is a comprehensive documentation.

For the Grid Engine development, we started some time ago to use so called Autodoc Headers that are interpreted by the adoc utility.

Adoc was originally written by Tobias Ferber and published unter the GPL.
It can create ASCII and Texinfo output, Texinfo can be used to publish the documentation in a lot of different output formats including HTML, PDF, GNU Info, DVI etc.

It's functionality is a little bit rudimentory and can not really compete with tools like doxygen or the javadoc tool, but it has the advantage, that it can be used to parse not only C or Java source code, but also any scripting languages like tcl, shell etc. using the same syntax.

The source code of adoc can be found in the directory `source/3rdparty/adoc`.
We modified it to support 3 levels of chapters/sections instead of only 2.

## Generating the documentation

The documentation can be generated automatically using the aimk script (`source/aimk`).
Calling `aimk -adoc` will

- compile the adoc tool
- generate 3 texinfo files:

- documentation of the library functions (`libs.texi`)
- documentation of the core system (`core.texi`)
- documentation of the testsuite (`testsuite.texi`)

The texinfo files can be used to generate a readable documentation in different output formats.

Examples:

- Generate HTML-Documentation:
`texi2html libs.texi`
Writes a file `libs.html` containing the documentation and a file `libs_toc.html` containing a table of contents with links to `libs.html`
- Generate Documentation in PDF Format:
`texi2pdf libs.texi`
Creates a file `libs.pdf`

## Writing documentation

Each function of C, TCL or shell code should have an autodoc header, especially these in function libraries.
The header can be inserted from a template (see`doc/devel/c-header`for C-Syntax, `doc/devel/sh-header`for TCL and shell syntax) or by using a tcl script defined as editor macro.

### Macro for automated insertion of autodoc headers

The distribution contains a small tcl script that may be used to automatically insert an autodoc header into a source file (see `doc/testsuite/adoc/doc_tool.tcl`).
It creates an autodoc header and automatically enters data for the category and function name, and the autodoc sections `NAME`, `SYNOPSIS`, `INPUTS` and `RESULT`.

It can be called as macro from the editor, if the editor has some means to include macros and the call of external programs.

The doc_tool.tcl has a variety of options to parse and modify source code for source code documentation purposes.

#### Insertion of autodoc headers in the vim editor

To install such a macro in the vim editor, edit the .vimrc file in your homedirectory and add the following line:

`map ^P :-r !doc_tool mode C output stdout no_warnings func <cword> %:p^M`

To enter the ^P and ^M, in vi press CTRL-V CTRL-p / CTRL-V CTRL-m.

If you place the cursor on a C function prototype and press F11, a new partly filled out autodoc header for the function will be inserted into the source file.

Newer vim versions (beginning with 6.0?) seem to use another syntax in the .vimrc file. Add the following line to your .vimrc file (install macro on F9, F11 as in the previous example didn't work for some reason:

`map <unique> <F9> :-r !doc_tool mode C output stdout no_warnings func <cword> %:p^M`

### Choosing a function identifier

The current structure of the source code documentation provides 3 individual files - documentation of libraries, the core system binaries and the testsuite.
Within each document, we have 3 hiearchical levels of sections - in texinfo: chapter, section and subsection.

The current documentation is structured not to reflext strictly the directory structure of the code in the section hierarchy, but the structure reflects categories of code within a certain directory based structure.

Each source file contains functions belonging to exactly one function category.

Example:
The module qsh has the function category `Interactive/qsh`: It is a tool to create an interactive job in Grid Engine, its name is qsh.

Possible function identifiers then are
`Interactive/qsh/--Interactive`
`Interactive/qsh/add2env()`
`...`

To enable the doc_tool.tcl to insert adoc headers with the correct function categoriy filled in automatically, there exists a file doc/devel/doc_tool.cat that maps directories and optionally filenames to function categories. One line contains one mapping.

Example:
`source/clients/qsh Interactive/qsh`
`source/clients/qrsh Interactive/qrsh`
`source/daemons/execd execd`
`source/daemons/execd/load_avg.* execd/load`
`source/daemons/execd/sge_load_sensor.* execd/load`
`...`

#### Function categories for the library documentation

- Eventclient

- Client

- commd

- commlib

- cull

- hash - cull specific hash table extensions, based on uti/hash
- list
- multitype

- gdi

- feature
- job_jatask
- queue
- range
- request
- security
- setup
- sge
- suser
- usermap

- rmon
- sched
- uti

- afsutil
- bitop
- dstring
- hostname
- htable - generic hash table implementation
- io
- language
- log
- os
- prog
- signal
- spool
- stdio
- stdlib
- string- string parsing functions
- time
- uidgid
- unistd

#### Function categories for the core system documentation

- commd
- common

- General
- Clients
- Daemons

- execd
- qacct
- qalter
- qconf
- qdel
- qhost
- qmaster
- qmod
- qmod
- qstat
- qsub
- schedd
- schadowd
- shepherd
- Interactive

- qsh

- qlogin
- qmake
- qrsh
- qtcsh

- Eventclient

- Server

#### Function categories for the testsuite documentation

- checks

- install
- loadcheck
- migration
- performance
- performance_cluster
- performance_scheduler
- qalter
- qconf
- qdel
- qmaster_size
- qmod
- qrsh
- qstat
- qsub
- shadowd_migrate

- library

- control
- file
- parser
- remote
- sge

- scripts

### Filling out the autodoc header

The following sections of an autodoc header should always be filled out:

- `NAME`
The function name and a short (one line) description
- `SYNOPSIS`
The function prototype, optionally preceeded by an include statement, if needed and compiler/linker options, see Examples for adoc headers
- `FUNCTION`
A complete function description.
- `INPUTS`
Description of all input parameters
- `RESULT`
Description of the result

These sections are optional :

- `EXAMPLE`
How to use the function
- `NOTES`
Notes for further development, hints for the usage etc.
- `BUGS`
adoc provides a BUGS section to document known bugs - please do not create BUGS sections - use IssueZilla instead.
- `SEE ALSO`
References to other functions, man pages etc.

### Introduction pages for Categories

A category usually reflects some hierarchy in the source code tree, often an individual binary, and deserves to have an introductory autodoc header before the detailed function descriptions.

Therefor an autodoc header should be written for each category that gives a short overview of the category and may refer to other autodoc headers giving more details.

The header of such an introductory page has the form
`/****** chapter/section/--topic *****************`

The hyphens (-) before the topic influence autodoc's sorting algorithm: The introductory page will always be the the first page of a section.

### Documentation of defines, global variables etc.

It is desirable to have documentation also for constants/defines, global variables, structures, typedefs etc.

The following special function identifiers are used for this type of documentation:
-topic-Defines
-topic-Global_Variables
-topic-Templates
-topic-Typedefs
-topic-Misc

Again, the hyphen before the identifiers name influence adoc's sorting algorithm.

## Examples for adoc headers

### An introduction header

`/****** Interactive/qmake/--qmake ***************************************`
`*`
`* NAME`
`* qmake -- Scheduled parallel distributed make`
`*`
`* SYNOPSIS`
`* qmake <sge options> -- <gmake options>`
`*`
`* FUNCTION`
`* Scheduled, parallel, distributed make.`
`* qmake is implemented based on GNU make 3.78.1 using the remote stub`
`* mechanism of gmake.`
`* qmake will start a parallel job and run make tasks as task in the`
`* parallel job using the Grid Engine qrsh -inherit command.`
`*`
`* INPUTS`
`* sge options - all options that can be specified with qsub command`
`* gmake options - all possible gmake options`
`*`
`* RESULT`
`* returncode from gmake or EXIT_FAILURE, if remote mechanism fails.`
`* On error, an appropriate error description is written to stderr.`
`*`
`* EXAMPLE`
`* qmake -cwd -v PATH -- -j 5`
`* Build sge system: aimk -qmake -parallel 10`
`*`
`* NOTES`
`* not yet internationalized`
`*`
`****************************************************************************`
`*/`

### A header describing defines

`/****** Interactive/qmake/-qmake-Defines ***************************************`
`*`
`* NAME`
`* Defines -- constant and macro definitions`
`*`
`* SYNOPSIS`
`* #define LOCK_SLEEP_TIME 500`
`* #define WAIT_SLOT_TIME 5`
`*`
`* FUNCTION`
`* LOCK_SLEEP_TIME - Defines how long qmake should (u)sleep`
`* after an unsuccessful try to get a lock to the`
`* hostfile`
`* WAIT_SLOT_TIME - Time to wait, if no slot is free in hostfile`
`*`
`****************************************************************************`
`*/`

### A function description

`/****** Interactive/qmake/remote_exit() ***************************************`
`*`
`* NAME`
`* remote_exit() -- exit qmake`
`*`
`* SYNOPSIS`
`* static void remote_exit(int code, const char *message,`
`* const char *reason);`
`*`
`* FUNCTION`
`* Outputs the error messages passed as parameters to stderr`
`* and then exits with the error code passed as parameter.`
`*`
`* INPUTS`
`* code - exit code`
`* message - message to output before exit, should describe the`
`* situation when error occurs`
`* reason - description of the error reason, e.g. result from`
`* system call strerror(errno)`
`*`
`* RESULT`
`* program exits`
`*`
`* EXAMPLE`
`* #include <stdlib.h>`
`* #include <string.h>`
`* #include <errno.h>`
`*`
`* ...`
`*`
`* if(write(filehandle, buffer, size) != size) {`
`* remote_exit(EXIT_FAILURE, "writing to file failed", strerror(errno));`
`* }*`
`****************************************************************************`
`*/`

### Description of a Cull object/class

`/****** gdi/range/--RN_Type ***************************************************`
`* NAME`
`* RN_Type - CULL range element`
`*`
`* ELEMENTS`
`* SGE_ULONG(RN_min)`
`* minimum or start value of an id range (e.g. 1)`
`*`
`* SGE_ULONG(RN_max)`
`* maximum or end value of an id range (e.g. 9)`
`*`
`* SGE_ULONG(RN_step)`
`* stepsize (e.g. 2)`
`*`
`* FUNCTION`
`* CULL element holding values which define a id range`
`* (e.g. 1-9:2 => 1, 3, 5, 7, 9).`
`* Lists of this CULL element are hold within a CULL job element`
`* (JB_Type) to hold job array task ids.`
`* Several functions may be used to access/modify/delete range`
`* elements and range lists. You may find them in the 'SEE ALSO'`
`* section. It is highly advised to use these access functions`
`* because they assure and require a defined structure of`
`* elements and lists.`
`*`
`* Range elements and lists stored in other CULL elements fullfill`
`* following conditions:`
`*`
`* - min <= max`
`* - step >= 1`
`* - real range elements (e.g. 1-9:2 instead of 1-10:2)`
`* - min-ids within range elements part of the same`
`* list are in ascending order: min_id(n) < min_id(n+1)`
`* (e.g. NOT 11-20:1; 1-9:2)`
`* - ids within range elements part of the same`
`* list are non-overlapping: max_id(n) < min_id(n+1)`
`* (e.g. 1-9:2; 11-20:1; 25-28:3)`
`*`
`* SEE ALSO`
`* gdi/range/RN_Type`
`* gdi/range/range_list_calculate_union_set()`
`* gdi/range/range_list_calculate_difference_set()`
`* gdi/range/range_list_calculate_intersection_set()`
`* gdi/range/range_list_compress()`
`* gdi/range/range_list_get_first_id()`
`* gdi/range/range_list_get_last_id()`
`* gdi/range/range_list_get_number_of_ids()`
`* gdi/range/range_list_initialize()`
`* gdi/range/range_list_insert_id()`
`* gdi/range/range_list_is_id_within()`
`* gdi/range/range_list_move_first_n_ids()`
`* gdi/range/range_list_print_to_string()`
`* gdi/range/range_list_remove_id()`
`* gdi/range/range_correct_end()`
`* gdi/range/range_get_all_ids()`
`* gdi/range/range_get_number_of_ids()`
`* gdi/range/range_is_overlapping()`
`* gdi/range/range_is_id_within()`
`* gdi/range/range_set_all_ids()`
`* gdi/range/range_sort_uniq_compress()`
`* gdi/job/JB_Type`
`******************************************************************************/`

### Documentation of a TCL function

#`****** library/control/ps_grep() ******`
`#`
`# NAME`
`# ps_grep -- call get_ps_info and return only expected ps information`
`#`
`# SYNOPSIS`
`# ps_grep { forwhat { host "local" } { variable ps_info } }`
`#`
`# FUNCTION`
`# This procedure will call the get_ps_info procedure. It will parse the`
`# get_ps_info result for the given strings and return only those process`
`# ids which match.`
`#`
`# INPUTS`
`# forwhat - search string (e.g. binary name)`
`# { host "local" } - host on which the ps command should be called`
`# { variable ps_info } - variable name to store the result (default ps_info)`
`#`
`# RESULT`
`# returns a list of indexes where the search string matches the ps output.`
`#`
`# EXAMPLE`
`#`
`# set myprocs [ ps_grep "execd" "fangorn" ]`
`#`
`# puts "execd's on fangorn index list: $myprocs"`
`#`
`# foreach elem $myprocs {`
`# puts $ps_info(string,$elem)`
`# }`
`#`
`# output of example:`
`#`
`# execd's on fangorn index list: 34 39 50 59 61`
`# 2530 140 1 259 S Sep12 1916 00:00:14 /sge_s/glinux/sge_execd`
`# 7700 142 1 339 S Sep13 2024 00:03:49 /vol2/bin/glinux/sge_execd`
`# 19159 0 1 0 S Sep14 1772 00:31:09 /vol/bin/glinux/sgeee_execd`
`# 24148 0 1 0 S Sep14 2088 00:06:23 bin/glinux/sge_execd`
`# 15085 0 1 0 S Sep14 1904 00:27:04 /vol2/glinux/sgeee_execd`
`#`
`# NOTES`
`# look at get_ps_info procedure for more information!`
`#`
`# SEE ALSO`
`# library/control/get_ps_info()`
`#*******************************`

### Documentation of a shell script

`#****** utilities/preemption/high_prolog.sh ***************************************`
`#`
`# NAME`
`# high_prolog.sh -- prolog for high priority queues`
`#`
`# SYNOPSIS`
`# high_prolog.sh`
`#`
`# FUNCTION`
`# If the number of slots occupied in queues belonging to the preemption`
`# mechanism exceeds the number of slots reserved, an appropriate number`
`# of low priority jobs/tasks are suspended.`
`# The jobid's of the jobs suspended are written to a job context variable`
`# "suspended" separated by a colon (:).`
`#`
`# If the number of slots occupied is equal or exceeds the number of`
`# reserved slots, the low priority queues are disabled.`
`#`
`# SEE ALSO`
`# utilities/preemption/--Preemption`
`# utilities/preemption/high_epilog.sh`
`# utilities/preemption/low_prolog.sh`
`# utilities/preemption/low_epilog.sh`
`#`
`#***************************************************************************`