nixos-render-docs: add structural includes, use for manual

this adds support for structural includes to nixos-render-docs.
structural includes provide a way to denote the (sub)structure of the
nixos manual in the markdown source files, very similar to how we used
literal docbook blocks before, and are processed by nixos-render-docs
without involvement of xml tooling. this will ultimately allow us to
emit the nixos manual in other formats as well, e.g. html, without going
through docbook at all.

alternatives to this source layout were also considered:

a parallel structure using e.g. toml files that describe the document
tree and links to each part is possible, but much more complicated to
implement than the solution chosen here and makes it harder to follow
which files have what substructure. it also makes it much harder to
include a substructure in the middle of a file.

much the same goes for command-line arguments to the converter, only
that command-lined arguments are even harder to specify correctly and
cannot be reasonably pulled together from many places without involving
another layer of tooling. cli arguments would also mean that the manual
structure would be fixed in default.nix, which is also not ideal.
This commit is contained in:
pennae 2023-02-11 08:48:06 +01:00
parent 8b7f0e559a
commit 1229e735ac
32 changed files with 427 additions and 431 deletions

View file

@ -21,8 +21,8 @@ which is often not what you want. By contrast, in the imperative
approach, containers are configured and updated independently from the
host system.
```{=docbook}
<xi:include href="imperative-containers.section.xml" />
<xi:include href="declarative-containers.section.xml" />
<xi:include href="container-networking.section.xml" />
```{=include=} sections
imperative-containers.section.md
declarative-containers.section.md
container-networking.section.md
```

View file

@ -0,0 +1,14 @@
# Administration {#ch-running}
This chapter describes various aspects of managing a running NixOS system, such as how to use the {command}`systemd` service manager.
```{=include=} chapters
service-mgmt.chapter.md
rebooting.chapter.md
user-sessions.chapter.md
control-groups.chapter.md
logging.chapter.md
cleaning-store.chapter.md
containers.chapter.md
troubleshooting.chapter.md
```

View file

@ -1,21 +0,0 @@
<part xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="ch-running">
<title>Administration</title>
<partintro xml:id="ch-running-intro">
<para>
This chapter describes various aspects of managing a running NixOS system,
such as how to use the <command>systemd</command> service manager.
</para>
</partintro>
<xi:include href="../from_md/administration/service-mgmt.chapter.xml" />
<xi:include href="../from_md/administration/rebooting.chapter.xml" />
<xi:include href="../from_md/administration/user-sessions.chapter.xml" />
<xi:include href="../from_md/administration/control-groups.chapter.xml" />
<xi:include href="../from_md/administration/logging.chapter.xml" />
<xi:include href="../from_md/administration/cleaning-store.chapter.xml" />
<xi:include href="../from_md/administration/containers.chapter.xml" />
<xi:include href="../from_md/administration/troubleshooting.chapter.xml" />
</part>

View file

@ -3,10 +3,10 @@
This chapter describes solutions to common problems you might encounter
when you manage your NixOS system.
```{=docbook}
<xi:include href="boot-problems.section.xml" />
<xi:include href="maintenance-mode.section.xml" />
<xi:include href="rollback.section.xml" />
<xi:include href="store-corruption.section.xml" />
<xi:include href="network-problems.section.xml" />
```{=include=} sections
boot-problems.section.md
maintenance-mode.section.md
rollback.section.md
store-corruption.section.md
network-problems.section.md
```

View file

@ -11,8 +11,8 @@ manual](https://nixos.org/nix/manual/#chap-writing-nix-expressions), but
here we give a short overview of the most important constructs useful in
NixOS configuration files.
```{=docbook}
<xi:include href="config-file.section.xml" />
<xi:include href="abstractions.section.xml" />
<xi:include href="modularity.section.xml" />
```{=include=} sections
config-file.section.md
abstractions.section.md
modularity.section.md
```

View file

@ -0,0 +1,27 @@
# Configuration {#ch-configuration}
This chapter describes how to configure various aspects of a NixOS machine through the configuration file {file}`/etc/nixos/configuration.nix`. As described in [](#sec-changing-config), changes to this file only take effect after you run {command}`nixos-rebuild`.
```{=include=} chapters
config-syntax.chapter.md
package-mgmt.chapter.md
user-mgmt.chapter.md
file-systems.chapter.md
x-windows.chapter.md
wayland.chapter.md
gpu-accel.chapter.md
xfce.chapter.md
networking.chapter.md
linux-kernel.chapter.md
subversion.chapter.md
```
```{=include=} chapters
@MODULE_CHAPTERS@
```
```{=include=} chapters
profiles.chapter.md
kubernetes.chapter.md
```
<!-- Apache; libvirtd virtualisation -->

View file

@ -1,31 +0,0 @@
<part xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="ch-configuration">
<title>Configuration</title>
<partintro xml:id="ch-configuration-intro">
<para>
This chapter describes how to configure various aspects of a NixOS machine
through the configuration file
<filename>/etc/nixos/configuration.nix</filename>. As described in
<xref linkend="sec-changing-config" />, changes to this file only take
effect after you run <command>nixos-rebuild</command>.
</para>
</partintro>
<xi:include href="../from_md/configuration/config-syntax.chapter.xml" />
<xi:include href="../from_md/configuration/package-mgmt.chapter.xml" />
<xi:include href="../from_md/configuration/user-mgmt.chapter.xml" />
<xi:include href="../from_md/configuration/file-systems.chapter.xml" />
<xi:include href="../from_md/configuration/x-windows.chapter.xml" />
<xi:include href="../from_md/configuration/wayland.chapter.xml" />
<xi:include href="../from_md/configuration/gpu-accel.chapter.xml" />
<xi:include href="../from_md/configuration/xfce.chapter.xml" />
<xi:include href="../from_md/configuration/networking.chapter.xml" />
<xi:include href="../from_md/configuration/linux-kernel.chapter.xml" />
<xi:include href="../from_md/configuration/subversion.chapter.xml" />
<xi:include href="../generated/modules.xml" xpointer="xpointer(//section[@id='modules']/*)" />
<xi:include href="../from_md/configuration/profiles.chapter.xml" />
<xi:include href="../from_md/configuration/kubernetes.chapter.xml" />
<!-- Apache; libvirtd virtualisation -->
</part>

View file

@ -40,7 +40,7 @@ configuration use `pkgs` prefix (variable).
To "uninstall" a package, simply remove it from
[](#opt-environment.systemPackages) and run `nixos-rebuild switch`.
```{=docbook}
<xi:include href="customizing-packages.section.xml" />
<xi:include href="adding-custom-packages.section.xml" />
```{=include=} sections
customizing-packages.section.md
adding-custom-packages.section.md
```

View file

@ -36,7 +36,7 @@ dropping you to the emergency shell. You can make a mount asynchronous
and non-critical by adding `options = [ "nofail" ];`.
:::
```{=docbook}
<xi:include href="luks-file-systems.section.xml" />
<xi:include href="sshfs-file-systems.section.xml" />
```{=include=} sections
luks-file-systems.section.md
sshfs-file-systems.section.md
```

View file

@ -3,14 +3,14 @@
This section describes how to configure networking components
on your NixOS machine.
```{=docbook}
<xi:include href="network-manager.section.xml" />
<xi:include href="ssh.section.xml" />
<xi:include href="ipv4-config.section.xml" />
<xi:include href="ipv6-config.section.xml" />
<xi:include href="firewall.section.xml" />
<xi:include href="wireless.section.xml" />
<xi:include href="ad-hoc-network-config.section.xml" />
<xi:include href="renaming-interfaces.section.xml" />
```{=include=} sections
network-manager.section.md
ssh.section.md
ipv4-config.section.md
ipv6-config.section.md
firewall.section.md
wireless.section.md
ad-hoc-network-config.section.md
renaming-interfaces.section.md
```
<!-- TODO: OpenVPN, NAT -->

View file

@ -12,7 +12,7 @@ NixOS has two distinct styles of package management:
`nix-env` command. This style allows mixing packages from different
Nixpkgs versions. It's the only choice for non-root users.
```{=docbook}
<xi:include href="declarative-packages.section.xml" />
<xi:include href="ad-hoc-packages.section.xml" />
```{=include=} sections
declarative-packages.section.md
ad-hoc-packages.section.md
```

View file

@ -19,16 +19,16 @@ install media, many are actually intended to be used in real installs.
What follows is a brief explanation on the purpose and use-case for each
profile. Detailing each option configured by each one is out of scope.
```{=docbook}
<xi:include href="profiles/all-hardware.section.xml" />
<xi:include href="profiles/base.section.xml" />
<xi:include href="profiles/clone-config.section.xml" />
<xi:include href="profiles/demo.section.xml" />
<xi:include href="profiles/docker-container.section.xml" />
<xi:include href="profiles/graphical.section.xml" />
<xi:include href="profiles/hardened.section.xml" />
<xi:include href="profiles/headless.section.xml" />
<xi:include href="profiles/installation-device.section.xml" />
<xi:include href="profiles/minimal.section.xml" />
<xi:include href="profiles/qemu-guest.section.xml" />
```{=include=} sections
profiles/all-hardware.section.md
profiles/base.section.md
profiles/clone-config.section.md
profiles/demo.section.md
profiles/docker-container.section.md
profiles/graphical.section.md
profiles/hardened.section.md
profiles/headless.section.md
profiles/installation-device.section.md
profiles/minimal.section.md
profiles/qemu-guest.section.md
```

View file

@ -143,35 +143,26 @@ let
''
cp -r --no-preserve=all $inputs/* .
declare -a convert_args
while read -r mf; do
if [[ "$mf" = *.chapter.md ]]; then
convert_args+=("--chapter")
else
convert_args+=("--section")
fi
substituteInPlace ./manual.md \
--replace '@NIXOS_VERSION@' "${version}"
substituteInPlace ./configuration/configuration.md \
--replace \
'@MODULE_CHAPTERS@' \
${lib.escapeShellArg (lib.concatMapStringsSep "\n" (p: "${p.value}") config.meta.doc)}
substituteInPlace ./nixos-options.md \
--replace \
'@NIXOS_OPTIONS_JSON@' \
${optionsDoc.optionsJSON}/share/doc/nixos/options.json
substituteInPlace ./development/writing-nixos-tests.section.md \
--replace \
'@NIXOS_TEST_OPTIONS_JSON@' \
${testOptionsDoc.optionsJSON}/share/doc/nixos/options.json
convert_args+=("from_md/''${mf%.md}.xml" "$mf")
done < <(find . -type f -name '*.md')
nixos-render-docs manual docbook-fragment \
nixos-render-docs manual docbook \
--manpage-urls ${manpageUrls} \
"''${convert_args[@]}"
mkdir ./generated
ln -s ${optionsDoc.optionsDocBook} ./generated/options-db.xml
ln -s ${testOptionsDoc.optionsDocBook} ./generated/test-options-db.xml
printf "%s" "${version}" > ./generated/version
chmod -R u+w .
nixos-render-docs manual docbook-section \
--manpage-urls ${manpageUrls} \
./generated/modules.xml \
--section \
--section-id modules \
--chapters ${lib.concatMapStrings (p: "${p.value} ") config.meta.doc}
xmllint --xinclude --output ./manual-combined.xml ./manual.xml
--revision ${lib.escapeShellArg revision} \
./manual.md \
./manual-combined.xml
${linterFunctions}

View file

@ -0,0 +1,14 @@
# Development {#ch-development}
This chapter describes how you can modify and extend NixOS.
```{=include=} chapters
sources.chapter.md
writing-modules.chapter.md
building-parts.chapter.md
bootspec.chapter.md
what-happens-during-a-system-switch.chapter.md
writing-documentation.chapter.md
nixos-tests.chapter.md
testing-installer.chapter.md
```

View file

@ -1,20 +0,0 @@
<part xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="ch-development">
<title>Development</title>
<partintro xml:id="ch-development-intro">
<para>
This chapter describes how you can modify and extend NixOS.
</para>
</partintro>
<xi:include href="../from_md/development/sources.chapter.xml" />
<xi:include href="../from_md/development/writing-modules.chapter.xml" />
<xi:include href="../from_md/development/building-parts.chapter.xml" />
<xi:include href="../from_md/development/bootspec.chapter.xml" />
<xi:include href="../from_md/development/what-happens-during-a-system-switch.chapter.xml" />
<xi:include href="../from_md/development/writing-documentation.chapter.xml" />
<xi:include href="../from_md/development/nixos-tests.chapter.xml" />
<xi:include href="../from_md/development/testing-installer.chapter.xml" />
</part>

View file

@ -5,9 +5,9 @@ NixOS tests are kept in the directory `nixos/tests`, and are executed
(using Nix) by a testing framework that automatically starts one or more
virtual machines containing the NixOS system(s) required for the test.
```{=docbook}
<xi:include href="writing-nixos-tests.section.xml" />
<xi:include href="running-nixos-tests.section.xml" />
<xi:include href="running-nixos-tests-interactively.section.xml" />
<xi:include href="linking-nixos-tests-to-packages.section.xml" />
```{=include=} sections
writing-nixos-tests.section.md
running-nixos-tests.section.md
running-nixos-tests-interactively.section.md
linking-nixos-tests-to-packages.section.md
```

View file

@ -47,7 +47,7 @@ Most of these actions are either self-explaining but some of them have to do
with our units or the activation script. For this reason, these topics are
explained in the next sections.
```{=docbook}
<xi:include href="unit-handling.section.xml" />
<xi:include href="activation-script.section.xml" />
```{=include=} sections
unit-handling.section.md
activation-script.section.md
```

View file

@ -83,7 +83,7 @@ Keep the following guidelines in mind when you create and add a topic:
## Adding a Topic to the Book {#sec-writing-docs-adding-a-topic}
Open the parent XML file and add an `xi:include` element to the list of
Open the parent CommonMark file and add a line to the list of
chapters with the file name of the topic that you created. If you
created a `section`, you add the file to the `chapter` file. If you created
a `chapter`, you add the file to the `part` file.

View file

@ -189,14 +189,14 @@ in {
```
:::
```{=docbook}
<xi:include href="option-declarations.section.xml" />
<xi:include href="option-types.section.xml" />
<xi:include href="option-def.section.xml" />
<xi:include href="assertions.section.xml" />
<xi:include href="meta-attributes.section.xml" />
<xi:include href="importing-modules.section.xml" />
<xi:include href="replace-modules.section.xml" />
<xi:include href="freeform-modules.section.xml" />
<xi:include href="settings-options.section.xml" />
```{=include=} sections
option-declarations.section.md
option-types.section.md
option-def.section.md
assertions.section.md
meta-attributes.section.md
importing-modules.section.md
replace-modules.section.md
freeform-modules.section.md
settings-options.section.md
```

View file

@ -470,6 +470,8 @@ In that case, `numpy` is chosen from the generic `python3Packages`.
The following options can be used when writing tests.
```{=docbook}
<xi:include href="../../generated/test-options-db.xml" xpointer="test-options-list"/>
```{=include=} options
id-prefix: test-opt-
list-id: test-options-list
source: @NIXOS_TEST_OPTIONS_JSON@
```

View file

@ -0,0 +1,11 @@
# Installation {#ch-installation}
This section describes how to obtain, install, and configure NixOS for first-time use.
```{=include=} chapters
obtaining.chapter.md
installing.chapter.md
changing-config.chapter.md
upgrading.chapter.md
building-nixos.chapter.md
```

View file

@ -1,18 +0,0 @@
<part xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="ch-installation">
<title>Installation</title>
<partintro xml:id="ch-installation-intro">
<para>
This section describes how to obtain, install, and configure NixOS for
first-time use.
</para>
</partintro>
<xi:include href="../from_md/installation/obtaining.chapter.xml" />
<xi:include href="../from_md/installation/installing.chapter.xml" />
<xi:include href="../from_md/installation/changing-config.chapter.xml" />
<xi:include href="../from_md/installation/upgrading.chapter.xml" />
<xi:include href="../from_md/installation/building-nixos.chapter.xml" />
</part>

View file

@ -602,11 +602,11 @@ With a partitioned disk.
## Additional installation notes {#sec-installation-additional-notes}
```{=docbook}
<xi:include href="installing-usb.section.xml" />
<xi:include href="installing-pxe.section.xml" />
<xi:include href="installing-kexec.section.xml" />
<xi:include href="installing-virtualbox-guest.section.xml" />
<xi:include href="installing-from-other-distro.section.xml" />
<xi:include href="installing-behind-a-proxy.section.xml" />
```{=include=} sections
installing-usb.section.md
installing-pxe.section.md
installing-kexec.section.md
installing-virtualbox-guest.section.md
installing-from-other-distro.section.md
installing-behind-a-proxy.section.md
```

View file

@ -0,0 +1,53 @@
# NixOS Manual {#book-nixos-manual}
## Version @NIXOS_VERSION@
<!--
this is the top-level structure file for the nixos manual.
the manual structure extends the nixpkgs commonmark further with include
blocks to allow better organization of input text. there are six types of
include blocks: preface, parts, chapters, sections, appendix, and options.
each type except `options`` corresponds to the docbook elements of (roughly)
the same name, and can itself can further include blocks to denote its
substructure.
non-`options`` include blocks are fenced code blocks that list a number of
files to include, in the form
```{=include=} <type>
<file-name-1>
<file-name-2>
<...>
```
`options` include blocks do not list file names but contain a list of key-value
pairs that describe the options to be included and how to convert them into
elements of the manual output type:
```{=include=} options
id-prefix: <options id prefix>
list-id: <variable list element id>
source: <path to options.json>
```
-->
```{=include=} preface
preface.md
```
```{=include=} parts
installation/installation.md
configuration/configuration.md
administration/running.md
development/development.md
```
```{=include=} chapters
contributing-to-this-manual.chapter.md
```
```{=include=} appendix
nixos-options.md
release-notes/release-notes.md
```

View file

@ -1,23 +0,0 @@
<book xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="book-nixos-manual">
<info>
<title>NixOS Manual</title>
<subtitle>Version <xi:include href="./generated/version" parse="text" />
</subtitle>
</info>
<xi:include href="preface.xml" />
<xi:include href="installation/installation.xml" />
<xi:include href="configuration/configuration.xml" />
<xi:include href="administration/running.xml" />
<xi:include href="development/development.xml" />
<xi:include href="./from_md/contributing-to-this-manual.chapter.xml" />
<appendix xml:id="ch-options">
<title>Configuration Options</title>
<xi:include href="./generated/options-db.xml"
xpointer="configuration-variable-list" />
</appendix>
<xi:include href="release-notes/release-notes.xml" />
</book>

View file

@ -0,0 +1,7 @@
# Configuration Options {#ch-options}
```{=include=} options
id-prefix: opt-
list-id: configuration-variable-list
source: @NIXOS_OPTIONS_JSON@
```

View file

@ -0,0 +1,11 @@
# Preface {#preface}
This manual describes how to install, use and extend NixOS, a Linux distribution based on the purely functional package management system [Nix](https://nixos.org/nix), that is composed using modules and packages defined in the [Nixpkgs](https://nixos.org/nixpkgs) project.
Additional information regarding the Nix package manager and the Nixpkgs project can be found in respectively the [Nix manual](https://nixos.org/nix/manual) and the [Nixpkgs manual](https://nixos.org/nixpkgs/manual).
If you encounter problems, please report them on the [`Discourse`](https://discourse.nixos.org), the [Matrix room](https://matrix.to/#nix:nixos.org), or on the [`#nixos` channel on Libera.Chat](irc://irc.libera.chat/#nixos). Alternatively, consider [contributing to this manual](#chap-contributing). Bugs should be reported in [NixOS GitHub issue tracker](https://github.com/NixOS/nixpkgs/issues).
::: {.note}
Commands prefixed with `#` have to be run as root, either requiring to login as root user or temporarily switching to it using `sudo` for example.
:::

View file

@ -1,42 +0,0 @@
<preface xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xml:id="preface">
<title>Preface</title>
<para>
This manual describes how to install, use and extend NixOS, a Linux
distribution based on the purely functional package management system
<link xlink:href="https://nixos.org/nix">Nix</link>, that is composed
using modules and packages defined in the
<link xlink:href="https://nixos.org/nixpkgs">Nixpkgs</link> project.
</para>
<para>
Additional information regarding the Nix package manager and the Nixpkgs
project can be found in respectively the
<link xlink:href="https://nixos.org/nix/manual">Nix manual</link> and the
<link xlink:href="https://nixos.org/nixpkgs/manual">Nixpkgs manual</link>.
</para>
<para>
If you encounter problems, please report them on the
<literal
xlink:href="https://discourse.nixos.org">Discourse</literal>,
the <link
xlink:href="https://matrix.to/#nix:nixos.org">Matrix room</link>,
or on the <link
xlink:href="irc://irc.libera.chat/#nixos">
<literal>#nixos</literal> channel on Libera.Chat</link>.
Alternatively, consider <link
xlink:href="#chap-contributing">
contributing to this manual</link>. Bugs should be
reported in
<link
xlink:href="https://github.com/NixOS/nixpkgs/issues">NixOS
GitHub issue tracker</link>.
</para>
<note>
<para>
Commands prefixed with <literal>#</literal> have to be run as root, either
requiring to login as root user or temporarily switching to it using
<literal>sudo</literal> for example.
</para>
</note>
</preface>

View file

@ -0,0 +1,25 @@
# Release Notes {#ch-release-notes}
This section lists the release notes for each stable version of NixOS and current unstable revision.
```{=include=} sections
rl-2305.section.md
rl-2211.section.md
rl-2205.section.md
rl-2111.section.md
rl-2105.section.md
rl-2009.section.md
rl-2003.section.md
rl-1909.section.md
rl-1903.section.md
rl-1809.section.md
rl-1803.section.md
rl-1709.section.md
rl-1703.section.md
rl-1609.section.md
rl-1603.section.md
rl-1509.section.md
rl-1412.section.md
rl-1404.section.md
rl-1310.section.md
```

View file

@ -1,30 +0,0 @@
<appendix xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="ch-release-notes">
<title>Release Notes</title>
<para>
This section lists the release notes for each stable version of NixOS and
current unstable revision.
</para>
<xi:include href="../from_md/release-notes/rl-2305.section.xml" />
<xi:include href="../from_md/release-notes/rl-2211.section.xml" />
<xi:include href="../from_md/release-notes/rl-2205.section.xml" />
<xi:include href="../from_md/release-notes/rl-2111.section.xml" />
<xi:include href="../from_md/release-notes/rl-2105.section.xml" />
<xi:include href="../from_md/release-notes/rl-2009.section.xml" />
<xi:include href="../from_md/release-notes/rl-2003.section.xml" />
<xi:include href="../from_md/release-notes/rl-1909.section.xml" />
<xi:include href="../from_md/release-notes/rl-1903.section.xml" />
<xi:include href="../from_md/release-notes/rl-1809.section.xml" />
<xi:include href="../from_md/release-notes/rl-1803.section.xml" />
<xi:include href="../from_md/release-notes/rl-1709.section.xml" />
<xi:include href="../from_md/release-notes/rl-1703.section.xml" />
<xi:include href="../from_md/release-notes/rl-1609.section.xml" />
<xi:include href="../from_md/release-notes/rl-1603.section.xml" />
<xi:include href="../from_md/release-notes/rl-1509.section.xml" />
<xi:include href="../from_md/release-notes/rl-1412.section.xml" />
<xi:include href="../from_md/release-notes/rl-1404.section.xml" />
<xi:include href="../from_md/release-notes/rl-1310.section.xml" />
</appendix>

View file

@ -2,68 +2,107 @@ import argparse
import json
from abc import abstractmethod
from collections.abc import MutableMapping, Sequence
from collections.abc import Mapping, MutableMapping, Sequence
from pathlib import Path
from typing import Any, cast, NamedTuple, Optional, Union
from xml.sax.saxutils import escape, quoteattr
import markdown_it
from markdown_it.token import Token
from markdown_it.utils import OptionsDict
from .docbook import DocBookRenderer
from . import options
from .docbook import DocBookRenderer, Heading
from .md import Converter
class RenderedSection:
id: Optional[str]
chapters: list[str]
def __init__(self, id: Optional[str]) -> None:
self.id = id
self.chapters = []
class BaseConverter(Converter):
_sections: list[RenderedSection]
def __init__(self, manpage_urls: dict[str, str]):
super().__init__(manpage_urls)
self._sections = []
def add_section(self, id: Optional[str], chapters: list[Path]) -> None:
self._sections.append(RenderedSection(id))
for chpath in chapters:
try:
with open(chpath, 'r') as f:
self._md.renderer._title_seen = False # type: ignore[attr-defined]
self._sections[-1].chapters.append(self._render(f.read()))
except Exception as e:
raise RuntimeError(f"failed to render manual chapter {chpath}") from e
@abstractmethod
def finalize(self) -> str: raise NotImplementedError()
class ManualDocBookRenderer(DocBookRenderer):
# needed to check correctness of chapters.
# we may want to use front matter instead of this kind of heuristic.
_title_seen = False
_toplevel_tag: str
def __init__(self, toplevel_tag: str, manpage_urls: Mapping[str, str],
parser: Optional[markdown_it.MarkdownIt] = None):
super().__init__(manpage_urls, parser)
self._toplevel_tag = toplevel_tag
self.rules |= {
'included_sections': lambda *args: self._included_thing("section", *args),
'included_chapters': lambda *args: self._included_thing("chapter", *args),
'included_preface': lambda *args: self._included_thing("preface", *args),
'included_parts': lambda *args: self._included_thing("part", *args),
'included_appendix': lambda *args: self._included_thing("appendix", *args),
'included_options': self.included_options,
}
def render(self, tokens: Sequence[Token], options: OptionsDict,
env: MutableMapping[str, Any]) -> str:
wanted = { 'h1': 'title' }
wanted |= { 'h2': 'subtitle' } if self._toplevel_tag == 'book' else {}
for (i, (tag, kind)) in enumerate(wanted.items()):
if len(tokens) < 3 * (i + 1):
raise RuntimeError(f"missing {kind} ({tag}) heading")
token = tokens[3 * i]
if token.type != 'heading_open' or token.tag != tag:
assert token.map
raise RuntimeError(f"expected {kind} ({tag}) heading in line {token.map[0] + 1}", token)
for t in tokens[3 * len(wanted):]:
if t.type != 'heading_open' or (info := wanted.get(t.tag)) is None:
continue
assert t.map
raise RuntimeError(
f"only one {info[0]} heading ({t.markup} [text...]) allowed per "
f"{self._toplevel_tag}, but found a second in lines [{t.map[0] + 1}..{t.map[1]}]. "
"please remove all such headings except the first or demote the subsequent headings.",
t)
# books get special handling because they have *two* title tags. doing this with
# generic code is more complicated than it's worth. the checks above have verified
# that both titles actually exist.
if self._toplevel_tag == 'book':
assert tokens[1].children
assert tokens[4].children
if (maybe_id := cast(str, tokens[0].attrs.get('id', ""))):
maybe_id = "xml:id=" + quoteattr(maybe_id)
return (f'<book xmlns="http://docbook.org/ns/docbook"'
f' xmlns:xlink="http://www.w3.org/1999/xlink"'
f' {maybe_id} version="5.0">'
f' <title>{self.renderInline(tokens[1].children, options, env)}</title>'
f' <subtitle>{self.renderInline(tokens[4].children, options, env)}</subtitle>'
f' {super().render(tokens[6:], options, env)}'
f'</book>')
return super().render(tokens, options, env)
def _heading_tag(self, token: Token, tokens: Sequence[Token], i: int, options: OptionsDict,
env: MutableMapping[str, Any]) -> tuple[str, dict[str, str]]:
(tag, attrs) = super()._heading_tag(token, tokens, i, options, env)
if self._title_seen:
if token.tag == 'h1':
assert token.map is not None
raise RuntimeError(
"only one title heading (# [text...]) allowed per manual chapter "
f"but found a second in lines [{token.map[0]}..{token.map[1]}]. "
"please remove all such headings except the first, split your "
"chapters, or demote the subsequent headings to (##) or lower.",
token)
# render() has already verified that we don't have supernumerary headings and since the
# book tag is handled specially we can leave the check this simple
if token.tag != 'h1':
return (tag, attrs)
self._title_seen = True
return ("chapter", attrs | {
return (self._toplevel_tag, attrs | {
'xmlns': "http://docbook.org/ns/docbook",
'xmlns:xlink': "http://www.w3.org/1999/xlink",
})
def _included_thing(self, tag: str, token: Token, tokens: Sequence[Token], i: int,
options: OptionsDict, env: MutableMapping[str, Any]) -> str:
result = []
# close existing partintro. the generic render doesn't really need this because
# it doesn't have a concept of structure in the way the manual does.
if self._headings and self._headings[-1] == Heading('part', 1):
result.append("</partintro>")
self._headings[-1] = self._headings[-1]._replace(partintro_closed=True)
# must nest properly for structural includes. this requires saving at least
# the headings stack, but creating new renderers is cheap and much easier.
r = ManualDocBookRenderer(tag, self._manpage_urls, None)
for (included, path) in token.meta['included']:
try:
result.append(r.render(included, options, env))
except Exception as e:
raise RuntimeError(f"rendering {path}") from e
return "".join(result)
def included_options(self, token: Token, tokens: Sequence[Token], i: int, options: OptionsDict,
env: MutableMapping[str, Any]) -> str:
return cast(str, token.meta['rendered-options'])
# TODO minimize docbook diffs with existing conversions. remove soon.
def paragraph_open(self, token: Token, tokens: Sequence[Token], i: int, options: OptionsDict,
env: MutableMapping[str, Any]) -> str:
@ -76,127 +115,113 @@ class ManualDocBookRenderer(DocBookRenderer):
return f"<programlisting>\n{escape(token.content)}</programlisting>"
def fence(self, token: Token, tokens: Sequence[Token], i: int, options: OptionsDict,
env: MutableMapping[str, Any]) -> str:
# HACK for temporarily being able to replace md-to-db.sh. pandoc used this syntax to
# allow md files to inject arbitrary docbook, and manual chapters use it.
if token.info == '{=docbook}':
return token.content
info = f" language={quoteattr(token.info)}" if token.info != "" else ""
return f"<programlisting{info}>\n{escape(token.content)}</programlisting>"
class DocBookSectionConverter(BaseConverter):
__renderer__ = ManualDocBookRenderer
class DocBookConverter(Converter):
def __renderer__(self, manpage_urls: Mapping[str, str],
parser: Optional[markdown_it.MarkdownIt]) -> ManualDocBookRenderer:
return ManualDocBookRenderer('book', manpage_urls, parser)
def finalize(self) -> str:
result = []
_base_paths: list[Path]
_revision: str
for section in self._sections:
id = "id=" + quoteattr(section.id) if section.id is not None else ""
result.append(f'<section {id}>')
result += section.chapters
result.append(f'</section>')
def __init__(self, manpage_urls: Mapping[str, str], revision: str):
super().__init__(manpage_urls)
self._revision = revision
return "\n".join(result)
class ManualFragmentDocBookRenderer(ManualDocBookRenderer):
_tag: str = "chapter"
def _heading_tag(self, token: Token, tokens: Sequence[Token], i: int, options: OptionsDict,
env: MutableMapping[str, Any]) -> tuple[str, dict[str, str]]:
(tag, attrs) = super()._heading_tag(token, tokens, i, options, env)
if token.tag == 'h1':
return (self._tag, attrs | { 'xmlns:xi': "http://www.w3.org/2001/XInclude" })
return (tag, attrs)
class DocBookFragmentConverter(Converter):
__renderer__ = ManualFragmentDocBookRenderer
def convert(self, file: Path, tag: str) -> str:
assert isinstance(self._md.renderer, ManualFragmentDocBookRenderer)
def convert(self, file: Path) -> str:
self._base_paths = [ file ]
try:
with open(file, 'r') as f:
self._md.renderer._title_seen = False
self._md.renderer._tag = tag
return self._render(f.read())
except Exception as e:
raise RuntimeError(f"failed to render manual {tag} {file}") from e
raise RuntimeError(f"failed to render manual {file}") from e
def _parse(self, src: str, env: Optional[MutableMapping[str, Any]] = None) -> list[Token]:
tokens = super()._parse(src, env)
for token in tokens:
if token.type != "fence" or not token.info.startswith("{=include=} "):
continue
typ = token.info[12:].strip()
if typ == 'options':
token.type = 'included_options'
self._parse_options(token)
elif typ in [ 'sections', 'chapters', 'preface', 'parts', 'appendix' ]:
token.type = 'included_' + typ
self._parse_included_blocks(token, env)
else:
raise RuntimeError(f"unsupported structural include type '{typ}'")
return tokens
def _parse_included_blocks(self, token: Token, env: Optional[MutableMapping[str, Any]]) -> None:
assert token.map
included = token.meta['included'] = []
for (lnum, line) in enumerate(token.content.splitlines(), token.map[0] + 2):
line = line.strip()
path = self._base_paths[-1].parent / line
if path in self._base_paths:
raise RuntimeError(f"circular include found in line {lnum}")
try:
self._base_paths.append(path)
with open(path, 'r') as f:
tokens = self._parse(f.read(), env)
included.append((tokens, path))
self._base_paths.pop()
except Exception as e:
raise RuntimeError(f"processing included file {path} from line {lnum}") from e
def _parse_options(self, token: Token) -> None:
assert token.map
items = {}
for (lnum, line) in enumerate(token.content.splitlines(), token.map[0] + 2):
if len(args := line.split(":", 1)) != 2:
raise RuntimeError(f"options directive with no argument in line {lnum}")
(k, v) = (args[0].strip(), args[1].strip())
if k in items:
raise RuntimeError(f"duplicate options directive {k} in line {lnum}")
items[k] = v
try:
id_prefix = items.pop('id-prefix')
varlist_id = items.pop('list-id')
source = items.pop('source')
except KeyError as e:
raise RuntimeError(f"options directive {e} missing in block at line {token.map[0] + 1}")
if items.keys():
raise RuntimeError(
f"unsupported options directives in block at line {token.map[0] + 1}",
" ".join(items.keys()))
try:
conv = options.DocBookConverter(
self._manpage_urls, self._revision, False, 'fragment', varlist_id, id_prefix)
with open(self._base_paths[-1].parent / source, 'r') as f:
conv.add_options(json.load(f))
token.meta['rendered-options'] = conv.finalize(fragment=True)
except Exception as e:
raise RuntimeError(f"processing options block in line {token.map[0] + 1}") from e
class Section:
id: Optional[str] = None
chapters: list[str]
def __init__(self) -> None:
self.chapters = []
class SectionAction(argparse.Action):
def __call__(self, parser: argparse.ArgumentParser, ns: argparse.Namespace,
values: Union[str, Sequence[Any], None], opt_str: Optional[str] = None) -> None:
sections = getattr(ns, self.dest)
if sections is None: sections = []
sections.append(Section())
setattr(ns, self.dest, sections)
class SectionIDAction(argparse.Action):
def __call__(self, parser: argparse.ArgumentParser, ns: argparse.Namespace,
values: Union[str, Sequence[Any], None], opt_str: Optional[str] = None) -> None:
sections = getattr(ns, self.dest)
if sections is None: raise argparse.ArgumentError(self, "no active section")
sections[-1].id = cast(str, values)
class ChaptersAction(argparse.Action):
def __call__(self, parser: argparse.ArgumentParser, ns: argparse.Namespace,
values: Union[str, Sequence[Any], None], opt_str: Optional[str] = None) -> None:
sections = getattr(ns, self.dest)
if sections is None: raise argparse.ArgumentError(self, "no active section")
sections[-1].chapters.extend(map(Path, cast(Sequence[str], values)))
class SingleFileAction(argparse.Action):
def __call__(self, parser: argparse.ArgumentParser, ns: argparse.Namespace,
values: Union[str, Sequence[Any], None], opt_str: Optional[str] = None) -> None:
assert isinstance(values, Sequence)
chapters = getattr(ns, self.dest) or []
chapters.append((Path(values[0]), Path(values[1])))
setattr(ns, self.dest, chapters)
def _build_cli_db_section(p: argparse.ArgumentParser) -> None:
def _build_cli_db(p: argparse.ArgumentParser) -> None:
p.add_argument('--manpage-urls', required=True)
p.add_argument("outfile")
p.add_argument("--section", dest="contents", action=SectionAction, nargs=0)
p.add_argument("--section-id", dest="contents", action=SectionIDAction)
p.add_argument("--chapters", dest="contents", action=ChaptersAction, nargs='+')
p.add_argument('--revision', required=True)
p.add_argument('infile', type=Path)
p.add_argument('outfile', type=Path)
def _build_cli_db_fragment(p: argparse.ArgumentParser) -> None:
p.add_argument('--manpage-urls', required=True)
p.add_argument("--chapter", action=SingleFileAction, required=True, nargs=2)
p.add_argument("--section", action=SingleFileAction, required=True, nargs=2)
def _run_cli_db_section(args: argparse.Namespace) -> None:
def _run_cli_db(args: argparse.Namespace) -> None:
with open(args.manpage_urls, 'r') as manpage_urls:
md = DocBookSectionConverter(json.load(manpage_urls))
for section in args.contents:
md.add_section(section.id, section.chapters)
with open(args.outfile, 'w') as f:
f.write(md.finalize())
def _run_cli_db_fragment(args: argparse.Namespace) -> None:
with open(args.manpage_urls, 'r') as manpage_urls:
md = DocBookFragmentConverter(json.load(manpage_urls))
for kind in [ 'chapter', 'section' ]:
for (target, file) in getattr(args, kind):
converted = md.convert(file, kind)
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(converted)
md = DocBookConverter(json.load(manpage_urls), args.revision)
converted = md.convert(args.infile)
args.outfile.write_text(converted)
def build_cli(p: argparse.ArgumentParser) -> None:
formats = p.add_subparsers(dest='format', required=True)
_build_cli_db_section(formats.add_parser('docbook-section'))
_build_cli_db_fragment(formats.add_parser('docbook-fragment'))
_build_cli_db(formats.add_parser('docbook'))
def run_cli(args: argparse.Namespace) -> None:
if args.format == 'docbook-section':
_run_cli_db_section(args)
elif args.format == 'docbook-fragment':
_run_cli_db_fragment(args)
if args.format == 'docbook':
_run_cli_db(args)
else:
raise RuntimeError('format not hooked up', args)

View file

@ -231,10 +231,11 @@ class DocBookConverter(BaseConverter):
def _decl_def_footer(self) -> list[str]:
return [ "</simplelist>" ]
def finalize(self) -> str:
def finalize(self, *, fragment: bool = False) -> str:
result = []
result.append('<?xml version="1.0" encoding="UTF-8"?>')
if not fragment:
result.append('<?xml version="1.0" encoding="UTF-8"?>')
if self._document_type == 'appendix':
result += [
'<appendix xmlns="http://docbook.org/ns/docbook"',