Commit graph

863 commits

Author SHA1 Message Date
pennae ba1f533134 nixos/make-option-docs: wrap md parser+renderer into Converter
the new rendering tool will consist of a number of different Converters,
each with a possibly specialized Renderer.
2023-01-26 00:38:06 +01:00
pennae 3e45994a3b nixos/make-options-doc: don't use inspect for optionsToDocbook
the rules are fixed, and we want to support all of them (or throw a
useful error message). this will also become the base for a generic
renderer system, so let's just list all the rules statically.
2023-01-26 00:38:06 +01:00
Robert Hensing d255493e8f nixos/optionsToDocbook.py: Add docbook ns for when the appendix tag is omitted 2023-01-26 00:32:56 +01:00
pennae 0175a91aa3 nixos/make-options-doc: split docbook conversion from mergeJSON
this restores mergeJSON to its former glory if…merging json, and
extracts the MD rendering into a new script that will run instead of the
py+nix+xslt pipeline we previously ran to convert options.json to docbook.

this change alone gives a noticable performance boost when building
docs (18s instead of 27s to build optionsDocBook).

no changes to rendered output, except for a single example in the
rsnapshot module that uses hard tabs for indentation instead of spaces.
this probably isn't important.

docbook warnings remain with mergeJSON since the other processing steps
output single files instead of directories. since we'll only keep the
check until 23.11 this is probably also not important to fix.

also contains a few improvements to error reporting in the MD renderers.
2023-01-26 00:32:56 +01:00
pennae 381dcd7f9d nixos/make-options-doc: remove program argument to xslt
this doesn't seem to be set anywhere, not from make-options-doc nor
anywhere else.
2023-01-26 00:32:56 +01:00
Andrew Hoff 6ae3e7695e nixos/virtualisation: add option for explicitly named network interfaces
Adds a new option to the virtualisation modules that enables specifying
explicitly named network interfaces in QEMU VMs. The existing
`virtualisation.vlans` is still supported for cases where the name of
the network interface is irrelevant.
2023-01-22 12:20:40 -05:00
pennae e2019c49d4 nixos/make-options-doc: use markdown-it-py for rendering
only whitespace changes (mostly empty descriptions rendered as literal
line breaks and trailing space toPretty generates, but that were dropped
by mistune).
2023-01-22 17:50:09 +01:00
pennae 248e1b957d nixos/make-options-doc: render related packages with md
don't generate docbook for related packages, generate markdown instead.
this could be extended further to not even generate markdown but have
mergeJSON handle all of the rendering. markdown will work fine for now
though.
2023-01-22 17:50:09 +01:00
pennae fa8a594c56 nixos/make-options-doc: deprecate \n\n parbreak
only whitespace changes to rendered outputs, all in the vicinity or body
of admonitions. previously admonitions would not receive paragraph
breaks even when they should have because the description postprocessing
did not match on their contents.
2023-01-22 17:50:09 +01:00
pennae c93e5dde67 nixos/make-options-doc: rearrange paras in related packages/defaults/examples
this mirrors what we will be able to create with markdown. no change to
rendered outputs.
2023-01-22 17:50:09 +01:00
pennae 3a5f1ae029 nixos/make-options-doc: render default/example contents through MD
removes some trailing whitespaces from the html output, no other changes.
2023-01-22 17:50:09 +01:00
pennae 2bd8129a47 nixos/make-options-doc: make whitespace more md-compatible
markdown-it-py creates different whitespace leaders/trailers than are
currently emitted, and when we convert examples and defaults to render
via markdown the spacing will change too. this has no effect on rendered
output.
2023-01-22 17:50:09 +01:00
pennae d1aa187c0e nixos/make-options-doc: don't escape link urls twice
mistune already does escaping. it does escaping for html, but the
difference is small enough that can just ignore that we're actually
targeting docbook here.
2023-01-22 17:50:09 +01:00
pennae 2cb43da587 nixos/make-options-doc: remove trailing whitespace from strings
this was done only to make the conversion to MD easier to verify. we no
longer need it, and not keeping whitespace does not affect rendered outputs.

stripping will have to stay for now because description postprocessing
would add empty paragraphs otherwise.
2023-01-22 17:50:09 +01:00
pennae df09c21fb2 nixos/documentation: deprecate docbook option docs
following the plan in https://github.com/NixOS/nixpkgs/pull/189318#discussion_r961764451

also adds an activation script to print the warning during activation
instead of during build, otherwise folks using the new CLI that hides
build logs by default might never see the warning.
2023-01-22 17:08:40 +01:00
Guillaume Girol 33afbf39f6 treewide: switch to nativeCheckInputs
checkInputs used to be added to nativeBuildInputs. Now we have
nativeCheckInputs to do that instead. Doing this treewide change allows
to keep hashes identical to before the introduction of
nativeCheckInputs.
2023-01-21 12:00:00 +00:00
Naïm Favier bff4bb5540
Merge pull request #208762 from ncfavier/link-manpages-options-doc 2023-01-09 11:57:01 +01:00
Naïm Favier fb1bc8de93
lib/modules: make mkAliasOptionModule emit DocBook
Follow-up to https://github.com/NixOS/nixpkgs/pull/208407

Removing `mdDoc` isn't enough, we need to emit actual DocBook.
2023-01-08 22:44:18 +01:00
Domen Kožar dce3b02d44
Merge pull request #209501 from sandydoo/improve/fix-commonmark-doc-isues
nixos/make-options-doc: skip re-escaping literal expressions
2023-01-08 11:14:57 +00:00
sandydoo 18df33f5bc
nixos/make-options-doc: skip re-escaping strings 2023-01-08 08:52:17 +00:00
pennae 9da5f12ecf modules: add mkPackageOptionMD
another transitional option factory, like mkAliasOptionModuleMD.
2023-01-05 02:33:13 +01:00
pennae 4c1cfbdb84 modules: add mkAliasOptionModuleMD
mkAliasOptionModule should not default to mdDoc descriptions because
that can break out-of-tree users of documentation infrastructure. add an
explicitly-MD variant for now, to be removed some time after the MD
transition is complete.
2023-01-05 02:33:13 +01:00
Naïm Favier 4fb500d629
nixos/doc: fix some manpage references 2023-01-03 14:03:35 +01:00
Naïm Favier f61e538c43
nixos/make-options-doc: link manpages
Add links to manpages without a link using the mapping defined in
`doc/manpage-urls.json`, as is already done for the nixpkgs and NixOS
manuals.
2023-01-03 14:03:35 +01:00
Jacek Galowicz 869545857f
Merge pull request #208354 from bjornfor/nixos-test-driver-quote-shell-args
nixos/test-driver: quote some shell command lines
2022-12-30 22:41:04 +01:00
Bjørn Forsman c876f0c146 nixos/test-driver: quote some shell command lines
This makes them robust against paths with whitespace.
2022-12-30 13:19:59 +01:00
Bjørn Forsman 4c45c3f8f2 nixos/test-driver: use ASCII single quotes everywhere
A few places used Unicode U+2018/U+2019 left/right single quotes (but
not always correctly balanced). Let's just use plain ASCII single quotes
everywhere.
2022-12-30 13:00:19 +01:00
Bjørn Forsman ce5644f658 nixos/test-driver: use f-strings instead of .format()
For readability.

Suggested-by: @tfc
2022-12-29 23:51:35 +01:00
Bjørn Forsman dba49a43a0 nixos/test-driver: add optional address arg to wait_for_{open,closed}_port
This is useful for testing servers configured to listen on specific
addresses.
2022-12-29 13:50:17 +01:00
Naïm Favier 3fc528ff7f
Merge pull request #207095 from ncfavier/linux-custom-kernel 2022-12-27 17:10:43 +01:00
K900 2d3cf010fe
Revert "treewide: use nativeBuildInputs with runCommand instead of inlining" 2022-12-26 21:05:35 +03:00
Jörg Thalheim ea415d1a38
Merge pull request #207038 from NixOS/make-disk-image-for-uefi
make-disk-image: documentation, UEFI variables recording, improved determinism
2022-12-26 11:02:28 +00:00
Sandro 5af3f865e8
Merge pull request #206775 from SuperSandro2000/runCommand-nativeBuildInputs 2022-12-25 21:42:05 +01:00
Raito Bezarius 22adcaa449 nixos/lib/make-disk-image: docs, UEFI vars recording, more determinism
- Extensive documentation in NixOS manual
- Deterministic mode that fixes various identifiers relative to disk
  partitions and filesystems in ext4 case
- UEFI variable recording
2022-12-24 19:15:29 +01:00
Robert Hensing 7bfb51f9cd
Merge pull request #207441 from hercules-ci/nixos-testing-python-no-more-specialArgs
nixos/testing-python.nix: Help users who need specialArgs
2022-12-24 17:19:32 +01:00
Jacek Galowicz 9938dec682
Merge pull request #170636 from Synthetica9/wait_before_entry
nixos/test-driver: add wait_before_entry
2022-12-23 19:46:23 +01:00
Robert Hensing d9283c04d5 nixos/testing-python.nix: Help users who need specialArgs 2022-12-23 18:41:52 +01:00
Naïm Favier 84d8b9a809
nixos/systemd: document what scriptArgs is for 2022-12-22 15:17:05 +01:00
Naïm Favier f677cbabe9
nixos/tests: remove minimal-kernel module
It's not used, doesn't build, and seems like the only reason to have
`manualConfig` take `stdenv` as an argument.
2022-12-22 01:42:15 +01:00
Sandro Jäckel 26f704b545
treewide: use nativeBuildInputs with runCommand instead of inlining 2022-12-18 23:36:40 +01:00
figsoda 6bb0dbf91f nixos: fix typos 2022-12-17 19:31:14 -05:00
Artturin 05a2dfd674 lib.replaceChars: warn about being a deprecated alias
replaceStrings has been in nix since 2015(nix 1.10)

so it is safe to remove the fallback

d6d5885c15
2022-12-15 22:25:51 +02:00
Naïm Favier 109f8b4657 nixos/make-options-doc: remove dead code
The logic for pretty-printing Nix values isn't needed any more because
`optionAttrSetToDocList` returns already rendered values.
2022-12-08 17:52:52 +01:00
Naïm Favier 6a117e2759 nixos/doc: render option values using lib.generators.toPretty
Render un`_type`d defaults and examples as `literalExpression`s using
`lib.generators.toPretty` so that consumers don't have to reinvent Nix
pretty-printing. `renderOptionValue` is kept internal for now intentionally.

Make `toPretty` print floats as valid Nix values (without a tilde).

Get rid of the now-obsolete `substSpecial` function.

Move towards disallowing evaluation of packages in the manual by
raising a warning on `pkgs.foo.{outPath,drvPath}`; later, this should
throw an error. Instead, module authors should use `literalExpression`
and `mkPackageOption`.
2022-12-08 17:52:52 +01:00
sandydoo 3564228a10
nixos/make-options-doc: improve CommonMark formatting
Render the `type` attribute in a code block to match the rest of the
attributes.
2022-12-07 14:39:26 +00:00
sandydoo 926afb6f1c
nixos/make-options-doc: pretty-print literals
Unlike the XML doc renderer, the AsciiDoc and CommonMark renderers don't
pretty-print certain complex types, like literal expressions, DocBook
literals, and derivations. These types are dumped into the documentation
as JSON.

This commit parses and unwraps these types when loading the
JSON-formatted NixOS options. The AsciiDoc and CommonMark renders have
also been combined into a single script to allow code reuse.
2022-12-07 14:06:56 +00:00
Artturi 645b5a2f9f
Revert "nixos/lib: add /home to pathsNeededForBoot" 2022-12-03 15:16:42 +02:00
Lin Jian 0c70035f24
nixos/lib: add /home to pathsNeededForBoot
Before this patch, if we:

  1. add a new user in the config,
  2. mount /home seperately,
  3. not set neededForBoot for /home,
  4. and run `nixos-rebuild boot`,

the newly added user's home will not be created after a reboot. This
is because when nixos/modules/config/update-users-groups.pl is running
in stage 2 to setup users, /home is not mounted.

This patch fixes this issue.
2022-12-01 00:49:46 +08:00
Zhaofeng Li 8b2d34fa5e test-driver: Allow configuring delay for send_{key,chars} 2022-11-19 06:55:25 +01:00
Robert Hensing 833f9d5e1f
Merge pull request #200646 from hercules-ci/options-markdown-and-errors
`nixosOptionsDoc`: add `markdownByDefault`, error handling
2022-11-14 18:58:54 +01:00
Andreas Rammhold 923e404389 nixos/lib/testing-python: remove unused with pkgs;
This with statement has no users anymore and can safely be removed.
2022-11-11 16:05:00 +01:00
Robert Hensing b106ff14ed nixosOptionsDoc: Report in which option an error occurs 2022-11-11 06:47:30 +01:00
Robert Hensing 429ba6c714 nixosOptionsDoc: Add markdownByDefault parameter 2022-11-11 06:29:44 +01:00
Domen Kožar e190302018 nixos options markdown: fix html escaping
\<foo\> will often be displayed like \<foo>, for example by mkdocs.

I've tested a number of markdown renderers and they render html escape
sequences fine.
2022-11-05 21:20:01 +00:00
Jörg Thalheim 951f81c0cc
nixos/systemd-unit-options: document correct wantedBy default for user units (#199007)
Until this commit, the documentation suggested `multi-user.target`
as `wantedBy` for all services.
Since `multi-user.target` is not available for user services,
propose a different default for those in the documentation.

Co-authored-by: Naïm Favier <n@monade.li>
Co-authored-by: Florian Warzecha <liketechnik@disroot.org>
2022-11-02 07:03:22 +00:00
Florian Klink eec71f9dee
Merge pull request #186314 from ck3d/unit-add-map-control
nixos: Add option to influence override strategies in systemd units
2022-10-28 17:56:49 +02:00
Christian Kögler 6d31353099 nixos: Add unit option overrideStrategy 2022-10-27 20:19:18 +02:00
Winter f540aeda6f nixos/make-options-doc: fix JSON generation on Darwin 2022-10-25 22:58:50 -04:00
Christian Kögler c4e0d766b7 nixos/unitGenerator: fix generation for nspawn files 2022-10-23 15:57:55 +02:00
github-actions[bot] c434165354
Merge master into staging-next 2022-10-21 00:05:50 +00:00
Daniel Olsen 3251123a77 nixos/lib.escapeSystemdPath: Implement the correct algorithm for escaping names in systemd units
Co-authored-by: ajs124 <git@ajs124.de>
2022-10-20 20:12:15 +02:00
Jan Tojnar 457f28f6f8 Merge branch 'master' into staging-next
; Conflicts:
;	pkgs/development/tools/codespell/default.nix

codespell 2.2.2 switched to pyproject & setuptools_scm:
https://github.com/codespell-project/codespell/pull/2523
2022-10-19 05:24:28 +02:00
Robert Hensing 14a822f72a
Merge pull request #196281 from hercules-ci/restore-nixos-test-dx
nixos: Restore test DX
2022-10-18 11:38:37 +02:00
Robert Hensing 6259b29f29
Merge pull request #194035 from Ma27/show-option-quoting
lib/options/showOption: fix quoting of attr-names that are not identifiers
2022-10-18 11:31:54 +02:00
Robert Hensing 1d9b913088 nixos/lib/testing: Delay nodes.machine.~config~ migration
Provide a window during which both solutions are valid without
warnings, in order to fight warning fatigue, and not to push 3rd
party repo maintainers to add unnecessary compat code.
2022-10-16 16:30:21 +02:00
Robert Hensing 611f247810 nixos/tests: Generalize nix-build file.nix hack to testing-python.nix 2022-10-16 16:30:20 +02:00
github-actions[bot] 3e675d06f5
Merge staging-next into staging 2022-10-09 12:02:02 +00:00
Maximilian Bosch 2480532bd1
nixos/doc: fix build
Now we even have options like
`services.listmonk.database.settings."app.notify_emails"` shown
correctly (i.e. with quotes).
2022-10-09 10:13:21 +02:00
talyz 4df4d2a8ea genJqSecretsReplacementSnippet: Allow dots in attribute names...
...and escape quotation marks and backslashes.
2022-10-09 08:12:19 +02:00
Artturin 09226fffcf nixosOptionsDoc: buildInputs -> nativeBuildInputs
to make strictDepsByDefault work
2022-10-07 19:26:22 +03:00
Robert Hensing 70ec3b9f54
Merge pull request #193498 from hercules-ci/nixos-doc-disambiguate-test-options
nixos/doc: disambiguate test option ids
2022-10-04 21:35:49 +01:00
Adam Joseph c46bdcbaf2 nixos/lib/qemu-common.nix: set qemuSerialDevice for isMips64 2022-10-04 07:48:12 +00:00
figsoda a1d50eecab
Merge pull request #193132 from figsoda/clean-up
treewide: clean up
2022-10-01 17:03:11 -04:00
Robert Hensing 216c5dc10d nixos/doc: Disambiguate test option ids
Changing this later on would break external links into the manual.
2022-09-29 12:41:59 +02:00
Robert Hensing 91879ce160 make-options-doc: Make optionIdPrefix configurable ("opt-") 2022-09-29 12:34:06 +02:00
Robert Hensing aed1deab05 nixos/testing: Fix release.nix tests evaluation
Fixes the problem introduced by 12b3066aae
which caused nixos/release.nix to return the wrong attributes, while
intending to only affect nixos/lib's runTest.
This also removes callTest from the test options, because callTest is
only ever invoked by all-tests.nix.
2022-09-29 10:49:09 +02:00
Franz Pletz 417b476db7
Merge pull request #187817 from intelfx/nixos-containertarballs-fix-pixz
make-system-tarball: use `pixz -t`
2022-09-29 03:26:20 +02:00
Robert Hensing 12b3066aae lib.nixos.runTest: Return the test itself
(NB: the lib in the flake)

Correcting a mistake I made when removing the test matrix code
from https://github.com/NixOS/nixpkgs/pull/191540
2022-09-28 18:34:36 +02:00
Robert Hensing 7f0d934f9a
Merge pull request #191540 from hercules-ci/nixosTest-modular
nixosTest: make modular
2022-09-28 10:27:45 +01:00
figsoda dc4b44a2f3 nixos/lib/make-single-disk-zfs-image.nix: remove unused binding 2022-09-26 22:19:06 -04:00
figsoda 84bc9ff334 nixos/lib/make-multi-disk-zfs-image.nix: remove unused binding 2022-09-26 22:18:55 -04:00
Robert Hensing 1c0b9c4a48 nixos/testing/network.nix: Add network config to specialisations 2022-09-24 17:42:52 +01:00
Robert Hensing 7cdc9bc340 nixos/testing: Improve interactive docs 2022-09-24 17:42:52 +01:00
Robert Hensing 666e969da0 nixos/testing: Add nodes.config backcompat to nodes module argument 2022-09-24 17:38:10 +01:00
Robert Hensing 6205d37747 nixos/testing: Improve option docs 2022-09-24 17:38:10 +01:00
Robert Hensing 4ce93fbae8 nixos/testing-python: Add interactive variant support to makeTest 2022-09-24 17:38:10 +01:00
Robert Hensing 52bfa318e8 nixos/testing: Support mypy through regular mechanisms
Rebase / forward port of 2c8bbf33fd
2022-09-24 17:38:10 +01:00
Robert Hensing 0af6e6b0e5 nixos/testing/meta.nix: Add options, some optional 2022-09-24 17:38:09 +01:00
Robert Hensing f01fec4099 nixos/testing/network.nix: Fix specialisations onlyShorthand 2022-09-24 17:38:09 +01:00
Robert Hensing 9886db059a nixos/testing: Embrace callTest
My conception of its input was wrong. It is quite a useful construct,
even if its name is a bit weird.
2022-09-24 17:38:09 +01:00
Robert Hensing 5297d584bc nixos/lib/eval-config: Document the use of baseModules 2022-09-24 17:38:09 +01:00
Robert Hensing 57ac9dd3ae nixos/lib/build-vms.nix: Remove
It has been replaced by the modular test framework in nixos/lib/testing.

If you are looking for a way to produce a VM-test-like configuration
outside of the test framework, use the nixos/lib/testing/nixos-test-base.nix
NixOS module, possibly in combination with { _module.args.nodes = .....; }.
2022-09-24 17:38:09 +01:00
Robert Hensing 24d1d74e4e nixos/testing: Extract nixos-test-base.nix NixOS module 2022-09-24 17:38:09 +01:00
Robert Hensing 38fb09e427 testing-python.nix: Replace makeTest implementation 2022-09-24 17:38:08 +01:00
Robert Hensing b0c781cc41 nixos/testing: Move entrypoint to nixos/lib + doc 2022-09-24 17:38:08 +01:00
Robert Hensing 9e4277b970 nixos/testing/network.nix: Fix for specialisations and specialArgs.name
Before this, it relied on being able to `imports` from the `name`
module argument, which wasn't really necessary and required a
potentially quite breaking change. We can revert that now.
2022-09-21 10:55:12 +01:00
Robert Hensing a958a4aa00 nixos/testing: Add pkgs parameter
This parameter is for packages to use in VMs, unlike hostPkgs.
2022-09-21 10:55:11 +01:00
Robert Hensing 124b0c4abc nixos/testing/network.nix: Avoid deprecated .config 2022-09-21 10:55:11 +01:00
Robert Hensing b3de22483c nixos/testing-python.nix: Add evalTest
This is a decomposition of the testing-python.nix and build-vms.nix
files into modules.

By refactoring the glue, we accomplish the following:

 - NixOS tests can now use `imports` and other module system features.
    - Network-wide test setup can now be reusable; example:
       - A setup with all VMs configured to use a DNS server
       - Split long, slow tests into multiple tests that import a
         common module that has most of the setup.
    - Type checking for the test arguments
    - (TBD) "generated" options reference docs
 - Aspects that had to be wired through all the glue are now in their
   own files.
    - Chief example: interactive.nix.
    - Also: network.nix

In rewriting this, I've generally stuck as close as possible to the
existing code; copying pieces of logic and rewiring them, without
changing the logic itself.

I've made two exceptions to this rule

 - Introduction of `extraDriverArgs` instead of hardcoded
   interactivity logic.

 - Incorporation of https://github.com/NixOS/nixpkgs/pull/144110
   in testScript.nix.

I might revert the latter and split it into a new commit.
2022-09-21 10:55:11 +01:00
Robert Hensing 5b52d552cb nixos/make-options-doc: Explain docbook to markdown migration 2022-09-17 13:38:23 +01:00