Commit graph

486 commits

Author SHA1 Message Date
Matthew Kenigsberg b00d2a8f3e
nixos/testing: remove unused function runInMachine
runInMachine and runInMachineWithX are not used for any tests and can be
removed
2021-09-08 06:25:20 -05:00
Graham Christensen 076f6e2d94 nixos/make-zfs-image: init
This is a private interface for internal NixOS  use. It is similar
to `make-disk-image` except it is much more opinionated about what
kind of disk image it'll make.

Specifically, it will always create *two* disks:

1. a `boot` disk formatted with FAT in a hybrid GPT mode.
2. a `root` disk which is completely owned by a single zpool.

The partitioning and FAT decisions should make the resulting images
bootable under EFI or BIOS, with systemd-boot or grub.

The root disk's zpools options are highly customizable, including
fully customizable datasets and their options.

Because the boot disk and partition are highly opinionated, it is
expected that the `boot` disk will be mounted at `/boot`. It is
always labeled ESP even on BIOS boot systems.

In order for the datasets to be mounted properly, the `datasets`
passed in to `make-zfs-image` are turned in to NixOS configuration
stored at /etc/nixos/configuration.nix inside the VM.
NOTE: The function accepts a system configuration in the `config`
argument. The *caller* must manually configure the system
in `config` to have each specified `dataset` be represented
by a corresponding `fileSystems` entry.

One way to test the resulting images is with qemu:

```sh
boot=$(find ./result/ -name '*.boot.*');
root=$(find ./result/ -name '*.root.*');

echo '`Ctrl-a h` to get help on the monitor';
echo '`Ctrl-a x` to exit';

qemu-kvm \
    -nographic \
    -cpu max \
    -m 16G \
    -drive file=$boot,snapshot=on,index=0,media=disk \
    -drive file=$root,snapshot=on,index=1,media=disk \
    -boot c \
    -net user \
    -net nic \
    -msg timestamp=on
```
2021-08-25 10:42:35 -04:00
David Arnold 2832896c5b
nixos/test: some test fixes in succession of #125992 2021-08-20 13:38:22 -05:00
David Arnold db614e11d6
nixos/tests/test-driver: better control test env symbols
Previous to this commit, the entire test driver environment was shared
with the actual python test environment.

This is a hefty api surface. This commit selectively exposes only those
symbols to the test environment that are actually meant to be used by
tests.
2021-08-19 23:55:26 -05:00
Maximilian Bosch 0362d572b2
Merge pull request #133675 from Ma27/test-driver-compat
nixos/test-driver: minor fixes for `nixos-build-vms(8)`
2021-08-16 14:07:01 +02:00
Robert Hensing fbafeb7ad5 treewide: runCommandNoCC -> runCommand
This has been synonymous for ~5y.
2021-08-15 17:36:41 +02:00
Maximilian Bosch 1ba3f7927f
nixos/testing: only create nixos-run-vms for nixos-build-vms(8)
This is the case when the test-script is empty. `nixos-build-vms(8)` is
primarily supposed to be used as tool to test changes or to reproduce
bugs (IMHO) where "just spinning up a few VMs" is the primary use-case.

In the ongoing discussion about these changes[1] it was suggested to
only expose it when needed (i.e. in the case I described above) to keep
the API surface as slim as possible.

[1] https://github.com/NixOS/nixpkgs/pull/133675#discussion_r688112485
2021-08-13 17:39:49 +02:00
Maximilian Bosch e1ec5acd31
nixos/test-driver: start interactive mode if testScript is empty
This is relevant for `nixos-build-vms(8)` which doesn't have a
test-script. In that case it's more intuitive to directly go into the
interactive mode which is IMHO more intuitive.
2021-08-12 23:01:03 +02:00
Maximilian Bosch 260d9cc7e1
nixos/testing: re-add nixos-run-vms script
Originally removed in 926fb93968. This
one is actually quite useful for `nixos-build-vms(8)`.
2021-08-12 22:50:29 +02:00
lewo 7aa78642c5
Merge pull request #125979 from blaggacao/nixos-test-ref/03-normalse-the-python-entrypoint
nixos/test-driver: normalize the python entrypoint
2021-08-08 10:24:47 +02:00
David Arnold 926fb93968
nixos/tests/test-driver: normalise test driver entrypoint(s)
Previously the driver was configured exclusively through convoluted
environment variables.

Now the driver's defaults are configured through env variables.

Some additional concerns are in the github comments of this PR.
2021-08-05 19:07:11 -05:00
David Arnold dc060ff8b2
nixos/testing: fix invalid node names detection
the use of python further restricts possible RFC1035 host labels since
dash is not allowed for use in python identifiers.

The previous implementation of this check was flawed, since it did not
check the `hostName` value that is actually used to construe the
identifier, but the node name, which can be anything, e.g. just `machine`.

The previous implementation, by further restricting RFC1035 labels, only
for the sake of testing seems to be an unacceptable restriction and should
be addressed separately.
2021-07-28 15:37:57 -05:00
Ingo Blechschmidt 5143ab9f74 tigervnc, tightvnc: add basic tests
Co-Authored-By: Ingo Blechschmidt <iblech@web.de>
2021-07-20 15:22:31 +02:00
misuzu 702a1c23a4 nixos/test-driver: allow overriding qemu binary in create_startcommand 2021-07-14 12:47:12 +03:00
Robert Hensing c1a75084ea
Merge pull request #127744 from cransom/disk-image-faster
make-disk-image: Reduce forking
2021-06-26 08:31:12 +02:00
Jörg Thalheim 48aec70ea1
nixos/test-driver: replace termlib with socat
telnetlib does not handle unicode, which is annoying when using systemctl.
Also this gives us a nice readline with history.
2021-06-22 09:08:57 +02:00
Casey Ransom c3f28aed94 make-disk-image: Reduce forking
The disk image calculator was using find + exec forking du for every
file in the disk image, making it very slow. Change du to accept files,
nul delimeted on stdin to speed it back up.

Before change:
nix-build nixos/tests/image-contents.nix  9.71s user 1.06s system 8% cpu 2:13.11 total

After change:
nix-build nixos/tests/image-contents.nix  9.93s user 1.23s system 21% cpu 51.601 total
2021-06-21 22:58:07 -04:00
David Arnold 8bbdff4581
nixosTest: Force system.nixos.revision constant
nixos tests are blended with other system configurations, hence
their settings must be either enforced or defaulted.

This particular setting is set via lib.nixosSystem as
`system.nixos.revision = final.mkIf (self ? rev) self.rev;` which would
mean that without this change no flake generated nixos could be blended
with nixos testing.

This setting was made previously constant in 
169c6b4b14 in order to avoid pointless
rebuilds of the testing VMs, but was set without enforcing it.
2021-06-21 11:08:26 -05:00
Kim Lindberger 410496d0f3
Merge pull request #127309 from jakobrs/nixos-normalise-null
Resolve evaluation failure when filesystem device option is null
2021-06-18 17:15:08 +02:00
aszlig bc0997489b
nixos/test-driver: Fix passing passthru attribute
Apparently this looks like it was forgotten when doing commit
3884ff70ba, which refactored the test
runner and driver a bit.

The passthru argument actually was correctly reintroduced in
setupDriverForTest, but the actual makeTest function didn't use it.

This fixes the nixpkgs tarball job, which previously failed with:

  attribute 'elkPackages' missing, at /build/source/pkgs/tools/misc/logstash/6.x.nix:58:30

Signed-off-by: aszlig <aszlig@nix.build>
Acked-by: David Arnold <dar@xoe.solutions>
Fixes: https://github.com/NixOS/nixpkgs/issues/127274
Merges: https://github.com/NixOS/nixpkgs/pull/127346
2021-06-18 14:50:15 +02:00
jakobrs 7a0c1728ac nixos/lib: Handle null device correctly 2021-06-18 08:32:45 +02:00
Jacek Galowicz a6d8488c58
Merge pull request #125973 from blaggacao/nixos-test-ref/02-normalise-test-driver-nix-expressions
nixos test ref/02 normalise test driver nix expressions
2021-06-16 18:59:23 +02:00
Kim Lindberger 26706834a5
Merge pull request #86967 from jakobrs/more-general-fsbefore
nixos/lib/utils: Add `fileSystems.<name>.depends` option and generalise fsBefore (fixes #86955)
2021-06-14 10:50:07 +02:00
David Arnold 3884ff70ba
nixos/tests/test-driver: cleanup nix expression
Less nesting, where that improves readability. More nesteing, where
that improves readability, but most importantly:

Expose individual functions separately so that they can be more easily
built directly, eg.:

`nix build --impure --expr '(import ./testing-python.nix {system = builtins.currentSystem;}).mkTestDriver'`
2021-06-13 15:47:51 -05:00
jakobrs b07602a604 nixos/lib, nixos/filesystems: Make fsBefore more stable, and add depends option 2021-06-08 18:51:31 +02:00
David Arnold 03bfa29891
nixos/tests/test-driver: cleanup "dead" code (USE_SERIAL)
At nixpkgs root:

`rg redirectSerial ./` does not result in any other match
nor does
`rg USE_SERIAL ./` except for an unrelated match in:

pkgs/tools/graphics/argyllcms/default.nix
2021-06-06 12:58:58 -05:00
talyz b7749c7671
nixos/test-driver: Run commands with error handling
Bash's standard behavior of not propagating non-zero exit codes
through a pipeline is unexpected and almost universally
unwanted. Default to setting `pipefail` for the command being run;
it can still be turned off by prefixing the pipeline with
`set +o pipefail` if needed.

Also, set `errexit` and `nonunset` options to make the first command
of consecutive commands separated by `;` fail, and disallow
dereferencing unset variables respectively.
2021-06-05 18:44:42 +02:00
Frederik Rietdijk 3edde6562e make-test-python: disallow aliases
When importing Nixpkgs within Nixpkgs, we should not consider aliases
to ensure we don't rely on them internally.

There are probably more places that need to be converted.
2021-06-03 11:03:31 -07:00
Domen Kožar f88be6e55c
Merge pull request #125372 from Synthetica9/shell_interact
nixos/tests/test-driver: add shell_interact
2021-06-03 11:46:01 +02:00
Patrick Hilhorst fd739c4dee
nixos/tests/test-driver: make it clear when shell is ready
Co-authored-by: Domen Kožar <domen@enlambda.com>
2021-06-03 11:20:26 +02:00
Patrick Hilhorst 9469433e34
nixos/tests/test-driver: document shell_interact 2021-06-02 14:49:59 +02:00
Patrick Hilhorst 5a589b5ba8
nixos/tests/test-driver: add shell_interact 2021-06-02 14:20:04 +02:00
Damien Diederen 7c2adb1d5c nixos/lib/make-ext4-fs: Fix: `resize2fs -M' can leave insufficient slack
The root filesystem resizing step, `resize2fs -M', does not provide any
control over the amount of slack left in the result.  It can produce an
arbitrarily tight fit, depending on how well the payload aligns with
ext4 data structures.

This is problematic, as NixOS must create a few files and directories
during its first boot, before the root is enlarged to match the size of
the containing SD card.

An overly tight fit can cause failures in the first stage:

    mkdir: can't create directory '/mnt-root/proc': No space left on device

or in the second stage:

    install: cannot create directory '/var': No space left on device

A previous version of `make-ext4-fs' (before PR #79368) was explicitly
"reserving" 16 MiB of free space in the final filesystem.  Manually
calculating the size of an ext4 filesystem is a perilous endeavor,
however, and the method it employed was apparently unreliable.

Reverting is consequently not a good option.

A solution would be to create some sort of "balloon" occupying inodes
and blocks in the image prior to invoking `resize2fs -M', and to remove
these temporary files/directories before the compression step.

This changeset takes the simpler approach of simply dropping the
resizing step.

Note that this does *not* result in a larger image in general, as the
current procedure does not truncate the `.img' file anyway.  In fact, it
has been observed to yield *smaller* compressed images---probably
because of some "noise" left after resizing.  E.g., before-vs-after:

    -r--r--r-- 2 root root 607M  1. Jan 1970  nixos-sd-image-21.11pre-git-x86_64-linux.img.zst

    -r--r--r-- 2 root root 606M  1. Jan 1970  nixos-sd-image-21.11pre-git-x86_64-linux.img.zst
2021-05-31 21:35:51 +02:00
Andreas Rammhold d07f52bf81
nixos/test-driver: mention the elapsed time when it times out
For now you had to know that the actions are retried for 900s when
seeing an error like

> Traceback (most recent call last):
>   File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 927, in run_tests
>     exec(tests, globals())
>   File "<string>", line 1, in <module>
>   File "<string>", line 31, in <module>
>   File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 565, in wait_for_file
>     retry(check_file)
>   File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 142, in retry
>     raise Exception("action timed out")
> Exception: action timed out

in your (hydra) build failure. Due to the absence of timestamps you were
left guessing if the machine was just slow, someone passed a low timeout
value (which they couldn't until now) or whatever might have happened.

By making this error a bit more descriptive (by including the elapsed
time) these hopefully become more useful.
2021-05-30 17:26:13 +02:00
Jacek Galowicz 5d05391846
Merge pull request #123823 from misuzu/test-driver-usb-boot-speedup
nixos/test-driver: use usb-ehci controller instead of piix3-usb-uhci
2021-05-21 20:47:47 +02:00
misuzu b2319b086c nixos/test-driver: use usb-ehci controller instead of piix3-usb-uhci
On my system this change offers ~5X speed up of
nixosTests.boot.biosUsb and nixosTests.boot.uefiUsb tests.
2021-05-20 22:33:08 +03:00
Robert Hensing a37d157601
Merge pull request #123052 from xoe-labs/da-test-vm-innteractive-log-switch
nixos/testing: add interactive serial stdout logs switch and dim them
2021-05-20 15:32:54 +02:00
Robert Hensing 76a7840f5f
Merge pull request #117275 from hercules-ci/nixosTest-remove-nixpkgs-commit-hash
nixosTest: Make system.nixos.revision constant
2021-05-20 10:40:59 +02:00
David Arnold 8ee31be5dd
nixos/testing: add interactive serial stdout logs switch and dim them 2021-05-14 20:36:00 -04:00
Robert Hensing 75c4fc1c8b nixos/testing-python.nix: Move makeWrapper to nativeBuildInputs 2021-05-09 03:04:03 +02:00
Robert Hensing 56d9637119
nixos/testing: Set up scope for testScript linter
Our test driver exposes a bunch of variables and functions, which
pyflakes doesn't recognise by default because it assumes that the test
script is executed standalone. In reality however the test driver script
is using exec() on the testScript.

Fortunately pyflakes has $PYFLAKES_BUILTINS, which are the attributes
that are globally available on all modules to be checked. Since we only
have one module, using this environment variable is fine as opposed to
my first approach to this, which tried to use the unstable internal API
of pyflakes.

The attributes are gathered by the main derivation of the test driver,
because we don't want to end up defining a new attribute in the test
driver module just to being confused why using it in a test will result
in an error.

Another way we could have gathered these attributes would be in
mkDriver, which is where the linting takes place. However, we do have a
different set of Python dependencies in scope and duplicating these will
again just cause confusion over having it at one location only.

Signed-off-by: aszlig <aszlig@nix.build>
Co-Authored-By: aszlig <aszlig@nix.build>
2021-05-09 02:26:51 +02:00
Robert Hensing 71087b2bc4
nixos/testing-python.nix: Expose driver
(cherry picked from commit a2c9220568648b4528154ebd8e657add243ed0b4)
2021-05-09 02:26:40 +02:00
aszlig c362a28fcf
nixos/testing: Switch from black to pyflakes
So far, we have used "black" for formatting the test code, which is
rather strict and opinionated and when used inline in Nix expressions it
creates all sorts of trouble.

One of the main annoyances is that when using strings coming from Nix
expressions (eg. store paths or option definitions from NixOS modules),
completely unrelated changes could cause tests to fail, since eg. black
wants lines to be broken.

Another downside of enforcing a certain kind of formatting is that it
makes the Nix expression code inconsistent because we're mixing two
spaces of indentation (common in nixpkgs) with four spaces of
indentation as defined in PEP-8. While this is perfectly fine for
standalone Python files, it really looks ugly and inconsistent IMO when
used within Nix strings.

What we actually want though is a linter that catches problems early on
before actually running the test, because this is *actually* helping in
development because running the actual VM test takes much longer.

This is the reason why I switched from black to pyflakes, because the
latter actually has useful checks, eg. usage of undefined variables,
invalid format arguments, duplicate arguments, shadowed loop vars and
more.

Signed-off-by: aszlig <aszlig@nix.build>
Closes: https://github.com/NixOS/nixpkgs/issues/72964
2021-05-09 02:26:37 +02:00
Michael Weiss 3c1a76611e
nixos/test-driver: Allow interactive testing on Wayland-only setups
On my system I have XWayland disabled and therefore only WAYLAND_DISPLAY
is set. This ensures that the graphical output will still be enabled on
such setups (both Wayland and X11 are supported by the viewer).
2021-05-04 16:23:02 +02:00
Robert Hensing 162b8fba12 testing-python.nix: document runTests pos argument 2021-05-03 13:33:41 +02:00
Robert Hensing b2f44e9aeb
Merge pull request #112504 from hercules-ci/fix-nixosTest-meta-position
nixosTest: fix meta.position
2021-05-03 11:50:57 +02:00
Alyssa Ross a8afbb45c1 treewide: use lib.warnIf where appropriate 2021-04-28 21:44:21 +00:00
Lassulus ee04d772e4
Merge pull request #120489 from samueldr/fix/make-disk-image-auto-size
Fix make disk image automatic size
2021-04-26 10:34:15 +02:00
Lassulus cdddbf59ea
Merge pull request #120251 from mschwaig/fix-make-disk-image-for-efi-2
make-disk-image: fix broken EFI image builds
2021-04-26 10:04:00 +02:00
Martin Schwaighofer f20ae954d5 make-disk-image: fix broken EFI image builds
Work around missing /dev files inside runInLinuxVM by creating a
symlink before calling nixos-enter.

This fixes https://github.com/NixOS/nixpkgs/issues/93381.
I ran into this issue when trying to create a VMware image that boots from EFI.

Thanks @colemickens for reporting this and @danielfullmer for fixing the same thing in in qemu-vm.nix (37676e77cb) and explaining what the issue was.
2021-04-26 01:12:10 +02:00