Commit graph

142 commits

Author SHA1 Message Date
Armando Santos fdf74c7741
maintainers/create-amis.sh: Add more AWS regions
Add all currently available AWS regions
2022-05-17 10:20:30 +01:00
Patryk Wychowaniec 56ab4f61bc
nixos/lxd: improve tests
- Make tests/lxd.nix use NixOS's lxdMeta & lxdImage to avoid relying on
  3rd party containers such as Alpine Linux for testing purposes.
- Merge tests/lxd-image.nix into tests/lxd.nix, since now both have a
  similar structure.
- Extract duplicated inline LXD configuration into a separate file,
- Add passthru.lxd-nftables & passthru.lxd-image-server.
2022-05-03 12:24:13 +02:00
Graham Christensen 555bc5335b openstack-image-zfs: start copying the channel now that we've mostly got the expressions down 2022-04-07 15:28:28 -04:00
Graham Christensen b4c495aeff openstack-image-zfs: make the generated configuration.nix valid
Drops the wrong expandOnBoot option (defaults to all) and mark ZFS as enabled.
2022-04-07 15:28:28 -04:00
Graham Christensen 9e3dab7d2e openstack-image-zfs: build a single-image ZFS root 2022-04-07 15:28:28 -04:00
Graham Christensen 1c0b76b5c4 openstack-image-zfs: don't support vpc type, default to qcow2 2022-04-07 15:28:28 -04:00
Graham Christensen a8f41adbb7 amazon-image: use make-multi-disk-zfs-image 2022-04-07 15:28:28 -04:00
Graham Christensen b8fe792394 openstack-image-zfs: init 2022-04-07 15:28:27 -04:00
Graham Christensen e9f015480c openstack-image: make it easy to disable copying the channel to improve iteration time 2022-04-07 15:28:27 -04:00
Graham Christensen b3dfff282b openstack-image: minor cleanups
Format and remove unnecessary `with lib;`.
2022-04-07 15:28:27 -04:00
polykernel 4a9d9928dc nixos/nix-daemon: use structural settings
The `nix.*` options, apart from options for setting up the
daemon itself, currently provide a lot of setting mappings
for the Nix daemon configuration. The scope of the mapping yields
convience, but the line where an option is considered essential
is blurry. For instance, the `extra-sandbox-paths` mapping is
provided without its primary consumer, and the corresponding
`sandbox-paths` option is also not mapped.

The current system increases the maintenance burden as maintainers have to
closely follow upstream changes. In this case, there are two state versions
of Nix which have to be maintained collectively, with different options
avaliable.

This commit aims to following the standard outlined in RFC 42[1] to
implement a structural setting pattern. The Nix configuration is encoded
at its core as key-value pairs which maps nicely to attribute sets, making
it feasible to express in the Nix language itself. Some existing options are
kept such as `buildMachines` and `registry` which present a simplified interface
to managing the respective settings. The interface is exposed as `nix.settings`.

Legacy configurations are mapped to their corresponding options under `nix.settings`
for backwards compatibility.

Various options settings in other nixos modules and relevant tests have been
updated to use structural setting for consistency.

The generation and validation of the configration file has been modified to
use `writeTextFile` instead of `runCommand` for clarity. Note that validation
is now mandatory as strict checking of options has been pushed down to the
derivation level due to freeformType consuming unmatched options. Furthermore,
validation can not occur when cross-compiling due to current limitations.

A new option `publicHostKey` was added to the `buildMachines`
submodule corresponding to the base64 encoded public host key settings
exposed in the builder syntax. The build machine generation was subsequently
rewritten to use `concatStringsSep` for better performance by grouping
concatenations.

[1] - https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md
2022-01-26 21:04:50 -05:00
Timothy DeHerrera f0aec20cd7
create-amis.sh: possible deprecation 2021-11-11 09:04:29 -07:00
Timothy DeHerrera ed4170733c
amis: enable setting ami boot mode on registration
This is important since legacy bios mode is still the default for Intel
and AMD based instances on AWS. That is, even if your image is setup to
use UEFI on the OS level, the AMI will still use BIOS unless the boot
mode is explicitly set during registration.
2021-11-10 17:38:58 -07:00
Timothy DeHerrera 4d765caecd create_amis.sh: fix logic for non-zfs amis 2021-11-09 15:56:04 -08:00
Maciej Krüger 1c31f8db6a
nixosTest.lxdImage: add lxdImage test 2021-11-03 07:49:54 +01:00
Maciej Krüger 3c3349e24b
lxdImage: enable docs & xlibs in standalone image 2021-11-03 07:49:54 +01:00
Maciej Krüger 9f66f9a669
release.lxdImage: add lxdImage to hydra 2021-11-03 07:49:52 +01:00
Robert Hensing a8166c9574 nixos/maintainers/scripts: Avoid copy in example 2021-10-17 23:57:42 +02:00
Robert Hensing 0699530f08
Merge pull request #136909 from ncfavier/cleanup-defaults-examples
nixos/doc: clean up defaults and examples
2021-10-04 20:37:42 +02:00
Naïm Favier 2ddc335e6f
nixos/doc: clean up defaults and examples 2021-10-04 12:47:20 +02:00
Timothy DeHerrera 1c0a20efcf create-amis.sh: fix typo 2021-10-03 19:03:28 -07:00
Timothy DeHerrera 2d67b946b7 create-amis.sh: use status message
The progress ID is fairly useless. Status message is more useful for
humans.
2021-10-03 19:03:28 -07:00
Timothy DeHerrera 407998d15a create-amis.sh: add support for the ZFS AMIs 2021-10-03 19:03:28 -07:00
Timothy DeHerrera 1ff82fec9a create-amis.sh: allow uploading private AMIs 2021-10-03 19:03:28 -07:00
Timothy DeHerrera 0543f2d2f6 create-amis.sh: make vars overridable from env 2021-10-03 19:03:28 -07:00
Graham Christensen 71b3d18181 amazon images: extend the image-info.json to have a disks object
Having a disks object with a dictionary of all the disks and their
properties makes it easier to process multi-disk images.

Note the rename of `label` to `system_label` is because `$label`i
is something of a special token to jq.
2021-08-25 10:42:35 -04:00
Graham Christensen bd38b059ea NixOS/amazonImageZfs: init
Introduce an AWS EC2 AMI which supports aarch64 and x86_64 with a ZFS
root.

This uses `make-zfs-image` which implies two EBS volumes are needed
inside EC2, one for boot, one for root. It should not matter which
is identified `xvda` and which is `xvdb`, though I have always
uploaded `boot` as `xvda`.
2021-08-25 10:42:35 -04:00
Luke Granger-Brown 87c3b7e767 amazonImage: make statically sized again
For reasons we haven't been able to work out, the aarch64 EC2 image now
regularly exceeds the output image size on hydra.nixos.org. As a
workaround, set this back to being statically sized again.

The other images do seem to build - it's just a case of the EC2 image
now being too large (occasionally non-determinstically).
2021-05-01 02:19:42 +00:00
lassulus 5aa4273e4f treewide: use auto diskSize for make-disk-image
(cherry picked from commit f3aa040bcb)
2021-04-24 14:49:07 -04:00
Luke Granger-Brown 4fb91cbafe Revert "treewide: use auto diskSize for make-disk-image"
This reverts commit f3aa040bcb.
2021-04-24 02:38:36 +00:00
Luke Granger-Brown f521b12b0e Revert "nixos/amazon-image: (temporarily) use fixed disk size again"
This reverts commit 6a8359a92a.
2021-04-24 02:38:25 +00:00
Luke Granger-Brown 6a8359a92a nixos/amazon-image: (temporarily) use fixed disk size again
As a temporary workaround for #120473 while the image builder is patched
to correctly look up disk sizes, partially revert
f3aa040bcb for EC2 disk images only.

We retain the type allowing "auto" but set the default back to the
previous value.
2021-04-24 00:43:47 +00:00
lassulus f3aa040bcb treewide: use auto diskSize for make-disk-image 2021-04-22 19:52:49 +02:00
AmineChikhaoui 606b49721f
add new Google Cloud image for the current release
update the create-gce.sh script with the ability to create public images
out of a GS object.
2021-03-21 14:04:09 -04:00
Graham Christensen 7092dd52f8
amazonImage: Upload disks as GP3 for cheaper & faster IO (#109027)
GP3 is always faster and cheaper than GP2, so sticking to GP2 is
leaving money on the table.

https://cloudwiry.com/ebs-gp3-vs-gp2-pricing-comparison/
2021-01-11 13:54:40 -05:00
Graham Christensen 38a394bdee
Merge pull request #102174 from grahamc/ami-root-use-gpt
AMI root partition table: use GPT to support >2T partitions
2020-10-30 16:14:37 -04:00
Graham Christensen d77ddf2a40
nixos.amazonAmi: use legacy+gpt disk images to support partitions >2T 2020-10-30 15:50:25 -04:00
Graham Christensen 74a577b293
create-amis: improve wording around the service name's IAM role
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2020-10-30 12:40:17 -04:00
Graham Christensen 2bf1fc0345
create-amis: allow customizing the service role name
The complete setup on the AWS end can be configured
with the following Terraform configuration. It generates
a ./credentials.sh which I just copy/pasted in to the
create-amis.sh script near the top. Note: the entire stack
of users and bucket can be destroyed at the end of the
import.

    variable "region" {
      type = string
    }
    variable "availability_zone" {
      type = string
    }

    provider "aws" {
      region = var.region
    }

    resource "aws_s3_bucket" "nixos-amis" {
      bucket_prefix = "nixos-amis-"
      lifecycle_rule {
        enabled = true
        abort_incomplete_multipart_upload_days = 1
        expiration {
          days = 7
        }
      }
    }

    resource "local_file" "credential-file" {
      file_permission = "0700"
      filename = "${path.module}/credentials.sh"
      sensitive_content = <<SCRIPT
    export service_role_name="${aws_iam_role.vmimport.name}"
    export bucket="${aws_s3_bucket.nixos-amis.bucket}"
    export AWS_ACCESS_KEY_ID="${aws_iam_access_key.uploader.id}"
    export AWS_SECRET_ACCESS_KEY="${aws_iam_access_key.uploader.secret}"
    SCRIPT
    }

    # The following resources are for the *uploader*
    resource "aws_iam_user" "uploader" {
      name = "nixos-amis-uploader"
    }

    resource "aws_iam_access_key" "uploader" {
      user = aws_iam_user.uploader.name
    }

    resource "aws_iam_user_policy" "upload-to-nixos-amis" {
      user = aws_iam_user.uploader.name

      policy = data.aws_iam_policy_document.upload-policy-document.json
    }

    data "aws_iam_policy_document" "upload-policy-document" {
      statement {
        effect = "Allow"

        actions = [
          "s3:ListBucket",
          "s3:GetBucketLocation",
        ]

        resources = [
          aws_s3_bucket.nixos-amis.arn
        ]
      }

      statement {
        effect = "Allow"

        actions = [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
        ]

        resources = [
          "${aws_s3_bucket.nixos-amis.arn}/*"
        ]
      }

      statement {
        effect = "Allow"
        actions = [
          "ec2:ImportSnapshot",
          "ec2:DescribeImportSnapshotTasks",
          "ec2:DescribeImportSnapshotTasks",
          "ec2:RegisterImage",
          "ec2:DescribeImages"
        ]
        resources = [
          "*"
        ]
      }
    }

    # The following resources are for the *vmimport service user*
    # See: https://docs.aws.amazon.com/vm-import/latest/userguide/vmie_prereqs.html#vmimport-role
    resource "aws_iam_role" "vmimport" {
      assume_role_policy = data.aws_iam_policy_document.vmimport-trust.json
    }

    resource "aws_iam_role_policy" "vmimport-access" {
      role = aws_iam_role.vmimport.id
      policy = data.aws_iam_policy_document.vmimport-access.json
    }

    data "aws_iam_policy_document" "vmimport-access" {
      statement {
        effect = "Allow"
        actions = [
          "s3:GetBucketLocation",
          "s3:GetObject",
          "s3:ListBucket",
        ]
        resources = [
          aws_s3_bucket.nixos-amis.arn,
          "${aws_s3_bucket.nixos-amis.arn}/*"
        ]
      }
      statement {
        effect = "Allow"
        actions = [
          "ec2:ModifySnapshotAttribute",
          "ec2:CopySnapshot",
          "ec2:RegisterImage",
          "ec2:Describe*"
        ]
        resources = [
          "*"
        ]
      }
    }

    data "aws_iam_policy_document" "vmimport-trust" {
      statement {
        effect = "Allow"
        principals {
          type = "Service"
          identifiers = [ "vmie.amazonaws.com" ]
        }

        actions = [
          "sts:AssumeRole"
        ]

        condition {
          test = "StringEquals"
          variable = "sts:ExternalId"
          values = [ "vmimport" ]
        }
      }
    }
2020-10-30 12:12:08 -04:00
Graham Christensen e253de8a77
create-amis.sh: log the full response if describing the import snapshot tasks fails 2020-10-30 12:08:01 -04:00
Graham Christensen f92a883ddb
nixos ec2/create-amis.sh: shellcheck: $ is not needed in arithmetic 2020-10-30 12:08:01 -04:00
Graham Christensen 7dac8470cf
nixos ec2/create-amis.sh: shellcheck: explicitly make the additions to block_device_mappings single strings 2020-10-30 12:08:00 -04:00
Graham Christensen a66a22ca54
nixos ec2/create-amis.sh: shellcheck: read without -r mangles backslashes 2020-10-30 12:08:00 -04:00
Graham Christensen baf7ed3f24
nixos ec2/create-amis.sh: shellcheck: SC2155: Declare and assign separately to avoid masking return values. 2020-10-30 12:07:59 -04:00
Graham Christensen f5994c208d
nixos ec2/create-amis.sh: shellcheck: quote state_dir reference 2020-10-30 12:07:59 -04:00
Graham Christensen c76692192a
nixos ec2/create-amis.sh: shellcheck: quote region references 2020-10-30 12:07:49 -04:00
Lassulus d08b81c3b5
Merge pull request #89116 from wagdav/fix-args-create-amis
nixos/maintainers/scripts/ec2/create-amis.sh: fix argument check
2020-08-22 16:47:54 +02:00
Jörg Thalheim 0cb79c953d nixos/ec2: remove dependency on NIX_PATH
This is required when migrating to flakes
2020-08-16 10:56:44 +00:00
zowoq 2b5659c700 nixos/maintainers/*: editorconfig fixes 2020-08-04 00:23:54 +10:00
David Wagner 3b1ed035c3 create-amis: fix argument check
Because this script enables `set -u` when no arguments are provided bash
exits with the error:

    $1: unbound variable

instead of the helpful usage message.
2020-05-28 17:41:45 +02:00