Passing `-l$NIX_BUILD_CORES` improperly limits the overall system load.
For a build machine which is configured to run `$B` builds where each
build gets `total cores / B` cores (`$C`), passing `-l $C` to make will
improperly limit the load to `$C` instead of `$B * $C`.
This effect becomes quite pronounced on machines with 80 cores, with
40 simultaneous builds and a cores limit of 2. On a machine with this
configuration, Nix will run 40 builds and make will limit the overall
system load to approximately 2. A build machine with this many cores
can happily run with a load approaching 80.
A non-solution is to oversubscribe the machine, by picking a larger
`$C`. However, there is no way to divide the number of cores in a way
which fairly subdivides the available cores when `$B` is greater than
1.
There has been exploration of passing a jobserver in to the sandbox,
or sharing a jobserver between all the builds. This is one option, but
relatively complicated and only supports make. Lots of other software
uses its own implementation of `-j` and doesn't support either `-l` or
the Make jobserver.
For the case of an interactive user machine, the user should limit
overall system load using `$B`, `$C`, and optionally systemd's
cpu/network/io limiting features.
Making this change should significantly improve the utilization of our
build farm, and improve the throughput of Hydra.
depsBuildBuild for 'ar: command not found'
BUILDCC for failure during 'checking whether the C compiler works...'
there's still a issue but it maybe wont cause reverse dependency issues?
so overall its still an improvement
```
cd texk/texlive/tl_support && make run-texlinks
make[1]: Entering directory '/build/texlive-20210325-source/WorkDir/texk/texlive/tl_support'
if test -s /nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/texlinks && test -s /nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/share/texmf-dist/web2c/fmtutil.cnf; then \
echo "running texlinks in `pwd` via: /nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/texlinks -v -f /nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/share/texmf-dist/web2c/fmtutil.cnf -e "" /nix/stor>
PATH="/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin:/nix/store/146229z6ingpiz0g1yxs2jpq1h546ms3-gcc-wrapper-11.3.0/bin:/nix/store/sc50c3qvzlw6dmfbhxw96j9yrj47pv1v-gcc-11.3.0/bin:/nix/store/20navlfxxadg2qvkqh3vc55z27divi40-glibc-2.35-163>
else :; fi
running texlinks in /build/texlive-20210325-source/WorkDir/texk/texlive/tl_support via: /nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/texlinks -v -f /nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/s>
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpsewhich: line 1: syntax error: unexpected word (expecting ")")
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 1: ^?ELF^B^A^A^B<B7>^A<80>: not found
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 1: ^]: not found
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 1: ^]A<B0>^B<B8>^B^A^B^Fp^]p^]Ap^]A^P^B^P^B^H^D^D<A8>^B<A8>^B@<A8>^B@: not found
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 1: ^]A: not found
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 2: @@<E0><9A>@8: not found
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 5: syntax error: unterminated quoted string
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin: no write permissions. Skipping...
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 1: ^?ELF^B^A^A^B<B7>^A<80>: not found
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 1: ^]: not found
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 1: ^]A<B0>^B<B8>^B^A^B^Fp^]p^]Ap^]A^P^B^P^B^H^D^D<A8>^B<A8>^B@<A8>^B@: not found
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 1: ^]A: not found
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 2: @@<E0><9A>@8: not found
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: line 5: syntax error: unterminated quoted string
make[1]: Leaving directory '/build/texlive-20210325-source/WorkDir/texk/texlive/tl_support'
/nix/store/j00iibma3wqn88f8bi9x2c9dns3ww98v-texlinks/bin/texlinks: line 252: /nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpsewhich: cannot execute binary file: Exec format error
/nix/store/j00iibma3wqn88f8bi9x2c9dns3ww98v-texlinks/bin/texlinks: line 298: /nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: cannot execute binary file: Exec format error
/nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin: no write permissions. Skipping...
/nix/store/j00iibma3wqn88f8bi9x2c9dns3ww98v-texlinks/bin/texlinks: line 192: /nix/store/1hs29hn23cdh2qbyws95plk1fyhqyzpx-texlive-bin-aarch64-unknown-linux-gnu-2021/bin/kpseaccess: cannot execute binary file: Exec format error
@nix { "action": "setPhase", "phase": "fixupPhase" }
```
post-installation fixup
strip is /nix/store/falx4gakl6k1q64aqr4f8mvy8vfqcqaq-cctools-binutils-darwin-949.0.1/bin/strip
patching script interpreter paths in /nix/store/q1haqqnh6i2677ihad4673gc3slna3sw-texlinks.sh
/nix/store/q1haqqnh6i2677ihad4673gc3slna3sw-texlinks.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/715y5lnw12vdcc3hh9sc9vbkdi6c83x6-bash-4.4-p23/bin/sh"
sed: couldn't open temporary file /nix/store/sedqPf2C0: Operation not permitted
In https://github.com/NixOS/nix/issues/5333 I noticed that adding
a minor change to `bash` derivation triggers rebuilds in CA for
`bash` (expected) and `texlive.combine` (unexpected).
regnat debugged derivation instability down to sort on outPath.
The change avoid sorting on outPaths and uses pname of the derivations.
Tested on `R` derivation.
Closes: https://github.com/NixOS/nix/issues/5333
Co-authored-by: Vincenzo Mantova <1962985+xworld21@users.noreply.github.com>
Motivations
- not well reproducible as they disable hashed mirrors (tarballs.nixos.org)
- only reachable by modifying the file pkgs.nix, thus unused
- the space saving is analogous to using substitutes
Rust 1.50.0 incorporated a Cargo change (rust-lang/cargo#8937) in
which cargo vendor erroneously changed permissions of vendored
crates. This was fixed in Rust
1.51.0 (rust-lang/cargo#9131). Unfortunately, this means that all
cargoSha256/cargoHashes produced during the Rust 1.50.0 cycle are
potentially broken.
This change updates cargoSha256/cargoHash tree-wide.
Fixes#121994.
The dvisvgm version provided by TeXLive now includes the fixes
mentioned in the attached issue and thus can be used again instead of
relying on upstream sources.
This is done because the xpdf provided by Nixpkgs is marked as
insecure. Granted, the version provided by TeXLive also has these
vulnerabilities, but it might be okay to treat them as irrelevant in
the context of TeX.
Building a simple LaTeX document without bibTeX with a table of contents yielded a document with an empty table of contents since LaTeX was only run once
In 85605c8a29 we got a really nice
performance improvement.
By using this implementation for texlive instantiation RSS is cut by
~2.6x with a similar wall time speedup.
Also change default package version from texlive year to a
revision. The problem with using texlive year is that it will change
on each of our update even if the package itself was not updated. The
texlive SVN revision number should not change unless the package is
updated. This will allow to share more store paths between different
texlive versions.
chktex binary doesn't work without its global resource file which
comes in its tlType="run" package. This commit moves chktex into a
separate tlType="bin" package so it is only used along with its
tlType="run" counterpart.
Without it, building a document fails with the following error:
pdflatex failed
index.tex: File `pdflscape.sty' not found.
index.tex:47: Emergency stop.
This helps kpathsea to find texmf.cnf in some cases. For example,
dvipng was trying to look for it in
/nix/store/<hash>-texlive-dvipng.bin-2019/ instead of
/nix/store/<hash>-texlive-combined-full-2019/.
Announced in [1], versioned tarballs allow to make sure that a
specific version is fetched. This does not guarantee that all previous
versions are retained on the main mirrors.
Logically, we would want to first try to download versioned tarballs
from any mirror and only then try the unversioned ones. But right now
we only have two mirrors and only some of the tarballs are versioned
in texlive-2019, so the order is changed to not hammer the weak
tug.org mirror.
[1] https://tug.org/pipermail/tex-live/2019-September/044086.html
The biber package is now at 2.14, but TextBibTeX is still required.
Few changes were needed for newly introduced scripts-extra path. This
broke some of our old tricks which were relying on having writable
script directories. The changes to the script locations made buildEnv
create symlinks to the script directories instead of directories of
symlinks to scripts. The changes to texlinks.sh and texlive/TeXLive
perl path were made because of this.
The shell script doesn't work very well in non-GNU environments like
darwin. This provides an implementation that uses just a single GNU tool
- gawk, thus reduces number of points of failure.
The `bash` pragma gives syntax highlighting on the file, and removing the
prompts allows a user to just copy-paste the entire block into the terminal to
run the update.
biber: 2.12 -> 2.13
Vendored potrace was moved from libs/potrace to
texk/dvisvgm/dvisvgm-src/libs/potrace. Unfortunately, it is not trivial
to remove it from there, so for now dvisvgm seems to be using its
vendored dependencies.
There ver very many conflicts, basically all due to
name -> pname+version. Fortunately, almost everything was auto-resolved
by kdiff3, and for now I just fixed up a couple evaluation problems,
as verified by the tarball job. There might be some fallback to these
conflicts, but I believe it should be minimal.
Hydra nixpkgs: ?compare=1538299
texlive attribute was accidentally added in attrset wrapped with
stdenv.lib.optionalAttrs (!stdenv.isDarwin)
Fixes: dbc2c1c4b8 ('texlive: add missing perl dependencies for latexindent')
Initial language.{dat,def} configuration files provided by
`texlive.hyphen-base` may declare languages that were not part of the
combined packages. Those are filtered out by a sed script that had few
problems:
1) The sed script was generated from a list of potentially non-unique
packages. Every repetition of a select and print clause would produce a
copy of a language declaration in the output file. This became a problem
for update to the 2018-final, the fmtutil would crash from too much
German hyphenation.
2) The select clauses were ambiguous: both '^% from hyphen' and
'^% from hyphen-welsh' will match a line 'from hyphen-welsh'.
mkUniqueOutPaths used to produce empty paths for dummy packages, this
version strips those out. This does not affect `pkgList.bin` at all, but
`pkgList.nonbin` is affected, so this is not exactly a refactoring. It
should not harm to have a cleaner `paths`.
Also, original comment said "here we deal with those dummy packages
needed for hyphenation filtering". This doesn't seem to be true, the
packages that were really filtered are actually metapackages that
represent collections. I also could not find any dummy packages even in
the originally committed version.
repstopdf is supposed to be a symlink to epstopdf. Then epstopdf looks
at "$0" to detect when restricted mode needs to be enabled. Unfortunately
our wrapper will drop all intermediate symlinks, which messes up "$0".
Restricted mode appears to be a security feature, so a test is
introduced to verify that the wrapper works as expected.
The automake file was patched but `automake` not run.
Also since the texk/web2c folder is not in autoconfig's
SUBDIRS the autoreconfHook has to be run in there.
Completely fixes#46376
This seems like a known issue as other distributions (ArchLinux here)
have patches fixing the issue.
This hopefully fixes more than one dependant builds for ZHF 18.09.
Otherwise they will have to be generated by users lazily,
which is probably not the right trade-off.
Noticed this because lack of formats was causing problems
on a few documents since they will (a TL2018 change, I believe)
attempt to be written to the nix build user's ~/.texlive2018.
This commit rebuilds texlive 2017 with the final release of 2017. As described
in these issues [1][2][3], the upstream CTAN mirrors are a continuously moving
rolling release without historical archives.
This particular FTP server is also a rolling release folling CTAN for the latest
version, but it has snapshots of the final texlive releases; it appears that the
2017 distribution has been unmodified since texlive-2018 was released earlier
this year.
Along the way, we needed to fix several issues:
- xindy: if $HOME is unset, it will try to mkdir /homeless-shelter, which fails
due to insufficient permissions.
- scheme-infraonly: this scheme had symlinks into other releases that were
read-only, so it couldn't patch and modify the scripts. This commit removes it
for now, but that's not a particularly satisfying solution. Ideas?
This also adds some documentation on the upgrade process to prepare for
texlive-2018 [4].
This commit also replaces the sha1 hashes with upstream's standard sha512 hashes.
It appears the motivation for the shorter hashes was to save disk space in the
derivations; in master, the size of this directory is 1012K; in this commit it
is 1600K. The difference is not particularly large, and the downsides to using
our own sha1 hashes are:
- More nix code to maintain
- Multi-step upgrade process for maintainers: the maintainer first has to
download all upstream tarballs by sha512 hash, then run the fix script, then
rebuild with sha1 hashes.
- Less transparent. If we use the upstream sha512 hashes, any user can
immediately verify that the hashes we're providing match upstream, or match
the snapshot in time.
- Easier to debug. Since upstream is rolling and packages may disappear or fail
to build, it's useful to be able to determine if the sha mismatch is because
of an update or not; if we have a sha1 mismatch and no tarball to pull, we
can't figure out which sha512sum would have produced that sha1.
- Less trust required. Due to the above, users don't have to trust the
content-addressed mirrors on IPFS and @veprbl's servers as much.
- Easier to cobble together a source distribution from a variety of sources. It
seems some FTP servers have more/less than others, or older/newer packages. If
we know what we're looking for beforehand and we're just missing a few
packages whose hashes match the advertised hashes upstream, it's easier to find.
[1] https://github.com/NixOS/nixpkgs/issues/24683
[2] https://github.com/NixOS/nixpkgs/issues/10026
[3] https://github.com/NixOS/nixpkgs/issues/34490
[4] https://github.com/NixOS/nixpkgs/issues/40232
toPath has confusing semantics and is never necessary; it can always
either just be omitted or replaced by pre-concatenating `/.`. It has
been marked as "!!! obsolete?" for more than 10 years in a C++
comment, hopefully removing it will let us properly deprecate and,
eventually, remove it.
