Skip to content

👷 Enable deptree hash-checking in the infra 🔗 - #420

Open
webknjaz wants to merge 4 commits into
pypa:unstable/v1from
webknjaz:integrity/runtime/deptree
Open

👷 Enable deptree hash-checking in the infra 🔗#420
webknjaz wants to merge 4 commits into
pypa:unstable/v1from
webknjaz:integrity/runtime/deptree

Conversation

@webknjaz

Copy link
Copy Markdown
Member

The main theme here is that this kinda fixes #279, perhaps not fully. I'm not planning to increase the scope, but people are welcome to send in follow-ups. We may want to discuss more in the future.

This specific piece of work became possible due to pip v26.1 [1] fixing the issue I reported back when @pradyunsg was first battle-testing the new resolvelib-based replacement in 2020, and it turned out to crash whenever the new resolver would
see constraint files with entries having --hash attached [2]. This was quite a popular use for the users of pip-tools which delayed the adoption here. It's finally been fixed this May [3] so we can use it here.

This version of pip got bundled into CPython and backported into 3.13 that our container uses. So I'm bumping pip to the corresponding version to match that in our pip-tools managed files too (since we've been downgrading pip to a version that doesn't for two months 🤣).

With that in mind, the central patch also includes hashes for whatever bits of the tree have been present in the constraint “lock” files already without re-resolving them or bumping anything else transitive. Plus it configures pip-tools to always include hashes from now on. There's also a small convenience commit before this for including the pip-tools config into the container image for future constraints management that wasn't there for some reason.

The last commit in the series bumps the “lock file” constraints separately+transitively:

$ podman run \
    --rm \
    -it \
    -v $(pwd)/requirements/:/requirements:rw,Z \
    --entrypoint=/bin/bash \
    --workdir=/requirements \
    ghcr.io/pypa/gh-action-pypi-publish:v1.14.2
> python3 -m pip install --upgrade 'pip < 26.1.1' pip-tools
> pip-compile --config=../.pip-tools.toml -o runtime-prerequisites.txt runtime-prerequisites.in
> pip-compile --config=../.pip-tools.toml -o runtime.txt runtime.in

Thanks to Pradyun for finally fixing the thing in pip upstream, and to
@sethmlarson for the initial exploratory PR, and to @graingert for poking me periodically about this ☮️

webknjaz added 3 commits July 31, 2026 22:52
This sorta fixes pypa#279, though not fully, perhaps. We may want to discuss
it more in the future.

It became possible to pip v26.1 [[1]] fixing the issue I reported when
Pradyun was first battle-testing the new resolvelib-based replacement
back in 2020 and it turned out to crash whenever the new resolver would
see constraint files with entries having `--hash` attached [[2]]. This
was quite a popular use for the users of `pip-tools` which delayed the
adoption. It's finally been fixed this May [[3]] so we can use it here.

This version of pip got bundled into CPython and backported into 3.13
that our container uses. So I'm bumping pip to the corresponding version
to match that in our pip-tools managed files too (since we've been
downgrading pip to a version that doesn't for two months LOL).

With that, the patch also includes hashes for whatever bits of the tree
have been present in the constraint "lock" files already without
re-resolving them or bumping anything else transitive.

Thanks to Pradyun for finally fixing the thing in pip upstream, and to
Seth, and Thomas for periodically poking me about this ☮️

[1]: https://pip.pypa.io/en/stable/news/#v26-1
[2]: pypa/pip#9243
[3]: pypa/pip#13887
$ podman run \
    --rm \
    -it \
    -v $(pwd)/requirements/:/requirements:rw,Z \
    --entrypoint=/bin/bash \
    --workdir=/requirements \
    ghcr.io/pypa/gh-action-pypi-publish:v1.14.2
> python3 -m pip install --upgrade 'pip < 26.1.1' pip-tools
> pip-compile --config=../.pip-tools.toml -o runtime-prerequisites.txt runtime-prerequisites.in
> pip-compile --config=../.pip-tools.toml -o runtime.txt runtime.in
@webknjaz webknjaz self-assigned this Jul 31, 2026
@webknjaz webknjaz added the dependencies Pull requests that update a dependency file label Jul 31, 2026
@webknjaz

Copy link
Copy Markdown
Member Author

I've tagged multiple people here mainly to see if anybody has concerns regarding bumping a lot of deep transitive deps together…

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the container build and dependency lockfiles to support hash-based dependency verification by generating pip-tools outputs with hashes, and aligns the build environment to newer pip behavior that supports hashed constraints.

Changes:

  • Enable pip-tools hash generation (generate-hashes = true) and regenerate requirements/*.txt with hashes.
  • Bump and hash-pin pip in runtime-prerequisites.* to a version compatible with hashed constraints.
  • Copy .pip-tools.toml into the Docker image to support in-container lockfile maintenance workflows.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
requirements/runtime.txt Regenerated lock/constraints file with --generate-hashes and updated pinned versions + hashes.
requirements/runtime-prerequisites.txt Regenerated pip prerequisite lock with hashes and updated pip pin.
requirements/runtime-prerequisites.in Document and require a pip version new enough to support hashed constraints.
Dockerfile Include .pip-tools.toml in the image; continues installing dependencies during image build.
.pip-tools.toml Enable hash generation in pip-tools configuration.
Suppressed comments (1)

Dockerfile:24

  • The image build installs from the unpinned .in files, while the hashed/pinned .txt files are only used as constraints. This likely does not enforce hash-checking during installation, which is the central goal described in the PR/issue. Consider installing directly from the hashed lock files (and optionally enabling --require-hashes) so the build fails if hashes don’t match.
  PIP_CONSTRAINT=requirements/runtime-prerequisites.txt \
    pip install --user --upgrade --no-cache-dir \
      -r requirements/runtime-prerequisites.in && \
  PIP_CONSTRAINT=requirements/runtime.txt \
    pip install --user --upgrade --no-cache-dir --prefer-binary \

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +2 to +3
# that prevented us from using `--require-hashes` in constraint files since
# roughly year 2020. But now we can get back to using it.
This shouldn't be important right now and can be addressed later.
run: python3 -m pip install build twine
env:
PIP_CONSTRAINT: test/requirements/runtime.txt
PIP_NO_REQUIRE_HASHES: 1 # FIXME: incomplete lock file explodes

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(looks like I'll have to deal w/ this before merging)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

use hash frozen requirements.txt files (configure pip-compile with --generate-hashes)

2 participants