pip-sync tries to uninstall itself (pip-tools)

[CaptainException]

When running pip-sync against a requirements.txt which includes a different version of pip-tools itself, it tries to remove it, which causes an OSError.

Environment Versions

• OS version: Windows 10
• Python version: 3.12.5
• pip version: 24.2 (venv) and 25.2 (machine)
• pip-tools version: 7.5.0

Steps to replicate

1. Delete .venv and recreate it
2. Install pip-tools in the new venv manually using pip install pip-tools (this will install version 7.5.0)
3. Run pip-sync against a requirements.txt which pins pip-tools to 7.4.1

Expected result

pip-sync does not modify packaging tools like itself and simply skips pip-tools unless it's not installed

Actual result

pip-sync tries to uninstall itself and hence tries removing the executable it's running from causing an OSError

[webknjaz]

I'd probably manage a bootstrapping env via pip install instead. I imagine this would not really be possible to solve, especially not on Windows which locks files like crazy.

[sirosen]

I agree that this isn't really possible to solve; I've been hesitant to write up a reply because I'm not sure how to reconcile the issue as filed with what pip-sync does.

Specifically:

Expected result

pip-sync does not modify packaging tools like itself and simply skips pip-tools unless it's not installed

That is not my expectation. pip-tools is a package installed into the environment.
If you try to sync it to some other version, you should expect it to try to replace itself.

We might be able to make it not error, which would be a valid thing to want. pip is able "update itself" and I don't see why pip-tools shouldn't also have that ability. But to do it, we would need to carefully rearrange the code (e.g., move a lot of logic into an atexit hook).

[webknjaz]

Yep, that sounds reasonable. I wasn't sure how difficult it would be.

[CaptainException]

Expected result

pip-sync does not modify packaging tools like itself and simply skips pip-tools unless it's not installed

That is not my expectation. pip-tools is a package installed into the environment. If you try to sync it to some other version, you should expect it to try to replace itself.

My expectation came from this part of the README:

Note: pip-sync will not upgrade or uninstall packaging tools like setuptools, pip, or pip-tools itself. Use python -m pip install --upgrade to upgrade those packages.

Is this currently true at all? Or does pip-sync always update everything it finds in the lockfile? If it does then this sentence should be removed from the README as it is very misleading.

pip is able "update itself" and I don't see why pip-tools shouldn't also have that ability.

Well yes it is able to, but it does not do it by default. Running pip install -r requirements.txt or pip install --upgrade pip will never update pip. You need to explicitly run python -m pip install --upgrade pip to modify pip itself.

We might be able to make it not error, which would be a valid thing to want. But to do it, we would need to carefully rearrange the code (e.g., move a lot of logic into an atexit hook).

I am not sure if that's enough, as then it would still uninstall itself. When pip-sync fails due to the OSError all you are left with is the pip-sync.exe itself. The rest of the piptools package is purged rendering it non-functional. Not erorring would be better, because then at least all the following deps will be installed but it still doesn't fix the main issue that arises from the self-reference.

PS: I was able to work around the issue by adding --constraint requirements.txt to pip install pip-tools in my npm script. But i still think pip-tools' behavior should be different. Or rather, it should be able to behave differently, the default behavior should obviously stay unchanged. But being able to skip itself (or maybe even having a --skip-package argument allowing you to ignore any specified package) would be useful.
At the very least the README should be updated to reflect the real behavior.

[sirosen]

My expectation came from this part of the README:

Note: pip-sync will not upgrade or uninstall packaging tools like setuptools, pip, or pip-tools itself. Use python -m pip install --upgrade to upgrade those packages.

Is this currently true at all? Or does pip-sync always update everything it finds in the lockfile? If it does then this sentence should be removed from the README as it is very misleading.

I think "will not upgrade or uninstall" is wrong, and it should be "will not uninstall".
(Unless I am misreading things, of course.)

pip-sync uninstalls non-matching packages which it finds in the environment. So it looks at what's installed (as you might see from pip freeze), excludes pip, pip-tools, and several others from the list, and marks what it finds for removal.

pip-tools won't uninstall itself in the process of trying to make an environment match requirements.txt constraints, but it definitely will treat itself as a valid target and try to upgrade itself.

I stand by the view that this is sensible behavior, and we could at least try to make it work. I'll now add to that a note of "and apparently not perfectly documented", so that's another improvement.

Running pip install -r requirements.txt or pip install --upgrade pip will never update pip. You need to explicitly run python -m pip install --upgrade pip to modify pip itself.

This is not accurate. pip can and does upgrade itself. python -m pip is just finding a slightly different entry-point to the pip package vs the script used for direct pip invocation. Both call the same code in the same package to do an install.

The main difference between the two invocations -- as far as a user of pip is concerned -- is that python -m pip finds the first python in your path, then looks for pip from there, whereas invoking pip directly looks for the first pip script/shim in your path. Most scenarios in which these don't point to the exact same code are some flavor of "broken environment", far beyond the scope of this discussion.

---

I don't think your reasoning against pip-tools upgrading itself is solid, since you're talking about what happens today -- which I agree is not good. And it looks to me like you have some incorrect information about how pip behaves which is misleading you. IMO the best resolution is to develop more correct behavior for this scenario, and that the result will be that pip-tools can upgrade or downgrade itself via pip-sync.

In any case, I agree that the docs are at best misleading and at worst outright wrong.
And we can treat this report partly as a doc enhancement request.

[CaptainException]

I think "will not upgrade or uninstall" is wrong, and it should be "will not uninstall". (Unless I am misreading things, of course.)
pip-sync uninstalls non-matching packages which it finds in the environment. So it looks at what's installed (as you might see from pip freeze), excludes pip, pip-tools, and several others from the list, and marks what it finds for removal.
pip-tools won't uninstall itself in the process of trying to make an environment match requirements.txt constraints, but it definitely will treat itself as a valid target and try to upgrade itself.

Only now do I understand how that sentence from the README is meant. It doesn't apply if the packaging tools are added to the lockfile, pip-sync only ignores packaging tools in a sense that it preserves them, even if they are not included in the lockfile.

I stand by the view that this is sensible behavior, and we could at least try to make it work. I'll now add to that a note of "and apparently not perfectly documented", so that's another improvement.

You are right. If pip-tools was able to update itself without erroring that would obviously be the best case as then you would end up exactly with what you expected - an environment which includes all dependencies pinned to the versions specified in the lockfile.

Running pip install -r requirements.txt or pip install --upgrade pip will never update pip. You need to explicitly run python -m pip install --upgrade pip to modify pip itself.

This is not accurate. pip can and does upgrade itself. python -m pip is just finding a slightly different entry-point to the pip package vs the script used for direct pip invocation. Both call the same code in the same package to do an install.
The main difference between the two invocations -- as far as a user of pip is concerned -- is that python -m pip finds the first python in your path, then looks for pip from there, whereas invoking pip directly looks for the first pip script/shim in your path. Most scenarios in which these don't point to the exact same code are some flavor of "broken environment", far beyond the scope of this discussion.

Are you implicating my environment is broken and it should be possible to just run pip install --upgrade pip? Because pip -V and python -m pip -V point to exactly the same location and pip still does not allow upgrading itself unless directly invoked as python module and specifically demands to run it like that in the error message: ERROR: To modify pip, please run the following command: <path to python.exe> -m pip install --upgrade pip.

I don't think your reasoning against pip-tools upgrading itself is solid, since you're talking about what happens today -- which I agree is not good. And it looks to me like you have some incorrect information about how pip behaves which is misleading you. IMO the best resolution is to develop more correct behavior for this scenario, and that the result will be that pip-tools can upgrade or downgrade itself via pip-sync.

Agreed. This would be the optimal behavior.

 
P.S.:
While trying to work around the issue I found two workarounds:

1. Installing the pinned version directly in my setup script by running pip install pip-tools --constraint requirements.txt --quiet. This way pip-syncwon't need to modify pip-tools as the version already matches.
2. Using the globally installed pip-tools and specifying the target environment like this: pip-sync --python-executable (Join-Path $env:VIRTUAL_ENV "\Scripts\python.exe"). This obviously only works the first time the setup script is ran unless modified to always grab the global python executable.

Maybe adding a short chapter on how you need to work with pip-tools when trying to bootstrap a repo would be a useful addition to the README. It could include the aforementioned workarounds.