I am trying to build an existing Dockerfile, which needs to be migrated from an older ubuntu version base image to a more up-to-date base image.
The existing Dockerfile contains commands such as
pip install pandas
There are many such pip commands, each of which triggers the following error message.
error: externally-managed-environment
This is not unexpected. Recent versions of Ubuntu produce this error when the user attempts to install pip packages without first activating a virtual environment.
This can be fixed by creating an activating an virtual environment. The disadvantage is that inside a Docker container, this shouldn't really be needed, since a container is its own isolated environment. In addition, it creates an additional layer which is slightly inconvenient. RUN python3 my_file.py no longer works directly, as the venv has to be activated first. (There are two ways to do this, the easiest of which is to do RUN /path/to/.venv/bin/python3 /path/to/my_file.py.)
The error could also be "fixed" by passing the --break-system-packages argument. I do not know in detail what the consequences of this are, so I do not know if this could be a recommended solution in this context.
There is a third possibility, which would be to install python3-pandas (assuming it exists). This is an apt package which provides an installation of pandas via apt. I would prefer not to use this method, since not all pip packages are available as apt packages. I aim to try and avoid a fragmented install whereby some packages are provided through one method and other packages are provided through a different method.
To review:
- What does the
--break-system-packagescommand line option do? How "safe" is this inside a Docker container? (Rather than frequently creating and destroying this particular container, it tends to persist for a significant period of time. Typically several weeks to a few months.) - If this isn't a suitable or recommended approach, is there a way that I can conveniently create a system-wide virtual environment, and somehow cause it to be "permanently" activated. (In other words, to create some kind of "transparent" virtual environment, which isn't noticeable to the user - so that running
python3 main.pywill runmain.pywith the virtual environment active, automatically. Can this be done?)
I am trying to build an existing Dockerfile, which needs to be migrated from an older ubuntu version base image to a more up-to-date base image.
The existing Dockerfile contains commands such as
pip install pandas
There are many such pip commands, each of which triggers the following error message.
error: externally-managed-environment
This is not unexpected. Recent versions of Ubuntu produce this error when the user attempts to install pip packages without first activating a virtual environment.
This can be fixed by creating an activating an virtual environment. The disadvantage is that inside a Docker container, this shouldn't really be needed, since a container is its own isolated environment. In addition, it creates an additional layer which is slightly inconvenient. RUN python3 my_file.py no longer works directly, as the venv has to be activated first. (There are two ways to do this, the easiest of which is to do RUN /path/to/.venv/bin/python3 /path/to/my_file.py.)
The error could also be "fixed" by passing the --break-system-packages argument. I do not know in detail what the consequences of this are, so I do not know if this could be a recommended solution in this context.
There is a third possibility, which would be to install python3-pandas (assuming it exists). This is an apt package which provides an installation of pandas via apt. I would prefer not to use this method, since not all pip packages are available as apt packages. I aim to try and avoid a fragmented install whereby some packages are provided through one method and other packages are provided through a different method.
To review:
- What does the
--break-system-packagescommand line option do? How "safe" is this inside a Docker container? (Rather than frequently creating and destroying this particular container, it tends to persist for a significant period of time. Typically several weeks to a few months.) - If this isn't a suitable or recommended approach, is there a way that I can conveniently create a system-wide virtual environment, and somehow cause it to be "permanently" activated. (In other words, to create some kind of "transparent" virtual environment, which isn't noticeable to the user - so that running
python3 main.pywill runmain.pywith the virtual environment active, automatically. Can this be done?)
1 Answer
Reset to default 2What does the --break-system-packages command line option do?
In super short, installs packages to /usr/ , which is managed by apt-get .
If this isn't a suitable or recommended approach, is there a way that I can conveniently create a system-wide virtual environment, and somehow cause it to be "permanently" activated.
Something along:
RUN python3 -m venv /venv
ENV VIRTUAL_ENV=/venv
ENV PATH=/venv/bin:$PATH
But it could fail for packages that need LD_LIBRARY_PATH or PATH, so just make sure to activate the venv in all environments. For exampe the following adds /etc/profile.d/activate sh file and runs login shell (see https://www.gnu./software/bash/manual/html_node/Bash-Startup-Files.html ).
RUN python3 -m venv /venv
RUN ln -s /venv/bin/activate /etc/profile.d/myactivate.sh
SHELL ["bash", "-lc"]
ENTRYPOINT ["bash", "-l"]
Use pip. If you are using python, consider just using python docker image.
The recommended practice is to create a virtual env and not conflict with system managers like apt-get, also in docker.
pip. Something likeRUN pip install .will install the current project, its dependencies as declared inpyproject.toml, and any Python entry point scripts. – David Maze Commented Feb 17 at 16:52