Simplifying Python 3 setup using venv
Without virtual environments, working with Python is a nightmare! In case of Ubuntu, the base operating system comes with Python preinstalled, and you might think that that\'s a nice thing. Except, that version of Python is old and pip is not even installed. So you install
pip only to find out that it is detecting and trying to update Python programs that are part of the base operating system and are supposed to be managed by your operating system's package manager.
This same pattern of unavailability and conflicts gets worse over time as you start working on multiple Python projects. Dependencies of one might conflict with the other. You may require a specific version of
request package for one project and a different version for another, and on and on. This frustration is well-depicted in this XKCD Comic:
Get 480GB NVMe extra storage for FREE
Celebrating our 9 year anniversary! Capacity is limited and some deals will sell out. Get yours before they are gone!
This post tries to address all these problems, and then some more. Obviously, most of this is aimmed at Linux users, and is tested on a Ubuntu 20.04 LTS server, but the principles can be generalized for macOS and Windows 10 too.
The goal is to achieve the following with regards to our Python 3 environment:
- Isolating it from the broader Operating system.
- Making it easy to configure and backup
- Making it easily upgradable
- Using methods recommended by the Official Documentation
We will be using
venv module which is the recommended way to create virtual environments
Python venv to the rescue
In Python 3.6 the problem of managing multiple dependencies has received the cleanest resolution -- Virtual Environments or venv. In my opinion, this has rendered a lot of third party options like conda, pyenv, etc less relevant, if not completely redundant.
venv is an official Python module that is meant for running an isolated Python environment. This isolated environment can include Python interpreter, pip, and setuptools. All these dependencies are packaged into a user-specified directory.
Creating a virtual environment or venv
To get started, let\'s install venv on our system first:
$ sudo apt install python3-venv
Let's create a virtual environment next. I will name this virtual environment,
my-venv and it will be created in the user\'s home directory for this specific case.
$ python3 -m venv $HOME/my-venv
For Ubuntu 20.04, we had to use
python3 in the command instead of just
python word is reserved for Python version 2. Same is true for macOS, but if you\'ve installed Python on Windows then you can just use
python to invoke Python 3 interpreter.
Activating and using a virtual environment or venv
To start using the virtual environment, we need to import a script located at
$HOME/my-venv/bin into our SHELL environment. This script will be add the virtual environment\'s directory to your shell\'s PATH environment such that when
pip or other packages are searched for the SHELL will search the virtual environment\'s directory first search the
my-venv directory to look for appropriate python program.
Depending on your SHELL the script you need to invoke will be different. For
zsh interpreters the script is named
[email protected]:~# source ~/my-venv/bin/activate (my-venv) [email protected]:~#
fish shell the script is called
activate.fish and there is a similar script for
tsch in the same directory. For Windows 10, the scripts will have a
.ps1 extensions for command prompt and powershell respectively. Use the command
echo $SHELL to know what shell your system is using.
Notice in the above shell, once the
venv is activated the prompt changed. It also has a copy of
pip install by default. You can verify this and even update it:
$ pip --version pip 20.0.2 from /root/my-venv/lib/python3.8/site-packages/pip (python 3.8)
To list all the packages installed, and to update them just do the following:
$ pip list Package Version ------------- ------- pip 20.0.2 pkg-resources 0.0.0 setuptools 44.0.0 $ pip install --upgrade pip pkg-resources setuptools $ pip list Package Version ------------- ------- pip 21.0.1 pkg-resources 0.0.0 setuptools 56.0.0
This won\'t update your system\'s copy of
pip or any other pip package you have installed system-wide. The version of Python being used is the same as the Python command used to create your virtual environment.
You can install any number of
pip packages inside this environment and use it for your Python projects. As long as the environment is active, the Python interpreter will prioritize these packages when your scripts call for a specific module, regardless of where you are in your filesystem. Once you are done working on the project, and want to switch gears simply type in
deactivate and the virtual environment will be disabled:
(my-venv) [email protected]:~# deactivate [email protected]:~#
Notice that the prompt is back to being normal. You can always reactivate the venv by running the
source command along with the
activate script. If your environment is broken simply remove the directory to get rid of it. Make sure that your Python project's file don't live in the same directory as the virtual environment.
Note on Updating Packages and Portability
Once you have installed necessary packages, it is always important to keep a snapshot of all the packages and their version numbers. This allows you to maintain and upgrade things gradually without the risk of things breaking down.
$ pip freeze > requirements.txt
By convention the
requirements.txt file is used to store all the packages. Here's a typical output:
click==7.1.2 cycler==0.10.0 joblib==1.0.1 kiwisolver==1.3.1 matplotlib==3.4.1 nltk==3.6.1 numpy==1.20.2 Pillow==8.2.0 pyparsing==2.4.7 python-dateutil==2.8.1 regex==2021.4.4 six==1.15.0 textblob==0.15.3 tqdm==4.60.0
It also allows you to:
- Recreate the environment quickly and exactly, even if the underlying packages update in the future. Reinstall the same packages using the following command:
$ pip install -r requirements.txt
- Keep track of the dependencies with your version control like git or GitHub.
- Reuse the file for other parts of your CI/CD like when creating a Docker image for your file.
That\'s it for Python environments! Happy computing!