How to create a
Devcontainer for your
Python project ๐ณ
The use case¶
You are assigned to setup a new repo for a team. The requirements are as follows:
So we need to align on:
- ๐ A specific Java version
- ๐ A specific Python version
- ๐ A specific
pysparkversion
โ otherwise we do not enjoy the guarantees we want in production code
๐ Issues
|
PP
|
Petra the Python Dev |
Corrupted Virtual Environment |
|
EO
|
Erik the old project maintainer |
Outdated project dependencies |
|
BW
|
Billy Windows |
Containerisation & going into production
e.g. Windows / Linux / MacOS |
|
PU
|
PySpark user |
Inconsistent environments
e.g. local โ CI โ prod โ team members |
|
JJ
|
Johnny Junior |
No formal specification of install steps
and missing docs ๐ |
๐ How to get a reproducible Dev environment?
Devcontainers to the rescue ⛑!¶
๐ณ Docker helps us create a formal definition of our environment.
๐ Devcontainers allow you to connect your editor (IDE) to that container.
๐ Reproducible means:
|
Downsides?
|
👷🏻♂️ Let's build a Devcontainer!¶
Letโs say we have a really simple project that looks like this:
$ tree .
.
โโโ README.md
โโโ requirements.txt
โโโ requirements-dev.txt
โโโ sales_analysis.py
โโโ test_sales_analysis.py
The .devcontainer folder¶
Your Devcontainer spec will live inside the .devcontainer folder.
There will be two main files:
devcontainer.jsonDockerfile
Create a new file called devcontainer.json:
{
"build": {
"dockerfile": "Dockerfile",
"context": ".."
}
}
So how does this Dockerfile look like?
FROM python:3.10
# Install Java
RUN apt update && \
apt install -y sudo && \
sudo apt install default-jdk -y
## Pip dependencies
# Upgrade pip
RUN pip install --upgrade pip
# Install production dependencies
COPY requirements.txt /tmp/requirements.txt
RUN pip install -r /tmp/requirements.txt && \
rm /tmp/requirements.txt
# Install development dependencies
COPY requirements-dev.txt /tmp/requirements-dev.txt
RUN pip install -r /tmp/requirements-dev.txt && \
rm /tmp/requirements-dev.txt
Opening the Devcontainer¶
The .devcontainer folder in place, now itโs time to open our Devcontainer.
Open up the command pallete (CMD + Shift + P) and select โDev Containers: Reopen in Containerโ:
Upon opening a repo with a valid .devcontainer folder, you are already notified:
Rebuilding allows you to get a fresh environment anytime you want:
What is happening under the hood 🚗¶
Besides starting the Docker image and attaching the terminal to it, VSCode is doing a couple more things:
- VSCode Server is being installed on your Devcontainer.
VSCode Server is installed as a service in the container itself so your VSCode installation can communicate with the container.
For example, install and run extensions. - Config is copied over.
Config like~/.gitconfigand~/.ssh/known_hostsare copied over to their respective locations in the container. - Filesystem mounts.
VSCode automatically takes care of mounting:- The folder you are running the Devcontainer from.
- Your VSCode workspace folder.
Opening your Devcontainer with the click of a button¶
Your entire project setup is now encapsulated in the Devcontainer. So actually we can add a Markdown button to open up the Devcontainer:
[
](
https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/godatadriven/python-devcontainer-template
)
Which basically means, open this URL:
vscode://
ms-vscode-remote.remote-containers/
cloneInVolume?
url=https://github.com/godatadriven/python-devcontainer-template
Just modify the GitHub URL after url= โ.
This renders the following button:
What kind of README would you rather like?
Extending the Devcontainer¶
We have built a working Devcontainer, that is great! But a couple things are still missing.
- Install a non-root user for extra safety and good-practice
- Pass in custom VSCode settings and install extensions by default
- Be able to access Spark UI (opening up port 4040)
- Run Continuous Integration (CI) in the Devcontainer
Let's see how.
*](https://godatadriven.com/wp-content/uploads/2022/10/running-pip-as-root.png)
So let's go ahead and create a user for this scenario.
# Add non-root user
ARG USERNAME=nonroot
RUN groupadd --gid 1000 $USERNAME && \
useradd --uid 1000 --gid 1000 -m $USERNAME
## Make sure to reflect new user in PATH
ENV PATH="/home/${USERNAME}/.local/bin:${PATH}"
USER $USERNAME
Add the following property to devcontainer.json:
"remoteUser": "nonroot"
That's great! When we now start the container we should connect as the user nonroot.
Passing custom VSCode settings¶
"customizations": {
"vscode": {
"extensions": [
"ms-python.python"
],
"settings": {
"python.testing.pytestArgs": [
"."
],
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"python.formatting.provider": "black",
"python.linting.mypyEnabled": true,
"python.linting.enabled": true
}
}
}
Accessing Spark UI¶
Since we are using pyspark, it would be nice to be able to access Spark UI.
"portsAttributes": {
"4040": {
"label": "SparkUI",
"onAutoForward": "notify"
}
},
"forwardPorts": [
4040
]
When we now run our code, we get a notification we can open Spark UI in the browser:
Resulting in the Spark UI like we know it:
Running our CI in the Devcontainer¶
There are two basic options:
- Build the Docker image within the CI/CD pipeline
- Prebuilding the image
Let's see about option number (1).
1. Build the Docker image within the CI/CD pipeline¶
Luckily, a GitHub Action was already setup for us to do exactly this:
To now build, push and run a command in the Devcontainer is as easy as:
name: Python app
on:
...
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout (GitHub)
uses: actions/checkout@v3
- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and run dev container task
uses: devcontainers/ci@v0.2
with:
imageName: ghcr.io/${{ github.repository }}/devcontainer
runCmd: pytest .
See below a trace of the executed GitHub Action:
Awesome!
The final Devcontainer definition¶
We built the following Devcontainer definitions.
First, devcontainer.json:
{
"build": {
"dockerfile": "Dockerfile",
"context": ".."
},
"remoteUser": "nonroot",
"customizations": {
"vscode": {
"extensions": [
"ms-python.python"
],
"settings": {
"python.testing.pytestArgs": [
"."
],
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"python.formatting.provider": "black",
"python.linting.mypyEnabled": true,
"python.linting.enabled": true
}
}
},
...
}
"portsAttributes": {
"4040": {
"label": "SparkUI",
"onAutoForward": "notify"
}
},
"forwardPorts": [
4040
]
And our Dockerfile:
FROM python:3.10
# Install Java
RUN apt update && \
apt install -y sudo && \
sudo apt install default-jdk -y
# Add non-root user
ARG USERNAME=nonroot
RUN groupadd --gid 1000 $USERNAME && \
useradd --uid 1000 --gid 1000 -m $USERNAME
## Make sure to reflect new user in PATH
ENV PATH="/home/${USERNAME}/.local/bin:${PATH}"
USER $USERNAME
## Pip dependencies
# Upgrade pip
RUN pip install --upgrade pip
# Install production dependencies
COPY --chown=nonroot:1000 requirements.txt /tmp/requirements.txt
RUN pip install -r /tmp/requirements.txt && \
rm /tmp/requirements.txt
# Install development dependencies
COPY --chown=nonroot:1000 requirements-dev.txt /tmp/requirements-dev.txt
RUN pip install -r /tmp/requirements-dev.txt && \
rm /tmp/requirements-dev.txt
Going further 🔮¶
-
๐ก Pro tip: mount your AWS/GCP/Azure credentials
- Devcontainer templates
- Devcontainer features
Remote Development ๐
e.g. GitHub Codespaces, VM's on Azure/GCP/AWS
... and much more (see references slide)
Concluding¶
๐ Devcontainers connect your IDE to a running ๐ณ Docker container.
โ reproducibility & isolation whilst getting a native experience.
๐ Reproducible means:
|
Now only VSCode, but open specification taking shape.
๐ค
๐ซต
Awesome resources¶
Associated blog post:
Spec:
- containers.dev. The official Devcontainer specification.
Docs:
- Dev Containers VSCode extension. The extension required to connect VSCode to a Devcontainer.
- Mounting file directories in Devcontainers.
- Add a non-root user to a container. More explanations & instructions for adding a non-root user to your
Dockerfileanddevcontainer.json. - Pre-building dev container images
Repo's:
- github.com/devcontainers/ci. Run your CI in your Devcontainers. Built on the Devcontainer CLI.
- github.com/devcontainers/images. A collection of ready-to-use Devcontainer images.
- github.com/manekinekko/awesome-devcontainers. A repo pointing to yet even more awesome resources.