Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to Debian Bookworm and Python 3.11 #4028

Merged
merged 1 commit into from
Oct 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 12 additions & 11 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
# vim: set expandtab sw=4
FROM golang:1.22.7-bullseye
FROM golang:1.22.7-bookworm

# Create a non-priviledged user to run browsers as (Firefox and Chrome do not
# like to run as root).
RUN chmod a+rx $HOME && useradd --uid 9999 --user-group --create-home browser

# Sort the package names!
# firefox-esr: provides deps for Firefox (we don't use ESR directly)
# openjdk-11-jdk: provides JDK/JRE to Selenium & gcloud SDK
# openjdk-17-jdk: provides JDK/JRE to Selenium & gcloud SDK
# python-crcmod: native module to speed up CRC checksum in gsutil
RUN apt-get update -qqy && apt-get install -qqy --no-install-suggests \
curl \
firefox-esr \
lsb-release \
openjdk-11-jdk \
python3.9 \
openjdk-17-jdk \
python3.11 \
python3-crcmod \
sudo \
tox \
Expand All @@ -35,13 +35,14 @@ RUN curl -sL https://deb.nodesource.com/setup_18.x | bash - && \
# Based on https://github.com/GoogleCloudPlatform/cloud-sdk-docker/blob/master/Dockerfile
RUN export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && \
echo "deb https://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" > /etc/apt/sources.list.d/google-cloud-sdk.list && \
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
apt-get update -qqy && apt-get install -qqy \
google-cloud-sdk \
google-cloud-sdk-app-engine-python \
google-cloud-sdk-app-engine-python-extras \
google-cloud-sdk-app-engine-go \
google-cloud-sdk-datastore-emulator && \
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list && \
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg && \
apt-get update -y && apt-get install -y \
google-cloud-cli \
google-cloud-cli-app-engine-python \
google-cloud-cli-app-engine-python-extras \
google-cloud-cli-app-engine-go \
google-cloud-cli-datastore-emulator && \
gcloud config set core/disable_usage_reporting true && \
gcloud config set component_manager/disable_update_check true && \
gcloud --version
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -229,15 +229,15 @@ gcc: apt-get-gcc
git: apt-get-git
jq: apt-get-jq
psmisc: apt-get-psmisc
python3: apt-get-python3.9
python3: apt-get-python3.11
tox: apt-get-tox
unzip: apt-get-unzip
wget: apt-get-wget

java:
@ # java has a different apt-get package name.
if [[ "$$(which java)" == "" ]]; then \
sudo apt-get install -qqy --no-install-suggests openjdk-11-jdk; \
sudo apt-get install -qqy --no-install-suggests openjdk-17-jdk; \
fi

gpg:
Expand Down
2 changes: 1 addition & 1 deletion api/query/cache/service/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Production deployment spec for query cache service.

FROM golang:1.22.7-bullseye as builder
FROM golang:1.22.7-bookworm as builder

RUN apt-get update
RUN apt-get install -qy --no-install-suggests git
Expand Down
2 changes: 1 addition & 1 deletion docs/upgrading-go.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Once you have confirmed that the desired version is available:
- tooling [Dockerfile](../Dockerfile) at the root of the repo
- searchcache [Dockerfile](../api/query/cache/service/Dockerfile)

The tooling image and the first stage of searchcache use the same Golang image. Check out the Golang [page](https://hub.docker.com/_/golang?tab=tags) on Docker Hub for the latest tags. Currently, we are using the `bullseye` [release](https://wiki.debian.org/DebianReleases) of Debian. As a result pick the `golang:<latest stable version>-bullseye` tag. If bullseye is superseded by a new version, you should change that as well.
The tooling image and the first stage of searchcache use the same Golang image. Check out the Golang [page](https://hub.docker.com/_/golang?tab=tags) on Docker Hub for the latest tags. Currently, we are using the `bookworm` [release](https://wiki.debian.org/DebianReleases) of Debian. As a result pick the `golang:<latest stable version>-bookworm` tag. If bookworm is superseded by a new version, you should change that as well.

## Step 3 - Change the version in go.mod

Expand Down
2 changes: 1 addition & 1 deletion results-processor/.python-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.9
3.11
4 changes: 2 additions & 2 deletions results-processor/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.9.20-bullseye
FROM python:3.11.9-bookworm

# Install runtime dependencies.
# python3-crcmod for faster gsutil checksum
Expand All @@ -17,7 +17,7 @@ RUN gcloud config set disable_usage_reporting false
RUN rm -f $HOME/.config/gcloud/gce

# Setup and activate virtualenv.
RUN virtualenv -p python3.9 /env
RUN virtualenv -p python3.11 /env
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH

Expand Down
8 changes: 4 additions & 4 deletions results-processor/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Basics

The results processor runs on Python 3.9. The entry point is a Flask web server
The results processor runs on Python 3.11. The entry point is a Flask web server
(`main.py`). In production, gunicorn is used as the WSGI (see `Dockerfile`) and
the container runs as a custom AppEngine Flex instance (see `app.yaml`).

Expand All @@ -10,7 +10,7 @@ We can create a virtualenv to recreate a setup close to production for daily
development.

```bash
virtualenv env -p python3.9
virtualenv env -p python3.11
. env/bin/activate
pip install -r requirements.txt
```
Expand Down Expand Up @@ -38,8 +38,8 @@ Dependabot is used to automatically update `requirements.txt`. To manually
update dependencies, run the following commands:

```bash
pip3.9 install --user pip-tools
python3.9 -m piptools compile requirements.in
pip3.11 install --user pip-tools
python3.11 -m piptools compile requirements.in
```

## Local debugging
Expand Down
10 changes: 2 additions & 8 deletions results-processor/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# This file is autogenerated by pip-compile with Python 3.9
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile requirements.in
Expand Down Expand Up @@ -53,14 +53,12 @@ grpcio==1.53.2
# via
# google-api-core
# grpcio-status
grpcio-status==1.44.0
grpcio-status==1.53.2
# via google-api-core
gunicorn==23.0.0
# via -r requirements.in
idna==3.7
# via requests
importlib-metadata==6.8.0
# via flask
itsdangerous==2.1.2
# via flask
jinja2==3.1.4
Expand Down Expand Up @@ -103,13 +101,9 @@ requests==2.32.3
# google-cloud-storage
rsa==4.8
# via google-auth
tomli==2.0.1
# via mypy
typing-extensions==4.7.1
# via mypy
urllib3==1.26.19
# via requests
werkzeug==3.0.3
# via flask
zipp==3.19.1
# via importlib-metadata
2 changes: 1 addition & 1 deletion results-processor/tox.ini
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[tox]
envlist = py39
envlist = py311
# We don't have or need setup.py for now.
skipsdist=True

Expand Down
Loading