chore: Add docker images for building the different platforms #11397

ermshiperete · 2024-05-08T18:26:23Z

This change adds the ability to build several docker images which can be used to build the different platforms (core, linux, web, android) and run tests. This can be used on a developer's machine but also on CI.

This change also updates the minimum node version to Node 20 as a temporary step until we try if we can use Node 22. This is necessary because the latest release of Node 18 still comes with the buggy npm that crashes the network. A better solution might have been to use nvm in the docker images (which would allow to easily update to the latest npm), but that would have required more changes and I decided to do it this way for now in order to finish this PR.

@keymanapp-test-bot skip

keymanapp-test-bot · 2024-05-08T18:26:26Z

User Test Results

Test specification and instructions

User tests are not required

Test Artifacts

Android
Developer
- Keyman Developer - build pending
- Compiler Regression Tests - build pending
- kmcomp.zip - build pending
iOS
- Keyman for iOS (simulator image)
- FirstVoices Keyboards for iOS (simulator image)
- TestFlight internal PR build version - 18.0.103 (0.11397.12008)
Keyboards
- Test Keyboards - build pending
Linux
macOS
- Keyman for macOS
Web
- KeymanWeb Test Home
Windows

This updates the minimum required node version to Node 20. The latest version of Node 18 still contains the npm bug (npm/cli#7072) whereas Node 20 got updated to a npm version that contains a fix. Node 20 is known to work with our current code, so this change updates to that as an intermediate step before we investigate if we can update to Node 22 as discussed for Keyman 18.

This change allows to build a docker image that can build Keyman for Linux with test coverage reports, and installs the necessary dependencies for running integration tests.

mcdurdin

This is looking good, let's also discuss how we can make this available from regular /build.sh, /android/build.sh etc as a parameter?

mcdurdin · 2024-08-27T07:59:27Z

resources/docker-images/README.md

+## Prerequisites
+
+You'll need Docker Buildx installed to successfully be able to build the
+container images. This is easiest achieved by installing the [official


Suggested change

container images. This is easiest achieved by installing the [official

container images. This is most easily achieved by installing the [official

mcdurdin · 2024-08-27T08:00:01Z

resources/docker-images/README.md

+By default this will create 64-bit images for building
+Keyman for Android, Keyman for Linux and Keyman for Web. These images
+are based on the Ubuntu 24.04 with Node 20 and Emscripten
+3.1.44 (for the exact versions, see [`minimum-versions.inc.sh`](../build/minimum-versions.inc.sh))


Suggested change

3.1.44 (for the exact versions, see [`minimum-versions.inc.sh`](../build/minimum-versions.inc.sh))

3.1.58 (for the exact versions, see

[`minimum-versions.inc.sh`](../build/minimum-versions.inc.sh))

mcdurdin · 2024-08-27T08:05:37Z

resources/docker-images/README.md

+  ```shell
+  # build 'Keyman Core' in docker
+  # keep build artifacts separate
+  mkdir -p $(git rev-parse --show-toplevel)/core/build/docker-core
+  docker run -it --rm -v $(git rev-parse --show-toplevel):/home/build/build \
+    -v $(git rev-parse --show-toplevel)/core/build/docker-core:/home/build/build/core/build \
+    keymanapp/keyman-core-ci:default \
+    core/build.sh --debug build
+  ```


Can we put this into a script under resources/docker-images, so we can e.g. ./resources/docker-images/run.sh core core/build.sh --debug build (then we'd need something for each of android,core,web,linux I guess)

(For example, we might also want to ./resources/docker-images/run.sh web common/web/build.sh)

mcdurdin · 2024-08-27T08:06:43Z

resources/docker-images/README.md

+
+  ```shell
+  # build 'Keyman Web' in docker
+  docker run --privileged -it --rm \


Is --privileged needed for web build?

mcdurdin · 2024-08-27T08:07:05Z

resources/docker-images/README.md

+- Keyman Core
+
+  ```shell
+  # build 'Keyman Core' in docker


I know this script is moving into run.sh but it should be:

Suggested change

# build 'Keyman Core' in docker

# test 'Keyman Core' in docker

mcdurdin · 2024-08-27T08:22:23Z

resources/docker-images/core/Dockerfile

+#### TMP until we properly figure out the dependencies needed for Core
+#### We should not need to install /tmp/control
+# Install dependencies
+ADD control /tmp/control
+RUN apt-get install -qy python3 python3-setuptools python3-coverage \
+  devscripts equivs libdatetime-perl meson pkgconf lcov gcovr xvfb \
+  xserver-xephyr metacity mutter dbus-x11 weston xwayland && \
+  (yes | mk-build-deps --install /tmp/control) || true && \
+  rm /tmp/control
+#### TMP END


mcdurdin · 2024-08-27T08:23:38Z

resources/docker-images/core/Dockerfile

+# Install node
+ARG NODE_MAJOR
+RUN set -eu; \
+  NODE_VERSION=$(curl -sSL https://unofficial-builds.nodejs.org/download/release/ | cut -d'>' -f2 | cut -d'/' -f1 | grep v${NODE_MAJOR} | sort -V | tail -1) && \
+  echo "Installing node version ${NODE_VERSION}" && \
+  curl -fsSLO --compressed "https://unofficial-builds.nodejs.org/download/release/${NODE_VERSION}/node-${NODE_VERSION}-linux-x64-glibc-217.tar.xz" && \
+  tar -xJf "node-${NODE_VERSION}-linux-x64-glibc-217.tar.xz" -C /usr/local --strip-components=1 --no-same-owner && \
+  ln -s /usr/local/bin/node /usr/local/bin/nodejs


Let's use https://nvm.sh instead

mcdurdin · 2024-08-27T08:24:24Z

resources/docker-images/core/Dockerfile

+  cd emsdk && \
+  ./emsdk install ${EMSCRIPTEN_VERSION} && \
+  ./emsdk activate ${EMSCRIPTEN_VERSION} && \


Once we have KEYMAN_USE_EMSDK we shouldn't need to do this first

mcdurdin · 2024-08-27T08:26:17Z

resources/docker-images/linux/run-tests.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+set -e
+


We could add a test to stop this if not running in a Docker image, for user safety

mcdurdin · 2024-08-27T08:28:07Z

resources/docker-images/web/Dockerfile

+# Install node
+ARG NODE_MAJOR
+RUN set -eu; \
+  NODE_VERSION=$(curl -sSL https://unofficial-builds.nodejs.org/download/release/ | cut -d'>' -f2 | cut -d'/' -f1 | grep v${NODE_MAJOR} | sort -V | tail -1) && \
+  echo "Installing node version ${NODE_VERSION}" && \
+  curl -fsSLO --compressed "https://unofficial-builds.nodejs.org/download/release/${NODE_VERSION}/node-${NODE_VERSION}-linux-x64-glibc-217.tar.xz" && \
+  tar -xJf "node-${NODE_VERSION}-linux-x64-glibc-217.tar.xz" -C /usr/local --strip-components=1 --no-same-owner && \
+  ln -s /usr/local/bin/node /usr/local/bin/nodejs
+
+# Install emscripten
+ARG EMSCRIPTEN_VERSION
+RUN echo "Installing emscripten version ${EMSCRIPTEN_VERSION}" && \
+  cd /usr/share && \
+  git clone https://github.com/emscripten-core/emsdk.git && \
+  cd emsdk && \
+  ./emsdk install ${EMSCRIPTEN_VERSION} && \
+  ./emsdk activate ${EMSCRIPTEN_VERSION} && \
+  echo "#!/bin/bash" > /usr/bin/bashwrapper && \
+  echo "export EMSCRIPTEN_BASE=/usr/share/emsdk/upstream/emscripten" >> /usr/bin/bashwrapper


nvm.sh and keyman_use_emsdk as per core's Dockerfile

Addresses code review comments.

- use KEYMAN_USE_NVM and KEYMAN_USE_EMSDK - pre-install node (where necessary) to prevent having to do it on each build - adjust to current `master` - fix a few bugs in the Dockerfile - get required node version from package.json (through `shellHelperFunctions.sh`)

darcywong00 · 2024-09-01T00:43:03Z

resources/docker-images/README.md

+By default this will create 64-bit images for building
+Keyman Core, Keyman for Android, Keyman for Linux and
+Keyman for Web. These images are based on the Ubuntu 24.04
+with Node 20 and Emscripten 3.1.58 (for the exact versions,


Some of the Dockerfiles below use Node 18?

ARG REQUIRED_NODE_VERSION=18

No, that's just the default value that doesn't get used. The node version get's passed in. I had to put something here, so I choose a somewhat sensible value that is different from what we currently have so that it's easier noticable if passing the value doesn't work.

Should be node 20 -- I think the req shifted since this PR started

Also add test action to build.sh which builds all images and then tests them by running configure,build,test on each. Also add new dependencies to web image which Playwright requires.

These changes bring us one step further, but the build is still failing.

Setting permissions is now done in the base image.

github-actions bot added linux/ common/ common/resources/ Build infrastructure docs chore labels May 8, 2024

keymanapp-test-bot bot added this to the A18S1 milestone May 8, 2024

darcywong00 modified the milestones: A18S1, A18S2 May 11, 2024

mcdurdin modified the milestones: A18S2, A18S3 May 24, 2024

mcdurdin modified the milestones: A18S3, A18S4 Jun 7, 2024

ermshiperete force-pushed the chore/linux/cicontainer branch 2 times, most recently from 90fe8aa to 1e10c81 Compare June 14, 2024 07:14

darcywong00 modified the milestones: A18S4, A18S5 Jun 21, 2024

ermshiperete added 8 commits July 4, 2024 13:14

docs(common): improve formatting of builder.md

2dffec3

change(common): add readme and build.sh

d554415

feat(common): add base CI image

5c337be

feat(linux): add CI image for building Keyman for Linux

72bc6c7

This change allows to build a docker image that can build Keyman for Linux with test coverage reports, and installs the necessary dependencies for running integration tests.

feat(web): add CI image for building Keyman Web

2767765

feat(android): add CI image for building Keyman for Android

23d8a01

feat(core): add CI image for building Keyman Core

8a4cdce

ermshiperete force-pushed the chore/linux/cicontainer branch from 1e10c81 to 8a4cdce Compare July 4, 2024 14:39

github-actions bot added the core/ Keyman Core label Jul 4, 2024

ermshiperete changed the base branch from master to fix/core/build July 4, 2024 14:40

ermshiperete marked this pull request as ready for review July 4, 2024 14:41

ermshiperete requested a review from mcdurdin as a code owner July 4, 2024 14:41

ermshiperete requested review from jahorton and darcywong00 as code owners July 4, 2024 14:41

ermshiperete removed the core/ Keyman Core label Jul 4, 2024

Base automatically changed from fix/core/build to master July 5, 2024 05:39

darcywong00 modified the milestones: A18S5, A18S6 Jul 8, 2024

darcywong00 modified the milestones: A18S6, A18S7 Jul 19, 2024

darcywong00 modified the milestones: A18S7, A18S8 Aug 2, 2024

darcywong00 modified the milestones: A18S8, A18S9 Aug 17, 2024

Merge branch 'master' into chore/linux/cicontainer

1ad55b1

mcdurdin requested changes Aug 27, 2024

View reviewed changes

ermshiperete added 2 commits August 28, 2024 14:15

feat(common): add script to run build and tests in Docker container

ad4bbdf

Addresses code review comments.

Merge branch 'master' into chore/linux/cicontainer

cc5fef0

darcywong00 modified the milestones: A18S9, A18S10 Aug 31, 2024

darcywong00 reviewed Sep 1, 2024

View reviewed changes

ermshiperete added 7 commits September 1, 2024 13:08

chore(linux): pre-install EMSCRIPTEN

1772fdf

Also add test action to build.sh which builds all images and then tests them by running configure,build,test on each. Also add new dependencies to web image which Playwright requires.

Merge branch 'master' into chore/linux/cicontainer

6b34c5c

chore(linux): fix build order

0d6f655

docs(android): document gradle version in minimum-versions

475fd73

chore(linux): attempt to fix Android build in Docker

caba09a

These changes bring us one step further, but the build is still failing.

chore(linux): remove temporary code for Core image

e23e112

chore(linux): cleanup

e54f822

Setting permissions is now done in the base image.

mcdurdin modified the milestones: A18S10, 19.0 Sep 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Add docker images for building the different platforms #11397

chore: Add docker images for building the different platforms #11397

ermshiperete commented May 8, 2024 •

edited

Loading

keymanapp-test-bot bot commented May 8, 2024 •

edited

Loading

mcdurdin left a comment

mcdurdin Aug 27, 2024

mcdurdin Aug 27, 2024

mcdurdin Aug 27, 2024

mcdurdin Aug 27, 2024

mcdurdin Aug 27, 2024

mcdurdin Aug 27, 2024

mcdurdin Aug 27, 2024

mcdurdin Aug 27, 2024

mcdurdin Aug 27, 2024

mcdurdin Aug 27, 2024

darcywong00 Sep 1, 2024

ermshiperete Sep 1, 2024

mcdurdin Sep 2, 2024

	container images. This is easiest achieved by installing the [official
	container images. This is most easily achieved by installing the [official

	3.1.44 (for the exact versions, see [`minimum-versions.inc.sh`](../build/minimum-versions.inc.sh))
	3.1.58 (for the exact versions, see
	[`minimum-versions.inc.sh`](../build/minimum-versions.inc.sh))

	# build 'Keyman Core' in docker
	# test 'Keyman Core' in docker

chore: Add docker images for building the different platforms #11397

Are you sure you want to change the base?

chore: Add docker images for building the different platforms #11397

Conversation

ermshiperete commented May 8, 2024 • edited Loading

keymanapp-test-bot bot commented May 8, 2024 • edited Loading

User Test Results

Test Artifacts

mcdurdin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ermshiperete commented May 8, 2024 •

edited

Loading

keymanapp-test-bot bot commented May 8, 2024 •

edited

Loading