building ue4-build-prerequisites fails at apt update

[superyu1337]

Output of the ue4-docker info command:

ue4-docker version:         0.0.114 (latest available version is 0.0.114)
Operating system:           Linux (Arch Linux, 6.11.4-zen2-1-zen)
Docker daemon version:      27.3.1
NVIDIA Docker supported:    No
Maximum image size:         No limit detected
Available disk space:       500.34 GiB
Total system memory:        62.7 GiB physical, 36 GiB virtual
CPU:                        16 physical, 32 logical ()

Additional details:

• Are you accessing the network through a proxy server? No

Build output:

➜  docker-root ue4-docker build 4.24.3              
[ue4-docker build] COMMAND-LINE INVOCATION:
[ue4-docker build] ['ue4-docker', '4.24.3']

[ue4-docker build] UNREAL ENGINE VERSION SETTINGS:
[ue4-docker build] Custom build:  No
[ue4-docker build] Release:       4.24.3
[ue4-docker build] Repository:    https://github.com/EpicGames/UnrealEngine.git
[ue4-docker build] Branch/tag:    4.24.3-release

[ue4-docker build] ADVANCED CONFIGURATION OPTIONS:
[ue4-docker build] credential_mode: "secrets"
[ue4-docker build] excluded_components: {"ddc": false, "debug": false, "templates": false}
[ue4-docker build] gitdependencies_args: "--exclude=Android --exclude=Mac --exclude=Win32 --exclude=Win64"

[ue4-docker build] LINUX CONTAINER SETTINGS
[ue4-docker build] Base OS image: nvidia/opengl:1.0-glvnd-devel-ubuntu22.04

[ue4-docker build] GENERAL SETTINGS
[ue4-docker build] Build targets: build-prerequisites minimal source
[ue4-docker build] Changelist override: (None specified)
[ue4-docker build] Not excluding any Engine components.

Retrieving the Git credentials that will be used to clone the UE4 repo
Username: superyu1337
Access token or password: 


[ue4-docker build] Building image "adamrehn/ue4-build-prerequisites:opengl-ubuntu22.04"...
[+] Building 22.1s (5/9)                                                                                                                                                                                                     docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 4.10kB                                                                                                                                                                                                 0.0s
 => WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 2)                                                                                                                                                         0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG ${BASEIMAGE} results in empty or invalid base image name (line 2)                                                                                                             0.0s
 => [internal] load metadata for docker.io/nvidia/opengl:1.0-glvnd-devel-ubuntu22.04                                                                                                                                                   1.3s
 => [internal] load .dockerignore                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                        0.0s
 => [1/6] FROM docker.io/nvidia/opengl:1.0-glvnd-devel-ubuntu22.04@sha256:c030a21588ddb89c245c8a2a24d6e68772cfa5939d312bab19f0833b90e9620a                                                                                            16.3s
 => => resolve docker.io/nvidia/opengl:1.0-glvnd-devel-ubuntu22.04@sha256:c030a21588ddb89c245c8a2a24d6e68772cfa5939d312bab19f0833b90e9620a                                                                                             0.0s
 => => sha256:1bc677758ad7fa4503417ae5be18809c5a8679b5b36fcd1464d5a8e41cb13305 30.43MB / 30.43MB                                                                                                                                       0.9s
 => => sha256:004619d771963a6538b82e9a6f657b6a5286a9ef59a7e88eddc268aeecfc6106 7.25MB / 7.25MB                                                                                                                                         0.9s
 => => sha256:cf44605dc541c5b92e23b49f974d95802af2f2da5b35af03846c9603962996a2 186B / 186B                                                                                                                                             0.7s
 => => sha256:c030a21588ddb89c245c8a2a24d6e68772cfa5939d312bab19f0833b90e9620a 1.79kB / 1.79kB                                                                                                                                         0.0s
 => => sha256:96334240cf5a2a45373deeec7b2f751a66741457b2cf3cf67480fd90b575fe0e 5.28kB / 5.28kB                                                                                                                                         0.0s
 => => sha256:df6e1db9485a3e9606c978d735853b66a824aaf7cb57437e92259e1ff9db005c 6.43kB / 6.43kB                                                                                                                                         0.9s
 => => sha256:0304682b62bc4f472af25e5e3be7b845d33a889ed17300eb41cae79cc345bec2 131.33MB / 131.33MB                                                                                                                                     4.6s
 => => sha256:f9f325c4255e261572a3145bf296593a52b884910ef1426ef2155faafb72b33d 298B / 298B                                                                                                                                             1.0s
 => => extracting sha256:1bc677758ad7fa4503417ae5be18809c5a8679b5b36fcd1464d5a8e41cb13305                                                                                                                                              1.2s
 => => sha256:5ace3e86e3122c2418e1f757c0e4525181a28b9ad3b8cdca9d1a10091fc62248 16.07MB / 16.07MB                                                                                                                                       2.0s
 => => extracting sha256:004619d771963a6538b82e9a6f657b6a5286a9ef59a7e88eddc268aeecfc6106                                                                                                                                              1.1s
 => => extracting sha256:cf44605dc541c5b92e23b49f974d95802af2f2da5b35af03846c9603962996a2                                                                                                                                              0.0s
 => => extracting sha256:df6e1db9485a3e9606c978d735853b66a824aaf7cb57437e92259e1ff9db005c                                                                                                                                              0.0s
 => => extracting sha256:0304682b62bc4f472af25e5e3be7b845d33a889ed17300eb41cae79cc345bec2                                                                                                                                              2.2s
 => => extracting sha256:f9f325c4255e261572a3145bf296593a52b884910ef1426ef2155faafb72b33d                                                                                                                                              0.0s
 => => extracting sha256:5ace3e86e3122c2418e1f757c0e4525181a28b9ad3b8cdca9d1a10091fc62248                                                                                                                                              3.5s
 => ERROR [2/6] RUN apt-get update && apt-get install -y --no-install-recommends   build-essential   ca-certificates   curl   git   git-lfs   gpg-agent   python3   python3-dev   python3-pip   shared-mime-info   software-propertie  4.3s
------                                                                                                                                                                                                                                      
 > [2/6] RUN apt-get update && apt-get install -y --no-install-recommends               build-essential                 ca-certificates                 curl            git             git-lfs                 gpg-agent               python3             python3-dev             python3-pip             shared-mime-info                software-properties-common              sudo            tzdata          unzip           xdg-user-dirs           xdg-utils               zip &&      rm -rf /var/lib/apt/lists/* && echo '' && echo 'RUN directive complete. Docker will now commit the filesystem layer to disk.' && echo 'Note that for large filesystem layers this can take quite some time.' && echo 'Performing filesystem layer commit...' && echo '':                                                                                                                                                                                                        
3.685 Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]                                                                                                                                                                       
3.688 Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
3.837 Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
3.865 Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
3.905 Err:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
3.905   At least one invalid signature was encountered.
4.078 Err:1 http://archive.ubuntu.com/ubuntu jammy InRelease
4.078   At least one invalid signature was encountered.
4.182 Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
4.182   At least one invalid signature was encountered.
4.295 Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
4.295   At least one invalid signature was encountered.
4.299 Reading package lists...
4.314 W: GPG error: http://security.ubuntu.com/ubuntu jammy-security InRelease: At least one invalid signature was encountered.
4.314 E: The repository 'http://security.ubuntu.com/ubuntu jammy-security InRelease' is not signed.
4.314 W: GPG error: http://archive.ubuntu.com/ubuntu jammy InRelease: At least one invalid signature was encountered.
4.314 E: The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed.
4.314 W: GPG error: http://archive.ubuntu.com/ubuntu jammy-updates InRelease: At least one invalid signature was encountered.
4.314 E: The repository 'http://archive.ubuntu.com/ubuntu jammy-updates InRelease' is not signed.
4.314 W: GPG error: http://archive.ubuntu.com/ubuntu jammy-backports InRelease: At least one invalid signature was encountered.
4.314 E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed.
------
Dockerfile:23
--------------------
  22 |     # Install our build prerequisites
  23 | >>> RUN apt-get update && apt-get install -y --no-install-recommends \
  24 | >>>              build-essential \
  25 | >>>              ca-certificates \
  26 | >>>              curl \
  27 | >>>              git \
  28 | >>>              git-lfs \
  29 | >>>              gpg-agent \
  30 | >>>              python3 \
  31 | >>>              python3-dev \
  32 | >>>              python3-pip \
  33 | >>>              shared-mime-info \
  34 | >>>              software-properties-common \
  35 | >>>              sudo \
  36 | >>>              tzdata \
  37 | >>>              unzip \
  38 | >>>              xdg-user-dirs \
  39 | >>>              xdg-utils \
  40 | >>>              zip && \
  41 | >>>      rm -rf /var/lib/apt/lists/* && echo '' && echo 'RUN directive complete. Docker will now commit the filesystem layer to disk.' && echo 'Note that for large filesystem layers this can take quite some time.' && echo 'Performing filesystem layer commit...' && echo ''
  42 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y --no-install-recommends \t\tbuild-essential \t\tca-certificates \t\tcurl \t\tgit \t\tgit-lfs \t\tgpg-agent \t\tpython3 \t\tpython3-dev \t\tpython3-pip \t\tshared-mime-info \t\tsoftware-properties-common \t\tsudo \t\ttzdata \t\tunzip \t\txdg-user-dirs \t\txdg-utils \t\tzip && \trm -rf /var/lib/apt/lists/* && echo '' && echo 'RUN directive complete. Docker will now commit the filesystem layer to disk.' && echo 'Note that for large filesystem layers this can take quite some time.' && echo 'Performing filesystem layer commit...' && echo ''" did not complete successfully: exit code: 100
[ue4-docker build] Error: failed to build image "adamrehn/ue4-build-prerequisites:opengl-ubuntu22.04".

Trying to build images keeps failing at the prerequisites. Could be related to #329, not sure.
Line that fails specifically seems to be this one.

I'm running on archlinux 6.11.4-zen2-1-zen, installed using pipx install ue4-docker.

[TBBle]

It doesn't look related to #329, it looks like the APT signatures are failing for some reason. It was working in March (seen in #353 (comment)) and the relevant files don't appear to have changed since 2022, and there's definitely a signature at the end of http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease so I don't immediately know what's going on here.

That said, this is over http, not https, so I wonder if there's some kind of intervening transparent proxy that is modifying the file in-flight?

What if you try running the official ubuntu:22.04 image from Docker Hub and apt-get update in that, does it work? If not, then something in your network environment is corrupting something on the wire, either the package update or the original image.

If ubuntu:22.04 works, then try running the nvidia/opengl:1.0-glvnd-devel-ubuntu22.04 image from Docker Hub and see if apt-get update in that works.

If the ubuntu image works but not the nvidia/opengl image, then something has changed in Ubuntu's infrastructure, e.g. old keys have expired and new keys added; the nvidia/opengl image hasn't been changed since May 2023, but the ubuntu image is being kept up-to-date. I don't think this is the case, because the files in question haven't changed since 2022, so could not have been resigned.

And just a quick sanity check: Is the system clock correct on your host?

[superyu1337]

And just a quick sanity check: Is the system clock correct on your host?

Actually this could be it, I was SSHing into my home machine from work through a vpn. And I run a windows dual boot at home which could mess with the system clock I guess.
I'll be home soon and test. I'll also try updating the base ubuntu:22.04 image.

[slonopotamus]

I've just retested, ue4-docker build --target=build-prerequisites builds properly on Linux

[superyu1337]

Issue seems to be related to my home machine for sure. On my office PC I'm able to run apt-get update within ubuntu:22.04 fine, but not on my home PC. I'll investigate further when I'm home.

[superyu1337]

I had my docker data set to an NTFS partition due to space constaints, im running docker on the default storage location now and ubuntu:22.04 is updating fine now... interesting.

[TBBle]

I'm guessing your setup is something like moby/moby#25328 which I suspect is not well (or at all) supported; either way, not something we can do anything about in ue4-docker.