randomly exec: "plugin-git": executable file not found in $PATH

[healdropper]

Hi, I'm running a woodpecker server and agents on next version (master). I barely saw this error when I only had one worker but since I increased it to 16 workers, It happens way more often.

I'm using the default clone step in all my pipelines and I 've already read this documentaion https://woodpecker-ci.org/docs/next/usage/pipeline-syntax#clone

Thanks

[healdropper]

Using a custom clone config

clone: git: image: woodpeckerci/plugin-git

randomly gives this error

[qwerty287]

Which backend do you use?

[healdropper]

Backend? Not sure what do you mean, sorry

[qwerty287]

How are the commands executed? Docker, Kubernetes, SSH or locally?

[healdropper]

Docker, both server and agents configured in the same docker-compose file

[anbraten]

Which os and architecture is it?

[healdropper]

Hetzner Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 5950X 16-Core Processor
CPU family: 25
Model: 33
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1

[azmeuk]

Which backend do you use?

Same message exec: "plugin-git": executable file not found in $PATH with the latest git woodpecker running on debian. Also same message fork/exec woodpeckerci/plugin-git: no such file or directory when trying to override the clone configuration with the documentation example.

Strangely enough, everything worked fine before I rebooted the system. Now every job fails to clone.

Debian GNU/Linux 11 (bullseye) x86_64
Kernel: 5.10.0-21-amd64
go version go1.20.1 linux/amd64
Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:46:54 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:46:54 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[azmeuk]

I could fix the issue by removing all the docker images and containers, then restarting the docker and woodpecker services. I am not sure what happened though.

docker rmi -f $(docker images -aq)
docker system prune -a --volumes
sudo systemctl restart docker

[6543]

hmm could be related to the default $PATH var used ... by docker

[6543]

but it should find it in any case:

plugin-git/docker/Dockerfile.multiarch

Line 19 in 4447028

ENTRYPOINT ["/bin/plugin-git"]

[6543]

https://stackoverflow.com/a/37904830 ; more

we might should try if moving to the "shell form" does fix it?

cc @anbraten
I can create a test image that has only this change ...
... @healdropper would you try that one?

[healdropper]

Yes @6543 I could test it our CI server for sure, let me know

[healdropper]

@6543 any update about this? Thank you

[healdropper]

I created an internal image with the proposed change, I let you know @6543

[healdropper]

It definitely improved the ratio it happens, but still occurs 1 out of 15 executions.