Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using gcp storage, multiple "Error 403: Access denied., forbidden" in "gcp.removeInProgressMetadata" #2012

Open
RangelReale opened this issue Dec 12, 2024 · 0 comments

Comments

@RangelReale
Copy link

Describe the bug
Using gcp storage, very frequently I get this "Error 403: Access denied., forbidden" from GCS, always in the "gcp.removeInProgressMetadata" step.

Looking at the code, it seems this is using "object.update" to clear object metadata.
The service account has "storageAdmin" permissions, calling the API directly in the exact same file to remove the metadata works every time.

Error Message
If applicable add error message to help explain your problem or console output enclose in triple back-ticks eg,

```console
{"http-method":"GET","http-path":"/cloud.google.com/go/secretmanager/@v/v1.14.1.mod","kind":"Internal Server Error","level":"error","module":"","msg":"googleapi: Error 403: Access denied., forbidden","operation":"download.VersionModuleHandler","ops":["download.VersionModuleHandler","pool.GoMod","protocol.GoMod","protocol.processDownload","gcslock.Stash","stash.Pool","stasher.Stash","gcp.save","gcp.removeInProgressMetadata"],"request-id":"13cb4103-3c37-423a-b5d6-f7a7df1bbdf5","time":"2024-12-12T18:21:20Z","version":""}
```

To Reproduce
It happens every time on our big monorepo.

Expected behavior
No error as the service account have the correct permission.

Environment (please complete the following information):

  • OS: Linux 64bit
  • Go version : 1.23.1
  • Proxy version : 0.15.4
  • Storage gcp :

Additional context
It looks like some kind of locking problem, maybe it is setting the metadata but the file isn't really there yet as GCS is eventually consistent? I saw some terraform threads about this message being really "object not found" instead of a permission error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant