[WIP] Support for @v/list and @latest endpoints in an offline environment #1676
Conversation
…mined for the @v/list and @latest endpoints
|
@cobalt77 I wanted to let you know that we're not ignoring this, I just need to take my time reviewing it. this feature has a relatively long history of discussion and I need to research all the relevant past PRs and issues so I can properly comment on it. I'll do that and come back to it early next week. |
There was a problem hiding this comment.
@cobalt77 except for my comment about mode.None, I think this is a good step toward enabling an "offline" mode that @nathanhack and others talked about in this thread
| @@ -74,19 +77,27 @@ func (p *protocol) List(ctx context.Context, mod string) ([]string, error) { | |||
| ctx, span := observ.StartSpan(ctx, op.String()) | |||
| defer span.End() | |||
|
|
|||
| // If the download mode is None, we won't be downloading anything from | |||
| // the internet anyway, so we should only examine local storage. | |||
| localOnly := p.df.Match(mod) == mode.None | |||
There was a problem hiding this comment.
@cobalt77 I think we need a different signal to determine when to not look to the internet. mode.None means that a 404 should be returned for this module, unless I'm misunderstanding something?
There was a problem hiding this comment.
Thought a lot about this. Let's say I have the following setup:
- Offline environment, and a self-hosted git server with some go modules
- Athens with file based storage and a dynamic config where all modules except my.git.server.com/* get download mode "None"
If I go get github.com/something, that will use download mode None and treat the contents of the athens cache as ground truth, and get me the latest of the available modules.
If I go get my.git.server.com/somethingelse, athens will go out to my git server for the module, and not assume that what's in cache is the latest.
This will work as expected with this change. But it changes what None does, and maybe there is a scenario where a 404 would be preferable. Or in online environments, athens treating the cache as ground truth sometimes but not all the time could lead to nondeterministic behavior or other weird issues.
What do you think about keeping None as-is (always 404 if it doesn't exist), and adding a new separate download mode that will do this?
| if err != nil { | ||
| return nil, errors.E(op, err) | ||
|
|
||
| localOnly := p.df.Match(mod) == mode.None |
What is the problem I am trying to address?
I am using Athens to use go modules in an offline, air-gapped environment. With download mode set to none, the @v/list and @latest api endpoints are still trying to go out to the internet to retrieve version information.
How is the fix applied?
When download mode is set to none and disk storage is used, @v/list will list only the modules available on disk, and @latest will return the latest version from the modules available on disk. It will not try to go out to the internet, since nothing can be downloaded anyway with download mode set to none.
So far these changes are working perfectly in my environment. I am able to develop using go modules as if I was online with no workflow changes or workarounds.
TODO: the implementation of @latest (sorting all the versions) could probably be moved somewhere else and/or simplified.
For reference, I am using the following system to synchronize the disk storage between my online and offline networks:
go mod downloaddoesn't really work, because (as far as I can tell) outdated semver-compatible dependencies need to be found before the go module system can decide that they can be updated. For example: your package directly requires foo v1.0.2. A dependency of yours requires foo v1.0.1. When online, go finds v1.0.1, but also knows that v1.0.2 is available, and the dependency's foo is updated to v1.0.2 automatically since it's semver compatible. But when offline, go still needs to be able to locate v1.0.1, which means we need both versions in our Athens disk cache.go mod downloadwill only give you v1.0.2.This works and it's relatively easy. It would be nice to be able to configure Athens to serve the default download cache disk format instead of needing to re-structure the files. Maybe that could be a potential improvement for this PR.
Reference issue
Closes #1868