Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Support for @v/list and @latest endpoints in an offline environment #1676

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[WIP] Support for @v/list and @latest endpoints in an offline environment #1676

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6a49bb8
When the download mode is set to none, only local storage will be exa…
kralicky Nov 5, 2020
b9e5ec4
Merge branch 'main' into feature/offline-disk-storage
arschles Nov 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ require (
github.com/aws/aws-sdk-go v1.32.7
github.com/bsm/redislock v0.4.2
github.com/codegangsta/negroni v1.0.0 // indirect
github.com/coreos/go-semver v0.2.0
github.com/fatih/color v1.7.0
github.com/go-playground/locales v0.12.1 // indirect
github.com/go-playground/universal-translator v0.16.0 // indirect
Expand Down Expand Up @@ -50,7 +51,7 @@ require (
go.opencensus.io v0.22.3
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6 // indirect
golang.org/x/lint v0.0.0-20200130185559-910be7a94367 // indirect
golang.org/x/mod v0.2.0
golang.org/x/mod v0.3.0
golang.org/x/net v0.0.0-20200222125558-5a598a2470a0 // indirect
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,8 +420,8 @@ golang.org/x/mod v0.1.0 h1:sfUMP1Gu8qASkorDVjnMuvgJzwFbTZSeXFiGBYAVdl4=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0 h1:KU7oHjnv3XNWfa5COkzUifxZmxp1TyI7ImMXqFxLwvQ=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180811021610-c39426892332/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
Expand Down
57 changes: 47 additions & 10 deletions pkg/download/protocol.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@ package download

import (
"context"
"encoding/json"
"io"
"regexp"
"sort"
"strings"
"sync"

Expand All @@ -13,6 +15,7 @@ import (
"github.com/gomods/athens/pkg/observ"
"github.com/gomods/athens/pkg/stash"
"github.com/gomods/athens/pkg/storage"
"golang.org/x/mod/semver"
)

// Protocol is the download protocol which mirrors
Expand Down Expand Up @@ -74,19 +77,27 @@ func (p *protocol) List(ctx context.Context, mod string) ([]string, error) {
ctx, span := observ.StartSpan(ctx, op.String())
defer span.End()

// If the download mode is None, we won't be downloading anything from
// the internet anyway, so we should only examine local storage.
localOnly := p.df.Match(mod) == mode.None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cobalt77 I think we need a different signal to determine when to not look to the internet. mode.None means that a 404 should be returned for this module, unless I'm misunderstanding something?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought a lot about this. Let's say I have the following setup:

  • Offline environment, and a self-hosted git server with some go modules
  • Athens with file based storage and a dynamic config where all modules except my.git.server.com/* get download mode "None"

If I go get github.com/something, that will use download mode None and treat the contents of the athens cache as ground truth, and get me the latest of the available modules.
If I go get my.git.server.com/somethingelse, athens will go out to my git server for the module, and not assume that what's in cache is the latest.

This will work as expected with this change. But it changes what None does, and maybe there is a scenario where a 404 would be preferable. Or in online environments, athens treating the cache as ground truth sometimes but not all the time could lead to nondeterministic behavior or other weird issues.

What do you think about keeping None as-is (always 404 if it doesn't exist), and adding a new separate download mode that will do this?


var strList, goList []string
var sErr, goErr error
var wg sync.WaitGroup
wg.Add(2)
wg.Add(1)

go func() {
defer wg.Done()
strList, sErr = p.storage.List(ctx, mod)
}()
if !localOnly {
wg.Add(1)

go func() {
defer wg.Done()
_, goList, goErr = p.lister.List(ctx, mod)
}()
}

go func() {
defer wg.Done()
_, goList, goErr = p.lister.List(ctx, mod)
strList, sErr = p.storage.List(ctx, mod)
}()

wg.Wait()
Expand Down Expand Up @@ -145,12 +156,38 @@ func (p *protocol) Latest(ctx context.Context, mod string) (*storage.RevInfo, er
const op errors.Op = "protocol.Latest"
ctx, span := observ.StartSpan(ctx, op.String())
defer span.End()
lr, _, err := p.lister.List(ctx, mod)
if err != nil {
return nil, errors.E(op, err)

localOnly := p.df.Match(mod) == mode.None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as my comment above

if localOnly {
sList, err := p.storage.List(ctx, mod)
if err != nil {
return nil, err
}
if len(sList) == 0 {
return nil, errors.E(op, errors.KindNotFound)
}
var sorted []string
for _, v := range sList {
sorted = append(sorted, semver.Canonical(v))
}
sort.Slice(sorted, func(i, j int) bool {
return semver.Compare(sorted[i], sorted[j]) == -1
})
latest := sorted[len(sorted)-1]
bytes, err := p.storage.Info(ctx, mod, latest)
if err != nil {
return nil, err
}
var revInfo storage.RevInfo
err = json.Unmarshal(bytes, &revInfo)
if err != nil {
return nil, err
}
return &revInfo, err
}

return lr, nil
revInfo, _, err := p.lister.List(ctx, mod)
return revInfo, err
}

func (p *protocol) Info(ctx context.Context, mod, ver string) ([]byte, error) {
Expand Down