Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding a better error message for download mode #1492

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
0224e53
Adding a better error message for download mode
arschles Dec 12, 2019
fa2631c
Fixing test
arschles Dec 12, 2019
834eba4
Making some progress toward removing the specific config error
arschles Dec 13, 2019
f45a03b
Fixing last test compile error
arschles Dec 13, 2019
4715103
Merge branch 'master' into config-msgs
arschles Dec 19, 2019
e928a0f
Merge branch 'master' into config-msgs
arschles Feb 14, 2020
32e7c28
Moving downloadModeURL to package-global
arschles Feb 14, 2020
a846557
using a valid HTTP error code instead of -1
arschles Feb 14, 2020
25b5860
removing the config.Validate function
arschles Feb 14, 2020
36b6d62
making the DownloadMode field required
arschles Feb 14, 2020
2c6b90b
validating that the port starts with ':'
arschles Feb 14, 2020
5b85142
Merge branch 'master' into config-msgs
arschles Feb 21, 2020
65662f5
Not passing an op to Validate functions
arschles Feb 21, 2020
e962a4d
Not checking for error strings in mode test
arschles Feb 21, 2020
1b29dcf
more doc for Validator
arschles Feb 21, 2020
d6a7cbd
removing struct tag for config port validation
arschles Feb 21, 2020
de71cb7
Merge branch 'master' into config-msgs
arschles Mar 10, 2020
cc70e12
Removing the unnecessary validator interface
arschles Mar 10, 2020
a5eda16
Joining error strings with newline+tabs instead of just a space
arschles Mar 10, 2020
198d574
s/retFile/df
arschles Mar 10, 2020
ee8f4d1
validating each download path in DownloadFile.Validate
arschles Mar 10, 2020
2cc9f84
Checking for custom or file modes
arschles Mar 10, 2020
9695c36
Checking top-level DownloadURL
arschles Mar 10, 2020
726b77c
Merge branch 'config-msgs' of github.com:arschles/athens into config-…
arschles Mar 10, 2020
4320101
validating the download path mode only in DownloadPath.Validate
arschles Mar 10, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions cmd/proxy/actions/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -130,8 +130,7 @@ func App(conf *config.Config) (http.Handler, error) {
lggr,
conf,
); err != nil {
err = fmt.Errorf("error adding proxy routes (%s)", err)
return nil, err
return nil, fmt.Errorf("Error initializing Athens:\n%s", err)
}

h := &ochttp.Handler{
Expand Down
16 changes: 15 additions & 1 deletion pkg/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,25 @@ type Config struct {
TLSKeyFile string `envconfig:"ATHENS_TLSKEY_FILE"`
SumDBs []string `envconfig:"ATHENS_SUM_DBS"`
NoSumPatterns []string `envconfig:"ATHENS_GONOSUM_PATTERNS"`
DownloadMode mode.Mode `envconfig:"ATHENS_DOWNLOAD_MODE"`
DownloadMode mode.Mode `validate:"required" envconfig:"ATHENS_DOWNLOAD_MODE" default:"sync"`
DownloadURL string `envconfig:"ATHENS_DOWNLOAD_URL"`
SingleFlightType string `envconfig:"ATHENS_SINGLE_FLIGHT_TYPE"`
RobotsFile string `envconfig:"ATHENS_ROBOTS_FILE"`
SingleFlight *SingleFlight
Storage *StorageConfig
}

// Validate implements Validator
func (c *Config) Validate() error {
const op errors.Op = "Config.Validate"
if !strings.HasPrefix(c.Port, ":") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will break people who are binding 0.0.0.0:<port> -- plus our code in ensurePortFormat will always add the ":", so I think it's best we remove this validation and if the PORT configuration that the user put is wrong, Go will tell them when the server doesn't boot up.

return errors.E(
op,
"Invalid configuration for Port. This value must start with ':'",
)
}
}

// EnvList is a list of key-value environment
// variables that are passed to the Go command
type EnvList []string
Expand Down Expand Up @@ -266,6 +277,9 @@ func validateConfig(config Config) error {
if err != nil {
return err
}
if err := config.Validate(); err != nil {
return err
}
switch config.StorageType {
case "memory":
return nil
Expand Down
8 changes: 8 additions & 0 deletions pkg/config/validator.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
package config

// Validator can validate a config struct. If you implement this,
// validate all of the configuration in your struct. It will
// automatically be called when Athens starts
type Validator interface {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely correct me if I'm wrong but:

Do we need a validator interface? I don't see any code actually dealing with the "interface" aspect of things. All the Validate functions are being called explicitly. AKA nothing is being abstracted here :)

I think we can make a type Validator func() error -- but even then it might be more for documentation than an actual use case.

Also the documentation here I think might be a little misleading:

If you implement this, validate all of the configuration in your struct. It will automatically be called when Athens starts

To the same point above, we need to explicitly call "Validate" and so there's nothing automatic happening right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I got ahead of myself here. My intention was to have the config.Validate function look for Validators in each of its fields, calling validate on each of them. Right now, Validate is called as necessary on each field that implements it, in various other parts of the code.

I'll leave Validator out of this PR

Validate() error
}
77 changes: 59 additions & 18 deletions pkg/download/mode/mode.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,33 @@ type Mode string

// DownloadMode constants. For more information see config.dev.toml
const (
Sync Mode = "sync"
Async Mode = "async"
Redirect Mode = "redirect"
AsyncRedirect Mode = "async_redirect"
None Mode = "none"
downloadModeErr = "download mode is not set"
invalidModeErr = "unrecognized download mode: %s"
Sync Mode = "sync"
Async Mode = "async"
Redirect Mode = "redirect"
AsyncRedirect Mode = "async_redirect"
None Mode = "none"
// This is the URL that logs will show when the DownloadMode
// config value is invalid
downloadModeURL = "https://docs.gomods.io/configuration/download/"
)

// Validate ensures that m is a valid mode. If this function returns nil, you are
// guaranteed that m is valid
func (m Mode) Validate() error {
const op errors.Op = "Mode.Validate"
switch m {
case Sync, Async, Redirect, AsyncRedirect, None:
return nil
default:
return errors.Config(
op,
"mode",
fmt.Sprintf("%s isn't a valid value.", m),
"https://docs.gomods.io/configuration/download/",
)
}
}

// DownloadFile represents a custom HCL format of
// how to handle module@version requests that are
// not found in storage.
Expand All @@ -44,6 +62,20 @@ type DownloadPath struct {
DownloadURL string `hcl:"downloadURL,optional"`
}

// Validate implements config.Validator
func (d DownloadPath) Validate() error {
const op errors.Op = "DownloadPath.Validate"
if d.DownloadURL == "" && (d.Mode == Redirect || d.Mode == AsyncRedirect) {
return errors.Config(
op,
fmt.Sprintf("DownloadURL (inside %s in the download file)", d.Pattern),
"You must set a value when the download mode is 'redirect' or 'async_redirect'",
"https://docs.gomods.io/configuration/download/",
)
}
return nil
}

// NewFile takes a mode and returns a DownloadFile.
// Mode can be one of the constants declared above
// or a custom HCL file. To pass a custom HCL file,
Expand All @@ -53,8 +85,8 @@ type DownloadPath struct {
func NewFile(m Mode, downloadURL string) (*DownloadFile, error) {
const op errors.Op = "downloadMode.NewFile"

if m == "" {
return nil, errors.E(op, downloadModeErr)
if err := m.Validate(); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this error is now less readable, because before it used to say download mode is not set but now it will say isn't a valid value (notice the space in the beginning because there's nothing)

I think in the Sprintf of the validate function we should pass a %q so it would say "" isn't a valid value or we should keep the old error here for better messaging.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about if we pass %q as you said, and also prefix the error with "Download mode ..."?
The result would then be Download mode "" isn't a valid value

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds great

return nil, err
}

if strings.HasPrefix(string(m), "file:") {
Expand All @@ -72,12 +104,11 @@ func NewFile(m Mode, downloadURL string) (*DownloadFile, error) {
return parseFile(bts)
}

switch m {
case Sync, Async, Redirect, AsyncRedirect, None:
return &DownloadFile{Mode: m, DownloadURL: downloadURL}, nil
default:
return nil, errors.E(op, errors.KindBadRequest, fmt.Sprintf(invalidModeErr, m))
retFile := &DownloadFile{Mode: m, DownloadURL: downloadURL}
marwan-at-work marked this conversation as resolved.
Show resolved Hide resolved
if err := retFile.Validate(); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I'm wrong but:

The old code that was removed here no longer works because in the old code we are checking the top level Mode, but in this validate function only the internal d.Paths[n].Mode is checked.

However, since top level Mode is now already checked above, then this Validate function is not necessary because retFile has no Paths to begin with.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marwan-at-work you're right. I left a few things out of this Validate and some tangential functions

  • DownloadFile.Validate needs to check DownloadURL for well-formedness
  • ^^ also needs to call DownloadPath.Validate for each d.Paths[n] - that does more than just validate its mode
  • Mode.Validate does not check for custom: and file: values

I've made changes to address all this

I've fixed all of that

return nil, err
}
return retFile, nil
}

// parseFile parses an HCL file according to the
Expand All @@ -93,19 +124,29 @@ func parseFile(file []byte) (*DownloadFile, error) {
if dig.HasErrors() {
return nil, errors.E(op, dig.Error())
}
if err := df.validate(); err != nil {
if err := df.Validate(); err != nil {
return nil, errors.E(op, err)
}
return &df, nil
}

func (d *DownloadFile) validate() error {
const op errors.Op = "downloadMode.validate"
// Validate validates the download file. It implements the
// config.Validator interface
func (d *DownloadFile) Validate() error {
const op errors.Op = "DownloadFile.Validate"
for _, p := range d.Paths {
if err := p.Mode.Validate(); err != nil {
return err
}
switch p.Mode {
case Sync, Async, Redirect, AsyncRedirect, None:
default:
return errors.E(op, fmt.Errorf("unrecognized mode for %v: %v", p.Pattern, p.Mode))
return errors.Config(
op,
fmt.Sprintf("mode (in pattern %v", p.Pattern),
fmt.Sprintf("%s is unrecognized", p.Mode),
"https://docs.gomods.io/configuration/download/",
)
}
}
return nil
Expand Down
54 changes: 42 additions & 12 deletions pkg/download/mode/mode_test.go
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
package mode

import (
"fmt"
"testing"

"github.com/gomods/athens/pkg/errors"
)

var testCases = []struct {
Expand Down Expand Up @@ -119,28 +120,57 @@ func TestMode(t *testing.T) {
}

func TestNewFile_err(t *testing.T) {
const op = errors.Op("downloadMode.NewFile")
tc := []struct {
name string
mode Mode
expected string
name string
mode Mode
hasErr bool
}{
{
name: "empty mode",
mode: "",
expected: downloadModeErr,
name: "empty mode",
mode: "",
hasErr: true,
},
{
name: "invalid mode",
mode: "invalidMode",
expected: fmt.Sprintf(invalidModeErr, "invalidMode"),
name: "invalid mode",
mode: "invalidMode",
hasErr: true,
},
}
for _, c := range tc {
t.Run(c.name, func(subT *testing.T) {
_, err := NewFile(c.mode, "github.com/gomods/athens")
if err.Error() != c.expected {
t.Fatalf("expected error %s from NewFile, got %s", c.expected, err.Error())
if c.hasErr && err == nil {
t.Errorf(
"Expected error for mode %s, but got none",
c.mode,
)
}
if !c.hasErr && err != nil {
t.Errorf(
"Expected no error for mode %s, but got %s",
c.mode,
err,
)
}
})
}
// loop through all of the valid modes
modeStrings := []string{
"sync",
"async",
"redirect",
"async_redirect",
"none",
}
for _, modeString := range modeStrings {
_, err := NewFile(Mode(modeString), "github.com/gomods/athens")
if err != nil {
t.Errorf(
"Expected no error for mode %s, got %s",
modeString,
err,
)
}
}
}
22 changes: 22 additions & 0 deletions pkg/errors/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package errors

import (
"fmt"
"strings"
)

// Config returns an error specifically tailored for reporting errors with configuration
// values. You can check for these errors by calling errors.Is(err, KindConfigError)
// (from the github.com/gomods/athens/pkg/errors package).
//
// Generally these kinds of errors should make Athens crash because it was configured
// improperly
func Config(op Op, field, helpText, url string) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this ever be called from outside the config package? why not just define it there and define it locally to minimize the API surface of this package?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope. I put it here just because I think it makes more sense to group a configuration error under the errors package. I'm willing to be convinced otherwise 😄

slc := []string{
fmt.Sprintf("There was a configuration error with %s. %s", field, helpText),
}
if url != "" {
slc = append(slc, fmt.Sprintf("Please see %s for more information.", url))
}
return E(op, KindConfigError, strings.Join(slc, " "))
marwan-at-work marked this conversation as resolved.
Show resolved Hide resolved
}
1 change: 1 addition & 0 deletions pkg/errors/errors.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ const (
KindRateLimit = http.StatusTooManyRequests
KindNotImplemented = http.StatusNotImplemented
KindRedirect = http.StatusMovedPermanently
KindConfigError = http.StatusUnprocessableEntity
)

// Error is an Athens system error.
Expand Down