-
Notifications
You must be signed in to change notification settings - Fork 214
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into travis/announcing-2.17
- Loading branch information
Showing
135 changed files
with
22,583 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
--- | ||
title: Docs | ||
cascade: | ||
type: docs | ||
|
||
# Redirect | ||
type: _default | ||
layout: redirect | ||
params: | ||
redirect: ./overview | ||
--- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
--- | ||
title: Checks | ||
|
||
# Redirect | ||
type: _default | ||
layout: redirect | ||
params: | ||
unlisted: true | ||
redirect: /2/tasks/troubleshooting/ | ||
--- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
--- | ||
title: Common Errors | ||
weight: 10 | ||
sitemap: | ||
priority: 1.0 | ||
--- | ||
|
||
Linkerd is generally robust, but things can always go wrong! You'll find | ||
information here about the most common things that cause people trouble. | ||
|
||
## When in Doubt, Start With `linkerd check` | ||
|
||
Whenever you see anything that looks unusual about your mesh, **always** start | ||
with `linkerd check`. It will check a long series of things that have caused | ||
trouble for others and make sure that your configuration is sane, and it will | ||
point you to help for any problems it finds. It's hard to overstate how useful | ||
this command is. | ||
|
||
## Common Errors | ||
|
||
{{< docs/section-toc >}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
--- | ||
title: Failfast | ||
description: Failfast means that no endpoints are available. | ||
--- | ||
|
||
If Linkerd reports that a given service is in the _failfast_ state, it | ||
means that the proxy has determined that there are no available endpoints | ||
for that service. In this situation there's no point in the proxy trying | ||
to actually make a connection to the service - it already knows that it | ||
can't talk to it - so it reports that the service is in failfast and | ||
immediately returns an error from the proxy. | ||
|
||
The error will be either a 503 or a 504; see below for more information, | ||
but if you already know that the service is in failfast because you saw | ||
it in the logs, that's the important part. | ||
|
||
To get out of failfast, some endpoints for the service have to | ||
become available. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
--- | ||
title: HTTP 502 Errors | ||
description: HTTP 502 means connection errors between proxies. | ||
--- | ||
|
||
The Linkerd proxy will return a 502 error for connection errors between | ||
proxies. Unfortunately it's fairly common to see an uptick in 502s when | ||
first meshing a workload that hasn't previously been used with a mesh, | ||
because the mesh surfaces errors that were previously invisible! | ||
|
||
There's actually a whole page on [debugging 502s](../../tasks/debugging-502s/). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
--- | ||
title: HTTP 503 and 504 Errors | ||
description: HTTP 503 and 504 mean overloaded workloads. | ||
--- | ||
|
||
503s and 504s show up when a Linkerd proxy is trying to make so many | ||
requests to a workload that it gets overwhelmed. | ||
|
||
When the workload next to a proxy makes a request, the proxy adds it | ||
to an internal dispatch queue. When things are going smoothly, the | ||
request is pulled from the queue and dispatched almost immediately. | ||
If the queue gets too long, though (which can generally happen only | ||
if the called service is slow to respond), the proxy will go into | ||
_load-shedding_, where any new request gets an immediate 503. The | ||
proxy can only get _out_ of load-shedding when the queue shrinks. | ||
|
||
Failfast also plays a role here: if the proxy puts a service into | ||
failfast while there are requests in the dispatch queue, all the | ||
requests in the dispatch queue get an immediate 504 before the | ||
proxy goes into load-shedding. | ||
|
||
To get out of failfast, some endpoints for the service have to | ||
become available. | ||
|
||
To get out of load-shedding, the dispatch queue has to start | ||
emptying, which implies that the service has to get more capacity | ||
to process requests or that the incoming request rate has to drop. |
36 changes: 36 additions & 0 deletions
36
linkerd.io/content/2.17/common-errors/protocol-detection.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
--- | ||
title: Protocol Detection Errors | ||
description: Protocol detection errors indicate that Linkerd doesn't understand the | ||
protocol in use. | ||
--- | ||
|
||
Linkerd is capable of proxying all TCP traffic, including TLS connections, | ||
WebSockets, and HTTP tunneling. In most cases where the client speaks first | ||
when a new connection is made, Linkerd can detect the protocol in use, | ||
allowing it to perform per-request routing and metrics. | ||
|
||
If your proxy logs contain messages like `protocol detection timed out after | ||
10s`, or you're experiencing 10-second delays when establishing connections, | ||
you're probably running a situation where Linkerd cannot detect the protocol. | ||
This is most common for protocols where the server speaks first, and the | ||
client is waiting for information from the server. It may also occur with | ||
non-HTTP protocols for which Linkerd doesn't yet understand the wire format of | ||
a request. | ||
|
||
You'll need to understand exactly what the situation is to fix this: | ||
|
||
- A server-speaks-first protocol will probably need to be configured as a | ||
`skip` or `opaque` port, as described in the [protocol detection | ||
documentation](../../features/protocol-detection/#configuring-protocol-detection). | ||
|
||
- If you're seeing transient protocol detection timeouts, this is more likely | ||
to indicate a misbehaving workload. | ||
|
||
- If you know the protocol is client-speaks-first but you're getting | ||
consistent protocol detection timeouts, you'll probably need to fall back on | ||
a `skip` or `opaque` port. | ||
|
||
Note that marking ports as `skip` or `opaque` has ramifications beyond | ||
protocol detection timeouts; see the [protocol detection | ||
documentation](../../features/protocol-detection/#configuring-protocol-detection) | ||
for more information. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
--- | ||
title: Features | ||
weight: 3 | ||
sitemap: | ||
priority: 1.0 | ||
--- | ||
|
||
Linkerd offers many features, outlined below. For our walkthroughs and guides, | ||
please see the [Linkerd task docs]({{< relref "../tasks" >}}). For a reference, | ||
see the [Linkerd reference docs]({{< relref "../reference" >}}). | ||
|
||
## Linkerd's features | ||
|
||
{{< docs/section-toc >}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
--- | ||
title: HTTP Access Logging | ||
description: Linkerd proxies can be configured to emit HTTP access logs. | ||
--- | ||
|
||
Linkerd proxies can be configured to generate an HTTP access log that records | ||
all HTTP requests that transit the proxy. | ||
|
||
The `config.linkerd.io/access-log` annotation is used to enable proxy HTTP | ||
access logging. Adding this annotation to a namespace or workload configures the | ||
proxy injector to set an environment variable in the proxy container that | ||
configures access logging. | ||
|
||
HTTP access logging is disabled by default because it has a performance impact, | ||
compared to proxies without access logging enabled. Enabling access logging may | ||
increase tail latency and CPU consumption under load. The severity of | ||
this performance cost may vary depending on the traffic being proxied, and may | ||
be acceptable in some environments. | ||
|
||
{{< note >}} | ||
The proxy's HTTP access log is distinct from proxy debug logging, which is | ||
configured separately. See the documentation on [modifying the proxy log | ||
level](../../tasks/modifying-proxy-log-level/) for details on configuring the | ||
proxy's debug logging. | ||
{{< /note >}} | ||
|
||
## Access Log Formats | ||
|
||
The value of the `config.linkerd.io/access-log` annotation determines the format | ||
of HTTP access log entries, and can be either "apache" or "json". | ||
|
||
Setting the `config.linkerd.io/access-log: "apache"` annotation configures the | ||
proxy to emit HTTP access logs in the [Apache Common Log | ||
Format](https://en.wikipedia.org/wiki/Common_Log_Format). For example: | ||
|
||
```text {class=disable-copy} | ||
10.42.0.63:51160 traffic.booksapp.serviceaccount.identity.linkerd.cluster.local - [2022-08-23T20:28:20.071809491Z] "GET http://webapp:7000/ HTTP/2.0" 200 | ||
10.42.0.63:51160 traffic.booksapp.serviceaccount.identity.linkerd.cluster.local - [2022-08-23T20:28:20.187706137Z] "POST http://webapp:7000/authors HTTP/2.0" 303 | ||
10.42.0.63:51160 traffic.booksapp.serviceaccount.identity.linkerd.cluster.local - [2022-08-23T20:28:20.301798187Z] "GET http://webapp:7000/authors/104 HTTP/2.0" 200 | ||
10.42.0.63:51160 traffic.booksapp.serviceaccount.identity.linkerd.cluster.local - [2022-08-23T20:28:20.409177224Z] "POST http://webapp:7000/books HTTP/2.0" 303 | ||
10.42.0.1:43682 - - [2022-08-23T20:28:23.049685223Z] "GET /ping HTTP/1.1" 200 | ||
``` | ||
|
||
Setting the `config.linkerd.io/access-log: json` annotation configures the proxy | ||
to emit access logs in a JSON format. For example: | ||
|
||
```json {class=disable-copy} | ||
{"client.addr":"10.42.0.70:32996","client.id":"traffic.booksapp.serviceaccount.identity.linkerd.cluster.local","host":"webapp:7000","method":"GET","processing_ns":"39826","request_bytes":"","response_bytes":"19627","status":200,"timestamp":"2022-08-23T20:33:42.321746212Z","total_ns":"14441135","trace_id":"","uri":"http://webapp:7000/","user_agent":"Go-http-client/1.1","version":"HTTP/2.0"} | ||
{"client.addr":"10.42.0.70:32996","client.id":"traffic.booksapp.serviceaccount.identity.linkerd.cluster.local","host":"webapp:7000","method":"POST","processing_ns":"30036","request_bytes":"33","response_bytes":"0","status":303,"timestamp":"2022-08-23T20:33:42.436964052Z","total_ns":"14122403","trace_id":"","uri":"http://webapp:7000/authors","user_agent":"Go-http-client/1.1","version":"HTTP/2.0"} | ||
{"client.addr":"10.42.0.70:32996","client.id":"traffic.booksapp.serviceaccount.identity.linkerd.cluster.local","host":"webapp:7000","method":"GET","processing_ns":"38664","request_bytes":"","response_bytes":"2350","status":200,"timestamp":"2022-08-23T20:33:42.551768300Z","total_ns":"6998222","trace_id":"","uri":"http://webapp:7000/authors/105","user_agent":"Go-http-client/1.1","version":"HTTP/2.0"} | ||
{"client.addr":"10.42.0.70:32996","client.id":"traffic.booksapp.serviceaccount.identity.linkerd.cluster.local","host":"webapp:7000","method":"POST","processing_ns":"42492","request_bytes":"46","response_bytes":"0","status":303,"timestamp":"2022-08-23T20:33:42.659401621Z","total_ns":"9274163","trace_id":"","uri":"http://webapp:7000/books","user_agent":"Go-http-client/1.1","version":"HTTP/2.0"} | ||
{"client.addr":"10.42.0.1:56300","client.id":"-","host":"10.42.0.69:7000","method":"GET","processing_ns":"35848","request_bytes":"","response_bytes":"4","status":200,"timestamp":"2022-08-23T20:33:49.254262428Z","total_ns":"1416066","trace_id":"","uri":"/ping","user_agent":"kube-probe/1.24","version":"HTTP/1.1"} | ||
``` | ||
|
||
## Consuming Access Logs | ||
|
||
The HTTP access log is written to the proxy container's `stderr` stream, while | ||
the proxy's standard debug logging is written to the proxy container's `stdout` | ||
stream. Currently, the `kubectl logs` command will always output both the | ||
container's `stdout` and `stderr` streams. However, [KEP | ||
3289](https://github.com/kubernetes/enhancements/pull/3289) will add support for | ||
separating a container's `stdout` or `stderr` in the `kubectl logs` command. |
Oops, something went wrong.