Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kafka exporter and receiver configuration #5703

Merged
merged 30 commits into from
Jul 24, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
f02e2fc
Add configs for collector and ingester
joeyyy09 Jul 3, 2024
a728c2e
Update collector config
joeyyy09 Jul 10, 2024
fc7da59
Update ingester config
joeyyy09 Jul 14, 2024
53173c5
Add initial kafka_test.go
joeyyy09 Jul 14, 2024
b2133cc
Add debug logs in the config
joeyyy09 Jul 15, 2024
2ccff53
Update conig files for UI
joeyyy09 Jul 15, 2024
c9c32cc
Update conig files
joeyyy09 Jul 16, 2024
802413a
Add kafka e2e setup
joeyyy09 Jul 18, 2024
819ee21
Add kafka e2e setup
joeyyy09 Jul 18, 2024
d02136e
Add Readme for Kafka
joeyyy09 Jul 18, 2024
3cf0151
Update cmd/jaeger/internal/integration/README.md
joeyyy09 Jul 19, 2024
d09eaa4
Update cmd/jaeger/internal/integration/README.md
joeyyy09 Jul 19, 2024
0acf969
Update cmd/jaeger/internal/integration/README.md
joeyyy09 Jul 19, 2024
17ebd78
Fixes
joeyyy09 Jul 19, 2024
dfcba73
Fix broken syntax
joeyyy09 Jul 19, 2024
fe794c1
Add injectStorageCleaner parameter
joeyyy09 Jul 19, 2024
6ab1285
fix merge conflicts
joeyyy09 Jul 19, 2024
66ef1f9
fix merge conflicts
joeyyy09 Jul 19, 2024
4d4a882
Modify E2EStorageIntegration struct
joeyyy09 Jul 20, 2024
7cc6a6f
Fixes
joeyyy09 Jul 20, 2024
29581a4
Fixes
joeyyy09 Jul 20, 2024
580bdc9
Update Lint
joeyyy09 Jul 20, 2024
fea62c6
Fix Lint
joeyyy09 Jul 20, 2024
284acc9
Add e2eCleanup in e2eInitialize
joeyyy09 Jul 21, 2024
4ba5917
Fixes
joeyyy09 Jul 24, 2024
06e68ac
Lint fixes
joeyyy09 Jul 24, 2024
07f903e
fix ui
yurishkuro Jul 24, 2024
46121f5
Merge branch 'main' into kafka-config
yurishkuro Jul 24, 2024
4d624d6
schedule clean-up after reader/writer are created
yurishkuro Jul 24, 2024
d7c70cf
remove duplicate call to cleanup
yurishkuro Jul 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions cmd/jaeger/internal/integration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,56 @@ flowchart LR
end
```

## Kafka Integration

The primary difference between the Kafka integration tests and other integration tests lies in the flow of data. In the standard tests, spans are written by the SpanWriter, sent through an RPC_client directly to a receiver, then to an exporter, and stored in an in-memory backend. Spans are read by the SpanReader, which queries the jaeger_query process accessing the storage backend. In contrast, the Kafka tests introduce Kafka as an intermediary. Spans go from the SpanWriter through an RPC_client to an OTLP receiver in the Jaeger Collector, exported to Kafka, received by the Jaeger Ingester, and then stored. For details, see the [Architecture](#KafkaArchitecture) section below.
joeyyy09 marked this conversation as resolved.
Show resolved Hide resolved


## Kafka Architecture

``` mermaid
flowchart LR
Test -->|writeSpan| SpanWriter
SpanWriter --> RPCW[RPC_client]
RPCW --> OTLP_Receiver[Receiver]
OTLP_Receiver --> CollectorExporter[Kafka Exporter]
CollectorExporter --> Kafka[Kafka]
Kafka --> IngesterReceiver[Kafka Receiver]
IngesterReceiver --> IngesterExporter[Exporter]
IngesterExporter --> StorageBackend[(In-Memory Store)]
Test -->|readSpan| SpanReader
SpanReader --> RPCR[RPC_client]
RPCR --> QueryProcess[jaeger_query]
QueryProcess --> StorageBackend
IngesterExporter -.-> StorageCleaner
joeyyy09 marked this conversation as resolved.
Show resolved Hide resolved
StorageCleaner --> StorageBackend
joeyyy09 marked this conversation as resolved.
Show resolved Hide resolved

subgraph Integration_Test_Executable
Test
SpanWriter
SpanReader
RPCW
RPCR
end

subgraph Jaeger Collector
OTLP_Receiver
CollectorExporter
end

subgraph Jaeger Ingester
IngesterReceiver
IngesterExporter
QueryProcess
StorageBackend
StorageCleaner[Storage Cleaner Extension]
end

subgraph Kafka Buffer
Kafka
joeyyy09 marked this conversation as resolved.
Show resolved Hide resolved
end
```

## Running tests locally

All integration tests can be run locally.
Expand Down
4 changes: 2 additions & 2 deletions plugin/storage/integration/integration.go
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ var fixtures embed.FS
// - in those functions it instantiates and populates this struct
// - it then calls RunAll.
//
// Some implementations may declate multuple tests, with different settings,
// Some implementations may declare multiple tests, with different settings,
// and RunAll() under different conditions.
type StorageIntegration struct {
SpanWriter spanstore.Writer
Expand Down Expand Up @@ -541,4 +541,4 @@ func (s *StorageIntegration) RunSpanStoreTests(t *testing.T) {
t.Run("GetTrace", s.testGetTrace)
t.Run("GetLargeSpans", s.testGetLargeSpan)
t.Run("FindTraces", s.testFindTraces)
}
}
Loading