Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add automated hole-punch tests #4549

Merged
merged 179 commits into from
Oct 25, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
179 commits
Select commit Hold shift + click to select a range
d6b0647
Initial routes working
thomaseizinger Jul 5, 2023
5bfb526
Some stuff works
thomaseizinger Jul 5, 2023
1fa84b6
Switch to debian and manual nft routes
thomaseizinger Jul 6, 2023
5781b34
Fix typo regarding IP forwarding
thomaseizinger Jul 17, 2023
a583cbf
Only assign single IP to interface
thomaseizinger Jul 17, 2023
b6f6587
Use default routes instead
thomaseizinger Jul 17, 2023
42453b8
Map to correct IP in NAT
thomaseizinger Jul 17, 2023
e63a8a8
Don't enable IP forwarding when not necessary
thomaseizinger Jul 17, 2023
93034f1
Introduce dedicated internet namespace
thomaseizinger Jul 17, 2023
5c99a98
Assign IPs in same subnet
thomaseizinger Jul 17, 2023
13e087b
WIP completely new approach
thomaseizinger Jul 18, 2023
49c3196
Return from client
thomaseizinger Aug 17, 2023
75db11f
Remove old scripts
thomaseizinger Aug 17, 2023
4d28d38
Add relay test module
thomaseizinger Aug 20, 2023
cc69536
Add hole punching tests to CI
thomaseizinger Aug 20, 2023
a072c94
Undo dockerignore change
thomaseizinger Aug 20, 2023
e19ddf0
Fix linter warnings
thomaseizinger Aug 20, 2023
37dddd4
feat(core): add logs for `OrTransport` when trying addresses
tcoratger Jul 5, 2023
7d2fff6
docs(swarm): document Swarm::external_addresses
mxinden Jul 7, 2023
e29b5a8
fix(swarm): external address candidate only after address translation
b-zee Jul 9, 2023
752bf13
fix(core/choice): log on dial and fix listen log
mxinden Jul 10, 2023
be11ca6
chore: Update CHANGELOG.md
dariusc93 Jul 10, 2023
4520a20
deps: bump tokio from 1.28.2 to 1.29.1
dependabot[bot] Jul 10, 2023
22551c1
deps: bump regex from 1.8.4 to 1.9.1
dependabot[bot] Jul 11, 2023
ac942f2
deps: bump smallvec from 1.10.0 to 1.11.0
dependabot[bot] Jul 11, 2023
42b3bc7
deps: bump serde_json from 1.0.99 to 1.0.100
dependabot[bot] Jul 11, 2023
4701040
deps: bump async-trait from 0.1.69 to 0.1.71
dependabot[bot] Jul 11, 2023
95a02cc
deps: bump syn from 2.0.23 to 2.0.25
dependabot[bot] Jul 11, 2023
d50280f
deps: bump webpki-roots from 0.23.1 to 0.24.0
dependabot[bot] Jul 11, 2023
0612907
deps: bump clap from 4.3.10 to 4.3.11
dependabot[bot] Jul 11, 2023
38756ec
ci: run interop tests on bigger hardware
thomaseizinger Jul 11, 2023
8bcf4a8
chore: prepare libp2p-swarm v0.43.1 and libp2p-kad v0.44.2
mxinden Jul 13, 2023
b693f17
fix(interop): install `wasm-opt` in `Dockerfile.chromium`
MarcoPolo Jul 13, 2023
3f1af3e
fix(examples/kv-store): set kademlia mode to `Server`
0x7CFE Jul 16, 2023
8c9ab56
fix: clippy useless_conversion and useless_vec
b-zee Jul 16, 2023
b036c61
deps: bump thiserror from 1.0.40 to 1.0.43
dependabot[bot] Jul 17, 2023
28ac449
chore: apply suggestions from clippy beta (1.72)
thomaseizinger Jul 18, 2023
a241e94
ci: automatically merge PRs of semver-compatible updates
thomaseizinger Jul 19, 2023
9ebe9e6
deps: bump prometheus-client from 0.21.1 to 0.21.2
dependabot[bot] Jul 19, 2023
a6e7972
deps: bump serde from 1.0.168 to 1.0.171
dependabot[bot] Jul 19, 2023
40157a8
deps: bump proc-macro2 from 1.0.63 to 1.0.66
dependabot[bot] Jul 19, 2023
1e0c5d3
deps: bump syn from 2.0.25 to 2.0.26
dependabot[bot] Jul 19, 2023
5aafef6
deps: bump axum from 0.6.18 to 0.6.19
dependabot[bot] Jul 19, 2023
7306e46
deps: bump quote from 1.0.29 to 1.0.31
dependabot[bot] Jul 19, 2023
63bed7c
deps: bump anyhow from 1.0.71 to 1.0.72
dependabot[bot] Jul 19, 2023
d851f07
deps: bump sec1 from 0.7.2 to 0.7.3
dependabot[bot] Jul 19, 2023
7923db0
deps: bump clap from 4.3.11 to 4.3.12
dependabot[bot] Jul 19, 2023
e6b4053
chore: update Rust version for clippy check to 1.71
thomaseizinger Jul 19, 2023
1bc484a
deps: bump lru from 0.10.1 to 0.11.0
dependabot[bot] Jul 19, 2023
2da30a3
deps: bump actions/upload-pages-artifact from 1.0.9 to 2.0.0
dependabot[bot] Jul 19, 2023
eed92ab
chore(gossipsub): include in `libp2p` meta crate when compiling for wasm
onur-ozkan Jul 20, 2023
4366f70
feat(relay): add functions to `Limit` to access its data
dariusc93 Jul 20, 2023
cdf3eca
chore(libp2p): expose `json` feature from `request-response`
Jul 20, 2023
a48a680
refactor(request-reponse): replace `serde_cbor` with `cbor4ii`
zeeshanlakhani Jul 20, 2023
db35c85
deps: bump syn from 2.0.26 to 2.0.27
dependabot[bot] Jul 24, 2023
e9dbd4a
deps: bump quote from 1.0.31 to 1.0.32
dependabot[bot] Jul 24, 2023
96cd90e
deps: bump rmp-serde from 1.1.1 to 1.1.2
dependabot[bot] Jul 24, 2023
ffbc1c4
deps: bump num-traits from 0.2.15 to 0.2.16
dependabot[bot] Jul 24, 2023
f85d866
deps: bump async-trait from 0.1.71 to 0.1.72
dependabot[bot] Jul 24, 2023
fd995fa
deps: bump tower-http from 0.4.1 to 0.4.3
dependabot[bot] Jul 24, 2023
ad372f1
deps: bump thiserror from 1.0.43 to 1.0.44
dependabot[bot] Jul 24, 2023
bbe211d
deps: bump either from 1.8.1 to 1.9.0
dependabot[bot] Jul 24, 2023
2501a6e
deps: bump serde from 1.0.171 to 1.0.175
dependabot[bot] Jul 24, 2023
d561017
deps: bump tempfile from 3.6.0 to 3.7.0
dependabot[bot] Jul 24, 2023
f2923d0
deps: bump asynchronous-codec from 0.6.1 to 0.6.2
dependabot[bot] Jul 24, 2023
e2a57f3
fix(kad): prevent simultaneous dials to peer
b-zee Jul 24, 2023
1394ac2
feat(identity): implement from_protobuf_encoding for RSA keys
mxinden Jul 24, 2023
ed47e93
fix(swarm): display cause of denied listen error
divagant-martian Jul 24, 2023
a614f1a
fix(connection-limits): correct connection tracking
AgeManning Jul 26, 2023
dea9604
chore: prepare identity v0.2.2 and swarm v0.43.2
mxinden Jul 26, 2023
ce3efcf
refactor(quic): rewrite quic using quinn
kpp Jul 28, 2023
50779db
docs: add maintainer handbook
thomaseizinger Jul 28, 2023
eda24b0
docs(readme): rename iroh to beetle
mxinden Jul 28, 2023
0d5b1ac
docs(readme): add Mina project as notable user
mxinden Jul 28, 2023
162238e
docs(readme): add safe network to notable users
mxinden Jul 28, 2023
33fa190
fix(identity): use test fixture for ed25519
mxinden Jul 29, 2023
af75537
chore: fix clippy
jxs Jul 31, 2023
a172032
deps: bump serde from 1.0.178 to 1.0.179
dependabot[bot] Jul 31, 2023
fe76e28
deps: bump syn from 2.0.27 to 2.0.28
dependabot[bot] Jul 31, 2023
e1c70d7
deps: bump webpki-roots from 0.24.0 to 0.25.1
dependabot[bot] Jul 31, 2023
939a5fe
chore(quic): prepare v0.9.0-alpha
mxinden Jul 31, 2023
2228677
fix(relay): export `RateLimiter` type
dariusc93 Jul 31, 2023
4fbae8b
feat(kad): implement common traits on `RoutingUpdate`
arsenron Jul 31, 2023
cf34483
chore(ci): bump `cargo semver-checks` to `0.22.1`
thomaseizinger Jul 31, 2023
97f2190
chore(roadmap): move QUIC quinn to done
mxinden Jul 31, 2023
bdacbe9
fix(examples/file-sharing): set Kademlia `Mode::Server`
mxinden Aug 1, 2023
928a183
docs: fix last OutboundQueryCompleted in kad doc
thomas-senechal Aug 2, 2023
cbb5621
fix(relay): remove unconditional `async-std` dependency
quininer Aug 2, 2023
7383a2d
deps(yamux): update yamux to `v0.12`
thomaseizinger Aug 2, 2023
8a01bdf
deps: bump Swatinem/rust-cache from 2.5.1 to 2.6.0
dependabot[bot] Aug 3, 2023
25259f3
feat(gossipsub): add getter function to obtain `TopicScoreParams`
ackintosh Aug 3, 2023
41414c8
deps: bump regex from 1.9.1 to 1.9.3
dependabot[bot] Aug 7, 2023
29224fc
deps: bump serde from 1.0.179 to 1.0.183
dependabot[bot] Aug 7, 2023
11bd29d
fix(kad): reduce noise of "remote supports our protocol" log
shamil-gadelshin Aug 7, 2023
ff0f16a
deps: bump axum from 0.6.19 to 0.6.20
dependabot[bot] Aug 7, 2023
6f9104c
deps: bump x509-parser from 0.15.0 to 0.15.1
dependabot[bot] Aug 7, 2023
8b6d65c
docs(roadmap): fix headings, update str0m, webtransport, autonatv2 & …
mxinden Aug 7, 2023
1e019f9
deps: bump r7kamura/rust-problem-matchers from 1.3.0 to 1.4.0
dependabot[bot] Aug 8, 2023
d5e36f0
feat: memory based connection limits
hanabi1224 Aug 8, 2023
097794d
fix(quic): allow listening on ipv4 and ipv6 separately
jxs Aug 8, 2023
db98022
chore(deps): bump github.com/libp2p/go-libp2p from 0.27.5 to 0.27.8 i…
dependabot[bot] Aug 9, 2023
e468e7a
chore: prepare libp2p v0.52.2
mxinden Aug 11, 2023
d83b33f
fix(quic): add support for reusing an existing socket for local dialing
arsenron Aug 11, 2023
2104001
fix(swarm): import libp2p-yamux dev-dependency via path
mxinden Aug 11, 2023
7a3a216
docs(roadmap): update to reflect ongoing wasm browser work
DougAnderson444 Aug 12, 2023
45b03b6
deps: bump Swatinem/rust-cache from 2.6.0 to 2.6.1
dependabot[bot] Aug 14, 2023
86ae11b
deps: bump tokio from 1.29.1 to 1.31.0
dependabot[bot] Aug 14, 2023
e49f7f9
deps: bump clap from 4.3.19 to 4.3.21
dependabot[bot] Aug 14, 2023
108b4e2
deps: bump async-trait from 0.1.72 to 0.1.73
dependabot[bot] Aug 14, 2023
a8d507c
deps: bump webpki-roots from 0.25.1 to 0.25.2
dependabot[bot] Aug 14, 2023
fb4d820
deps: bump log from 0.4.19 to 0.4.20
dependabot[bot] Aug 14, 2023
24a1a9d
feat(quic): promote to stable release
mxinden Aug 17, 2023
353882a
deps(identity): update ed25519-dalek to 2.0
jxs Aug 18, 2023
44cff59
Add redis client
thomaseizinger Aug 20, 2023
5349cf1
Merge branch 'master' into feat/hole-punching-tests
thomaseizinger Aug 20, 2023
e6fb4b5
Import quic from `libp2p`
thomaseizinger Aug 20, 2023
c38b80c
Modernize binaries
thomaseizinger Aug 20, 2023
da34611
Install openvswitch
thomaseizinger Aug 20, 2023
16f6aed
Remove DNS
thomaseizinger Aug 20, 2023
121da4a
WIP Use redis
thomaseizinger Aug 20, 2023
7daa046
Rewrite to approach based on docker-compose
thomaseizinger Sep 12, 2023
1ce8d24
It works!
thomaseizinger Sep 12, 2023
73c7525
Remove need for setup script for hosts
thomaseizinger Sep 12, 2023
ef15043
Move main dockerfile to root dir
thomaseizinger Sep 12, 2023
1f5a753
Refactor a single binary
thomaseizinger Sep 13, 2023
06ae18c
Switch to alpine docker container
thomaseizinger Sep 13, 2023
a6cc6b9
Make routers shut down faster
thomaseizinger Sep 13, 2023
87bd67b
Have routers output tcpdump
thomaseizinger Sep 13, 2023
422cd46
Reduce hole-punch-tests to clients
thomaseizinger Sep 14, 2023
cc1d4f9
Remove compose setup as it is being moved to test-plans
thomaseizinger Sep 14, 2023
008fc85
Deduplicate code
thomaseizinger Sep 14, 2023
1c45c87
Install bind-utils in container
thomaseizinger Sep 15, 2023
203cffb
Use correct package name
thomaseizinger Sep 15, 2023
bb251a4
Add more required tools
thomaseizinger Sep 15, 2023
919f49e
Make redis timeout configurable
thomaseizinger Sep 19, 2023
aa3c5aa
Dialer to emit rtt to hole-punched peer
thomaseizinger Sep 19, 2023
a8df437
Add derive feature flag
thomaseizinger Sep 19, 2023
3387579
Report 5 RTTs
thomaseizinger Sep 19, 2023
15b783e
Keep connections alive indefinitely
thomaseizinger Sep 19, 2023
ab82fa5
Fixes
thomaseizinger Sep 19, 2023
31e9ed9
Directly set logger
thomaseizinger Sep 25, 2023
164d717
Install tcpdump in image
thomaseizinger Sep 25, 2023
13a6854
Disable choice logs
thomaseizinger Sep 25, 2023
d99fa2f
Make logs less noisy
thomaseizinger Sep 25, 2023
c11710a
Merge branch 'master' into feat/hole-punching-tests
thomaseizinger Sep 25, 2023
5eba2c5
Use correct CI
thomaseizinger Sep 25, 2023
9cff06c
Undo `.dockerignore` change
thomaseizinger Sep 25, 2023
0a230e9
Remove stale .gitignore
thomaseizinger Sep 25, 2023
26862e9
Don't use deprecated code
thomaseizinger Sep 25, 2023
b06be43
Reference correct action
thomaseizinger Sep 25, 2023
c9c7cca
Don't run in parallel
thomaseizinger Sep 26, 2023
51488d1
Merge branch 'master' into feat/hole-punching-tests
thomaseizinger Oct 16, 2023
cb507c3
Change to new `SwarmBuilder`
thomaseizinger Oct 16, 2023
c3d4eb4
Update CI action
thomaseizinger Oct 16, 2023
aefaf5e
Merge branch 'master' into feat/hole-punching-tests
thomaseizinger Oct 18, 2023
9ea9fa9
Install tc
thomaseizinger Oct 24, 2023
ebffd74
Log hole-punch attempt
thomaseizinger Oct 24, 2023
6ecd7f4
Merge remote-tracking branch 'origin/feat/hole-punching-tests' into f…
thomaseizinger Oct 24, 2023
78d095e
Delete dead code in `dcutr`
thomaseizinger Oct 24, 2023
b1296c2
Add more logging
thomaseizinger Oct 24, 2023
af125c6
Fix compile error
thomaseizinger Oct 24, 2023
a12e9b7
Don't sleep before sending first hole-punch packet
thomaseizinger Oct 24, 2023
b6869fa
Refactor loop
thomaseizinger Oct 24, 2023
0dfd9d0
Combine log filters
thomaseizinger Oct 24, 2023
43238f9
Update log filter
thomaseizinger Oct 24, 2023
e2af98e
Fail early if relay connection fails
thomaseizinger Oct 25, 2023
c24940c
Merge branch 'master' into feat/hole-punching-tests
thomaseizinger Oct 25, 2023
1d1dd45
Simplify code using new API
thomaseizinger Oct 25, 2023
e940da5
Better logging
thomaseizinger Oct 25, 2023
86add0f
Update Rust version
thomaseizinger Oct 25, 2023
9bbb6f3
Re-connect to relay if it fails
thomaseizinger Oct 25, 2023
1ead41e
De-duplicate code
thomaseizinger Oct 25, 2023
5b72ab3
Add changelog for quic
thomaseizinger Oct 25, 2023
06a0acf
Merge branch 'master' into feat/hole-punching-tests
thomaseizinger Oct 25, 2023
2c23215
fix(hole-punch): Add support for arm64 architecture (#4687)
diegomrsantos Oct 25, 2023
ba78b24
Merge remote-tracking branch 'origin/feat/hole-punching-tests' into f…
thomaseizinger Oct 25, 2023
1d894e9
Merge branch 'master' into feat/hole-punching-tests
thomaseizinger Oct 25, 2023
f4076b6
Merge branch 'master' into feat/hole-punching-tests
thomaseizinger Oct 25, 2023
b2a8a84
Merge branch 'master' into feat/hole-punching-tests
mergify[bot] Oct 25, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/workflows/interop-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,20 @@ jobs:
s3-access-key-id: ${{ vars.TEST_PLANS_BUILD_CACHE_KEY_ID }}
s3-secret-access-key: ${{ secrets.TEST_PLANS_BUILD_CACHE_KEY }}
worker-count: 16
run-holepunching-interop:
name: Run hole-punch interoperability tests
runs-on: ${{ fromJSON(github.repository == 'libp2p/rust-libp2p' && '["self-hosted", "linux", "x64", "4xlarge"]' || '"ubuntu-latest"') }}
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- name: Build image
run: docker buildx build --load -t rust-libp2p-head . -f hole-punching-tests/Dockerfile
- name: Run tests
uses: libp2p/test-plans/.github/actions/run-interop-hole-punch-test@master
with:
test-filter: rust-libp2p-head
extra-versions: ${{ github.workspace }}/hole-punching-tests/version.json
s3-cache-bucket: libp2p-by-tf-aws-bootstrap
s3-access-key-id: ${{ vars.TEST_PLANS_BUILD_CACHE_KEY_ID }}
s3-secret-access-key: ${{ secrets.TEST_PLANS_BUILD_CACHE_KEY }}
worker-count: 1
15 changes: 15 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ members = [
"examples/relay-server",
"examples/rendezvous",
"examples/upnp",
"hole-punching-tests",
"identity",
"interop-tests",
"misc/allow-block-list",
Expand Down
17 changes: 17 additions & 0 deletions hole-punching-tests/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[package]
name = "hole-punching-tests"
version = "0.1.0"
edition = "2021"
publish = false
license = "MIT"

[dependencies]
anyhow = "1"
env_logger = "0.10.0"
futures = "0.3.28"
libp2p = { path = "../libp2p", features = ["tokio", "dcutr", "identify", "macros", "noise", "ping", "relay", "tcp", "yamux", "quic"] }
log = "0.4"
redis = { version = "0.23.0", default-features = false, features = ["tokio-comp"] }
tokio = { version = "1.29.1", features = ["full"] }
serde = { version = "1.0.188", features = ["derive"] }
serde_json = "1.0.107"
19 changes: 19 additions & 0 deletions hole-punching-tests/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# syntax=docker/dockerfile:1.5-labs
FROM rust:1.73.0 as builder

# Run with access to the target cache to speed up builds
WORKDIR /workspace
ADD . .

# Build the relay as a statically-linked binary. Unfortunately, we must specify the `--target` explicitly. See https://msfjarvis.dev/posts/building-static-rust-binaries-for-linux/.
RUN --mount=type=cache,target=./target \
--mount=type=cache,target=/usr/local/cargo/registry \
RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --package hole-punching-tests --target $(rustc -vV | grep host | awk '{print $2}')

RUN --mount=type=cache,target=./target \
mv ./target/$(rustc -vV | grep host | awk '{print $2}')/release/hole-punching-tests /usr/local/bin/hole-punching-tests

FROM alpine:3
COPY --from=builder /usr/local/bin/hole-punching-tests /usr/bin/hole-punch-client
RUN --mount=type=cache,target=/var/cache/apk apk add bind-tools jq curl tcpdump iproute2-tc
ENV RUST_BACKTRACE=1
Loading