From 3b431fb3c8d34e51b417519540462861e361f69f Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Mon, 19 Jun 2023 21:38:22 +0200 Subject: [PATCH 1/7] Add release blogpost --- src/_blog/rust-libp2p-052-release.md | 233 +++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) create mode 100644 src/_blog/rust-libp2p-052-release.md diff --git a/src/_blog/rust-libp2p-052-release.md b/src/_blog/rust-libp2p-052-release.md new file mode 100644 index 00000000..25c4a4d3 --- /dev/null +++ b/src/_blog/rust-libp2p-052-release.md @@ -0,0 +1,233 @@ +--- +tags: +- libp2p +- rust +title: rust-libp2p v0.52 release spotlight +description: Walking through the rust-libp2p v0.52 release +date: 2023-06-19 +permalink: "/2023-06-19-rust-libp2p-052" +translationKey: '' +header_image: /rust-libp2p-2022-header.png +author: Thomas Eizinger +--- + +# rust-libp2p v0.52 release spotlight + +With the `v0.52` release, the `rust-libp2p` maintainer team tried something new. +Breaking changes - even in the presence of an obvious or easy upgrade path - are a burden for our users. +They easily trigger a cascade of breaking changes for downstream users. + +Coming up to the `v0.52` release, we consciously held back and batched up breaking changes and released several features as patch releases instead. +As a result, `v0.52` is unfortunately quite large. +This blogpost aims to highlight some of the more exciting changes: + +- Automatic kademlia client/server mode +- Type-safe `/p2p` multiaddresses +- More consistent event/command naming +- Improve ergonomics around stream-upgrade errors +- Simpler noise interface + +## Automatic kademlia client/server mode + +Let's get the biggest one out the way first, I promise the other points are easier explained but equally exciting. +The **tl;dr** is: Healthier Kademlia routing tables and an improved developer experience. + +If you don't know about Kademlia's client/server mode, checkout the [specs](https://github.com/libp2p/specs/tree/master/kad-dht#client-and-server-mode). + +With the `v0.52` release, `rust-libp2p` automatically configures Kademlia in client or server mode depending on our external addresses. +If we have a confirmed, external address, we will operate in server-mode, otherwise client-mode. +This is entirely configuration-free (yay!) although follow-up work is under-way to allow setting this manually in certain situations: [#4074](https://github.com/libp2p/rust-libp2p/issues/4074). + +We can now do the following: + +1. As soon as we learn about an external address (e.g. via AutoNAT), we activate server mode of Kademlia. +2. Activating server-mode means we allow inbound requests, this is a change in our set of supported protocols. +3. The change is detected automatically and reported to all protocols as `ConnectionEvent::LocalProtocolsChange`. +4. `libp2p-identify` picks up this change and pushes it to all connected remote nodes. +5. Remote nodes can instantly put us into their routing table. + +To implement this, several other features/issues had to be fixed. +If you are interested in the details, read on: + +- Changes to the supported protocols are now detected at runtime and communicated to all protocols: [#3651](https://github.com/libp2p/rust-libp2p/pull/3651). + + Previously, a protocol could retrieve the supported protocols via `PollParameters::supported_protocols`. + This list however was computed at start-up and was static. + Now, `ConnectionEvent` has two new variants: + + ```rust + pub enum ConnectionEvent<'a> { + // existing variants omitted ... + + /// The local [`ConnectionHandler`] added or removed support for one or more protocols. + LocalProtocolsChange(ProtocolsChange<'a>), + /// The remote [`ConnectionHandler`] now supports a different set of protocols. + RemoteProtocolsChange(ProtocolsChange<'a>), + } + + pub enum ProtocolsChange<'a> { + Added(ProtocolsAdded<'a>), + Removed(ProtocolsRemoved<'a>), + } + ``` + + `ProtocolsAdded` and `ProtocolsRemoved` are iterators over a (new) type called `StreamProtocol`. +- Protocols are now enforced to be valid UTF-8 strings: [#3746](https://github.com/libp2p/rust-libp2p/pull/3746). + + This was always the case in the specs but the `rust-libp2p` implementation was lagging behind here and improperly represented them as bytes internally. +- Local changes to our protocols (i.e. a node going from Kademlia server to client mode) are now immediately pushed to the remote via the [`/ipfs/id/push/1.0.0`](https://github.com/libp2p/specs/tree/master/identify#identifypush) protocol: [#3980](https://github.com/libp2p/rust-libp2p/pull/3980). +- Simplify the scoring mechanism of external addresses: [#3954](https://github.com/libp2p/rust-libp2p/pull/3954) + +Not only does this work out-of-the-box and thus improves the developer experience of `rust-libp2p`, it should also result in a much more useful and up-to-date routing table for all users of the Kademlia DHT. + +## Type-safe `/p2p` multiaddresses + +The `/p2p` protocol of multiaddresses specifies the identity of a peer in the form of a `PeerId` such as `12D3KooWETLZBFBfkzvH3BQEtA1TJZPmjb4a18ss5TpwNU7DHDX6`. +Yet for the longest time, the type-definition of the `/p2p` protocol looked like this: + +```rust +pub enum Protocol<'a> { + // omitted other variants ... + P2p(Multihash), +} +``` + +Every `PeerId` is a valid `Multihash` but not vice-versa. +Dang! +That is a lot of "impossible" errors that the type-system should avoid. + +[Thanks](https://github.com/multiformats/rust-multihash/issues/322) [to](https://github.com/libp2p/rust-libp2p/pull/3514) [a](https://github.com/libp2p/rust-libp2p/pull/3656) [lot](https://github.com/libp2p/rust-libp2p/pull/3350) [of](https://github.com/multiformats/rust-multihash/pull/272) [work](https://github.com/multiformats/rust-multiaddr/pull/83), it now looks like this: + + +```rust +pub enum Protocol<'a> { + // omitted other variants ... + P2p(PeerId), +} +``` + +## More consistent event/command naming + +Naming is hard and humans are creatures of habit. +Thus, once familiar with certain names, it is often hard to see how they make absolutely no sense at all to newcomers. + +In the `v0.52` release we renamed several associated types and enums which hopefully make the message-passing system implemented in `rust-libp2p` easier to grasp. + +A quick recap: +- A `NetworkBehaviour` represent a protocol's state across all peers and connections. +- A `ConnectionHandler` represents a protocol's state for a single connection to a peer. +- Multiple `NetworkBehaviour`s are composed into a tree using `#[derive(NetworkBehaviour)]` and run inside a `Swarm` + +`NetworkBehaviour`s can emit events to the `Swarm`. +This used to be called `OutEvent`. +Now, its aptly named `ToSwarm`: + +```rust +pub trait NetworkBehaviour { + type ToSwarm; + + // functions and other types omitted ... +} +``` + +Returning one of these events was previously done with a `NetworkBehaviourAction::GenerateEvent`, a type-name so long you were grateful for autocomplete. +These actions are essentially commands that are issued **to** the **swarm**. +What could possibly be a good name for that? +I present: + +```rust +pub enum ToSwarm { + GenerateEvent(TOutEvent), + + // other variants omitted .. +} +``` + +We followed the same strategy for `ConnectionHandler`. +A `ConnectionHandler` can receive messages **from** a `NetworkBehaviour` via `ToSwarm::NotifyHandler` (gosh, that was so much easier to write, why didn't we do this earlier?) and send message **to** its `NetworkBehaviour`. +The associated types defining these messages are now called `ToBehaviour` and `FromBehaviour`, representing _where_ the message is going / coming from. +Previously, they carried the generic names `InEvent` and `OutEvent` which had me utterly confused when I first started working on `rust-libp2p`. + +To wrap it all up, the `ConnectionHandlerEvent::Custom` variant is now called `ConnectionHandlerEvent::NotifyBehaviour`, making it clear what happens to types returned here. + +## Improved ergonomics around stream errors + +Oh, isn't this one of my favourites! + +Ever wondered why `ConnectionHandlerUpgrErr` had what felt like 10 layers of nested enums? +So did we and went ahead and fixed this in [#3882](https://github.com/libp2p/rust-libp2p/pull/3882). + +This is what it looked like before: + +```rust +pub enum ConnectionHandlerUpgrErr { + Timeout, + Timer, + Upgrade(UpgradeError), +} + +pub enum UpgradeError { + Select(NegotiationError), + Apply(E), +} + +pub enum NegotiationError { + ProtocolError(ProtocolError), + Failed, +} + +pub enum ProtocolError { + IoError(io::Error), + InvalidMessage, + InvalidProtocol, + TooManyProtocols, +} +``` + +Now, it looks like this: + +```rust +pub enum StreamUpgradeError { + Timeout, + Apply(TUpgrErr), + NegotiationFailed, + Io(io::Error), +} +``` + +But that is not it! + +Previously, we would give you one of these `ConnectionHandlerUpgrErr` when an _inbound_ stream failed. +But, what exactly does `ProtocolError::InvalidMessage` for example mean for an inbound stream? +How would we even figure out, which protocol (read `ConnectionHandler`) this should be dispatched to if we failed to negotiate the protocols? + +Well, you might have guessed it. +It is impossible to dispatch this to the correct one, so we just informed all protocols. +But that was pretty useless. +If we don't know, which stream a protocol belongs to, we shouldn't just inform all of them. +Thus, in [#3605](https://github.com/libp2p/rust-libp2p/pull/3605) we stopped this which removes several "impossible" error paths. + +## Simpler noise interface + +Since its introduction, the `libp2p-noise` crate supported a wide range of handshake patterns. +The libp2p specs however only documents the `XX` handshake: [libp2p/specs/noise](https://github.com/libp2p/specs/tree/master/noise#handshake-pattern). + +Supporting additional patterns introduced significant complexity to the codebase. +We decided to [deprecate](https://github.com/libp2p/rust-libp2p/pull/3768) and [remove](https://github.com/libp2p/rust-libp2p/pull/3511) them. +This significantly reduced the complexity of the `libp2p-noise` implementation (-1000 LoC!). +Additionally, it allowed us to finally adopt the [naming conventions](https://github.com/libp2p/rust-libp2p/issues/2217) we have been pursuing across the workspace for `libp2p-noise`. + +Using the noise handshake is now as simple as: + +```rust +let config = libp2p::noise::Config::new(&keypair).unwrap(); +``` + +## Closing + +Well done for making it to the end! + +As always, a detailed log of changes for every release can be found in our repository: [CHANGELOG.md](https://github.com/libp2p/rust-libp2p/blob/master/CHANGELOG.md). +If you are struggling with an upgrade, feel free to tag Max (@mxinden) or myself (@thomaseizinger) in a PR and we'll try to help. + +Happy coding! From b8e8bba0ff8cf3e652fa30accb77318d1188334c Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Tue, 20 Jun 2023 08:55:25 +0200 Subject: [PATCH 2/7] Update src/_blog/rust-libp2p-052-release.md Co-authored-by: Max Inden --- src/_blog/rust-libp2p-052-release.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/_blog/rust-libp2p-052-release.md b/src/_blog/rust-libp2p-052-release.md index 25c4a4d3..8198d8d4 100644 --- a/src/_blog/rust-libp2p-052-release.md +++ b/src/_blog/rust-libp2p-052-release.md @@ -19,7 +19,7 @@ They easily trigger a cascade of breaking changes for downstream users. Coming up to the `v0.52` release, we consciously held back and batched up breaking changes and released several features as patch releases instead. As a result, `v0.52` is unfortunately quite large. -This blogpost aims to highlight some of the more exciting changes: +This blogpost highlights the most exciting changes: - Automatic kademlia client/server mode - Type-safe `/p2p` multiaddresses From ea22402721ac61205fb73e4a4ec8de6875a545a6 Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Tue, 20 Jun 2023 08:55:43 +0200 Subject: [PATCH 3/7] Update src/_blog/rust-libp2p-052-release.md Co-authored-by: Max Inden --- src/_blog/rust-libp2p-052-release.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/_blog/rust-libp2p-052-release.md b/src/_blog/rust-libp2p-052-release.md index 8198d8d4..94f980e9 100644 --- a/src/_blog/rust-libp2p-052-release.md +++ b/src/_blog/rust-libp2p-052-release.md @@ -78,7 +78,7 @@ If you are interested in the details, read on: - Local changes to our protocols (i.e. a node going from Kademlia server to client mode) are now immediately pushed to the remote via the [`/ipfs/id/push/1.0.0`](https://github.com/libp2p/specs/tree/master/identify#identifypush) protocol: [#3980](https://github.com/libp2p/rust-libp2p/pull/3980). - Simplify the scoring mechanism of external addresses: [#3954](https://github.com/libp2p/rust-libp2p/pull/3954) -Not only does this work out-of-the-box and thus improves the developer experience of `rust-libp2p`, it should also result in a much more useful and up-to-date routing table for all users of the Kademlia DHT. +Not only does this work out-of-the-box and thus improves the developer experience of `rust-libp2p`, it should also result in a much more useful and up-to-date routing table for all nodes on a Kademlia DHT. ## Type-safe `/p2p` multiaddresses From 82c5595b1d4dc5f77ac7614b46e8212a0bfbeac1 Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Tue, 20 Jun 2023 09:17:21 +0200 Subject: [PATCH 4/7] Add serde req-res protocol --- src/_blog/rust-libp2p-052-release.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/_blog/rust-libp2p-052-release.md b/src/_blog/rust-libp2p-052-release.md index 94f980e9..22c89341 100644 --- a/src/_blog/rust-libp2p-052-release.md +++ b/src/_blog/rust-libp2p-052-release.md @@ -26,6 +26,7 @@ This blogpost highlights the most exciting changes: - More consistent event/command naming - Improve ergonomics around stream-upgrade errors - Simpler noise interface +- Request-response protocols using serde ## Automatic kademlia client/server mode @@ -223,6 +224,19 @@ Using the noise handshake is now as simple as: let config = libp2p::noise::Config::new(&keypair).unwrap(); ``` +## Request-response protocols using serde + +A simple, yet impactful contribution was made by [@dgarus](https://github.com/dgarus) in [#3952](https://github.com/libp2p/rust-libp2p/pull/3952): + +- `libp2p::request_response::cbor::Behaviour` +- `libp2p::request_response::json::Behaviour` + +These two, new type-aliases come with a pre-configured CBOR/JSON codec and only need to be parameterised by the `Req` and `Res` types. +The only requirement is that the types implement `serde::{Serialize,Deserialize}`. +Defining a request-response protocol has never been easier in `rust-libp2p`! + +As with any code that serializes data structures, be aware that any changes to it can easily be breaking and thus not backwards-compatible with older versions. + ## Closing Well done for making it to the end! From d7e1f9f54a0520c118879b7061dab96de1496e78 Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Tue, 20 Jun 2023 09:26:30 +0200 Subject: [PATCH 5/7] Reword intro --- src/_blog/rust-libp2p-052-release.md | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/src/_blog/rust-libp2p-052-release.md b/src/_blog/rust-libp2p-052-release.md index 22c89341..b01bd02e 100644 --- a/src/_blog/rust-libp2p-052-release.md +++ b/src/_blog/rust-libp2p-052-release.md @@ -13,13 +13,8 @@ author: Thomas Eizinger # rust-libp2p v0.52 release spotlight -With the `v0.52` release, the `rust-libp2p` maintainer team tried something new. -Breaking changes - even in the presence of an obvious or easy upgrade path - are a burden for our users. -They easily trigger a cascade of breaking changes for downstream users. - -Coming up to the `v0.52` release, we consciously held back and batched up breaking changes and released several features as patch releases instead. -As a result, `v0.52` is unfortunately quite large. -This blogpost highlights the most exciting changes: +We are continuously working to make `rust-libp2p` better and easier to use. +To give you a taste of the `v0.52` release, this post highlights its most exciting changes: - Automatic kademlia client/server mode - Type-safe `/p2p` multiaddresses From d0886a7ef8d3a9b126ad5af28eeb5a7a27012341 Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Tue, 20 Jun 2023 10:50:19 +0200 Subject: [PATCH 6/7] Add list of external contributors --- src/_blog/rust-libp2p-052-release.md | 42 ++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/src/_blog/rust-libp2p-052-release.md b/src/_blog/rust-libp2p-052-release.md index b01bd02e..4fe4d572 100644 --- a/src/_blog/rust-libp2p-052-release.md +++ b/src/_blog/rust-libp2p-052-release.md @@ -232,6 +232,48 @@ Defining a request-response protocol has never been easier in `rust-libp2p`! As with any code that serializes data structures, be aware that any changes to it can easily be breaking and thus not backwards-compatible with older versions. +## Thanks! + +A big thanks to all people that contributed to this release. +In alphabetical order: + +- [@AgeManning](https://github.com/AgeManning) +- [@arpankapoor](https://github.com/arpankapoor) +- [@b0xtch](https://github.com/b0xtch) +- [@b-zee](https://github.com/b-zee) +- [@creativcoder](https://github.com/creativcoder) +- [@dariusc93](https://github.com/dariusc93) +- [@dgarus](https://github.com/dgarus) +- [@DougAnderson444](https://github.com/DougAnderson444) +- [@drHuangMHT](https://github.com/drHuangMHT) +- [@elenaf9](https://github.com/elenaf9) +- [@felinira](https://github.com/felinira) +- [@galargh](https://github.com/galargh) +- [@hanabi1224](https://github.com/hanabi1224) +- [@helloimalemur](https://github.com/helloimalemur) +- [@jsantell](https://github.com/jsantell) +- [@kckeiks](https://github.com/kckeiks) +- [@leviathanbeak](https://github.com/leviathanbeak) +- [@linoscope](https://github.com/linoscope) +- [@melekes](https://github.com/melekes) +- [@nathanielc](https://github.com/nathanielc) +- [@obi1kenobi](https://github.com/obi1kenobi) +- [@oblique](https://github.com/oblique) +- [@ozkanonur](https://github.com/ozkanonur) +- [@ozwaldorf](https://github.com/ozwaldorf) +- [@PopBogdan97](https://github.com/PopBogdan97) +- [@shamil-gadelshin](https://github.com/shamil-gadelshin) +- [@StemCll](https://github.com/StemCll) +- [@stonecharioteer](https://github.com/stonecharioteer) +- [@stormshield-pj50](https://github.com/stormshield-pj50) +- [@tcoratger](https://github.com/tcoratger) +- [@tthebst](https://github.com/tthebst) +- [@umgefahren](https://github.com/umgefahren) +- [@uwejan](https://github.com/uwejan) +- [@vnermolaev](https://github.com/vnermolaev) +- [@wizeguyy](https://github.com/wizeguyy) +- [@yellowred](https://github.com/yellowred) + ## Closing Well done for making it to the end! From 71789104b028d1a1bc19dc65af1ee0244ec2d676 Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Tue, 20 Jun 2023 13:48:44 +0200 Subject: [PATCH 7/7] Add hole punching and identify change --- src/_blog/rust-libp2p-052-release.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/src/_blog/rust-libp2p-052-release.md b/src/_blog/rust-libp2p-052-release.md index 4fe4d572..accc1ae3 100644 --- a/src/_blog/rust-libp2p-052-release.md +++ b/src/_blog/rust-libp2p-052-release.md @@ -22,6 +22,7 @@ To give you a taste of the `v0.52` release, this post highlights its most exciti - Improve ergonomics around stream-upgrade errors - Simpler noise interface - Request-response protocols using serde +- Hole-punching for QUIC ## Automatic kademlia client/server mode @@ -45,6 +46,11 @@ We can now do the following: To implement this, several other features/issues had to be fixed. If you are interested in the details, read on: +- Simplify the scoring mechanism of external addresses: [#3954](https://github.com/libp2p/rust-libp2p/pull/3954) + + As a consequence, the observed address reported by identify is no longer considered an _external_ address but just an address candidate. + Checkout the [changelog-entry](https://github.com/libp2p/rust-libp2p/blob/master/protocols/identify/CHANGELOG.md#0430) for a way of restoring the old behaviour. + - Changes to the supported protocols are now detected at runtime and communicated to all protocols: [#3651](https://github.com/libp2p/rust-libp2p/pull/3651). Previously, a protocol could retrieve the supported protocols via `PollParameters::supported_protocols`. @@ -72,7 +78,6 @@ If you are interested in the details, read on: This was always the case in the specs but the `rust-libp2p` implementation was lagging behind here and improperly represented them as bytes internally. - Local changes to our protocols (i.e. a node going from Kademlia server to client mode) are now immediately pushed to the remote via the [`/ipfs/id/push/1.0.0`](https://github.com/libp2p/specs/tree/master/identify#identifypush) protocol: [#3980](https://github.com/libp2p/rust-libp2p/pull/3980). -- Simplify the scoring mechanism of external addresses: [#3954](https://github.com/libp2p/rust-libp2p/pull/3954) Not only does this work out-of-the-box and thus improves the developer experience of `rust-libp2p`, it should also result in a much more useful and up-to-date routing table for all nodes on a Kademlia DHT. @@ -232,6 +237,14 @@ Defining a request-response protocol has never been easier in `rust-libp2p`! As with any code that serializes data structures, be aware that any changes to it can easily be breaking and thus not backwards-compatible with older versions. +## Hole-punching for QUIC + +Despite still being in alpha, our QUIC implementation can now establish direct connections across certain NATs, i.e. hole-punching! +This was contributed by [@arpankapoor](https://github.com/arpankapoor) in [#3964](https://github.com/libp2p/rust-libp2p/pull/3964). + +You don't need to configure anything special besides the `dcutr` behaviour. +See the [dcutr-example](https://github.com/libp2p/rust-libp2p/tree/libp2p-v0.52.0/examples/dcutr) in our repository for details. + ## Thanks! A big thanks to all people that contributed to this release.