Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: federated catalog node resolver sample #346

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ a standalone runtime or as part of a connector. This also includes demonstration
node resolver, which resolves the participant connectors in the dataspace and crawls these connectors to compile
a set of all offered catalogs.

All federated catalog samples are located in the `federated-catalog` directory.
All federated catalog samples are located in the [`federated-catalog`](./federated-catalog/README.md) directory.

## Contributing

Expand Down
5 changes: 3 additions & 2 deletions federated-catalog/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ participants in the dataspace. To accomplish this, the FC utilizes crawlers
that periodically crawl the catalogs from each participant and store this list
of catalogs in a local cache.
By maintaining this locally cached version of catalogs, it eliminates the need to query
each participant individually, resulting in faster and more reliable queries.
each participant individually, resulting in faster and more reliable queries. Refer to the
[eclipse-edc/FederatedCatalog](https://github.com/eclipse-edc/FederatedCatalog) for further details.


The following samples shows how to
Expand Down Expand Up @@ -41,6 +42,6 @@ exposes a catalog API that serves the list of catalogs.
---
### Different Implementations of Node Resolver

### [FC sample 03](): Resolve Target Catalog Nodes from static participant file
### [FC sample 03](./fc-03-static-node-directory/README.md): Resolve Target Catalog Nodes from static participant file
This sample demonstrates a Catalog Node resolver, that implements TargetNodeDirectory. It resolves the Target Catalog
Nodes from a static participant file containing the DSP endpoints of the participants.
5 changes: 3 additions & 2 deletions federated-catalog/fc-00-basic/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Any further dependencies will be added in the later samples based on their use c
### fixed-node-resolver
The Federated Catalog requires a list of Target Catalog Nodes, which are essentially the DSP endpoints of the dataspace participants.
The catalog crawler then crawls these listed endpoints to collect their offered catalogs.
This list of Target Nodes is resolved by a Catalog Node Resolver which implements the [TargetNodeDirectory](https://github.com/eclipse-edc/FederatedCatalog/blob/main/spi/crawler-spi/src/main/java/org/eclipse/edc/crawler/spi/TargetNodeDirectory.java).
This list of Target Nodes is resolved by a Catalog Node Resolver which implements the `TargetNodeDirectory`.
Check out [eclipse-edc/FederatedCatalog](https://github.com/eclipse-edc/FederatedCatalog/tree/main) for further information on this topic.
farhin23 marked this conversation as resolved.
Show resolved Hide resolved


Expand All @@ -47,7 +47,8 @@ of the federated catalogs that we are going to build in sample

When the federated catalog boots up, the crawler begins periodically invoking the Target Nodes returned by the
Node Resolver and collecting the catalogs offered by these nodes. To test whether our federated catalogs
(which we will build in later samples: [fc-01-embedded](../fc-01-embedded) and [fc-02-standalone](../fc-02-standalone)) can successfully request and retrieve these catalogs, we need at least one connector with a contract offer.
(which we will build in later samples: [fc-01-embedded](../fc-01-embedded) and [fc-02-standalone](../fc-02-standalone))
can successfully request and retrieve these catalogs, we need at least one connector with a contract offer.

Therefore, in this section, we will start a connector and then create a contract
for this connector. In the future samples, we will refer to it as `participant-connector`.
Expand Down
50 changes: 16 additions & 34 deletions federated-catalog/fc-03-static-node-directory/README.md
Original file line number Diff line number Diff line change
@@ -1,43 +1,37 @@
# Target Node Resolver - Static Node Directory
The Federated Catalog requires a list of Target Catalog Nodes (TCN), which are essentially the participant connectors in the dataspace.
The catalog crawler then crawls the DSP endpoints of these listed nodes, and stores the consolidated set of catalogs in a Fderated Catalog Cache (FCC).
The catalog crawler then crawls the DSP endpoints of these listed nodes, and stores the consolidated set of catalogs in a Federated Catalog Cache (FCC).


This list of Target Catalog Nodes, represented by [TargetNodes](https://github.com/eclipse-edc/FederatedCatalog/blob/main/spi/crawler-spi/src/main/java/org/eclipse/edc/crawler/spi/TargetNode.java),
is provided by the [TargetNodeDirectory](https://github.com/eclipse-edc/FederatedCatalog/blob/main/spi/crawler-spi/src/main/java/org/eclipse/edc/crawler/spi/TargetNodeDirectory.java).
This TargetNodeDirectory serves as a 'phone book', maintaining specific information about the
This list of Target Catalog Nodes, represented by `TargetNodes`,
is provided by the `TargetNodeDirectory`.
This `TargetNodeDirectory` serves as a 'phone book', maintaining specific information about the
dataspace participants. It accepts an initial list of participants (e.g. list of participants'
IDs), and resolves this input to a list of TargetNodes.

The initial participant list may vary in its source and format depending on specific use cases.
To accommodate these variations, different implementations of the TargetNodeDirectory can be
adapted to customize the resolution process of Target Nodes from the provided participant list.
In this sample, we will build a Catalog Node Resolver that reads the participants' data from a
static file, [participants.json](./target-node-resolver/resources/participants.json)
static file, [participants.json](./target-node-resolver/src/main/resources/participants.json)
and resolves it into TargetNodes.


The code in this sample has been organized into several Java modules:

- `target-node-resolver`: contains `CatalogNodeDirectory`, an implementation of
`TargetNodeDirectory`, which accepts the [`participants.json`](./target-node-resolver/resources/participants.json)
`TargetNodeDirectory`, which accepts the [`participants.json`](./target-node-resolver/src/main/resources/participants.json)
and returns a list of TargetNodes.
- `embedded|standalone-fc-with-node-resolver`: the embedded/ standalone federated catalog that will be using the `catalog-node-resolver`.




## Implement the Catalog Node Resolver

### Participant file
To keep things straightforward, in this sample we will store our participant list in a static
json file, [participant.json](./target-node-resolver/resources/participants.json).
This `participant.json`
stores the [TargetNode](https://github.com/eclipse-edc/FederatedCatalog/blob/main/spi/crawler-spi/src/main/java/org/eclipse/edc/crawler/spi/TargetNode.java)
json file, [participant.json](./target-node-resolver/src/main/resources/participants.json), that contains the `TargetNode`
properties of the dataspace participants.
As we will be using the `participant-connector` from [fc-00-basic](../fc-00-basic),
currently the `participant.json`
contains only the TargetNode properties of the `participant-connector`, including its DSP endpoint.
In this case, the file contains the properties of the `participant-connector` from [fc-00-basic](../fc-00-basic).

```json
{
"name": "https://w3id.org/edc/v0.0.1/ns/",
Expand All @@ -46,20 +40,13 @@ contains only the TargetNode properties of the `participant-connector`, includin
"supportedProtocols": ["dataspace-protocol-http"]
}
```

However, in a real-world scenario, this participant file should not directly
include these [`TargetNodes`](https://github.com/eclipse-edc/FederatedCatalog/blob/main/spi/crawler-spi/src/main/java/org/eclipse/edc/crawler/spi/TargetNode.java)
properties. Instead, it should contain some form of participant node identifier
(e.g. participants' DIDs).
Additionally, instead of relying on a static file,
more complex implementations like centralized participant registry can also be adapted.
However, this solution is intended for use only within the sample scope; in production, it must be managed in different way.

### Target Node Resolver

#### TargetNodeResolver
The [CatalogNodeResolver](./target-node-resolver/src/main/java/org/eclipse/edc/sample/extension/fc/CatalogNodeDirectory.java)
The [CatalogNodeDirectory](./target-node-resolver/src/main/java/org/eclipse/edc/sample/extension/fc/CatalogNodeDirectory.java)
implements TargetNodeDirectory and overrides its `getAll()` method.
In our implementation, this method maps the array of Json objects from the [`participant.json`](./target-node-resolver/resources/participants.json)
In our implementation, this method maps the file content of [`participant.json`](./target-node-resolver/src/main/resources/participants.json)
to a list of TargetNodes.

```java
Expand All @@ -68,7 +55,7 @@ public class CatalogNodeDirectory implements TargetNodeDirectory {
@Override
public List<TargetNode> getAll() {
try {
return objectMapper.readValue(participantListFile, new TypeReference<>() {});
return objectMapper.readValue(participantFileContent, new TypeReference<>() {});
} catch (IOException e) {
throw new RuntimeException(e);
}
Expand All @@ -80,9 +67,7 @@ During the preparation phase of a crawler run, the FC ExecutionManager invokes t
to obtain the list of TargetNodes.
The crawler requests the DSP endpoints of the participants and stores the
aggregated catalogs in a Federated Catalog Cache (FCC).
In this example we are using the default implementation of an FCC, [InMemoryFederatedCatalogCache](https://github.com/eclipse-edc/FederatedCatalog/blob/main/core/federated-catalog-core/src/main/java/org/eclipse/edc/catalog/store/InMemoryFederatedCatalogCache.java).


In this example we are using the in-memory implementation of an FCC.

## Run Federated Catalog with Node Resolver

Expand All @@ -96,14 +81,13 @@ Before requesting each of the federated catalog APIs, make sure the `partcipant-

### Run standalone-fc with Node Resolver
Apply the following steps to run a standalone federated catalog that uses the implemented static `target-node-resolver`.

#### Build the standalone-fc JAR
Execute this command in project root:

```bash
./gradlew federated-catalog:fc-03-static-node-directory:standalone-fc-with-node-resolver:build
```


#### Run the standalone-fc

To run the federated catalog, execute the following command
Expand All @@ -124,7 +108,6 @@ curl -d @federated-catalog/fc-01-embedded/resources/empty-query.json \
-s | jq
```


### Run embedded-FC with Node Resolver
Apply the following steps to run an embedded federated catalog connector that uses the implemented static `target-node-resolver`.

Expand All @@ -135,7 +118,6 @@ Execute this command in project root:
./gradlew federated-catalog:fc-03-static-node-directory:embedded-fc-with-node-resolver:build
```


#### Run the fc-connector

To run the federated catalog, execute the following command
Expand All @@ -154,4 +136,4 @@ To get the combined set of catalogs, use the following request:
curl -d @federated-catalog/fc-01-embedded/resources/empty-query.json \
-H 'content-type: application/json' http://localhost:29195/api/catalog/v1alpha/catalog/query \
-s | jq
```
```
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,24 @@
import org.eclipse.edc.crawler.spi.TargetNode;
import org.eclipse.edc.crawler.spi.TargetNodeDirectory;

import java.io.File;
import java.io.IOException;
import java.util.List;

public class CatalogNodeDirectory implements TargetNodeDirectory {

private final ObjectMapper objectMapper;
private final File participantListFile;
private final String participantFileContent;

public CatalogNodeDirectory(File participantListFile) {
this.objectMapper = new ObjectMapper();
this.participantListFile = participantListFile;

public CatalogNodeDirectory(ObjectMapper objectMapper, String participantFileContent) {
this.objectMapper = objectMapper;
this.participantFileContent = participantFileContent;
}

@Override
public List<TargetNode> getAll() {
try {
return objectMapper.readValue(participantListFile, new TypeReference<>() {});
return objectMapper.readValue(participantFileContent, new TypeReference<>() {});
} catch (IOException e) {
throw new RuntimeException(e);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,29 +15,34 @@
package org.eclipse.edc.sample.extension.fc;

import org.eclipse.edc.crawler.spi.TargetNodeDirectory;
import org.eclipse.edc.runtime.metamodel.annotation.Inject;
import org.eclipse.edc.runtime.metamodel.annotation.Provider;
import org.eclipse.edc.spi.system.ServiceExtension;
import org.eclipse.edc.spi.system.ServiceExtensionContext;
import org.eclipse.edc.spi.types.TypeManager;

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;

public class CatalogNodeDirectoryExtension implements ServiceExtension {
@Inject
private TypeManager typeManager;

private File participantListFile;
@Provider
public TargetNodeDirectory federatedCacheNodeDirectory() {
String participantsFilePath = "participants.json";

@Override
public void initialize(ServiceExtensionContext context) {
var participantsFilePath = "federated-catalog/fc-03-static-node-directory/target-node-resolver/resources/participants.json";
ClassLoader classLoader = getClass().getClassLoader();
try (InputStream participantFileInputStream = classLoader.getResourceAsStream(participantsFilePath)) {
if (participantFileInputStream == null) {
throw new RuntimeException("Participant list file does not exist: " + participantsFilePath);
}

participantListFile = new File(participantsFilePath).getAbsoluteFile();
if (!participantListFile.exists()) {
context.getMonitor().warning("Path '%s' does not exist.".formatted(participantsFilePath));
}
}
String participantFileContent = new String(participantFileInputStream.readAllBytes(), StandardCharsets.UTF_8);

@Provider
public TargetNodeDirectory federatedCacheNodeDirectory() {
return new CatalogNodeDirectory(participantListFile);
return new CatalogNodeDirectory(typeManager.getMapper(), participantFileContent);
} catch (IOException e) {
throw new RuntimeException("Failed to read and map participant list file: " + participantsFilePath, e);
}
}

}
4 changes: 0 additions & 4 deletions system-tests/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,6 @@ plugins {
`java-library`
}

tasks.test {
workingDir = rootProject.projectDir
}

dependencies {
testImplementation(libs.edc.junit)
testImplementation(libs.edc.json.ld.lib)
Expand Down
Loading