From 6e0c33f2a148d301f6345d896436bde61ecea6cb Mon Sep 17 00:00:00 2001 From: Joshua Baskaran Date: Tue, 19 Nov 2024 09:37:11 +0100 Subject: [PATCH 1/2] Add comprehensive README for Local EGA TSD Proxy service Adds detailed documentation covering: - Service overview and core features - Configuration and prerequisites - Build and deployment instructions - API endpoints and authentication flows - Development guidelines and monitoring This documentation will help new developers understand the service's architecture and assist with deployment and maintenance. --- services/localega-tsd-proxy/README.md | 164 ++++++++++++++++++++++++++ 1 file changed, 164 insertions(+) create mode 100644 services/localega-tsd-proxy/README.md diff --git a/services/localega-tsd-proxy/README.md b/services/localega-tsd-proxy/README.md new file mode 100644 index 00000000..10385edd --- /dev/null +++ b/services/localega-tsd-proxy/README.md @@ -0,0 +1,164 @@ +# Local EGA TSD Proxy Service + +The Local EGA TSD Proxy service is a component of the FEGA-Norway stack that facilitates secure file transfers between users and TSD (Services for Sensitive Data) storage. It implements authentication and authorization using both ELIXIR AAI and CEGA credentials, and manages file operations through the TSD File API. + +## Features + +- ELIXIR AAI (OpenID Connect) authentication support: +- GA4GH Passport & Visa validation +- Secure file upload/download operations +- Resumable file transfers +- File operation event publishing to RabbitMQ +- Service health monitoring +- Redis-based caching +- PostgreSQL integration for credential mapping + +## Prerequisites + +- JDK 21 +- Docker (for containerization) +- PostgreSQL database +- Redis instance +- RabbitMQ server with SSL support +- TSD API access credentials +- ELIXIR AAI client credentials +- CEGA authentication endpoint access + +## Configuration + +The service is configured through environment variables and the `application.yaml` file. Key configuration areas include: + +### SSL Configuration +```yaml +server.ssl: + enabled: ${SSL_ENABLED:true} + key-store-type: PKCS12 + key-store: file:${SERVER_KEYSTORE_PATH:/etc/ega/ssl/server.cert} + key-store-password: ${SERVER_CERT_PASSWORD} +``` + +### Database Configuration +```yaml +spring.datasource: + url: jdbc:postgresql://${DB_INSTANCE:postgres}:${DB_PORT:5432}/${POSTGRES_DB:postgres} + username: ${POSTGRES_USER:postgres} + password: ${POSTGRES_PASSWORD} +``` + +### Redis Configuration +```yaml +spring.data.redis: + host: ${REDIS_HOSTNAME:redis} + port: ${REDIS_PORT:6379} + database: ${REDIS_DB:0} +``` + +### RabbitMQ Configuration +```yaml +spring.rabbitmq: + host: ${BROKER_HOST:public-mq} + port: ${BROKER_PORT:5671} + virtual-host: ${BROKER_VHOST:/} + username: ${BROKER_USERNAME:admin} + password: ${BROKER_PASSWORD:guest} + ssl: + enabled: ${BROKER_SSL_ENABLED:true} +``` + +## Building + +The service uses Gradle as its build system. To build the service: + +```bash +# Build the JAR +./gradlew build + +# Build Docker image +./gradlew buildDockerImage +``` + +## Running + +### Using Docker + +```bash +docker run -p 8080:8080 \ + --env-file \ + localega-tsd-proxy +``` + +### Using Java + +```bash +java -jar build/libs/localega-tsd-proxy.jar +``` + +## API Endpoints + +### Authentication +- `GET /token` - Retrieve access token +- `GET /user` - Retrieve user information + +### File Operations +- `PATCH /stream/{fileName}` - Upload file +- `GET /stream/{fileName}` - Download file +- `GET /files` - List files +- `DELETE /files` - Delete file + +### Resumable Uploads +- `GET /resumables` - List resumable uploads +- `DELETE /resumables` - Delete resumable upload + +### Monitoring +- `GET /heartbeat` - Service health check + +## Authentication Flow + +1. Users authenticate using either: + - ELIXIR AAI OpenID Connect + - CEGA username/password +2. For ELIXIR AAI: + - GA4GH Passports are validated + - Visas are checked for access permissions +3. For CEGA: + - Credentials are validated against CEGA auth endpoint + - Password hashes are verified (BCrypt or crypt) +4. Upon successful authentication, EGA username is mapped to ELIXIR ID + +## File Transfer Flow + +1. Files are uploaded through resumable chunks +2. Each chunk's checksum is validated +3. Upon successful upload: + - File metadata is recorded + - Event is published to RabbitMQ + - File is stored in TSD storage + +## Security Features + +- SSL/TLS encryption for all communications +- OAuth 2.0 / OpenID Connect integration +- GA4GH Passport & Visa validation +- Secure credential storage and validation +- Checksum verification for file integrity +- Secure file transfer protocols + +## Development + +### Project Structure + +- `controllers/` - REST API endpoints +- `services/` - Business logic implementation +- `aspects/` - Cross-cutting concerns (AOP) +- `dto/` - Data transfer objects +- `authentication/` - Authentication components +- `config/` - Application configuration + +### Adding New Features + +When adding new features: +1. Follow the existing package structure +2. Implement appropriate tests +3. Use AOP for cross-cutting concerns +4. Update configuration as needed +5. Document changes in code \ No newline at end of file From 1bad467bde5c0891cc5e9ca75a7a213f49c1adb6 Mon Sep 17 00:00:00 2001 From: Joshua Baskaran Date: Wed, 20 Nov 2024 13:14:59 +0100 Subject: [PATCH 2/2] Add note on the FEGA-Norway static web pages --- services/localega-tsd-proxy/README.md | 36 +++++++++++++++++++-------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/services/localega-tsd-proxy/README.md b/services/localega-tsd-proxy/README.md index 10385edd..25311a20 100644 --- a/services/localega-tsd-proxy/README.md +++ b/services/localega-tsd-proxy/README.md @@ -4,7 +4,7 @@ The Local EGA TSD Proxy service is a component of the FEGA-Norway stack that fac ## Features -- ELIXIR AAI (OpenID Connect) authentication support: +- ELIXIR AAI (OpenID Connect) authentication support - GA4GH Passport & Visa validation - Secure file upload/download operations - Resumable file transfers @@ -12,6 +12,7 @@ The Local EGA TSD Proxy service is a component of the FEGA-Norway stack that fac - Service health monitoring - Redis-based caching - PostgreSQL integration for credential mapping +- Serve the FEGA-Norway static web pages ## Prerequisites @@ -29,6 +30,7 @@ The Local EGA TSD Proxy service is a component of the FEGA-Norway stack that fac The service is configured through environment variables and the `application.yaml` file. Key configuration areas include: ### SSL Configuration + ```yaml server.ssl: enabled: ${SSL_ENABLED:true} @@ -38,6 +40,7 @@ server.ssl: ``` ### Database Configuration + ```yaml spring.datasource: url: jdbc:postgresql://${DB_INSTANCE:postgres}:${DB_PORT:5432}/${POSTGRES_DB:postgres} @@ -46,6 +49,7 @@ spring.datasource: ``` ### Redis Configuration + ```yaml spring.data.redis: host: ${REDIS_HOSTNAME:redis} @@ -54,6 +58,7 @@ spring.data.redis: ``` ### RabbitMQ Configuration + ```yaml spring.rabbitmq: host: ${BROKER_HOST:public-mq} @@ -96,33 +101,37 @@ java -jar build/libs/localega-tsd-proxy.jar ## API Endpoints ### Authentication + - `GET /token` - Retrieve access token - `GET /user` - Retrieve user information ### File Operations + - `PATCH /stream/{fileName}` - Upload file - `GET /stream/{fileName}` - Download file - `GET /files` - List files - `DELETE /files` - Delete file ### Resumable Uploads + - `GET /resumables` - List resumable uploads - `DELETE /resumables` - Delete resumable upload ### Monitoring + - `GET /heartbeat` - Service health check ## Authentication Flow 1. Users authenticate using either: - - ELIXIR AAI OpenID Connect - - CEGA username/password + - ELIXIR AAI OpenID Connect + - CEGA username/password 2. For ELIXIR AAI: - - GA4GH Passports are validated - - Visas are checked for access permissions + - GA4GH Passports are validated + - Visas are checked for access permissions 3. For CEGA: - - Credentials are validated against CEGA auth endpoint - - Password hashes are verified (BCrypt or crypt) + - Credentials are validated against CEGA auth endpoint + - Password hashes are verified (BCrypt or crypt) 4. Upon successful authentication, EGA username is mapped to ELIXIR ID ## File Transfer Flow @@ -130,9 +139,9 @@ java -jar build/libs/localega-tsd-proxy.jar 1. Files are uploaded through resumable chunks 2. Each chunk's checksum is validated 3. Upon successful upload: - - File metadata is recorded - - Event is published to RabbitMQ - - File is stored in TSD storage + - File metadata is recorded + - Event is published to RabbitMQ + - File is stored in TSD storage ## Security Features @@ -143,6 +152,10 @@ java -jar build/libs/localega-tsd-proxy.jar - Checksum verification for file integrity - Secure file transfer protocols +## Static Web Pages + +- The FEGA-Norway static web pages are maintained in a separate repository: [FEGA-Norway-webpages](https://github.com/ELIXIR-NO/FEGA-Norway-webpages) + ## Development ### Project Structure @@ -157,8 +170,9 @@ java -jar build/libs/localega-tsd-proxy.jar ### Adding New Features When adding new features: + 1. Follow the existing package structure 2. Implement appropriate tests 3. Use AOP for cross-cutting concerns 4. Update configuration as needed -5. Document changes in code \ No newline at end of file +5. Document changes in code