pixels-to-cloud

pixels-to-cloud is a photo-editor app that aims to inspire developers to create high-performance, scalable, distributed systems. By leveraging technologies such as gRPC and DigitalOcean Object Storage (Spaces), this project showcases the potential of modern tech stacks to build robust and efficient application architectures.

Here is the overview of the architecture:

Prerequisites

• Rust
  • Includes Cargo: Rust package manager
• DigitalOcean Account
  • To create and access DO Object Storages: Spaces
• Rclone
  • Spaces sync across different geographies

Setting Up DigitalOcean Spaces & Rclone

To create three DigitalOcean Spaces instances in NYC, AMS, and SYD, follow these steps:

1. Log in to Your DigitalOcean Account:

   • Go to DigitalOcean and log in.

2. Navigate to Spaces Object Storage:

   • Access the Spaces section: DigitalOcean Spaces.

3. Create Spaces across three Data Centers:

   • Choose data center regions:
     • NYC: "New York(nyc3)"
     • AMS: "Amsterdam(ams3)"
     • SYD: "Sydney(syd1)"
   • Name your Spaces (e.g., pixbox-nyc, pixbox-ams, pixbox-syd).
   • Adjust settings as needed and click "Create a Spaces Bucket".
   • For more details, refer to the DigitalOcean Spaces Documentation.

4. Generate Spaces Access Keys:

   • Go to the API section to generate new Spaces Keys.
   • Copy the Access Key and Secret Key and store them securely.

5. Configure Spaces Access using Rclone:

   • Run rclone config to set up the rclone configuration
   • Your configuration file should be similar to the one below:

   # Rclone configuration example
   [pixbox-ams]
   type = s3
   provider = DigitalOcean
   env_auth = false
   access_key_id = <Access Key>
   secret_access_key = <Secret>
   endpoint = ams3.digitaloceanspaces.com
   acl = 
   
   [pixbox-syd]
   type = s3
   provider = DigitalOcean
   env_auth = false
   access_key_id = <Access Key>
   secret_access_key = <Secret>
   endpoint = syd1.digitaloceanspaces.com
   acl = 
   
   [pixbox-nyc]
   type = s3
   provider = DigitalOcean
   env_auth = false
   access_key_id = <Access Key>
   secret_access_key = <Secret>
   endpoint = nyc3.digitaloceanspaces.com
   acl = 

Run the demo

git clone https://github.com/digitalocean/pixels-to-cloud.git
cd pixels-to-cloud

There are three main components:

• client
• server
• synchronizer

For ease of usage, we have created a Makefile with all the necessary commands to build/run the project.

Start the Server

# The server will start listening on localhost:9001
make run
# The above make command translates to
cargo run --release --bin server

Start the Rclone Synchronizer

./scripts/spaces-sync.sh <path to the directory to be synced>
# Example
./scripts/spaces-sync.sh ./images/edited 

Using the Client

# The client is used to upload the image to be edited to the server.
make upload
# The above make command translates to
cargo run --bin client upload --imgpath </path/to/your/image>

Acknowledgments

• Photon: Image Processing Library in WebAssembly/Rust
• gRPC Community
• Cape Breton: Photo Courtesy - Margaret MacAfee

References

• Postman: What is gRPC?
• gRPC over HTTP/2 in Rust
• Tonic: gRPC Client and Server Implementation