Kinesis Parser with Protobuf and Redis

This application aims to do a very simple process as below:

1. read streams from a given kinesis stream (configurable)
2. parse them based on given protobuf schemes
3. write processed records into a db (Redis)
4. print saved records to the stdout once it has reached the max read config

Architecture

The project is developed using Python 3.12 and Redis. Check the architecture if you are interested to read mode about design choices.

Requirments

You need to have the following packages on your system to be able to run/develop the project

1. Docker
2. (Optionally) docker-compose

How to run

There are two ways you can run the project, but before digging into that, first ensure you know what you are doing.

Check the configuration

The project is designed to be easily extendable to different environments by using separate configuration files located under config/env/. These files include parameters for setting up the project, including the AWS region and the kinesis stream name, as well as the parameters for the application such as the number of retries if the rate limit exceeded. Please ensure you have checked your ENV file before moving to the next steps.

Automatic

This is an easy way to run the project locally to see how it work. This approach doesn't give you much flexibity unless you dig into changing some config files and the docker-compose.yaml file. Since this step launches a Redis docker container on your machine, first ensure there won't be any conflicts with your server.

If this sounds good to you, simply follow commands:

$ # (optionally) stop the redis server if it is already running
$ service redis-server stop
$
$ # (optionally) build the project first
$ docker build . -t kinesis
$
$ # start the service
$ docker compose up

Manual

If for any reason (like if you have a redis running on your host that can't be stopped) you'd rather to do everything manually, follow the steps below

$ # export required environments
$ export ENV=local  # local is the only config file for now
$ export AWS_ACCESS_KEY_ID=""
$ export AWS_SECRET_ACCESS_KEY=""
$
$ # make sure you have redis accessible
$ # Update the relevant config file if it has a different configuration than the local
$ redis-cli ping
$
$ # build the container
$ docker build . -t kinesis
$
$ # run the container
$ docker run \
    -v ./:/app/ \
    -e ENV="$ENV" \
    -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
    -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
    kinesis

How to maintain

The project mounts your filesystem to the Docker container to perform tasks. You can simply get the bash access into the machine and check the commands available there

$ docker run -v ./:/app/ -it kinesis /bin/bash
$
$ # check available commands
$ task -l

task: Available tasks for this project:
* clean:           Cleanup the project
* compile:         Compile Protobuff schemes to Python
* format:          Format code via isort-black
* install:         Install dependencies via poetry
* lint:            Run linter via ruff
* run:             Run the application
* sync_lock:       Sync poetry.lock with pyproject.toml
* test:            Run tests via pytest
* update:          Update dependencies via poetry

Poetry

Tasks

The project uses Taskfile to manage project commands. The container runs these commands once it is launched

Tests

At the time of writing this doc, the project has a test coverage of 100%. If you are interested to see the report, you can check the coverage report.