K8s setup for reNgine #1457
K8s setup for reNgine #1457
Conversation
…a, and Nginx This commit adds the Kubernetes service configurations for Redis, Postgres, Web, Ollama, and Nginx. These services are essential for the application's functionality and communication within the Kubernetes cluster. Each service is defined with its respective port and targetPort, and is associated with the corresponding app label. The services are created with the necessary metadata and specifications to ensure proper networking and connectivity. The added service configurations include: - Redis service: Exposes port 6379 for Redis communication. - Postgres service: Exposes port 5432 for Postgres communication. - Web service: Exposes port 8000 for the web application. - Ollama service: Exposes port 11434 for the Ollama application. - Nginx service: Exposes ports 80 and 443 for HTTP and HTTPS traffic. These service configurations are crucial for enabling communication between different components of the application and facilitating external access to the services when needed.
|
Woohoo @0xtejas! 🎉 You've just dropped some hot new code! 🔥 Hang tight while we review this! You rock! 🤘 |
|
I need additional hands on this to improvise a few more things:
|
|
The boot order for the services are: postgres, redis, web, celery and celery-beat. |
|
@yogeshojha you'll also have to release packages (docker images) for celery and celery-beat. In the current setup, it is using the ones that I built and pushed to GHCR. |
|
@0xtejas I am reviewing this, is this ready? |
|
No @yogeshojha I need some more help on the other points that I have listed out above. I'll try accomplishing if possible and update the checkboxes. |
|
There seems to be an issue with the current PR, the celery pod gets evicted after an hour or so. Also, the current resource requests and limits will get the pod killed with OOM. If we remove it, the issue should be resolved. However, the former issue is not yet addressed. |
…with fallback with custom certs - Changed the service type from LoadBalancer to ClusterIP in the nginx service configuration. - Updated the containerPort in the nginx deployment configuration from 8082 to 80. - Added new files for cert-manager configuration: certificate.yaml and cluster-issuer.yml. - Created an ingress configuration for nginx with SSL redirection and rewrite rules.
|
Hey, thanks for your contribution! 🙏 We appreciate the time and effort you put into this PR. Sadly this is not the right fit for reNgine at the moment. While we couldn't merge it this time, we value your interest in improving reNgine. Feel free to reach out if you have any questions. Thanks again! |
|
Sorry closed the PR by mistake. I have a few more changes left and an investigation to conduct into why the pod celery gets evicted once it reaches the end. |
|
Almost, everything is done. I still cannot figure out why the celery pod gets OOMKilled. I'd appreciate it if you could check. Overall it works. I did not implement a replication method for DBs. This means if a user has more than 1 replica then they'll have to figure out how to setup data replication. We can revisit this advanced stuff in later issues/pr. |
|
I've confirmed that the celery pod used to get killed cuz of less resource in the node. We will have to figure out a proper resource request and limitation for it to work in the |
|
@yogeshojha, can you please take a look when you get time? I've added all the manifest required. Some say the correct way to distribute K8s is using the HELM Chart, but I'm not very familiar with doing it. |
|
Hi @0xtejas , Yes you are right the correct way to distribute in the K8s environment for the public one is via the Helm chart. Helm is the current industry standard and would have the most community support, making it hard to refuse. Let me know any thoughts over this @yogeshojha |
|
You are right, @ajutamangdev. I did further research following my last comment. This is the case. However, I'm not familiar with the Helm Chart. Until then, this could be a stepping stone towards K8s deployments. |
This pull request introduces several Kubernetes configurations for deploying various services, including Celery, Nginx, PostgreSQL, Redis, and a web application. The changes include deployment configurations, persistent volume claims, services, and secrets.
Deployment Configurations:
k8s/celery-beat/deployment.yml: Added deployment configuration forcelery-beatwith environment variables and volume mounts.k8s/celery/deployment.yml: Added deployment configuration forcelerywith environment variables, resource requests, and volume mounts.k8s/nginx/deployment.yml: Added deployment configuration fornginxwith volume mounts for configuration, certificates, and static files.k8s/ollama/deployment.yml: Added deployment configuration forollamawith volume mounts.k8s/web/deployment.yml: Added deployment configuration forwebwith environment variables, ports, and init containers for database migration and static file collection.Persistent Volume Claims:
k8s/celery-beat/pvc.yml: Added persistent volume claims forcelery-beatincludinggithub-repos-pvc,wordlist-pvc,scan-results-pvc,gf-patterns-pvc,nuclei-templates-pvc, andtool-config-pvc.k8s/pvc.yml: Added persistent volume claims for shared storage and static files withnfs-rwx-storage.k8s/ollama/pvc.yml: Added persistent volume claim forollamadata storage.Services:
k8s/celery-beat/service.yml: Added service configuration forcelery-beatwith TCP port 5672.k8s/nginx/service.yml: Added service configuration fornginxwith LoadBalancer type and ports for HTTP and HTTPS.k8s/ollama/service.yml: Added service configuration forollamawith ClusterIP type and port 11434.k8s/redis/service.yml: Added service configuration forrediswith port 6379.k8s/postgres/service.yml: Added service configuration forpostgreswith port 5432.Secrets:
k8s/postgres/secret.yml: Added secret configuration for PostgreSQL credentials and domain name.ConfigMaps:
k8s/nginx/configmap.yml: Added ConfigMap fornginxconfiguration, including SSL settings and proxy settings.StatefulSets:
k8s/postgres/statefulset.yml: Added StatefulSet configuration forpostgreswith environment variables and volume mounts for data storage.These changes collectively set up the necessary infrastructure for deploying and managing the services in a Kubernetes environment.