Onboard Katio to Kubernetes services hosted by other cloud vendors

[Fei-Guo]

### Tasks
- [ ] https://github.com/Azure/kaito/pull/357
- [ ] https://github.com/Azure/kaito/pull/360
- [ ] https://github.com/Azure/kaito/pull/364
- [ ] https://github.com/Azure/kaito/pull/518
- [ ] https://github.com/Azure/kaito/pull/602
- [x] Test: verify Kaito works on AWS using self-provisioned gpu-nodes
- [x] Test: verify Kaito works on AWS with Karpenter integration
- [ ] Add installation script for AWS
- [ ] Update docs
- [ ] add/update tests
- [ ] https://github.com/Azure/kaito/pull/650

[Fei-Guo]

We will start from deploying Kaito in AWS since AWS does support Karpenter, making the integration easier.

[javaniecampbell]

Hey all, just quick question does this feature enhancement will it include self-hosted kubernetes, I checked a few places but wasn't sure so I figured maybe this could be right place to see if this is considered?

The consideration is that some will need for self-host community, home-labs and companies, etc that need the llms to be ran locally.

[Fei-Guo]

Hey all, just quick question does this feature enhancement will it include self-hosted kubernetes, I checked a few places but wasn't sure so I figured maybe this could be right place to see if this is considered?

The consideration is that some will need for self-host community, home-labs and companies, etc that need the llms to be ran locally.

You can run Kaito in selfmanaged k8s if you already add GPU nodes in the cluster (with proper gpu driver and k8s plugin installed). In this case, you can just add those nodes in the Kaito workspace CR as preferrednodes in the Resource spec. Kaito will skip provisioning gpu nodes and just run inference workload in the existing nodes.