/area vertical-pod-autoscaler

Thanks for submitting this!

This makes sense and is nice and simple, as you mention, it's very analogous to the min flags.

One possible alternative that we didn't talk about during the SIG meeting was to use the ContainerResourcePolicy.MaxAllowed field to cap the recommendations. Though the downside to using this is that you would need every single VPA in the cluster to have this field set.

Regarding this proposal, as we talked about, the downside is that you need to manually set this flag, this value could change over the lifetime of your cluster as different sized nodes are adopted.

If you were to set this "max" value dynamically you would want to make this adjustment in:

• the updater to make sure it can only evict a pod if there is enough room in the cluster
• the admission-controller to trim down and cap recommendations on new or restarted pods to make sure they can fit in what is available in the cluster

However, doing this statically cluster wide is a great first step and seems totally reasonable to me!

That would be very helpful. Usually, the configuration of the cluster and its VPA installation will be done by the same (human or technical) operators. It would be more difficult to change all VPA resources, especially if they are controlled by third-party (technical) operators that we cannot influence in all aspects. Then, the alternative, if this isn't becoming a global VPA option, is to create a mutating web hook, but then everybody would have to do that. If VPA would offer that feature out-of-the-box, this would be great. After all, maxAllowed seems to have predominantly this use case: to make the recommender not recommend anything that wouldn't fit. The definition of the cluster and its worker pools is usually always handled by other people than the developers of the workload and their VPA resources - at least that's a pattern we see. As for dynamically following the adjustment/larger/largest nodes, I am doubtful too much intelligence will pay off. In our case it wouldn't as we a.) use worker pool node scale-from-zero, i.e. VPA cannot see the possible available machine sizes at all if it just checks the nodes and b.) it is challenging to understand which nodes are really suitable (taints/tolerations, affinities/anti-affinities, etc.). The last point could be overcome, the first one never. Therefore static configuration for VPA's global maxAllowed sounds simple and reasonable - more intelligence may help, but isn't guaranteed to always help.

Here’s how I see the options for supporting this:

1. Iterate through the cluster nodes and find the maximum allocatable resources per node. The downside is that VPA won’t be able to account for "upcoming nodes" (nodes that are expected to join the cluster via the cluster autoscaler). I've started working on this approach here: [WIP] VPA: Takes maximum node size into account (upper-bound) #7345.
2. Iterate through both the current cluster nodes and the "upcoming nodes."
3. Implement a mutating webhook on VPA that sets a maxAllowed value per VPA resource (as suggested by the issuer).
4. Use a fixed value, similar to how pod-recommendation-min-cpu-millicores and pod-recommendation-min-memory-mb are handled.

cc @adrianmoisey

Related issue: #6487

After spending some time, it is not straightforward to add Pod-level flags for max allowed cpu and memory recommendations.

I already raised the concern in the initial issue description:

PS: However, for min allowed recommendation the flags seem to be per Pod and the recommendations are distributed proportionally based on the Pod's containers count. See

autoscaler/vertical-pod-autoscaler/pkg/recommender/logic/recommender.go

Lines 70 to 74 in 5aaeb83

	fraction := 1.0 / float64(len(containerNameToAggregateStateMap))
	minResources := model.Resources{
	model.ResourceCPU: model.ScaleResource(model.CPUAmountFromCores(*podMinCPUMillicores*0.001), fraction),
	model.ResourceMemory: model.ScaleResource(model.MemoryAmountFromBytes(*podMinMemoryMb*1024*1024), fraction),
	}

I am not sure if the same thing would work well for Pods that have multiple containers under VPA. For example, for a Pod with 2 containers, one of the containers could be limited at 50% of the configured max allowed value.

As end users of VPA, it would be great if end users can specify Pod-level maximum allowed values because scheduling is based on Pod resource requests. In this way, end user can make sure that VPA won't recommended values that make Pods unschedulable.
On the other side, recommendations in VPA are on container-level. VPA can be enabled for > 1 container in a Pod. The question is how VPA is expected to distribute the Pod-level max allowed recommendation across the many containers in the Pod. Some approaches are - distribute the configured max allowed recommendation values evenly; distribute proportionally (proportionally based on what?). My conclusion is that VPA should not do that as it does not know about the workload.

Hence, I would propose to introduce container-level max allowed recommendation flags. Yes, with this approach end user can theoretically end up again with unschedulable Pods due to VPA recommendations above the Node's allocatable.

Example: Cluster Node's allocatable is 7Gi, while a Pod has 2 containers and both of the containers have a recommendation of 4Gi each. The Pod's memory request will be 8Gi and it will exceed the Node's allocatable of 7Gi.

However, I think the container-level max allowed proposal is good enough because:

• it is not a common case autoscaling to be enabled for many containers in a Pod. Even it is, usually there is a one main container while the others are not likely to need upscaling like the main one
• it is not a perfect solution but it is a practical one that will address the majority of the cases caused by the well-known VPA limitation
• even with the VPA's maxAllowed field it is not possible containers of a Pod to grow up to a max value but at the same time to ensure that the Pod remain schedulable.

That's my proposal is to add flags on container-level:

• --container-recommendation-max-memory-mb
• --container-recommendation-max-cpu-millicores

I would even propose to deprecate the existing Pod-level flags for min allowed recommendations in favor of container-level equivalents.

Let me know what you think.

I like the proposal, but I worry about this:

Example: Cluster Node's allocatable is 7Gi, while a Pod has 2 containers and both of the containers have a recommendation of 4Gi each. The Pod's memory request will be 8Gi and it will exceed the Node's allocatable of 7Gi.

I guess if one were to run a recommender for various Pods it could work.
For example:

1. recommender-1 - VPA recommender for Pods with single containers
2. recommender 2- VPA recommender for Pods with 2 containers

I am wondering if the Pod level resources KEP could actually help out here.
I haven't read the final proposal thoroughly yet, but the VPA could look at a combination of all containers metrics, and set a Pod level request. And in this case, it could keep the Pod's recommendation within the limits of an option, such as --pod-recommendation-max-memory-mb

I like the proposal, but I worry about this:

Example: Cluster Node's allocatable is 7Gi, while a Pod has 2 containers and both of the containers have a recommendation of 4Gi each. The Pod's memory request will be 8Gi and it will exceed the Node's allocatable of 7Gi.

I guess if one were to run a recommender for various Pods it could work. For example:

1. recommender-1 - VPA recommender for Pods with single containers
2. recommender 2- VPA recommender for Pods with 2 containers

I am wondering if the Pod level resources KEP could actually help out here. I haven't read the final proposal thoroughly yet, but the VPA could look at a combination of all containers metrics, and set a Pod level request. And in this case, it could keep the Pod's recommendation within the limits of an option, such as --pod-recommendation-max-memory-mb

From my understanding, pod-level resource limits are optional, so we can't depend on them being enforced. However, we should account for them to ensure the total resource usage of all containers stays within the pod's defined limits.

Here's an approach we could consider—let me know your thoughts:

Assume a pod has a memory limit of 1 Gi, for example, and contains 3 containers.

1. The recommender sums up the recommended memory for each container.

2. If the total recommended memory is less than or equal to the pod's memory limit, no adjustment is needed.

3. If the total exceeds the pod's memory limit, we normalize the recommendations. Each container would get a proportion of the pod's memory limit based on its recommendation:

   Adjusted Memory for Container i = $\left( \frac{x_i}{x_1 + x_2 + x_3} \right) \times \text{Pod Memory Limit}$

   Here, $x_i$ represents the memory recommendation for container i.

@raywainman @adrianmoisey , I created #7560.

I will be happy to receive a feedback! :)