VPA support for pod-level resource specifications

[voelzmo]

Which component are you using?:
/area vertical-pod-autoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:
With k8s 1.32, a new feature gate is released in alpha: PodLevelResources, allowing users to specify resources on Pod level instead of on Container level.
More info on how this works in the KEP

Describe the solution you'd like.:
VPA should support this and autoscale Pods instead of individual Containers.
This is just a placeholder/umbrella issue, we need a proper enhancement proposal doc for this.

[omerap12]

I’d be happy to open a KEP for this, as I have some ideas to contribute. Specifically, I believe the VPA should support both approaches. That means, if we have PodLevelResources, the scaling should apply at the pod level; otherwise, it should scale at the container level. The main challenge lies in scaling multi-container pods with PodLevelResources. I shared an idea on this topic here: comment.