Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider adding option to hide irrelevant CPU and RAM stats inside container #1538

Open
jjyyxx opened this issue Sep 17, 2024 · 6 comments
Open
Labels
Containers 📦 Light-weight virtualization suites enhancement Extension or improvement to existing feature Linux 🐧 Linux related issues

Comments

@jjyyxx
Copy link

jjyyxx commented Sep 17, 2024

Inside containers hosted on a server with a lot of cores and RAM, with the actual core and RAM limit constrained via cgroup settings (cpuset and memory), the htop's CPU and RAM meters are quite unhelpful, in the sense that it could not reflect the container's actual situation (and sometimes even annoying when the terminal size is small, leaving no space for processes). In my case, the server has 2x Intel(R) Xeon(R) Platinum 8480+ (224 threads) and 2TiB memory, but the container is limited to 16 threads and 32GiB memory.

This two constraints can be queried via

  • /sys/fs/cgroup/cpuset/cpuset.cpus and /sys/fs/cgroup/memory/memory.limit_in_bytes for cgroup v1
  • /sys/fs/cgroup/cpuset.cpus and /sys/fs/cgroup/memory.max for cgroup v2

Ideally, htop could provide container-aware option such that only the core utilization within the cpuset, and the actual memory limit (and usage) is displayed.

@BenBE BenBE added enhancement Extension or improvement to existing feature labels Sep 17, 2024
@fasterit
Copy link
Member

What containerization solution are you using?

@jjyyxx
Copy link
Author

jjyyxx commented Sep 17, 2024

The containers are allocated with web ui of certain proprietary cluster management system, but the underlying containers are most likely managed by docker with NVIDIA Container Toolkit. Cgroup v1 is used, and I could check /sys/fs/cgroup/cpuset/cpuset.cpus (showing 208-223) and /sys/fs/cgroup/memory/memory.limit_in_bytes (showing 34359738368).

@fasterit
Copy link
Member

The cgroup stuff seems to be quite messy still cf. kubernetes/kubernetes#119669 .
What does cat /proc/self/cgroup say in your case?

@jjyyxx
Copy link
Author

jjyyxx commented Sep 17, 2024

It shows

12:devices:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
11:memory:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
10:rdma:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
9:pids:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
8:hugetlb:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
7:perf_event:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
6:cpuset:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
5:blkio:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
4:freezer:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
3:net_cls,net_prio:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
2:cpu,cpuacct:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
1:name=systemd:/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope
0::/system.slice/docker-c18d38b354eab53da0111aa259a7b247b29f261eb6cfef946f7653ba18453271.scope

Or, if universal support is difficult at the moment, is it practical to add two customizable meters:

  1. Show a subset of CPU cores as configured, or better, read from a file;
  2. Show memory usage and limit from one or two user-configured files periodically?

@fasterit fasterit added Linux 🐧 Linux related issues Containers 📦 Light-weight virtualization suites labels Sep 17, 2024
@divinity76
Copy link
Contributor

divinity76 commented Dec 23, 2024

if you have both lxc and lxcfs,
lxc makes it easy to see this issue for yourself (lxc launch will download ubuntu24.04 image automatically):

apt install lxcfs lxc;
lxc launch ubuntu:24.04 htopcontainer;
lxc exec htopcontainer apt update;
lxc exec htopcontainer apt install htop;
lxc exec htopcontainer htop;

@divinity76
Copy link
Contributor

ps if lxcfs is enabled and the kernel is running cgroups v1 (eg running an old kernel, or running a modern kernel with parameters SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE=1 systemd.unified_cgroup_hierarchy=0 )
then htop only sees the container cpu usage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Containers 📦 Light-weight virtualization suites enhancement Extension or improvement to existing feature Linux 🐧 Linux related issues
Projects
None yet
Development

No branches or pull requests

4 participants