Allow Running Trace Pods Against Containers In a Different Namespace

[seanmalloy]

As a cluster administrator I would like to be able to trace application pods in end user(tenant) namespaces, but I do not want to allow end users(tenants) to run privileged pods in their namespace.

For example as a cluster administrator I want to be able to run the below command, but I do not want any service account in the myapp namespace to be able to run privileged pods. Running the below command requires that the default service account in the myapp namespace is allowed to run privileged containers.

kubectl trace run pod/router-default-7467bc9b7c-9db7s -n myapp -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); }'

Instead I want kubectl-trace to create a trace pod in a namespace that only cluster administrators have access to, but be able to specify a "pod to trace" in an end user(tenant) namespace.

One possible option would be to add a new CLI option --trace-job-namesapce. In this example a trace would be run against the pod router-default-7467bc9b7c-9db7s in the myapp namespace, but the trace pod/job would be running in the tracejobs namespace.

kubectl trace run pod/router-default-7467bc9b7c-9db7s -n myapp --trace-job-namespace tracejobs -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); }'

This feature would be use for k8s clusters running in environments where strict audit requirements must be adhered to. For example running in a cluster with pod security policies enabled, or an OpenShift cluster with uses security context constraints.

[seanmalloy]

I'm curious if people have a different solution to the problem. Also I'm not dead set on the name of the --trace-job-namespace CLI option name.

Any feedback would be greatly appreciated.

[fntlnz]

HI @seanmalloy - thanks for opening this. Did you try to use the --namespace option?

[seanmalloy]

HI @seanmalloy - thanks for opening this. Did you try to use the --namespace option?

Yes, I believe -n is the same as --namespace. Everything works using this option, but in my example I don't want to allow a service account in the myapp namespace to run priveliged pods.

[fntlnz]

Ah sorry I didn't understand the use case, what you actually want to do is a separation of the two!
I think this is a good idea!

[seanmalloy]

@fntlnz thanks for confirming. I'm interested in contributing this feature, but I can't start working on it right away. I'm hoping to start working on it by the end of 2020.

I'll comment here again when I actually start working on it for real.

Thanks!

[dalehamel]

Heya! What you describe above with --trace-job-namespace is pretty much identical to what we are looking for as well @seanmalloy

In our case, we want to be able to have a one-time setup of a single namespace to run all traces within a given cluster. This way if you are using something like workload identity, you only have to create the service account binding once. This goes pretty naturally with the work we are doing on #85 internally, as we would like to have this be a one-time operation when the operator is installed.

I definitely see the use case for privilege separation, but there are other benefits too:

• If resource limits are set per namespace, running traces alongside the application can make resource planning unpredictable. Having a single namespace for trace jobs allows for this to be contained
• For garbage collection, keeping everything in one namespace is much easier than tracking down traces potentially scattered about a cluster
• Calling it --trace-job-namespace lets us very clearly delineate which namespace is for the target pod being traced, and which is for the tracejob

[seanmalloy]

I haven't had time to start working on this. If someone else wants to submit a PR to add this feature then go for it.

[dalehamel]

Looking at the code, it actually does look like the existing flag for --namespace is where to create the trace job:

https://github.com/iovisor/kubectl-trace/blob/master/pkg/tracejob/job.go#L208

So I think the feature request here is actually for how the selector for the trace itself should work (what pod to trace). Regardless, we'll look into clarifying this. @zqureshi has been doing some work on selector syntax for workloads.

EDIT: so for now, it looks like both are the same

kubectl-trace/pkg/cmd/run.go

Line 308 in bb25231

Namespace: o.namespace,