Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing activity from spy_users chisel #1631

Open
jbarszcz opened this issue May 13, 2020 · 3 comments
Open

Missing activity from spy_users chisel #1631

jbarszcz opened this issue May 13, 2020 · 3 comments

Comments

@jbarszcz
Copy link

jbarszcz commented May 13, 2020

I noticed a strange behaviour of the spy_users chisel. After

  1. entering a container shell
  2. changing directory via cd command
  3. exiting container shell
  4. entering shell again

it stops showing commands executed inside the container. Sometimes two iterations of this process are required to reproduce this behaviour, but I noticed that it happenes every time.

Terminal 1:
sudo sysdig -w output.scap

Terminal 2:

ubuntu@host:~/$ docker exec -it nginx bash
root@9ba4f934fa5e:/# cd /home
root@9ba4f934fa5e:/home# exit
ubuntu@host:~/$ docker exec -it nginx bash
root@9ba4f934fa5e:/# cd /home
root@9ba4f934fa5e:/home# cat /etc/passwd
root@9ba4f934fa5e:/home# ls -ltr

Terminal 1

^C
ubuntu@host:~/$ sudo sysdig -pc -c spy_users -r output.scap

output:

18001 09:22:27 ubuntu@host) docker exec -it nginx bash
18680 09:22:29 root@nginx) cd /home
18001 09:22:32 ubuntu@host) docker exec -it nginx bash

The cat /etc/passwd and ls -ltr commands did not appear as the result of spy_users chisel. However, events of executing this commands are visibile in the output.scap.

The cd directory does not matter, it is /home in the example but any other directory should cause the same problem. In my proof of concept the spy_users chisel stopped working after one "iteration" of docker exec/cd/exit commands, but sometimes it stopps after two iterations. Moreover, the container image should not matter as well, since I have encountered this issue using multiple different container images.

System: Ubuntu 18.4. Tested on two separate environments (AWS and local VM).

ubuntu@host:~$ sysdig --version
sysdig version 0.26.7
@jbarszcz jbarszcz reopened this May 13, 2020
@UweKrause
Copy link

UweKrause commented May 26, 2020

I have the same problem, with a modified version of the chisel but even with the original.
My observation was that logging "just stops working" after about 2 hours.

@jbarszcz: Your reproduction steps show that it is not depending on time (what I thought first), but on what is happening. Your report alone saved me some gray hairs and made it possible to try to hunt this thing down.

When running sysdig -D -c spy_users, one can see, that sysdig does not crash or something.
Sysdig just somehow decides to not print() the events after the steps you described.
In the debug-log one can see that the events are still recognized.

Same sysdig version (0.28.7), full updated ubuntu 18.04.

root@host:~# cat /proc/version
Linux version 4.15.0-99-generic (buildd@lcy01-amd64-013) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #100-Ubuntu SMP Wed Apr 22 20:32:56 UTC 2020

@UweKrause
Copy link

when shortening the filter to (evt.type=execve and evt.dir=<) the log keeps logging.
(Of course no cd event)

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants