Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tracing instrumentation using tracetools #294

Conversation

christophebedard
Copy link
Member

@christophebedard christophebedard commented Oct 10, 2024

With this instrumentation, rmw_zenoh_cpp is supported out-of-the-box by tools that process ROS 2 trace data, such as Eclipse Trace Compass.

It's pretty straightforward except for a few things:

  1. Getting the rmw_publisher_t * for the rmw_publish tracepoint and rmw_service_t * for the rmw_send_response tracepoint
    • Add them to PublisherData and ServiceData, respectively.
  2. Getting the source timestamp for the rmw_publish and rmw_send_response tracepoints
    • Getting the source timestamp of a message being published is needed for tracking messages from publisher to subscription or replies from service to client.
    • Since Add timestamp to rmw_publish tracepoint ros2_tracing#74, we're getting the source timestamp for a published message from rmw. This means we don't need to instrument the underlying middleware itself, which we used to have to do.
    • Remove create_map_and_set_sequence_num(), since it's already pretty simple and it makes extracting the source timestamp easier.
  3. Getting the/a subscription rmw GID for the rmw_subscription_init tracepoint
    • See the comment before the tracepoint.

I'm open to suggestions.


action-ros-ci-repos-override: https://gist.githubusercontent.com/christophebedard/18561011c03603652ad49701412e5f41/raw/9e9d0bc9bbf0c36b7d5419bc5797ef17aa90d6fa/ros2.repos

@christophebedard
Copy link
Member Author

christophebedard commented Oct 10, 2024

Oh, looks like this repo aims to support Iron, Jazzy, and Rolling on the rolling branch. I will deal with that after a first round of review.

@christophebedard christophebedard force-pushed the christophebedard/add-tracing-instrumentation branch from c2d6771 to a1db0ea Compare November 17, 2024 23:09
@christophebedard
Copy link
Member Author

@Yadunund @clalancette any interest in having this?

@Yadunund
Copy link
Member

@christophebedard we'd love to have this! We're sort of freezing the rolling branch at the moment to focus effortts on getting the #276 merged before we add new features. So let's wait for that to get in before merging this PR . By then Iron would have ben sunsetted so we won't have to worry about failing Iron CI job.

@Yadunund
Copy link
Member

Yadunund commented Dec 9, 2024

@christophebedard do you mind resolving the conflicts after rebasing to the latest rolling branch?

@christophebedard christophebedard force-pushed the christophebedard/add-tracing-instrumentation branch from a1db0ea to 66cdb1f Compare December 9, 2024 14:55
@christophebedard
Copy link
Member Author

I rebased, but now the tracing tests in test_tracetools sometimes time out. It seems to happen during rclcpp::shutdown(), which might be related to #329.

@christophebedard christophebedard force-pushed the christophebedard/add-tracing-instrumentation branch from 66cdb1f to 11a0ea5 Compare December 16, 2024 21:11
@christophebedard
Copy link
Member Author

christophebedard commented Dec 16, 2024

I rebased & resolved the new merge conflicts.

I also added client/service instrumentation, see ros2/ros2_tracing#145.

I rebased, but now the tracing tests in test_tracetools sometimes time out. It seems to happen during rclcpp::shutdown(), which might be related to #329.

I ran the tests quite a few times and didn't get any failures. Not sure what changed in the code, other than the instrumentation.

@christophebedard christophebedard force-pushed the christophebedard/add-tracing-instrumentation branch from 11a0ea5 to 9ddd3cb Compare December 18, 2024 17:09
Copy link
Collaborator

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've left 3 more minor things to fix, then I'm happy with it.

rmw_zenoh_cpp/src/rmw_zenoh.cpp Outdated Show resolved Hide resolved
rmw_zenoh_cpp/src/rmw_zenoh.cpp Outdated Show resolved Hide resolved
rmw_zenoh_cpp/src/rmw_zenoh.cpp Show resolved Hide resolved
@christophebedard christophebedard force-pushed the christophebedard/add-tracing-instrumentation branch from 1587c67 to 1b65002 Compare December 18, 2024 23:14
Copy link
Collaborator

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more fix, then I'm happy with this.

rmw_zenoh_cpp/src/rmw_zenoh.cpp Outdated Show resolved Hide resolved
christophebedard and others added 2 commits December 19, 2024 10:10
With the recent changes to get data out of it, and the change
to zenoh-cpp, it can mostly be replaced with a couple of
inline pieces of code.  However, we do add in a new function
to help us explicitly get system clock time from std::chrono
in nanoseconds.

Signed-off-by: Chris Lalancette <[email protected]>
rmw_zenoh_cpp/src/rmw_zenoh.cpp Outdated Show resolved Hide resolved
Signed-off-by: Christophe Bedard <[email protected]>
@christophebedard christophebedard force-pushed the christophebedard/add-tracing-instrumentation branch from 4fa479f to ebc61f9 Compare December 19, 2024 22:57
Copy link
Collaborator

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. Thanks for all of the iteration, and for being patient with us as we revamped other things. I'm going to go ahead and merge now.

@clalancette clalancette merged commit 79fc5ce into ros2:rolling Dec 20, 2024
6 checks passed
@christophebedard christophebedard deleted the christophebedard/add-tracing-instrumentation branch December 20, 2024 15:15
@cybaol
Copy link

cybaol commented Dec 21, 2024

@christophebedard TRACETOOLS_TRACEPOINT seems break jazzy build (replicated on archlinux).

@christophebedard
Copy link
Member Author

Make sure to build the tracetools package (from the ros2/ros2_tracing repo) from source. If you rely on the binaries, rmw_zenoh will indeed not build.

@christophebedard
Copy link
Member Author

Oh, I guess this does break Jazzy builds. This will only work with Rolling (built from source for now).

The new service/client tracepoints would need to be excluded on Jazzy.

@christophebedard
Copy link
Member Author

So CI passed for this PR because the build_and_test_binary_jazzy job wasn't actually testing Jazzy: https://github.com/ros2/rmw_zenoh/actions/runs/12422154058/job/34683299857. It's my fault, I set action-ros-ci-repos-override: to use the ros2_tracing branch that enables tracing tests for rmw_zenoh_cpp (ros2/ros2_tracing#140). The same job is indeed now failing on rolling: https://github.com/ros2/rmw_zenoh/actions/runs/12432407339/job/34711738348.

Since this repo doesn't use distro-specific branches yet, could we add distro-specific #defines`? Since I broke the build, I've opened a PR with a quick-and-dirty solution: #355.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants