Rpr job seems running multiple tests in parallel

[MichaelOrlov]

Description

Rpr and Jpr jobs seems running multiple tests in parallel

Expected Behavior

Rpr and Jpr jobs run tests sequentially

Actual Behavior

Reference to the failing Rpr job https://build.ros2.org/job/Rpr__rosbag2__ubuntu_noble_amd64/477/testReport/junit/(root)/projectroot/test_record_all__rmw_fastrtps_cpp/
From the log I see that in the published_messages_from_multiple_topics_are_recorded test, we've got more messages that we expected.

rosbag2/rosbag2_transport/test/rosbag2_transport/test_record_all.cpp

Lines 38 to 55 in 8f1527e

	TEST_F(RecordIntegrationTestFixture, published_messages_from_multiple_topics_are_recorded)
	{
	auto array_message = get_messages_arrays()[0];
	array_message->float32_values = {{40.0f, 2.0f, 0.0f}};
	array_message->bool_values = {{true, false, true}};
	std::string array_topic = "/array_topic";
	auto string_message = get_messages_strings()[0];
	string_message->string_value = "Hello World";
	std::string string_topic = "/string_topic";
	rosbag2_test_common::PublicationManager pub_manager;
	pub_manager.setup_publisher(array_topic, array_message, 2);
	pub_manager.setup_publisher(string_topic, string_message, 2);
	rosbag2_transport::RecordOptions record_options =
	{true, false, false, {}, {}, {}, {"/rosout"}, {}, {}, "rmw_format", 100ms};
	auto recorder = std::make_shared<rosbag2_transport::Recorder>(

And I see that the Rosbag2 recorder subscribed to the
[INFO] [1733734416.350351887] [rosbag2_recorder]: Subscribed to topic '/composable_recorder_test_string_topic'
However, we publish on this topic only from the other test recorder_can_automatically_start_recording_after_composition

rosbag2/rosbag2_transport/test/rosbag2_transport/test_composable_recorder.cpp

Lines 268 to 276 in 8f1527e

	TEST_P(
	ComposableRecorderIntegrationTests,
	recorder_can_automatically_start_recording_after_composition) {
	const size_t num_messages_to_publish = 5;
	auto string_message = get_messages_strings()[0];
	string_message->string_value = "Hello World";
	const std::string test_topic_name = "/composable_recorder_test_string_topic";
	rosbag2_test_common::PublicationManager pub_manager;

To Reproduce

N/A
** Steps to reproduce the behavior, e.g.

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. Logs print error '...' **

System (please complete the following information)

• OS: Ubuntu 22.04, 24.04
• ROS 2 Distro: Rolling, Jazzy
• Install Method: CI, Rpr job
• Version: Rolling

Additional context

** Add any other context about the problem here **

[MichaelOrlov]

@claraberendsen @clalancette @cottsay This is an example of the Rpr job failure that I mentioned on today's ROS PMC meeting with suspicion that the Rpr job runs multiple tests at a time.

[clalancette]

@claraberendsen @clalancette @cottsay This is an example of the Rpr job failure that I mentioned on today's ROS PMC meeting with suspicion that the Rpr job runs multiple tests at a time.

Based on the logs, I don't think that it is running tests in parallel. First of all, within a package we never run tests in parallel; they are always run sequentially by ctest. However, there is some possibility that colcon is running multiple packages in parallel. But if you look in the logs, you see this:

Invoking '. /opt/ros/rolling/setup.sh && PYTHONIOENCODING=utf_8 PYTHONUNBUFFERED=1 colcon test --build-base build_isolated --install-base install_isolated --test-result-base test_results --event-handlers console_direct+ --executor sequential --test-result-base /tmp/ws/test_results' in '/tmp/ws'

Which is very clearly using the sequential executor. That means it will not run packages in parallel, but one-at-a-time.

My guess is that something else is going on here; maybe a previous test in the same package isn't properly cleaning up after itself.