Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Builder refactor with Python and the ability to layer builds #23

Merged
merged 5 commits into from
Mar 26, 2022

Conversation

shuhaowu
Copy link
Member

@shuhaowu shuhaowu commented Mar 14, 2022

This is a port of the bash script to Python, with the added ability to overlay multiple directories on top of each other.

Since this is a relatively large PR that reworks the repo. I've updated the docs significantly with the design as well as a detailed usage guide (to customize the image builder). Please read it first as it will hopefully make reviewing the rest of the code a lot easier.

Stuff I don't want to do now but will do in future PRs as they're not super urgent:

  • Create a proper command line builder instead of a hard-coded one
  • Move qemu-user-static and pause_after configurations into the command line tool.
  • Create command line shortcuts to cleanup the host system if something goes wrong
  • Create command line shortcuts to chroot into the target image.

@shuhaowu shuhaowu marked this pull request as draft March 14, 2022 02:52
@LanderU
Copy link
Collaborator

LanderU commented Mar 14, 2022

@shuhaowu, thank you! Let us know if you need any help!

@shuhaowu
Copy link
Member Author

Thanks! I should be good. Hopefully at some point this week I'll have a bit of time to finalize this.

Makefile Show resolved Hide resolved
@shuhaowu shuhaowu changed the title [EXTREMELY WIP] Python builder refactor Builder refactor with Python and the ability to layer builds Mar 25, 2022
@shuhaowu shuhaowu force-pushed the python-builder branch 3 times, most recently from 49a4f6d to 5228d13 Compare March 25, 2022 16:32
Copy link
Member Author

@shuhaowu shuhaowu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LanderU @carlossvg This PR is ready to review. I also updated the description of this PR to link to the docs, as well as document what's deferred to a future PR.

I'm testing the image right now (dding to an SD card takes a bit of time... 🕐). It should work, and I'll report back either way.

"focal-rt-galactic",
])

b.build()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to convert this script into a proper command line app. However, it's not that urgent and it will take a bit more time. I would like to get this merged first so we can iterate on other things that needs to be done.

Comment on lines +227 to +230
shutil.copy(
"/etc/resolv.conf",
os.path.join(self.chroot_path, "etc", "resolv.conf")
)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not 100% certain if this works on Ubuntu. (I'm testing my builder primarily on Ubuntu for now). I think it should because the container is setup via systemd-nspawn but it is not in an network namespace.

Comment on lines +28 to +32
# TODO: same as qemu_user_static_path. This should be specified via
# command-line arguments as opposed to build configuration.
# Uncomment this if you want to pause the builder after a particular stage to
# debug/experiment.
# pause_after = cleanup_chroot
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we have a command line tool, this and qemu_user_static_path will move out of here.

@@ -0,0 +1,3 @@
#!/bin/bash

# At this point in time, the rootfs at CHROOT_PATH should be fully setup and usable as a cross compile toolchain sysroot.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a place holder file for now

@shuhaowu shuhaowu requested review from carlossvg and LanderU March 25, 2022 16:38
@shuhaowu shuhaowu marked this pull request as ready for review March 25, 2022 16:38
@shuhaowu
Copy link
Member Author

The image works after testing.

@carlossvg
Copy link
Collaborator

carlossvg commented Mar 26, 2022

@shuhaowu This is great work, thanks a lot! I just have a couple of suggestions for the follow-up PRs:

  • create a directory for the ROS 2 scripts: ros2_build_profiles/galactic, ros2_build_profiles/rolling
  • Same for rt-focal, i,e: ubuntu_build_profiles/rt-focal, ubuntu_build_profiles/focal (for no RT version)
  • Rename ros2-rt-rpi4 to something generic so we can put here different script types. i.e. ros2-rt-utils, rt-configuration-scripts, rt-configuration-tools, etc. Here we can add scripts to isolate CPUs, CPU affinities, etc
  • Rename phase1-* scripts to something more descriptive. i.e host-build and host-post-build.

@carlossvg
Copy link
Collaborator

@shuhaowu BTW, I tested the image and all is working for me. LGTM

@shuhaowu
Copy link
Member Author

Thanks for the review. I created a few follow up issues (#25 and #26) for us to discuss and work on. I'm going to merge this for now. If we have something more to say about it we can fix in a follow up PR

@shuhaowu shuhaowu merged commit 64d7a13 into ros-realtime:master Mar 26, 2022
@shuhaowu shuhaowu deleted the python-builder branch March 26, 2022 14:25
output_filename = ubuntu-20.04.4-rt-ros2-arm64+raspi.img

[env]
ROS_DISTRO = galactic
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can cause an error if you already source a ROS 2 ws, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only used for the builder script. It's not for the image. That said, we do have this script which probably should be updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants