Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chromium tarball is much too big: split source? #763

Open
rwmacleod opened this issue Oct 15, 2023 · 6 comments
Open

chromium tarball is much too big: split source? #763

rwmacleod opened this issue Oct 15, 2023 · 6 comments

Comments

@rwmacleod
Copy link
Contributor

rwmacleod commented Oct 15, 2023 •
edited
Loading

@nrpt-m mentioned that with upstream adding Rust in chromium 117, the tarball is ~3 GB.
Even before this, chromium-116.0.5845.187.tar.xz is 1.5 GB and 13 GB unpacked.
It seemed hard to believe that all of that data would be required to build chromium.
Using ncdu, the top level directories that are > 100 MiB are:

    9.7 GiB [##########] /third_party                                                                                                                                                         
  822.8 MiB [          ] /chrome
  420.8 MiB [          ] /components
  288.4 MiB [          ] /buildtools 
  199.6 MiB [          ] /tools
  126.0 MiB [          ] /ios
  110.9 MiB [          ] /net
  104.3 MiB [          ] /ui
  101.0 MiB [          ] /ash

and the largest third-party componets are:

    1.6 GiB [##########] /llvm  
    1.5 GiB [######### ] /angle
    1.0 GiB [######    ] /swiftshader
  649.2 MiB [###       ] /dawn                                                                                                                                                                
  454.5 MiB [##        ] /devtools-frontend
  373.9 MiB [##        ] /catapult

Is it possible to use llvm and rust from YP/OE and does anyone think that it would be a good idea to look at other large directories and remove them or split them into a seperate tarball if not needed. Does anyone object to such a change assuming that it's not too painful to implement and maintain?
@rakuco
Copy link
Collaborator

rakuco commented Oct 17, 2023

The tarball publishing process upstream already creates a tarball called "chromium-lite" which removes some directories, but this list is not very up-to-date and actively maintained.

As a first step one could see if this tarball is considerably smaller than the one we use and whether one can actually build Chromium from it.

As for using LLVM and Rust from YP/OE and the rest of the suggestion: do you mean creating a tarball of our own?

@MaxIhlenfeldt
Copy link
Collaborator

https://groups.google.com/a/chromium.org/g/chromium-packagers/c/oE60kVFFMyQ/m/xxF0DCnkAQAJ seems interesting:

Since distributions link against their own system libraries, and have
their own toolchains, we can exclude:

  • Debian sysroot(s)
  • Toolchain components (Rust, Clang, LLVM etc.)
  • third_party/instrumented_libs

I assume we don't want to actually use these third-party tarballs in our recipe, but maybe the -lite tarballs will get a little more attention again:

We are hoping that these changes will be accepted into the '-lite'
tarballs upstream, as these are no longer significantly smaller
than the standard Chromium tarball.

@MaxIhlenfeldt
Copy link
Collaborator

https://chromium-review.googlesource.com/c/chromium/tools/build/+/6072713 has landed, and this already affect the latest releases. e.g. https://storage.cloud.google.com/chromium-browser-official/chromium-131.0.6778.139-lite.tar.xz is only 1.9 GB. I'll give building with that a try. @rakuco if that works, should we switch to the lite tarballs?

@MaxIhlenfeldt
Copy link
Collaborator

Hm, it seems there are some licenses missing in the lite tarball:

WARNING: chromium-x11-131.0.6778.139-r0 do_populate_lic: Could not copy license file /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/blink/LICENSE to /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/license-destdir/cortexa15t2hf-neon/chromium-x11/LICENSE.11: [Errno 2] No such file or directory: '/home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/blink/LICENSE'
WARNING: chromium-x11-131.0.6778.139-r0 do_populate_lic: Could not copy license file /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/lottie/LICENSE to /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/license-destdir/cortexa15t2hf-neon/chromium-x11/LICENSE.12: [Errno 2] No such file or directory: '/home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/lottie/LICENSE'
WARNING: chromium-x11-131.0.6778.139-r0 do_populate_lic: Could not copy license file /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/icu/LICENSE to /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/license-destdir/cortexa15t2hf-neon/chromium-x11/LICENSE.119: [Errno 2] No such file or directory: '/home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/icu/LICENSE'
WARNING: chromium-x11-131.0.6778.139-r0 do_populate_lic: Could not copy license file /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/snappy/src/COPYING to /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/license-destdir/cortexa15t2hf-neon/chromium-x11/COPYING.12: [Errno 2] No such file or directory: '/home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/snappy/src/COPYING'
WARNING: chromium-x11-131.0.6778.139-r0 do_populate_lic: Could not copy license file /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/native_client_sdk/src/libraries/third_party/newlib-extras/README to /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/license-destdir/cortexa15t2hf-neon/chromium-x11/README: [Errno 2] No such file or directory: '/home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/native_client_sdk/src/libraries/third_party/newlib-extras/README'
WARNING: chromium-x11-131.0.6778.139-r0 do_populate_lic: Could not copy license file /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/libjpeg_turbo/LICENSE.md to /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/license-destdir/cortexa15t2hf-neon/chromium-x11/LICENSE.md.2: [Errno 2] No such file or directory: '/home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/libjpeg_turbo/LICENSE.md'
WARNING: chromium-x11-131.0.6778.139-r0 do_populate_lic: Could not copy license file /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/freetype/src/docs/FTL.TXT to /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/license-destdir/cortexa15t2hf-neon/chromium-x11/FTL.TXT.0: [Errno 2] No such file or directory: '/home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/freetype/src/docs/FTL.TXT'
WARNING: chromium-x11-131.0.6778.139-r0 do_populate_lic: Could not copy license file /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/libxml/src/Copyright to /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/license-destdir/cortexa15t2hf-neon/chromium-x11/Copyright.0: [Errno 2] No such file or directory: '/home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/libxml/src/Copyright'
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/blink/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/lottie/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/material_components_ios/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/material_font_disk_loader_ios/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/material_internationalization_ios/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/material_roboto_font_loader_ios/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/material_sprited_animation_view_ios/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/material_text_accessibility_ios/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/motion_animator_objc/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/motion_interchange_objc/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/ios/third_party/motion_transitioning_objc/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/native_client_sdk/src/libraries/third_party/newlib-extras/README [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/freetype/src/docs/FTL.TXT [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/icu/LICENSE [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/libjpeg_turbo/LICENSE.md [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/libxml/src/Copyright [license-checksum]
ERROR: chromium-x11-131.0.6778.139-r0 do_populate_lic: QA Issue: chromium-x11: LIC_FILES_CHKSUM points to an invalid file: /home/gitlab-runner/yocto-chromium/builds/ci-chromium-x11-qemuarm-master/tmp/work/cortexa15t2hf-neon-oe-linux-gnueabi/chromium-x11/131.0.6778.139/chromium-131.0.6778.139/third_party/snappy/src/COPYING [license-checksum]

The ios ones and the native_client_sdk one we can probably just remove from LIC_FILES_CHECKSUM (which reminds me that I wanted to do the same with the android stuff, oops). But e.g. libxml is definitely used on Linux, so its license should be in the tarball.

It seems this part of publish_tarball.py removes e.g. third_party/libxml/src, which includes the license file. I'll file an upstream issue.

@MaxIhlenfeldt
Copy link
Collaborator

Issue is at https://issues.chromium.org/issues/384781867, and CL to fix it at https://chromium-review.googlesource.com/c/chromium/tools/build/+/6105714.

@MaxIhlenfeldt
Copy link
Collaborator

CL landed and seems to have fixed the issue, but there's no new 131 tarball yet. We should try again once there is, or maybe as part of the next update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rakuco @rwmacleod @MaxIhlenfeldt