This layer provides LAT (Linux Assembly Tool).
Use this tool to perform a number of tasks to help manage your images,
such as building and publishing RPM packages, generating images from
package feeds for specific hardware, generating an updated SDK, and
verifying the sanity of the SDK build. In addition, you can add or
remove packages, and specify any pre- and post-build (do_build)
instructions for the build, which lets you customize your image to
meet your needs.
This tool uses *.yaml input files to define many aspects of the build,
packages and package-related configuration for a variety of uses.
It provides validation for the *.yaml input files that define your intended
project output to help identify any incorrect or misspelled options.
With the exampleyamls positional argument, you can generate a list
of different sample *.yaml files, which you can use as a basis for
customizing your builds. These reusable files provide a data reference
for all your image customizations.
LAT is a tool that assembles various types of images using binary package feeds. Currently it supports three types of package feeds.
- rpm: Yocto RPM feeds
- deb: Yocto DEB feeds
- external-debian: Debian package feeds
$ mkdir <project>
$ cd <project>
$ git clone https://github.com/Wind-River/meta-lat.git
$ git clone git://git.yoctoproject.org/meta-virtualization
$ git clone git://git.openembedded.org/meta-openembedded
$ git clone git://git.pokylinux.org/poky.git
$ . <project>/poky/oe-init-build-env <build>
cat << ENDOF >> conf/local.conf
INIT_MANAGER = "systemd"
require conf/distro/ostree.conf
require conf/distro/lat.conf
MACHINE = "intel-x86-64"
PACKAGE_CLASSES = "package_rpm"
PACKAGE_FEED_URIS = "http://<web-server-url>/lat"
ENDOF
Edit conf/bblayers.conf to include other layers
BBLAYERS ?= " \
<project>/oe-core/meta \
<project>/meta-openembedded/meta-filesystems \
<project>/meta-openembedded/meta-python \
<project>/meta-openembedded/meta-oe \
<project>/meta-openembedded/meta-xfce \
<project>/meta-openembedded/meta-gnome \
<project>/meta-openembedded/meta-multimedia \
<project>/meta-openembedded/meta-networking \
<project>/meta-virtualization \
<project>/meta-lat \
"
$ bitbake world && bitbake package-index
$ ls tmp*/deploy/rpm/*/repodata/repomd.xml -1
tmp/deploy/rpm/corei7_64/repodata/repomd.xml
tmp/deploy/rpm/intel_x86_64/repodata/repomd.xml
tmp/deploy/rpm/noarch/repodata/repomd.xml
Above ls only list required repodata
$ bitbake lat-sdk
Setup a web server (such as httpd, apache2), and create a symlink to deploy dir
$ ln -snf path-to-build/tmp/deploy /var/www/html/lat
Then http:///lat is accessible
wget http://<web-server-url>/lat/sdk/lat-sdk-poky-intel-x86-64-1.0.sh
x86-64
Check sdk.host.manifest and sdk.target.manifest
$ ./lat-sdk-poky-intel-x86-64-1.0.sh -y -d <dir>
$ cd <workdir>
$ . <dir>/environment-setup-*-linux
$ appsdk -h
usage: appsdk [-h] [-d] [-q] [--log-dir LOGDIR] {gensdk,checksdk,genrpm,publishrpm,genimage,geninitramfs,gencontainer,genyaml,exampleyamls} ...
Wind River Linux Assembly Tool
positional arguments:
{gensdk,checksdk,genrpm,publishrpm,genimage,geninitramfs,gencontainer,genyaml,exampleyamls}
Subcommands. "appsdk <subcommand> --help" to get more info
gensdk Generate a new SDK
checksdk Sanity check for SDK
genrpm Build RPM package
publishrpm Publish RPM package
genimage Generate images from package feeds for specified machines
geninitramfs Generate Initramfs from package feeds for specified machines
gencontainer Generate Container Image from package feeds for specified machines
genyaml Generate Yaml file from Input Yamls
exampleyamls Deploy Example Yaml files
optional arguments:
-h, --help show this help message and exit
-d, --debug Enable debug output
-q, --quiet Hide all output except error messages
--log-dir LOGDIR Specify dir to save debug messages as log.appsdk regardless of the logging level
Use appsdk <subcommand> --help to get help
$ appsdk gensdk -h
usage: appsdk gensdk [-h] [-f FILE] [-o OUTPUT]
optional arguments:
-h, --help show this help message and exit
-f FILE, --file FILE An input yaml file specifying image information. Default to image.yaml in current directory
-o OUTPUT, --output OUTPUT
The path of the generated SDK. Default to deploy/AppSDK.sh in current directory
$ appsdk gensdk -f input.yaml
$ appsdk checksdk
appsdk genrpm -h
usage: appsdk genrpm [-h] -f FILE -i INSTALLDIR [-o OUTPUTDIR] [--pkgarch PKGARCH]
optional arguments:
-h, --help show this help message and exit
-f FILE, --file FILE A yaml or spec file specifying package information
-i INSTALLDIR, --installdir INSTALLDIR
An installdir serving as input to generate RPM package
-o OUTPUTDIR, --outputdir OUTPUTDIR
Output directory to hold the generated RPM package
--pkgarch PKGARCH package arch about the generated RPM package
appsdk publishrpm -h
usage: appsdk publishrpm [-h] -r REPO [rpms [rpms ...]]
positional arguments:
rpms RPM package paths
optional arguments:
-h, --help show this help message and exit
-r REPO, --repo REPO RPM repo path
$ appsdk exampleyamls -h
usage: appsdk exampleyamls [-h] [-o OUTDIR]
optional arguments:
-h, --help show this help message and exit
-o OUTDIR, --outdir OUTDIR
Specify output dir, default is current working directory
$ appsdk exampleyamls
appsdk - INFO: Deploy Directory: path-to-outdir/exampleyamls
+-------------------+------------------------------------------+
| Yaml Type | Name |
+===================+==========================================+
| Image | lat-image-base-intel-x86-64.yaml |
| | core-image-minimal-intel-x86-64.yaml |
| | core-image-sato-intel-x86-64.yaml |
| | initramfs-ostree-image-intel-x86-64.yaml |
| | |
+-------------------+------------------------------------------+
| Feature | feature/debug-tweaks.yaml |
| | feature/package_management.yaml |
| | feature/set_root_password.yaml |
| | feature/startup-container.yaml |
| | feature/vboxguestdrivers.yaml |
| | feature/xfce_desktop.yaml |
| | |
+-------------------+------------------------------------------+
| System Definition | sysdef/add-system-user.yaml |
| Yamls | sysdef/add-user-home.yaml |
| | sysdef/contains-container-base.yaml |
| | sysdef/set-dns.yaml |
| | sysdef/set-hostname.yaml |
| | sysdef/set-ntp.yaml |
| | sysdef/update-containers.yaml |
| | |
+-------------------+------------------------------------------+
appsdk - INFO: Then, run genimage or genyaml with Yaml Files:
appsdk genimage <Image>.yaml <Feature>.yaml
Or
appsdk genyaml <Image>.yaml <Feature>.yaml
Image Yamls:
-
These image Yamls does not list all installed packages, it refers PACKAGE_INSTALL from Yocto (bitbake -e
)
-
For core-image-minimal, core-image-sato, lat-image-base, the image type is ostree_repo and ustart, they are used by 'appsdk genimage' only
-
For conainer-base, the image type is container, it is used by 'appsdk gencontainer' only
-
For initramfs-ostree-image, the image type initramfs, it is used by 'appsdk geninitramfs' only
Feature Yamls:
- Attention: these feature Yamls could not random combination with image Yamls, such as vboxguestdrivers.yaml and xfce_desktop.yaml does not work on initramfs and container image
$ appsdk genimage -h
usage: appsdk genimage [-h] [-t {wic,vmdk,vdi,ostree-repo,ustart,all}] [-o OUTDIR] [-w WORKDIR] [-n NAME] [-u URL] [-p PKG] [--pkg-external PKG_EXTERNAL]
[--rootfs-post-script ROOTFS_POST_SCRIPT] [--rootfs-pre-script ROOTFS_PRE_SCRIPT] [--env ENV] [--no-clean] [--no-validate] [-g GPGPATH]
[input [input ...]]
positional arguments:
input Input yaml files that the tool can be run against a package feed to generate an image
optional arguments:
-h, --help show this help message and exit
-t {wic,vmdk,vdi,ostree-repo,ustart,all}, --type {wic,vmdk,vdi,ostree-repo,ustart,all}
Specify image type, it overrides 'image_type' in Yaml
-o OUTDIR, --outdir OUTDIR
Specify output dir, default is current working directory
-w WORKDIR, --workdir WORKDIR
Specify work dir, default is current working directory
-n NAME, --name NAME Specify image name, it overrides 'name' in Yaml
-u URL, --url URL Specify extra urls of rpm package feeds
-p PKG, --pkg PKG Specify extra package to be installed
--pkg-external PKG_EXTERNAL
Specify extra external package to be installed
--rootfs-post-script ROOTFS_POST_SCRIPT
Specify extra script to run after do_rootfs
--rootfs-pre-script ROOTFS_PRE_SCRIPT
Specify extra script to run before do_rootfs
--env ENV Specify extra environment to export before do_rootfs: --env NAME=VALUE
--no-clean Do not cleanup previously generated rootfs in workdir
--no-validate Do not validate parameters in Input yaml files
-g GPGPATH, --gpgpath GPGPATH
Specify gpg homedir, it overrides 'gpg_path' in Yaml, default is /tmp/.lat_gnupg
--ostree-remote-url OSTREE_REMOTE_URL
Specify ostree remote url, it overrides 'ostree_remote_url' in Yaml, default is None
$ appsdk genimage input.yaml --type all
appsdk - INFO: Deploy Directory: path-to-outdir/deploy
+------------------+-----------------------------------------------------------+
| Type | Name |
+==================+===========================================================+
| Image Yaml File | core-image-small-intel-x86-64.yaml |
+------------------+-----------------------------------------------------------+
| Ostree Repo | ostree_repo |
+------------------+-----------------------------------------------------------+
| WIC Image | core-image-small-intel-x86-64.wic -> core-image- |
| | small-intel-x86-64-20200908060827.rootfs.wic |
+------------------+-----------------------------------------------------------+
| WIC Image Doc | core-image-small-intel-x86-64.wic.README.md |
+------------------+-----------------------------------------------------------+
| WIC Image | core-image-small-intel-x86-64.qemuboot.conf -> |
| Qemu Conf | core-image-small- |
| | intel-x86-64-20200908060827.qemuboot.conf |
+------------------+-----------------------------------------------------------+
| VDI Image | core-image-small-intel-x86-64.wic.vdi -> core- |
| | image-small-intel-x86-64-20200908060827.rootfs.wic.vdi |
+------------------+-----------------------------------------------------------+
| VMDK Image | core-image-small-intel-x86-64.wic.vmdk -> core- |
| | image-small-intel-x86-64-20200908060827.rootfs.wic.vmdk |
+------------------+-----------------------------------------------------------+
| Ustart Image | core-image-small-intel-x86-64.ustart.img.gz -> |
| | core-image-small- |
| | intel-x86-64-20200908060827.ustart.img.gz |
+------------------+-----------------------------------------------------------+
| Ustart Image Doc | core-image-small-intel-x86-64.ustart.img.gz.README.md |
+------------------+-----------------------------------------------------------+
If no option --type, it generates ostree repo and ustart image by default
$ appsdk geninitramfs -h
It is similar with 'appsdk genimage -h', the differ is no -t/--type option
If image name is 'initramfs-ostree-image' (by default), it will replace existed
initrd used by the generation of wic/ustart image (appsdk genimage)
$ appsdk geninitramfs
appsdk - INFO: Deploy Directory: path-to-outdir/deploy
+-------+----------------------------------------------------------------------+
| Image | initramfs-ostree-image-intel-x86-64.cpio.gz -> initramfs-ostree- |
| | image-intel-x86-64-20200908062501.rootfs.cpio.gz |
+-------+----------------------------------------------------------------------+
$ appsdk gencontainer -h
It is similar with 'appsdk genimage -h', the differ is no -t/--type and
-g/--gpg-path options
$ appsdk gencontainer
appsdk - INFO: Deploy Directory: path-to-outdir/deploy
+------------------------+-----------------------------------------------------+
| Docker Image | container-base-intel-x86-64.docker-image.tar.bz2 -> |
| | container-base-intel-x86-64-20201012135125.docker- |
| | image.tar.bz2 |
+------------------------+-----------------------------------------------------+
| OCI Image Rootfs | container-base-intel-x86-64.rootfs-oci |
+------------------------+-----------------------------------------------------+
| Container Image Doc | container-base-intel-x86-64.container.README.md |
+------------------------+-----------------------------------------------------+
| Yaml file for genimage | container-base-intel-x86-64.startup-container.yaml |
| to load and run | |
+------------------------+-----------------------------------------------------+
$ appsdk genyaml -h
It is similar with 'appsdk genimage -h', the differ is '--type {wic,vmdk,vdi, ostree-repo,container,initramfs,ustart,all}'
$ appsdk genyaml exampleyamls/core-image-small-intel-x86-64.yaml exampleyamls/feature/xfce_desktop.yaml
appsdk - INFO: Input YAML File: exampleyamls/core-image-small-intel-x86-64.yaml
appsdk - INFO: Input YAML File: exampleyamls/feature/xfce_desktop.yaml
appsdk - INFO: Save Yaml FIle to : path-to-outdir/core-image-small-intel-x86-64.yaml
Customize order: Default setting < Section in Yaml < Command option
-
Section in Yaml overrides default setting If the section in Yaml is missing, use default setting to replace
-
Install default packages or not Use include-default-packages section in Yaml, if set 1, install default packages to image; if set 0, the packages section in Yaml overrides default packages. The default include-default-packages is 1.
-
Command option overrides section in Yaml The options --gpgpath, --type, --name, --ostree-remote-url, override the sections in input.yaml
-
Command option append section in Yaml The options --url, --pkg, --pkg-external, --rootfs-post-script, --rootfs-pre-script, --env, append value to the sections in input.yaml. If values are duplicated, only one copy is used.
-
Set environment variable for rootfs generation For option --env and section 'environments', if the same environment variable is set multiple times (such as NAME=VALUE1, NAME=VALUE2), only the last set (NAME=VALUE2) works
-
Customize rootfs by script For option --rootfs-pre-script/--rootfs-post-script and section rootfs-pre-scripts/rootfs-post-scripts which allows you to run a script using the pseudo context to customize rootfs. The 'pre' means run script before rootfs generation, the 'post' means run script after rootfs generation. Define an environment variable IMAGE_ROOTFS for the location of the rootfs install directory
-
Collect the following sections from multiple Yamls, the duplication is allowed:
- packages
- external-packages
- environments
- rootfs-pre-scripts
- rootfs-post-scripts
But duplication is not allowed for other sections such as section name, machine and so on.
Input yaml format:
name: core-image-small
machine: intel-x86-64
image_type:
- ostree-repo
- ustart
- vdi
- vmdk
- wic
package_feeds:
- http://XXXX/repos/rpm/corei7_64
- http://XXXX/repos/rpm/intel_x86_64
- http://XXXX/repos/rpm/noarch
ostree:
OSTREE_CONSOLE: console=ttyS0,115200 console=tty1
OSTREE_FDISK_BLM: '2506'
OSTREE_FDISK_BSZ: '200'
OSTREE_FDISK_FSZ: '32'
OSTREE_FDISK_RSZ: '4096'
OSTREE_FDISK_VSZ: '0'
OSTREE_GRUB_PW_FILE: $OECORE_NATIVE_SYSROOT/usr/share/bootfs/boot_keys/ostree_grub_pw
OSTREE_GRUB_USER: root
ostree_osname: latos
ostree_remote_url: ''
ostree_skip_boot_diff: '2'
ostree_use_ab: '0'
wic:
OSTREE_FLUX_PART: fluxdata
OSTREE_WKS_BOOT_SIZE: ''
OSTREE_WKS_EFI_SIZE: --size=32M
OSTREE_WKS_FLUX_SIZE: ''
OSTREE_WKS_ROOT_SIZE: ''
remote_pkgdatadir: http://XXXX/repos/rpm
features:
image_linguas: 'en-us' # Specifies the list of locales to
# install into the image during the
# root filesystem construction process.
pkg_globs: '*-src, *-dev, *-dbg' # Install complementary packages
# based upon the list of currently
# installed packages
# e.g. *-src, *-dev, *-dbg
gpg:
gpg_path: /tmp/.lat_gnupg # Specify gpg homedir, dir length is no
# more than 80 chars, make sure the
# permission on dir
grub:
BOOT_GPG_NAME: SecureBootCore
BOOT_GPG_PASSPHRASE: SecureCore
BOOT_KEYS_DIR: $OECORE_NATIVE_SYSROOT/usr/share/bootfs/boot_keys
ostree:
gpg_password: windriver
gpgid: Wind-River-Linux-Sample
gpgkey: $OECORE_NATIVE_SYSROOT/usr/share/genimage/rpm_keys/RPM-GPG-PRIVKEY-Wind-River-Linux-Sample
packages:
- ca-certificates
- glib-networking
- grub-efi
- i2c-tools
- intel-microcode
- iucode-tool
- kernel-modules
- lmsensors
- os-release
- ostree
- ostree-upgrade-mgr
- packagegroup-core-base-utils
- packagegroup-core-boot
- rtl8723bs-bt
- run-postinsts
- systemd
external-packages: []
include-default-packages: '0'
rootfs-pre-scripts:
- echo "run script before do_rootfs in $IMAGE_ROOTFS"
rootfs-post-scripts:
- echo "run script after do_rootfs in $IMAGE_ROOTFS"
environments:
- KERNEL_PARAMS="key=value"
- NO_RECOMMENDATIONS="0"
Here's a simple example of how to use appsdk.
1. Download source and build.
e.g.
wget http://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz
tar xzvf hello-2.10.tar.gz
cd hello-2.10
./configure $CONFIGURE_FLAGS
make
make DESTDIR=/path/to/install-hello install
2. Generate RPM package
2a) Create yaml file for hello as below.
"""
name: hello
version: '2.10'
release: r0
summary: Hello World Program From Gnu
license: GPLv3
description: |
A simple hello world program that only does one thing.
It's from GNU.
post_install: |
#!/bin/sh
echo "This is the post install script of hello program"
echo "It only prints some message."
"""
Note that user could also specify directories and files like below.
e.g.
"""
dirs:
- /usr (0755, root, root)
- /usr/local (0755, root, root)
- /usr/local/bin (0755, root, root)
files:
- /usr/local/bin/hello (0755, root, root)
"""
The path is a must, the permission and owner part is optional.
For example, we could just use '- /usr' instead of the full
'- /usr (0755, root, root)'.
By default, genrpm packages all files, but user could use 'dirs' and 'files'
to select the directories and files they want to package.
2b) appsdk genrpm -f hello.yaml -i /path/to/install-hello
3. Publish the RPM package
3a) Publish the RPM to repo
mkdir -p /path/to/http_service_data/third_party_repo
appsdk publishrpm -r /path/to/http_service_data/third_party_repo deploy/rpms/corei7_64/hello-2.10-r0.corei7_64.rpm
3b) [OPTIONAL] Setup http service
python3 -m http.server 8888 --directory /path/to/http_service_data
Use browser to access http://HOST_IP:8888/third_party_repo/
4. Use the RPM package on target
4a) Setup RPM repo
cat > /etc/yum.repos.d/test.repo <<EOF
[appsdk-test-repo]
name=appsdk test repo
baseurl=http://HOST_IP:8888/third_party_repo/
gpgcheck=0
EOF
4b) Enter development mode, get a writable filesystem
ostree admin unlock --hotfix
4c) Install hello package
dnf install hello
4d) Run hello program
e.g.
root@intel-x86-64:~# hello
Hello, world!
5. Use the RPM package as input of 'appsdk genimage' and 'appsdk gensdk`
5a) Modify yaml file
Add 'http://HOST_IP:8888/third_party_repo/' to package_feeds section.
Add 'hello' to external-packages section.
e.g.1 Manually edit image-with-hello-intel-x86-64.yaml
name: image-with-hello
package_feeds:
- http://<web-server-url>/repos/rpm/noarch/
- http://<web-server-url>/repos/rpm/corei7_64/
- http://<web-server-url>/repos/rpm/intel_x86_64/
- http://HOST_IP:8888/third_party_repo
external-packages:
- hello
e.g.2 Run genyaml to generate image-with-hello-intel-x86-64.yaml
genyaml --url http://HOST_IP:8888/third_party_repo --pkg-external hello --name image-with-hello
5b) appsdk genimage --type all image-with-hello-intel-x86-64.yaml
appsdk gensdk -f image-with-hello-intel-x86-64.yaml
Check environment-setup-*-wrs-linux for the exported variables.
The goal of the System Definition is to provide a way to define and implement system configuration changes that goes beyond those configurations shipped in individual software packages. These can include, but are not limited to:
- System configurations, such as hostname and IP
- Containers images and configurations
- User accounts and configurations
- Vendor data
- Cloud data
Only apply system definition on ustart/wic/vmdk/vdi image (generated by appsdk genimage), the initramfs image and container image do not support it
-
The current Linux Assembly Tool YAML schema should be enhanced with a 'system' top level list tag. List elements can be 'run_once', 'run_on_upgrade', 'run_always', 'files' and 'contains'.
-
The 'system: files' tag should be a list of 'file' elements.
-
The 'system: files: file' tag should each define a mode, source and destination.
-
The 'system: run_once' tag should be a list of script file sources. Each of these will be copied to a location in /etc where system definition tools will run them once after installation.
-
The 'system: run_on_upgrade' tag should be a list of script file sources. Each of these will be copied to a location in /etc where system definition tools will run them once after installation, and once after each upgrade. In order to track script updates there will be a new directory in /etc for each upgrade containing the copies of the scripts included in that upgrade bundle.
-
The 'system: run_always' tag should be a list of script file sources. Each of these will be copied to a location in /etc where system definition tools will run each script on each reboot.
-
The 'system: contains' tag should be a list of system YAML files and acts as an include for other system specification files. In this way we can define a nesting of systems. For example this would allow us to define a Xen top level system along with the contained VMs. We will only support one level of nesting, so a contained YAML can not definie a 'contains' of its own. The 'contains' tag will only be valid for image types ('ustart', 'wic', 'vmdk' and 'vdi'), for example a 'container' image must not use the 'contains' tag, and should throw an ERROR.
Run 'appsdk exampleyamls' to get a set of pre-canned scripts are made available as part of the Linux Assembly Tool with the ability to add new ones as time and need demands.
$ tree exampleyamls/sysdef/
exampleyamls/sysdef/
|-- add-system-user.yaml
|-- add-user-home.yaml
|-- contains-container-base.yaml
|-- files
| |-- docker_daemon.jason
| |-- sudoers_sudo
| `-- dns.conf
|-- run_always.d
| '-- 10_start_containers.sh
|-- run_on_upgrade.d
| |-- 10_update_containers.sh
| `-- containers.dat
|-- run_once.d
| |-- 10_add_system_user.sh
| |-- 20_add_user_home.sh
| |-- 30_set_hostname.sh
| `-- 40_set_ntp.sh
|-- set-dns.yaml
|-- set-hostname.yaml
|-- set-ntp.yaml
`-- update-containers.yaml
The first set of scripts includes:
-
the script that adds a new user to the system See exampleyamls/sysdef/add-system-user.yaml
-
the script that adds a new user and home directory See exampleyamls/sysdef/add-user-home.yaml
-
the script that sets the hostname based on an system attribute such as the MAC address See exampleyamls/sysdef/set-hostname.yaml
-
a set of the scripts that pulls listed containers (from containers.dat) from a public registrey and runs them. This should replace the existing 'include-container-images' functionality. See exampleyamls/sysdef/update-containers.yaml
-
the script that set the DNS resolver to a given IP See exampleyamls/sysdef/set-dns.yaml
-
the script that updates the address of the ntp server See exampleyamls/sysdef/set-ntp.yaml
-
The yaml that generates an image which contains a sub container image, a nest build will be run ahead of the main build See contains-container-base.yaml
The customer should refer these yamls and scripts to manipulate their own, take exampleyamls/sysdef/add-system-user.yaml for example, the yaml inclues a run_once script exampleyamls/sysdef/run_once.d/10_add_system_user.sh, in which it calls 'useradd' at target first boot to ceate a system user with username 'system-user' and password '123456'. Customer could create their own user and password base on the example
A new tool 'sysdef.sh' is written to implement the runtime functionality of the System Definition.
$ sysdef.sh -h
usage: sysdef.sh [-f] [-v] |run-once|run-on-upgrade|run-always [script1] [script2] [...]
sysdef.sh [-f] [-v] run-all
sysdef.sh [-v] list
-f: ignore stamp, force to run
-v: verbose
- In order to run sysdef.sh automatically when included in an image, a systemd service 'sysdef.service' ensures the sysdef.sh be run at a suitable time during boot. Maximum 3 times rerun if the service failed.
$ systemctl status sysdef.service
* sysdef.service - A tool to implement the runtime functionality of the System Definition.
Loaded: loaded (/usr/lib/systemd/system/sysdef.service; enabled; vendor preset: disabled)
Active: active (exited) since Mon 2020-11-23 07:53:30 UTC; 1h 8min ago
Process: 329 ExecStart=/usr/bin/sysdef.sh run-all (code=exited, status=0/SUCCESS)
Main PID: 329 (code=exited, status=0/SUCCESS)
- Log (/var/log/syslog) is captured for the tool and each of the scripts it runs.
$ grep sysdef /var/log/syslog
2020-11-23T07:52:54.216341+00:00 intel-x86-64 sysdef.sh[329]: Start run-once 10_add_system_user.sh
2020-11-23T07:52:54.220970+00:00 intel-x86-64 sysdef.sh[329]: Run run-once 10_add_system_user.sh success
2020-11-23T07:52:54.222146+00:00 intel-x86-64 sysdef.sh[329]: Start run-once 20_add_user_home.sh
...
-
It handles failures gracefully. If a single script fails the remaining scripts should still be run.
-
It makes use of stamp files to prevent run once scripts from being run multiple times. The scripts in each type (run_once/run_always/ run_on_upgrade)) are in alphanumeric order, allowing the user to specify the order in which scripts are run.
/etc/sysdef/
run_once.d/
10_add_system_user.sh
10_add_system_user.sh.stamp
20_add_user_home.sh
20_add_user_home.sh.stamp
30_set_hostname.sh
30_set_hostname.sh.stamp
40_set_ntp.sh
40_set_ntp.sh.stamp
run_always.d/
10_start_containers.sh
run_on_upgrade.d/
15082020025600//
10_update_containers.sh
10_update_containers.sh.stamp
containers.dat
20201123074928/
10_update_containers.sh
10_update_containers.sh.stamp
containers.dat
-
Support to run manually, it provides options '-f' to ignore the stamp files (not by default though).
List all scripts in run:
$ sysdef.sh list
run-once
10_add_system_user.sh
10_add_system_user.sh.stamp
20_add_user_home.sh
20_add_user_home.sh.stamp
30_set_hostname.sh
30_set_hostname.sh.stamp
40_set_ntp.sh
40_set_ntp.sh.stamp
run-on-upgrade(20201123074928)
10_update_containers.sh
10_update_containers.sh.stamp
containers.dat
run-always
10_start_containers.sh
Rerun all sysdef scripts manually:
$ sysdef.sh -f run-all
Start run-once 10_add_system_user.sh
useradd: user 'system-user' already exists
Run run-once 10_add_system_user.sh failed
...
Rerun specific sysdef scripts manually:
$ sysdef.sh -f run-on-upgrade 10_update_containers.sh
Start run-on-upgrade(20201123074928) 10_update_containers.sh
...
Run run-on-upgrade(20201123074928) 10_update_containers.sh success
- Load docker image by docker load
- Import filesystem tarball by docker import
- Pull docker image from registry
- Run multiple containers from one image
- Run container with user define docker run option
- Run container with user define docker command
- Start container automatically while system boot
- Stop container gracefully while system shutdown
- Update containers while system upgrade
Provides two scripts and one data file in exampleyamls
exampleyamls/sysdef/run_always.d
10_start_containers.sh
exampleyamls/sysdef/run_on_upgrade.d
containers.dat
10_update_containers.sh
Each line in containers.dat records how to load/import/pull and run a container The format of line is:
The '<container-name>' is mandatory, it is the name of container (docker run --name <container-name> XXX);
If 'load=<docker-image-tarball>' is set, use 'docker load' to add image tarball;
If 'import=<fs-tarball>' is set, use 'docker import' to add filesystem tarball;
If no 'load=<docker-image-tarball>' and no 'import=<fs-tarball>', use 'docker pull' to add image;
The 'image=<container-image-name>' is optional, if not set, use '<container-name>' by default;
The 'run-opt=<docker-run-opt>' is optional, if not set, use '-itd' by default (docker run -itd XXX);
The 'run-cmd=<docker-run-cmd>' is optional, if not set, default is empty
Examples:
ubuntu
-> docker pull ubuntu
-> docker run --name ubuntu ubuntu
-> systemctl start [email protected]
ubuntu-bash image=ubuntu run-opt="-p 2224:22 -it" run-cmd=/bin/bash
-> docker pull ubuntu
-> docker run -p 2224:22 -it -d --name ubuntu-bash ubuntu /bin/bash
-> systemctl start [email protected]
core-image-full import=/var/docker-images/core-image-full-intel-x86-64.tar.bz2 run-cmd=/bin/sh
-> docker import /var/docker-images/core-image-full-intel-x86-64.tar.bz2 core-image-full
-> docker run -itd --name core-image-full core-image-full /bin/sh
-> systemctl start [email protected]
container-base load=/var/docker-images/container-base-intel-x86-64.docker-image.tar.bz2
-> docker load -i /var/docker-images/container-base-intel-x86-64.docker-image.tar.bz2
-> docker run -itd --name container-base
-> systemctl start [email protected]
Usually the following Steps are required for customer
-
Run 'appsdk exampleyamls' to prepare yamls and scripts
-
If use docker to load image or import filesystem tarball, make these files be available on host or web server. If use docker to pull image, this step is ignored
-
Manipulate Yaml file:
-
Set image type with "ustart" or "wic" to avoid the yaml was incorrectly parsed by geninitramfs or gencontainer
-
Use "packages" tag to install dependency packages 'startup-container' and 'docker'
-
If use docker to load image or import filesystem tarball, make use of "system:file" tag or "rootfs-post-scripts" tag to copy files to target rootfs. If use docker to pull image, this step is ignored
-
Use "system:run_always" tag and "system:run_on_upgrade" tag to install scripts 10_start_containers.sh and 10_update_containers.sh from exampleyamls to target rootfs
-
Use "rootfs-post-scripts" tag to add settings as described above to containers.dat, one line for one container
- Use 3)'s yaml file as input, run 'appsdk genimage' to generate image
An example from exampleyamls/sysdef/update-containers.yaml which pulls ubuntu image from public registry, and run two containers from the ubuntu image, one with default option and command, one with user define option and command.
# - At a boot after upgrade, pulls listed containers (from containers.dat)
# from a public registrey and runs them.
# - At each boot, start listed containers (from containers.dat)
# - Two containers in containers.dat: hello-world and ubuntu
# - Add a docker_daemon.jason to set private insecure registries of Wind River
image_type:
- ostree-repo
- ustart
packages:
- startup-container
- docker
rootfs-post-scripts:
- echo "ubuntu" >> $IMAGE_ROOTFS/etc/sysdef/run_on_upgrade.d/containers.dat
- echo "ubuntu-bash image=ubuntu run-opt="-p 2224:22 -it" run-cmd=/bin/bash" >> $IMAGE_ROOTFS/etc/sysdef/run_on_upgrade.d/containers.dat
system:
- run_on_upgrade:
- exampleyamls/sysdef/run_on_upgrade.d/10_update_containers.sh
- run_always:
- exampleyamls/sysdef/run_always.d/10_start_containers.sh
- files:
- file:
src: exampleyamls/sysdef/files/docker_daemon.jason
dst: /etc/docker/daemon.json
mode: 644
- file:
src: exampleyamls/sysdef/run_on_upgrade.d/containers.dat
dst: /etc/sysdef/run_on_upgrade.d/
mode: 644
An example from exampleyamls/feature/startup-container.yaml, there are three images, one is a docker image which was generated by 'appsdk gencontainer', it will be used by docker load; one is a filesystem tarball which is downloaded from web server, it will be used by docker import; one is a docker image which is downloaded from private regristry by command 'skopeo copy', it will be used by docker load
packages:
- startup-container
- docker
rootfs-post-scripts:
- echo "container-base load=/var/docker-images/container-base-intel-x86-64.docker-image.tar.bz2 image=container-base-intel-x86-64" >> $IMAGE_ROOTFS/etc/sysdef/run_on_upgrade.d/containers.dat
- echo "ubuntu-tar load=/var/docker-images/ubuntu.docker-image.tar.bz2" >> $IMAGE_ROOTFS/etc/sysdef/run_on_upgrade.d/containers.dat
- echo "core-image-full import=/var/docker-images/core-image-full-intel-x86-64.tar.bz2 run-cmd=/bin/sh" >> $IMAGE_ROOTFS/etc/sysdef/run_on_upgrade.d/containers.dat
- skopeo copy --src-tls-verify=false --insecure-policy docker://pek-lpdfs01:5000/ubuntu docker-archive:$IMAGE_ROOTFS/var/docker-images/ubuntu.docker-image.tar.bz2:ubuntu-tar
system:
- run_on_upgrade:
- exampleyamls/sysdef/run_on_upgrade.d/10_update_containers.sh
- run_always:
- exampleyamls/sysdef/run_always.d/10_start_containers.sh
- files:
- file:
src: deploy/container-base-intel-x86-64.docker-image.tar.bz2
dst: /var/docker-images/
mode: 644
- file:
src: http://xxx/core-image-full-intel-x86-64.tar.bz2
dst: /var/docker-images/
mode: 644
Using the Wind River Linux Assembly Tool - Run appsdk from build dir Compare with appsdk in SDK:
-
The appsdk in the build uses local package feed of the same build.
-
If set PACKAGE_FEED_URIS and PACKAGE_FEED_URIS, the remote package feed will be saved as target yum repo. It will not be used by appsdk to generate image, but be used by dnf at target run time. Please make sure the remote package feed is available on web server
-
The appsdk in the build does not provide subcommand gensdk and checksdk
intel-x86-64
$ . ./oe-init-build-env
$ bitbake appsdk-native && bitbake package-index build-sysroots
$ bitbake world appsdk-native && bitbake package-index build-sysroots
$ tmp/sysroots/x86_64/usr/bin/appsdk -h
The bash completion of appsdk requires host bash support for complete -D, which was introduced in bash 4.2
$ bash --rcfile tmp/sysroots/x86_64/environment-appsdk-native
$ appsdk <TAB>
-d exampleyamls genimage genrpm -h --log-dir -q
--debug gencontainer geninitramfs genyaml --help publishrpm --quiet
Using the Wind River Linux Assembly Tool - create OSTree debian-based image
intel-x86-64
See meta-lat/data/debian/README.txt for details
Then http://<web-server-url>/debian is accessible
Prepare all the needed layers.
$ . ./oe-init-build-env
$ echo 'DEBIAN_CUSTOMIZE_FEED_URI = "http://<web-server-url>/debian"' >> conf/local.conf
(If not set, default is http://ftp.us.debian.org/debian)
$ echo 'DEFAULT_DEBIAN_MIRROR = "http://ftp.cn.debian.org/debian"' >> conf/local.conf
$ bitbake appsdk-native && bitbake lat-sdk && bitbake package-index build-sysroots
$ bitbake world appsdk-native && bitbake lat-sdk && bitbake package-index build-sysroots
$ ./lat-sdk-poky-intel-x86-64-1.0.sh -y -d stx
$ cd stx
$ sudo su
$ . environment-setup-corei7-64-wrs-linux
Provides three yaml files, edit yamls file for further customization
$ appsdk exampleyamls --pkg-type external-debian
+-----------+-------------------------------------------------+
| Yaml Type | Name |
+===========+=================================================+
| Image | debian-container-base-intel-x86-64.yaml |
| | debian-image-base-intel-x86-64.yaml |
| | debian-initramfs-ostree-image-intel-x86-64.yaml |
| | |
+-----------+-------------------------------------------------+
$ appsdk --log-dir log gencontainer exampleyamls/debian-container-base-intel-x86-64.yaml
$ appsdk --log-dir log geninitramfs exampleyamls/debian-initramfs-ostree-image-intel-x86-64.yaml
$ appsdk --log-dir log genimage exampleyamls/debian-image-base-intel-x86-64.yaml
$ appsdk --log-dir log genimage exampleyamls/debian-image-base-intel-x86-64.yaml --type iso
This layer depends on the poky, meta-openembedded and meta-virtualization.
This layer depends on:
URI: git://git.pokylinux.org/poky.git
branch: master
revision: HEAD
URI: git://git.openembedded.org/meta-openembedded
branch: master
revision: HEAD
layers: meta-oe
meta-networking
meta-filesystems
meta-python
meta-xfce
meta-gnome
meta-multimedia
URI: git://git.yoctoproject.org/meta-virtualization
branch: master
revision: HEAD