runtime_probe

Overview

Runtime Probe is essentially a command line tool that consumes the probe syntax and outputs the probe result.

This command line tool will gradually replace fields in HWID (i.e. less fields will be encoded into HWID) as it reflects the status of a device in a more timely approach. We are not aiming to be a daemon service, instead, we pose ourselves as a one-shot call tool, just like the ping command, the D-Bus method introduced later is also on-demand and expected to exit immediately after one call.

Currently, the reported data is purely hardware information. For example, the model of storage, the remaining capacity of battery, the screen resolution..etc.

To serve clients that are not able to call the command line directly, we offer a simple, restricted D-Bus interface with only one method org.chromium.RuntimeProbe.ProbeCategories. This D-Bus method follows the Chrome OS D-Bus Best Practices, proper minijail0, secomp policy are applied inside dbus/org.chromium.RuntimeProbe.conf.

Motivation

To better reflect hardware configuration on users’ system, we need a tool in release image that is able to reflect the truth of the moment in the field. Traditionally, this need has been satisfied by factory's probe framework, because we assume rare components replacement after mass production. However, we have seen more and more requests from partners to switch components after devices left the factory process.

This work is also related to the evolution of HWID. Instead of going into details on the complicated plan, we would like to use an example to let the reader get a high level concept on how this work helps cases in HWID.

Let's look into a typical scenario here: After years' love into the Chromebook, a user has one of the DRAM broken in slots, and would like to replace it. However, the original part is already EOL (End Of Life), ODM has no choice but to pick another part from the AVL (Approved Vendor List), while this new DRAM is installed, the original probe result (HWID) in factory is violated. Hence, even after the factory process, we would like to get the probe result to reflect the truth of the moment under release image. Runtime is used to convey the concept of dynamic.

Contribute to Runtime Probe

Environment

Runtime Probe has a significant different logic when working with cros_debug=0, in order to test the behavior under so, you might try to setup a device like the following:

• CCD Unlock

• Build locally a test image for your device and flash it to ensure your USE flags are aligned. Image is suggested to build with rootfs verification disabled in build_image step.

• Flash the latest dev-signed firmware, for Googler, it might be easier to get it from GoldenEye. Suggest to use the following for flashing while keeping the VPD: futility update --force -i $FILE

• Switch on/off developer mode

  Besides the developer doc describing how to enter/leave developer mode, following is another useful hack to prevent some long running check while switching mode for the first time.

  • Before switching on developer mode
    • touch /mnt/stateful_partition/.developer_mode
    • sync
  • Before switching off developer mode
    • rm /mnt/stateful_partition/.developer_mode
    • sync

• Hack¹ to keep developer tools (For example: python, emerge for cros deploy) while mocking cros_debug=0

  • Switch off developer mode
  • sed -i 's/CROS_DEBUG=/CROS_DEBUG=1 #/g' /sbin/chromeos_startup
  • Reboot

• Toggle the cros_debug to 0

  • cd /usr/share/vboot/bin/
  • ./make_dev_ssd.sh --save_config /tmp/foo --partitions 2
  • sed -i 's/cros_debug//g' /tmp/foo.2
  • ./make_dev_ssd.sh --set_config /tmp/foo --partitions 2

• Toggle the cros_debug to 1

  • cd /usr/share/vboot/bin/
  • ./make_dev_ssd.sh --save_config /tmp/foo --partitions 2
  • sed -i 's/cros_secure/cros_secure cros_debug/g' /tmp/foo.2
  • ./make_dev_ssd.sh --set_config /tmp/foo --partitions 2

Development

• cros_workon-$BOARD start runtime_probe
• emerge-$BOARD runtime_probe
• cros deploy $ACTIVE_DUT runtime_probe
• On the device:
  • runtime_probe --vebosity_level=3

Testing

Runtime Probe provides CLI and a D-Bus interface, and our goal is to make sure using Runtime Probe by both interface works correctly. The main runtime_probe binary executed via CLI is free of minijail and hence is useful for testing the correctness of probe function. D-Bus call to Runtime Probe is the main entry point we expect in most use cases. It is guarded by minijail, and the main goal is to ensure the integrity of this entry point.

Via CLI

Simply run the following:

localhost ~ # runtime_probe [--verbosity_level=<level>]

This command will produce the probe result in json format, also it verifies the correctness of probe results and tests the sandbox args of each probe function (if cros_debug is set to 1).

Via D-Bus call

The following script tests the D-Bus entry of runtime_probe remotely. Besides the output of the following script, the protocol buffer would also be shown in /var/log/messages by changing the verbosity_level in runtime_probe/init/runtime_probe.conf to 3. (And deploy to DUT by emerge-${BOARD} runtime_probe && cros deploy "$ACTIVE_DUT" runtime_probe)

#!/bin/sh

PROTO_DIR_PATH="$HOME/trunk/src/platform2/system_api/dbus/runtime_probe/"
PROTO_PATH="$PROTO_DIR_PATH/runtime_probe.proto"

# Executing the following commands in cros_sdk or change the proto paths above

CATEGORIES="categories:battery\ncategories:vpd_cached\ncategories:storage"
PROTO_BYTES=$(echo -e "$CATEGORIES" | \
  protoc --encode runtime_probe.ProbeRequest --proto_path="$PROTO_DIR_PATH" "$PROTO_PATH" | \
  hexdump -v -e '/1 "%d,"')

RAW_HEX_PROBE_RESULT=$(ssh root@$ACTIVE_DUT sudo -u chronos \
  dbus-send --system --print-reply=literal \
  --type=method_call --dest=org.chromium.RuntimeProbe \
  /org/chromium/RuntimeProbe org.chromium.RuntimeProbe.ProbeCategories \
  array:byte:"$PROTO_BYTES")

echo "$RAW_HEX_PROBE_RESULT" | sed -e '1d;$d' | \
  xxd -r -p | \
  protoc --decode runtime_probe.ProbeResult --proto_path="$PROTO_DIR_PATH" "$PROTO_PATH"

Sample output:

storage {
  name: "generic"
  values {
    path: "/sys/class/block/nvme0n1"
    sectors: 1234567890
    size: 123456789012
    type: "NVMe"
    pci_vendor: 9876
    pci_device: 5432
    pci_class: 67890
  }
}
...
probe_config_checksum: "DA39A3EE5E6B4B0D3255BFEF95601890AFD80709"

Per probe function

The probe statement of Runtime Probe could be customized for any supported probe function. The following script tests each function via debugd and use the corresponding sandbox environment for each probe function.

#!/bin/sh
# on DUT with cros_debug = 1
# "generic_battery" could be replace with any supported probe function.
cat << EOF > per_function.json
{"CUSTOM_CATEGORY": {"CUSTOM_COMPONENT_NAME": {"eval": {"some_new_probe_function": {}}}}}
EOF
runtime_probe --config_file_path=./per_function.json

On DUT with cros_debug = 0 we cannot use --config_file_path option. However, we could leverage /etc/runtime_probe/$MODEL/probe_config.json like the following:

#!/bin/sh
# backup probe_config.json
cat << EOF > /etc/runtime_probe/$MODEL/probe_config.json
{"CUSTOM_CATEGORY": {"CUSTOM_COMPONENT_NAME": {"eval": {"some_new_probe_function": {}}}}}
EOF
runtime_probe  # or via D-Bus as described earlier

On Recovery image

• Get the following from the image folder of a Live, where xxx is the channel [beta, dev, stable] and yyy is the board name and zzz is the version info.
  • xxx-channel_yyy_zzz_ChromeOS-firmware-...tar.bz2
    • image-yyy.bin
  • xxx-channel_yyy_zzz_ChromeOS-recovery-...tar.xz
    • recovery_image.bin
• Flash the dev-signed firmware, image-yyy.bin by either flashrom or servo board.
• Flash to the USB and trigger a recovery install [ Ctrl + Alt + D ]
• Switch to developer mode and login into console [ Ctrl + Alt + F2 ]
• Disable rootfs verification
  • cd /usr/share/vboot/bin/
  • ./make_dev_ssd.sh --remove_rootfs_verification --partitions 2
• Reboot the system
• Enable Developer Console login in normal mode
  • Delete the following lines in /etc/init/boot-splash.conf
    • if is_developer_end_user; then
    • fi
• Login as root and change the password to non-empty (might just use test0000)
• Reboot the system and disable developer switch
• You should be able to login as root with the password set before
• Clear the /var/log/message for better check
  • echo 0 > /var/log/messages
• Acts as chronos
  • su chronos
• Either execute a script or type manually on DUT

#!/bin/sh
PROTO_BYTES="10,2,1,2"
PROBO_BYTES="10,1,3"  # for devices report vpd_cached only
dbus-send --system --print-reply \
--type=method_call --dest=org.chromium.RuntimeProbe \
/org/chromium/RuntimeProbe org.chromium.RuntimeProbe.ProbeCategories \
array:byte:"$PROTO_BYTES"

• 10,2,1,2 is the generated ProbeRequest, which should depend on the version of image. For example 10,1,3 for devices only reporting vpd_cached. These value could be encoded by protoc --encode command mentioned in via D-Bus call.
• Check the /var/log/message if the ProbeResult dumps every field in the protocol buffer.
• Again this ProbeResult could be decoded from local machine in a way similar to the test script described in via D-Bus call

Via Tast

Tast is a golang-based test framework. Currently tast-tests for Runtime Probe check if the probe result matches the cros labels we decoded from the HWID of DUT. Please refer to cros_runtime_probe_*.go under platform tests.

Note that Runtime Probe uses the probe statements at /etc/runtime_probe/$MODEL/probe_config.json on DUT. The names of probed components are directly under the probe categories (e.g. component "MODEL_COMPONENT1" under category "battery"). This is an example for such component name.

On the other hand, we will compare the above probed component name with one described in HWID DB file which is used to generate HWID string. This is an example for the same component name described in HWID DB. These component names will be cros labels and be passed from -varsfiles option of tast. We can manually create one and run tast tests with it in cros_sdk.

#!/bin/sh
# in cros_sdk
cat << EOF > labels.yaml
autotest_host_info_labels: '["model:MODEL", "hwid_component:battery/MODEL_COMPONENT1", "hwid_component:storage/MODEL_COMPONENT2"]'
EOF
tast -verbose=true -logtime=false run -build=true -logtime=false -varsfile=labels.yaml "$ACTIVE_DUT" '(!disabled && "group:runtime_probe")'

In the output there will be the probed component names and expected component names for clarification.

Example output:

Using SSH key ...
Using SSH dir ...
Writing results to /tmp/tast/results/20200220-172233
Connecting to $ACTIVE_DUT
Getting architecture from target
Building local_test_runner, cros, remote_test_runner, cros
Built in 2.778s
Pushing executables to target
Pushed executables in 4.104s (sent 9.4 MB)
Getting data file list from target
Got data file list with 0 file(s)
Getting DUT info
Software features supported by DUT: ...
Getting initial system state
[01:22:42.048] Devserver status: [[http://127.0.0.1:28082 UP]]
[01:22:42.048] Found 0 external linked data file(s), need to download 0
Started test platform.CrosRuntimeProbeBattery
[01:22:42.173] Probed battery:MODEL_COMPONENT1
[01:22:42.173] Skip known generic probe result
Completed test platform.CrosRuntimeProbeBattery in 89ms with 0 error(s)
Ran 1 local test(s) in 481ms
Starting /tmp/tast/build/host/remote_test_runner locally
Ran 0 remote test(s) in 26ms
Collecting system information
--------------------------------------------------------------------------------
platform.CrosRuntimeProbeBattery  [ PASS ]
--------------------------------------------------------------------------------
Results saved to /tmp/tast/results/20200220-172233

Documents

• Design doc: go/cros-probe
• Consolidated summary for all supported component types and probing fields: go/cros-runtime-probe-fields

Useful Reference

• minijail0 manpage (man 1 minijail0 in cros_sdk)
• docs/sandboxing.md
• debugd/src/

Notes

Footnotes

1. Subject to change based on the other program, after this quick hack, you will not able to switch to developer mode easily. ↩