gimlet-seq: Record why the power state changed

[hawkw]

Currently, when the Gimlet CPU sequencer changes the system's power
state, no information about why the power state changed is recorded by
the SP firmware. A system may power off or reboot for a variety of
reasons: it may be requested by the host OS over IPCC, by the control
plane over the management network, or triggered by the thermal task due
to an overheat condition. This makes debugging an unexpected reboot or
power off difficult, as the SP ringbuffers and other diagnostics do not
indicate why an unexpected power state change occurred. See #1950 for a
motivating example.

This commit resolves this as described in #1950 by adding a new field to
the SetState variant in the drv-gimlet-seq-server ringbuffer, so
that the reason a power state change occurred can be recorded. A new IPC
function, Sequencer.set_state_with_reason, is added to the cpu_seq
IPC API. This is equivalent to Sequencer.set_state but with the
addition of a StateChangeReason argument in addition to the desired
power state, and the sequencer task will record the provided reason in
its ringbuffer. This way, we can distinguish between the various reasons
a power state change may have occurred when debugging such issues.

All Hubris-internal callers of Sequencer.set_state are updated to
instead use Sequencer.set_state_with_reason. In particular,
host-sp-comms will record a variety of different StateChangeReasons,
allowing us to indicate whether the host requested a normal
power-off/reboot, the host OS panicked or failed to boot, or the host
CPU reset itself. Other callers like control-plane-agent and thermal
are simpler and just say "it was the control plane" or "overheat",
respectively. For backwards compatibility with existing callers of
Sequencer.set_state via hiffy, the set_state IPC is left as-is,
and will be recorded in the ringbuffer with StateChangeReason::Other.
Since all Hubris tasks now use the new API, Other basically just
means hiffy.

The StateChangeReason enum also generates counters, so that the total
number of power state changes can be tracked.

Also, while I was here, I've changed the Trace::SetState entry in the
drv-gimlet-seq-server ringbuf from a tuple-like enum variant to a
struct-like enum variant with named fields. This entry includes two
PowerState fields, one recording the previous power state and the
other recording the new power state that has been set. IMHO, using a
tuple-like variant to represent this is a bit unfortunate, as in
Humility, we'll see two values of the same type and it's not immediately
obvious which is the previous state and which is the new state. This
must be determined based on the order of the fields in the ringbuf
entry, which requires referencing the Hubris code to determine.

I felt like it was nicer to just use a struct-like variant with named
fields for this. That way, the semantic meaning of the two PowerStates
is actually encoded in the debug info, and Humility can just indicate
which is the previous state and which is the new state when displaying
the ring buffer. I also think it's a bit nicer to name the timestamp
field --- otherwise, it just looks like some arbitrary integer, and you
need to look at the code to determine that it's the timestamp of the
power state change.

Fixes #1950

[hawkw]

argh, I forgot that Gimletlet also implements the sequencer API

[hawkw]

I'm very open to suggestions about the naming of these variants, if anyone dislikes the ones I came up with...

[jgallagher]

Thanks for this! I'm not sure what to suggest about your (very good) question about hiffy backwards compatibility - might be worth checking with mfg and host-os folks who are probably the biggest users of that?

[hawkw]

@jgallagher Alright, I think all the review feedback has been addressed and I'd love another review whenever you've got the chance!

[jgallagher]

LGTM 👍