58 lines
3.0 KiB
ReStructuredText
58 lines
3.0 KiB
ReStructuredText
.. _module-pw_snapshot-design_discussion:
|
|
|
|
=================
|
|
Design Discussion
|
|
=================
|
|
There were a handful of key requirements going into the design of pw_snapshot:
|
|
|
|
* **Pre-established file format** - Building and maintaining tooling to support
|
|
parsing binary snapshot data is a high maintenance burden that detracts from
|
|
the appeal of a pre-existing widely known/supported format.
|
|
* **Incremental writing** - Needing to build an entire snapshot before
|
|
committing it as a finished file is a big limitation on embedded devices where
|
|
RAM is often very constrained. It is important that a snapshot can be built in
|
|
smaller in-memory segments that can be committed incrementally to a larger
|
|
sink (e.g. UART, off-chip flash).
|
|
* **Extensible** - Pigweed doesn't know everything users might want to capture
|
|
in a snapshot. It's important that users have ways to include their own
|
|
information into snapshots with minimal friction.
|
|
* **Relatively compact** - It's important that snapshots can contain useful
|
|
information even when they are limited to a few hundred bytes in size.
|
|
|
|
Why Proto?
|
|
==========
|
|
Protobufs are widely used and supported across many languages and platforms.
|
|
This greatly reduces the encode/decode tooling maintenance introduced by using
|
|
custom or unstructured formats. While using a format like JSON provides
|
|
similarly wide tooling support, encoding the same information as a proto
|
|
significantly reduces the final file size.
|
|
|
|
While protobuffer messages aren't truly streamable (i.e. can be written without
|
|
any intermediate buffers) due to how message nesting works, a large message can
|
|
be incrementally written as long as there's enough buffer space for encoding the
|
|
largest single sub-message in the proto.
|
|
|
|
Why overlay multiple protos?
|
|
============================
|
|
Proto 2 supported a feature called "extensions" that explicitly allowed this
|
|
behavior. While proto 3 removed this feature, it doesn't disallow the old
|
|
behavior of serializing two 'overlayed' protos to the same data stream. Proto 3
|
|
recommends using an "Any" proto instead of extensions, as it is more explicit
|
|
and eliminates the issue of collisions in proto messages. Unfortunately, proto
|
|
'Any' messages introduce unacceptable overhead. For a single integer that would
|
|
encode to a few bytes using extensions, an Any submessage quickly expands to
|
|
tens of bytes.
|
|
|
|
pw_snapshot's proto format takes advantage of "extensions" from proto 2 without
|
|
explicitly relying on the feature. To reduce the risk of colissions and maximize
|
|
encoding efficiency, certain ranges are reserved to allow Pigweed to grow while
|
|
ensuring downstream customers have equivalent flexibility when using the
|
|
Snapshot proto format.
|
|
|
|
Why no file header?
|
|
===================
|
|
Right now it's assumed that anything that is storing or transferring a
|
|
serialized snapshot implicitly tracks its size (and a checksum, if desired).
|
|
While a container format might be introduced independently, pw_snapshot focuses
|
|
on treating an encoded snapshot as raw serialized proto data.
|