docs: extend consensus encoding adr with tech design
What changed, and why it matters
This commit only updates documentation. It expands a design document explaining how a new consensus encoding crate works and removes a TODO comment from another document. No code, tests, build files, or configuration were changed, so it cannot introduce or fix a security vulnerability.
No security action needed. Review as normal documentation if desired.
Security signals we found
No strong security signals were identified.
Evidence from the diff
The diff is limited to two Markdown files under docs/. It adds a ‘Technical Design’ section to docs/adr/0001_consensus_encoding.md describing sans-I/O encoder/decoder traits and example caller signatures, and removes a TODO comment from docs/primitives.md. There are no executable changes.
Changed components
docs/adr/0001_consensus_encoding.mddocs/primitives.mdInspect captured patch +78 / −2
diff --git a/docs/adr/0001_consensus_encoding.md b/docs/adr/0001_consensus_encoding.md
index 6eb161a4..48bb1553 100644
--- a/docs/adr/0001_consensus_encoding.md
+++ b/docs/adr/0001_consensus_encoding.md
@@ -63,6 +63,83 @@ Extract consensus encoding/decoding logic into a new crate which is sans-I/O and
Option #3, dedicated `consensus_encoding` crate. While it is new code, it is heavily based on [`push_decode`] and slimmed down to just what is required in the workspace. It supports no-std, std, non-I/O, and could support async contexts. It should be relatively cheap to maintain given the limited domain.
+## Technical Design
+
+The `consensus_encoding` crate provides traits and utilities for encoding and decoding Bitcoin data types in a consensus-consistent way. This crate implements a **sans-I/O** architecture, designed to work efficiently in `no_std` environments while supporting both synchronous and asynchronous I/O when needed.
+
+The implementation is heavily based on the more general [`push_decode`] crate written by Martin Habovštiak. But where `push_decode` is flexible to handle many different encoding scenarios, `consensus_encoding` has been slimmed down to only handle what is required by the Bitcoin ecosystem.
+
+Sans-I/O architecture separates data codecs from I/O operations. Instead of reading directly from `io::Read` traits or writing to `io::Write` traits, the core types work with byte slices and provide iterator-like interfaces for consuming data. Same codec logic works for sync I/O, async I/O, or other use cases such as hash engines. So unlike traditional "pull decoders" that read data on-demand, this crate uses a "push" approach where *decoders* consume data in chunks and maintain internal state. The *caller* drives the process by managing buffers and I/O to push bytes into the decoder. And on the other side, *encoders* produce data in chunks which a caller pulls in order to write to a sink.
+
+Encoding is generally infallible (e.g. encoding to `Vec` never fails). Decoding on the other hand has to deal with many failure scenarios due to not owning the bytes it's consuming. This complicates the interface as decoders provide specific errors for failure modes (`UnexpectedEof`, `InvalidData`, etc.). But I/O errors are handled by the caller functions, keeping the core codec logic I/O-agnostic.
+
+### Encoders
+
+```rust
+pub trait Encodable {
+ type Encoder<'s>: Encoder
+ where
+ Self: 's;
+
+ fn encoder(&self) -> Self::Encoder<'_>;
+}
+
+pub trait Encoder {
+ fn current_chunk(&self) -> &[u8];
+ fn advance(&mut self) -> bool;
+}
+```
+
+A Bitcoin type implements `Encodable` in order to produce `Encoder` instances for its type. So for example a `Transaction` type is made `Encodable` and linked to a `TransactionEncoder` type which implements `Encoder`. The `Encodable` trait makes use of Rust's [Generic Associated Type (GAT)] feature which ties `Encoder` lifetimes to the instance of the type they are encoding. This allows the encoder to avoid any copy or clones of bytes, instead referencing them directly, which is often powerful in the Bitcoin context where bytes are already in encoded form.
+
+Once a caller has an `Encoder` instance, it pulls bytes out with calls to `Encoder::current_chunk`. The caller bounces between `Encoder::current_chunk` and `Encoder::advance` until the encoder is exhausted which is signaled by `Encoder::advance` returning `false`. While we considered making these a single method, the "immutable accessor method plus mutable advance state method" greatly simplifies lifetime management when encoders are combined. And encoder composition is very common in Bitcoin types which are generally composed internally of other Bitcoin types.
+
+### Decoders
+
+```rust
+pub trait Decodable {
+ type Decoder: Decoder<Output = Self>;
+ fn decoder() -> Self::Decoder;
+}
+
+pub trait Decoder: Sized {
+ type Output;
+ type Error;
+
+ fn push_bytes(&mut self, bytes: &mut &[u8]) -> Result<bool, Self::Error>;
+ fn end(self) -> Result<Self::Output, Self::Error>;
+ fn read_limit(&self) -> usize;
+}
+```
+
+Similar to encoding, Bitcoin types implement `Decodable` in order to generate `Decoder`s. Unlike encoding, no GAT is required since the decoder is taking ownership of the bytes.
+
+The caller interface is significantly more complex than encoding since `Decoder`s are fallible, so both `Decoder::push_bytes` and `Decoder::end` return a `Result`. A caller pushes bytes through `Decoder::push_bytes` until it returns `false`. That is the signal for the caller to now consume the `Decoder` with `Decoder::end` and get the output type. The `bytes` parameter of `Decoder::push_bytes` is mutable to allow the decoder to "consume" bytes by advancing the slice. Any un-used bytes remain in the `bytes` buffer.
+
+The `Decoder::read_limit` method has no encoding parallel. It is another complexity due to decoders not having a priori knowledge about the amount of bytes they will be dealing with. But this helper function supplies hints to callers which allows them to better manage their buffer for the decoder, avoiding both inefficient under-reads and unnecessary over-reads.
+
+### Callers
+
+Library consumers can always define callers for their specific needs (e.g. async), but callers for the most common use cases are provided by the crate as free functions. Here are the signatures of the provided callers which obviously connect codecs to the standard library I/O.
+
+```rust
+#[cfg(feature = "std")]
+pub fn encode_to_writer<T, W>(object: &T, mut writer: W) -> Result<(), std::io::Error>
+where
+ T: Encodable + ?Sized,
+ W: std::io::Write,
+{}
+
+#[cfg(feature = "std")]
+pub fn decode_from_read<T, R>(mut reader: R) -> Result<T, ReadError<<T::Decoder as Decoder>::Error>>
+where
+ T: Decodable,
+ R: std::io::BufRead,
+{}
+```
+
+The keen eye will catch how the decode caller requires the use of a `std::io::BufRead` instead of just `std::io::Read`. While the crate also supports `std::io::Read`, `std::io::BufRead` mitigates a lot of the complexity for decoder buffer management and will almost always be more performant.
+
## Links
* Initial implementation in [#4912].
@@ -71,3 +148,4 @@ Option #3, dedicated `consensus_encoding` crate. While it is new code, it is hea
[#4912]: <https://github.com/rust-bitcoin/rust-bitcoin/pull/4912>
[#5160]: <https://github.com/rust-bitcoin/rust-bitcoin/pull/5160>
[`push_decode`]: <https://github.com/Kixunil/push_decode>
+[Generic Associated Type (GAT)]: <https://blog.rust-lang.org/2022/10/28/gats-stabilization/>
diff --git a/docs/primitives.md b/docs/primitives.md
index e2330e89..98a169dc 100644
--- a/docs/primitives.md
+++ b/docs/primitives.md
@@ -10,8 +10,6 @@ GitHub discussion: https://github.com/rust-bitcoin/rust-bitcoin/discussions/4856
Pull encoding done in: https://github.com/rust-bitcoin/rust-bitcoin/pull/4912
-<!-- TODO: Add a consensus-encoding.md file? -->
-
### Remove `hashes` from the public API.
Required due to [C-STABLE](https://rust-lang.github.io/api-guidelines/necessities.html#c-stable).
Why this scored 15/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.