HTTPserver: support "chunked" Transfer-Encoding
What changed, and why it matters
This commit adds support for HTTP 'chunked' transfer encoding to Bitcoin Core's built-in HTTP server. Before this change, the server only understood requests with a plain Content-Length body. The patch implements chunk-size parsing, chunk-extension handling, trailer skipping, and size limits. It also includes tests for valid, oversized, malformed, and delayed chunked requests. There is no direct evidence in the commit that this fixes a known security vulnerability, but adding chunked parsing is a sensitive network-facing change that could affect availability or request handling if implemented incorrectly.
Review the chunked parser for HTTP request smuggling or desynchronization issues, especially interactions between Content-Length and Transfer-Encoding, trailer parsing limits, and chunk-extension edge cases. Verify that the server rejects ambiguous or simultaneous Content-Length + chunked requests. Run the new tests and consider fuzzing the parser with malformed chunked input.
Security signals we found
New network-facing HTTP parser code for chunked transfer encoding
Addition of size-limit checks against MAX_BODY_SIZE during chunked parsing
Handling of chunk extensions and trailers, which are common sources of parser smuggling bugs
CRLF termination validation for chunks
Removal of prior TODO indicating chunked support was unimplemented
Evidence from the diff
The change modifies HTTPRequest::LoadBody() in src/httpserver.cpp to detect Transfer-Encoding: chunked and parse chunked bodies per RFC 9112/RFC 7230. It reads hex chunk sizes, strips chunk extensions, enforces MAX_BODY_SIZE per-chunk, appends chunk data, consumes CRLF terminators, and skips optional trailers. The existing Content-Length path is preserved in an else branch. Tests are added in src/test/httpserver_tests.cpp covering normal chunked requests, oversized chunks, chunk extensions, invalid hex sizes, improper termination, and delayed completion. The commit removes a TODO about implementing chunked transfer encoding.
Changed components
src/httpserver.cppsrc/httpserver.hsrc/test/httpserver_tests.cppHTTPRequest::LoadBodyBitcoin Core HTTP RPC serverInspect captured patch +195 / −18
diff --git a/src/httpserver.cpp b/src/httpserver.cpp
index 1e785a5b..dffe7915 100644
--- a/src/httpserver.cpp
+++ b/src/httpserver.cpp
@@ -875,30 +875,96 @@ bool HTTPRequest::LoadHeaders(LineReader& reader)
bool HTTPRequest::LoadBody(LineReader& reader)
{
// https://httpwg.org/specs/rfc9112.html#message.body
+ auto transfer_encoding_header = m_headers.FindFirst("Transfer-Encoding");
+ if (transfer_encoding_header && ToLower(transfer_encoding_header.value()) == "chunked") {
+ // Transfer-Encoding: https://datatracker.ietf.org/doc/html/rfc7230.html#section-3.3.1
+ // Chunked Transfer Coding: https://datatracker.ietf.org/doc/html/rfc7230.html#section-4.1
+ // see evhttp_handle_chunked_read() in libevent http.c
+ while (reader.Remaining() > 0) {
+ auto maybe_chunk_size = reader.ReadLine();
+ if (!maybe_chunk_size) return false;
+
+ // Allow (but ignore) Chunk Extensions
+ // See https://www.rfc-editor.org/rfc/rfc9112.html#name-chunk-extensions
+ std::string_view chunk_size_noext{maybe_chunk_size.value()};
+ const auto semicolon_pos = chunk_size_noext.find(';');
+ if (semicolon_pos != chunk_size_noext.npos) {
+ chunk_size_noext.remove_suffix(chunk_size_noext.size() - semicolon_pos);
+ }
- // No Content-length or Transfer-Encoding header means no body, see libevent evhttp_get_body()
- // TODO: we must also implement Transfer-Encoding for chunk-reading
- auto content_length_values{m_headers.FindAll("Content-Length")};
- if (content_length_values.empty()) return true;
+ const auto chunk_size{ToIntegral<uint64_t>(util::TrimStringView(chunk_size_noext), /*base=*/16)};
+ if (!chunk_size) throw std::runtime_error("Cannot parse chunk length value");
+
+ if ((m_body.size() > MAX_BODY_SIZE) ||
+ (*chunk_size > MAX_BODY_SIZE - m_body.size()))
+ throw ContentTooLargeError("Chunk will exceed max body size");
+
+ // Last chunk has size 0
+ if (*chunk_size == 0) {
+ // Allow (but ignore) Chunked Trailer section, by
+ // reading CRLF-terminated lines until we read an empty line,
+ // which indicates the end of this request.
+ // See https://httpwg.org/specs/rfc9112.html#rfc.section.7.1.2
+ const size_t trailer_start{reader.Consumed()};
+ while (true) {
+ auto maybe_trailer = reader.ReadLine();
+ if (reader.Consumed() - trailer_start > MAX_HEADERS_SIZE) {
+ throw std::runtime_error("HTTP chunked trailer exceeds size limit");
+ }
+ if (!maybe_trailer) return false;
+ if (maybe_trailer->empty()) break;
+ }
+ // Complete request has been parsed, reader is now pointing
+ // to beginning of next request or end of the buffer.
+ return true;
+ }
- // Duplicate Content-Length headers are allowed only if they all have the same value
- // https://www.rfc-editor.org/rfc/rfc7230#section-3.3.3
- const auto& first_content_length_value{content_length_values[0]};
- for (size_t i = 1; i < content_length_values.size(); ++i) {
- if (content_length_values[i] != first_content_length_value) throw std::runtime_error("Differing Content-Length values");
- }
+ // We are still expecting more data for this chunk
+ if (reader.Remaining() < *chunk_size) {
+ return false;
+ }
- const auto content_length{ToIntegral<uint64_t>(first_content_length_value)};
- if (!content_length) throw std::runtime_error("Cannot parse Content-Length value");
+ // Pack chunk onto body
+ m_body += reader.ReadLength(*chunk_size);
- if (*content_length > MAX_BODY_SIZE) throw ContentTooLargeError("Max body size exceeded");
+ // Even though every chunk size is explicitly declared,
+ // they are still terminated by a CRLF we don't need,
+ // just consume it here.
+ auto crlf = reader.ReadLine();
+ if (!crlf) {
+ // CRLF not found before end of buffer: it has not been received by our socket yet.
+ return false;
+ }
+ // CRLF was found but there was unexpected data after the chunk_sized chunk
+ if (!crlf.value().empty()) throw std::runtime_error("Improperly terminated chunk");
+ }
- // Not enough data in buffer for expected body
- if (reader.Remaining() < *content_length) return false;
+ // We read all the chunks but never got the last chunk, wait for client to send more
+ return false;
+ } else {
+ // No Content-length or Transfer-Encoding header means no body, see libevent evhttp_get_body()
+ auto content_length_values{m_headers.FindAll("Content-Length")};
+ if (content_length_values.empty()) return true;
+
+ // Duplicate Content-Length headers are allowed only if they all have the same value
+ // https://www.rfc-editor.org/rfc/rfc7230#section-3.3.3
+ const auto& first_content_length_value{content_length_values[0]};
+ for (size_t i = 1; i < content_length_values.size(); ++i) {
+ if (content_length_values[i] != first_content_length_value) throw std::runtime_error("Differing Content-Length values");
+ }
- m_body = reader.ReadLength(*content_length);
+ const auto content_length{ToIntegral<uint64_t>(first_content_length_value)};
+ if (!content_length) throw std::runtime_error("Cannot parse Content-Length value");
- return true;
+ if (*content_length > MAX_BODY_SIZE) throw ContentTooLargeError("Max body size exceeded");
+
+ // Not enough data in buffer for expected body
+ if (reader.Remaining() < *content_length) return false;
+
+ m_body = reader.ReadLength(*content_length);
+
+ return true;
+ }
}
util::Expected<void, std::string> HTTPServer::BindAndStartListening(const CService& to)
diff --git a/src/httpserver.h b/src/httpserver.h
index 092c5935..01a97922 100644
--- a/src/httpserver.h
+++ b/src/httpserver.h
@@ -215,7 +215,7 @@ constexpr size_t MAX_HEADERS_SIZE{8192};
//! Maximum size of an HTTP request body
constexpr uint64_t MAX_BODY_SIZE{32_MiB};
-//! Thrown when a request body exceeds MAX_BODY_SIZE
+//! Thrown when a request body exceeds MAX_BODY_SIZE (or *will* exceed, in chunked transfer)
//! so the server can reply with more specific code 413 (content too large) vs general 400 (bad request)
struct ContentTooLargeError : std::runtime_error {
using std::runtime_error::runtime_error;
diff --git a/src/test/httpserver_tests.cpp b/src/test/httpserver_tests.cpp
index bb9da546..1c90cdc2 100644
--- a/src/test/httpserver_tests.cpp
+++ b/src/test/httpserver_tests.cpp
@@ -374,6 +374,117 @@ BOOST_AUTO_TEST_CASE(http_request_tests)
BOOST_CHECK(req.LoadHeaders(reader));
BOOST_CHECK(!req.LoadBody(reader));
}
+ {
+ // Support "chunked" transfer. Chunk lengths are ascii-encoded hex integers, whitespace ignored
+ HTTPRequest req;
+ std::string_view ok_chunked = "GET / HTTP/1.0\n"
+ "Transfer-Encoding: chunked\n"
+ "\n"
+ "10\n"
+ R"({"method":"getbl)""\n"
+ " a \n"
+ R"(ockcount"})""\n"
+ "0\n"
+ "\n";
+ LineReader reader(ok_chunked, MAX_HEADERS_SIZE);
+ BOOST_CHECK(req.LoadControlData(reader));
+ BOOST_CHECK(req.LoadHeaders(reader));
+ BOOST_CHECK(req.LoadBody(reader));
+ BOOST_CHECK_EQUAL(req.m_body, R"({"method":"getblockcount"})");
+ }
+ {
+ // Prevent "chunked" transfer from exceeding size limit
+ HTTPRequest req;
+ std::string_view excessive_chunk_size = "GET / HTTP/1.0\n"
+ "Transfer-Encoding: chunked\n"
+ "\n"
+ "10\n"
+ R"({"method":"getbl)""\n"
+ "20000000\n"
+ R"(ockcount"})""\n"
+ "0\n"
+ "\n";
+ LineReader reader(excessive_chunk_size, MAX_HEADERS_SIZE);
+ BOOST_CHECK(req.LoadControlData(reader));
+ BOOST_CHECK(req.LoadHeaders(reader));
+ BOOST_CHECK_EXCEPTION(req.LoadBody(reader), http_bitcoin::ContentTooLargeError, HasReason{"Chunk will exceed max body size"});
+ }
+ {
+ // Allow (but ignore) Chunk Extensions
+ HTTPRequest req;
+ std::string_view ok_chunked = "GET / HTTP/1.0\n"
+ "Transfer-Encoding: chunked\n"
+ "\n"
+ "10;sha256=715790e8a3b09d704ac9641f42d183a5ebc5fd939663de23da548519ac2165e5\n"
+ R"({"method":"getbl)""\n"
+ " a ; compressed\n"
+ R"(ockcount"})""\n"
+ "0;why;would;anyone;do;this;\n"
+ "Expires: Wed, 21 Oct 2026 07:28:00 GMT\n"
+ "\n";
+ LineReader reader(ok_chunked, MAX_HEADERS_SIZE);
+ BOOST_CHECK(req.LoadControlData(reader));
+ BOOST_CHECK(req.LoadHeaders(reader));
+ BOOST_CHECK(req.LoadBody(reader));
+ BOOST_CHECK_EQUAL(req.m_body, R"({"method":"getblockcount"})");
+ // Chunk Trailer was cleared
+ BOOST_CHECK_EQUAL(reader.Remaining(), 0);
+ }
+ {
+ // Invalid "chunked" transfer, using roman numerals instead of hex for chunk length
+ HTTPRequest req;
+ std::string_view invalid_chunked = "GET / HTTP/1.0\n"
+ "Transfer-Encoding: chunked\n"
+ "\n"
+ "XVI\n"
+ R"({"method":"getbl)""\n"
+ "X\n"
+ R"(ockcount"})""\n"
+ "0\n"
+ "\n";
+ LineReader reader(invalid_chunked, MAX_HEADERS_SIZE);
+ BOOST_CHECK(req.LoadControlData(reader));
+ BOOST_CHECK(req.LoadHeaders(reader));
+ BOOST_CHECK_EXCEPTION(req.LoadBody(reader), std::runtime_error, HasReason{"Cannot parse chunk length value"});
+ }
+ {
+ // Invalid "chunked" transfer, missing chunk termination \n
+ HTTPRequest req;
+ std::string_view invalid_chunked = "GET / HTTP/1.0\n"
+ "Transfer-Encoding: chunked\n"
+ "\n"
+ "10\n"
+ R"({"method":"getbl)"
+ "a\n" // interpreted as extra data at the end of `0x10`-sized chunk
+ R"(ockcount"})"
+ "0\n"
+ "\n";
+ LineReader reader(invalid_chunked, MAX_HEADERS_SIZE);
+ BOOST_CHECK(req.LoadControlData(reader));
+ BOOST_CHECK(req.LoadHeaders(reader));
+ BOOST_CHECK_EXCEPTION(req.LoadBody(reader), std::runtime_error, HasReason{"Improperly terminated chunk"});
+ }
+ {
+ // End of buffer reached without chunk termination, caller must wait for more data to arrive
+ HTTPRequest req;
+ std::string delayed_chunked = "GET / HTTP/1.0\n"
+ "Transfer-Encoding: chunked\n"
+ "\n"
+ "10\n"
+ R"({"method":"getbl)""\n"
+ "a\n"
+ R"(ockcount"})";
+ LineReader reader1(delayed_chunked, MAX_HEADERS_SIZE);
+ BOOST_CHECK(req.LoadControlData(reader1));
+ BOOST_CHECK(req.LoadHeaders(reader1));
+ BOOST_CHECK(!req.LoadBody(reader1));
+ // more data arrives!
+ delayed_chunked += "\n0\n\n";
+ LineReader reader2(delayed_chunked, MAX_HEADERS_SIZE);
+ BOOST_CHECK(req.LoadControlData(reader2));
+ BOOST_CHECK(req.LoadHeaders(reader2));
+ BOOST_CHECK(req.LoadBody(reader2));
+ }
}
BOOST_AUTO_TEST_CASE(http_server_socket_tests)
Why this scored 38/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.