Memory leak during `Response` forwarding

[sebadob]

Version

v1.9.0

Platform

Linux 6.19.14-200.fc43.x86_64

Summary

I am not sure if this is actually a leak, or if I screwed up and I am missing something.

The situation: I started to create a reverse proxy based on hyper. I am now at a point where I started to do some benchmarks and memory profiling to evaluate the whole thing and get an idea of whats happening in detail. In a specific scenario, when I am forwarding a Response<_> that I got from upstream, the Parts are leaking memory and I am not getting the memory back even after the client received the complete and correct response.

This happens when I accept HTTP/2 connections that are then forwarded to an HTTP/1.1 upstream. When I receive the h1 response from upstream, and I simply forward the Response<_> (after custom business logic, removing hop-by-hop headers, and so on ...), The size of the HeaderMap is leaked in memory. I can resolve this immediately when I parts.headers.drain() after the resp.into_parts(), and force a new allocation for each value with HeaderValue::from_bytes(value.as_bytes())?. I can forward the HeaderName as-is, but if I don't allocate a fresh HeaderValue for each entry, memory is leaked.

It also does not happen for any other combination, like:

• h2 listener with h2 upstream
• h1 listener with h1 upstream
• h1 listener with h2 upstream

All of these are fine. The leaking one is only h2 listener with h1 upstream.

So my current "fix" after I got a response from upstream is something like this:

if upstream.protocol == UpstreamProtocol::Http11 && ctx.protocol == Protocol::Http2 {
    let (mut parts, body) = resp.into_parts();

    let mut resp = Response::builder().status(parts.status).body(body)?;
    let headers = resp.headers_mut();
    headers.reserve(parts.headers.len());

    for (name, value) in parts.headers.drain() {
        let Some(name) = name else { continue };
        let owned = HeaderValue::from_bytes(value.as_bytes())?;
        headers.insert(name, owned);
    }

    Ok(resp)
} else {
    Ok(resp)
}

This fixes the leak immediately, but it would be nice to not need this additional allocation for each header value, and it's definitely very unexpected (for me at least). Unfortunately, the project is not ready yet to be open-souced and the full code is not publicly available yet. But, after lots of debugging and profiling, and after I thought initially that the body stream would be the issue, I found this one.

So the question now is: Did I screw up and miss something, and is this to be expected, or is this actually a bug?

Code Sample

Expected Behavior

Expected behavior would be that "h2 proxy -> h1 upstream" behaves the same as the other 3 combinations.

Actual Behavior

"h2 proxy -> h1 upstream" and then forwarding the h1 upstream response to the original client leaks memory without re-allocating all HeaderValues.

Additional Context

No response

[seanmonstar]

The size of the HeaderMap is leaked in memory.

Could you clarify a little here? Do you mean hashmap-like memory of the HeaderMap, or the bytes memory of the underlying names and values, or something else?

[sebadob]

What I am doing is basically taking the Response like I get it from the hyper Client, and forwarding it as-is. I guess it is about the Bytes part, because only the bytes-backed HeaderValue is causing this. What I now also found out is this leak seems to be bound somehow to the underlying h2 remote connection. When this is closed, the memory seems to be freed up again. So it's most probably not a "real" leak. There is just something piling up, even though I am not sure where exactly.

I tried to dig a bit deeper with heaptrack, and it shows the memory pile-up in the actual connection handler for the client.
I only moved the anonymous task into a named function for debugging. In the following spawned task, heaptrack shows all the memory allocations:

let (sender, conn) = client::conn::http1::handshake(io).await?;
let conn = Box::new(conn);

task::spawn(async move {
    if let Err(err) = conn.await {
        debug!("upstream h1 conn err: {:?}", err);
    }
});

This would also make a lot of sense to me, since the Bytes are allocated in the client part. It seems that after the h2 connection on the public-facing side gets them the reference is never dropped. Could that be the case?

[0x676e67]

This patch may have already fixed the issue you described. Although it hasn't been released yet, would you like to give it a try? #4042

Okay, I don't quite understand your question.

[sebadob]

This patch may have already fixed the issue you described. Although it hasn't been released yet, would you like to give it a try? #4042

I just tested against main, and unfortunately, it is the exact same behavior.

Just to give you some numbers from a very minimal benchmarking / comparison example, and I do 10_000_000 requests against my proxy:

When I do the header re-allocation like shown above before I am forading an h1 upstram requests to h2 downstream, the application never exceeds 26MiB and stays very consistently there. When I remove the HeaderValue re-allocation, after 10_000_000 requests it is already at 75MiB, and it only goes up with each single request. This happens even with the most minimal response headers you can have, which are only the date and a Content-Length: 1. When I return a typical number of response headers, I am very quickly in the GB range for consumed memory.

Is there any other help / data I can provide, apart from providing the whole code? If I have the time, I could maybe set up a very minimal, fully-working test example.

[0x676e67]

You need to provide a complete example; otherwise, no one will know what you're talking about.

[sebadob]

I created a very minimal example that produces exactly the mentioned behavior:

https://github.com/sebadob/hyper-mem-leak

[seanmonstar]

No, I understand the issue. I think what you describe does happen. I mean, if I correctly understand this being about HTTP/2. HeaderValues will get stored in the connection's dynamic table (I think, it's been a while since I looked). If I'm right, then yea, the bytes backing that value would live until the connection dropped, or the value was evicted from the dynamic table.

[sebadob]

Yeah it feels like that. With heaptrack I can see the memory being freed up again after the connection is dropped. The only thing that made me curious is that it does not behave this way when the upstream is H2C / HTTP2.

For all combinations, in my testing setup the proxy will consume ~20-25mb of memory during testing, apart from this specific one. So, if it stores the Bytes, why is it different when the upstream uses a different protocol? I mean, the HeaderValues are just the same Bytes under the hood, right? Only that memory does not pile up in that situation.

[0x676e67]

To be honest, I haven't been able to reproduce this issue in Ubuntu 26. It seems unbelievable—could it be a problem with the system's memory allocator? Memory usage is slightly lower when using realloc.

• cargo run -r -- realloc

Memory 34MB

• cargo run -r

Memory 45MB

[0x676e67]

To be honest, I haven't been able to reproduce this issue in Ubuntu 26. It seems unbelievable—could it be a problem with the system's memory allocator? Memory usage is slightly lower when using realloc

I mean, is it impossible to reproduce GB-level memory usage in these examples? Or is the difference only a few dozen MB?

[sebadob]

Memory fragmentation was my first guess as well, so I compared malloc and jemalloc, and also jemalloc with only a single arena. They all behave the same. I am running Fedora right now. This tiny example code produces quite a bit lower numbers than my "real" application, now sure why atm.

I ran the tests again, and heaptracked both versions. The h1 version shows the expected behavior. However, the h2 version seems to leak as well, just a lot less and slower, and in smaller steps. I did not even notice it before. I guess it was because it's kind of hidden by the allocator when you don't run it for long enough, or you just use OS tools to look at the total allocated memory instead of the actually used portions of it. The allocator requests areas in chunks from the OS. I guess that's what hid the h2 leakage from me before.

I ran both tests for ~6.5 minutes now. The heaptrack UI is partly german (sorry), but I guess you get it anyway. Having all of the other stuff from my app not shadowing anything makes the h1 graph look even a bit more weird. It has the pretty rapid increase at the beginning (probably full allocator requests from the OS), and then at some point it suddenly slows down and kind of looks comparable to the h2 version, just with quite a bit more memory.

The reasons for both leaks even show different sources. The h1 version shows BytesMut::reverse_inner() as the big part, while for the h2 version it's about the HeaderMap.

Without realloc -> h1 upstream

With realloc -> h2 upstream

With realloc for 15 minutes

Because I wanted to see if h2 upstreams kind of show this breakpoint like we can see it for the h1 version, I ran the test with realloc again for 15 minutes. I also increased the concurrency from 100 to 500 connections, trying to amplify the behavior a bit. Because heaptrack makes everything super slow, it was only 13.5 mil requests over this time. The graphs would rise a lot faster without heap tracking.

Apart from some weird spikes in the memory allocation, memory usage is consistently increasing. After the initial jump (oha does some kind of warmup, probably when it really started), total usage increased only by 10mb over 15minutes withn 15k req/s.

The h1 breakpoint

After my last test when I did an additional screenshot with the legend shown for usage over time, I saw what caused this breakpoint for h1 upstream (no realloc). It's entirely about the BytesMut for whatever reason. It first increases rapidly, and then it stays completely consistent over the whole duration. From that moment on, we see the same rise over time as for h2 upstreams (with realloc), where only HeaderMap is leaking something, though it's a pretty low amount.

So I guess we have 2 different things here:

1. The BytesMut for h1 upstreams. Increases a lot and pretty quickly, but then stays consistent. The overall usage is pretty high in comparison, but at least it has a ceiling. This is tied to the number of total connections in my other testing.
2. The HeaderMap leakage. This is the same for both variants. I just did not notice it earilier, because it's leaking a lot slower, but it seems it doesn't stop.

---

So I am not sure if this is to be expected, or if we can do something about it. I did not have the time yet to take a look at the code of hyper and h2, but maybe I can help out in the future.

[sebadob]

Because I was curious, I also did another very quick test with h1 only, so h1 proxy -> h1 upstream. The BytesMut issue is not showing up, as expected, but the HeaderMap increase with the try_reserve_one() over time seems to be a thing here as well, which I did not notice before:

[sebadob]

Sorry for the spam, this is the last one, I promise.

For comparison, I did track direct h1 and h2 requests without any proxying or upstream reqeusts. Only acting as a "normal" webserver. Everything looks perfectly fine:

h1

h2

---

But, here's the catch. When I do the upstream request, and then even fully ignore it, and build a completely independent response, we see the HeaderMap leak again. So it's probably nothing in the server, but in the client part?