Fix proxy panic when origin returns compressed HTML/CSS responses

[ChristianPavilonis]

Summary

Fixes a pre-existing bug where the first-party proxy panics when the origin server returns compressed (gzip, brotli, zstd) HTML or CSS content. This bug surfaced when mocktioneer was deployed to our Coolify instance, which serves responses with Content-Encoding: zstd or Content-Encoding: br when browsers send Accept-Encoding: gzip, deflate, br, zstd.

Problem

The proxy forwards the client's Accept-Encoding header to the origin server. When a browser requests content, it sends:
Accept-Encoding: gzip, deflate, br, zstd
The origin respects this and returns compressed content. However, finalize_proxied_response calls take_body_str() directly on the response body, which:

1. Assumes the body is valid UTF-8
2. Panics via .expect() when it encounters compressed bytes (which are not valid UTF-8)
   This worked in curl testing (which doesn't send Accept-Encoding by default) but failed 100% of the time in browsers.

Solution

1. Added decompress_body helper - Decompresses response bodies based on Content-Encoding header, supporting gzip, deflate, and brotli.
2. Filter Accept-Encoding header - Instead of forwarding the client's encoding preferences, we now always send Accept-Encoding: gzip, deflate, br to the origin, ensuring we only receive encodings we can handle.
3. Proper error handling - Changed finalize_proxied_response to return Result<Response, ...> and use String::from_utf8() with proper error handling instead of the panicking take_body_str().
4. Strip Content-Encoding from response - Since we decompress before rewriting, the final response is uncompressed (CDN can re-compress for clients).

Future Improvements

• Add zstd support - We could add the zstd crate and handle Content-Encoding: zstd in decompress_body. This would allow the origin to use the most efficient compression.
• Replace take_body_str() usage elsewhere - The Fastly SDK's take_body_str() method calls .expect() internally and will panic on invalid UTF-8. We should audit other usages in the codebase and replace with take_body().into_bytes() + String::from_utf8() for graceful error handling.

@aram356 should we add zstd suppor and replace the panicking functions?

[jevansnyc]

I think in context of this we also need to think about possibilities of running out of memory using take_body_str() for larger responses. This will kill the request and throw a 503

[aram356]

🤔 We have functions to deal with proxying compressed content from publisher domains. See publisher.rs

I think best try to use the existing functions and refactor to be consistent for proxying any content either from creative or publisher domain.

[jevansnyc]

I agree to be consistent and totally fine using existing functions, just not sure I've ever really considered super large pub domain response (I don't see where creative could overflow memory allocation), so more just wanted to clarify that Fastly Compute has a 128MB memory limit by default (but it can be increased), and while there's not much HTML in a single request that would hit this there may be buffering happening elsewhere trying to hold / pull the response together?

[aram356]

Good enough for now if you address the changes. I think there is still opportunity to refactor so please create another ticket to consolidate proxying logic from publisher and creative

[aram356]

🔧 Let's do size check

if self.buffer.len() + chunk.len() > MAX_REWRITABLE_BODY_SIZE 

[aram356]

🔧 Clone is expensive. Do we need all settings?

[aram356]

⛏️ Should be const

[aram356]

⛏️ This code inside if statement and code starting on line 180 looks similar

[aram356]

⛏️ This code inside if statement and code starting on line 206 looks similar