Refactor Protobuf-YAML conversion #295
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
One noteworthy case is unknown fields on a nested known message
Handle unknown values directly instead of augmenting the existing protobuf schema. This also fixes the issue of nested messages only displaying known fields, in case an existing .proto was provided.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ehlo! I updated mitmproxy after a long while, and was pleased at the improvements. Especially the ability to include a known
.protofor mitmproxy to use for formatting gRPC messages is very handy. Plus an addon and keybind to invalidate content-views and reload an updated.proto, very nifty.However, during my iterative schema engineering, I noticed that nested partially-defined messages lacked all unknown fields, and started debugging.
Turns out that the schema augmentation process in
raw_to_proto.rsis only done for the root known message, by looking at the unknown fields, defining them and re-parsing the protobuf. Nested messages on those fields will get an empty descriptor that gets augmented by the samecreate_descriptor_proto()as the root message, but nested known messages won't, andmessage.unknown_fields_dyn()gets ignored in those cases.I hacked together a dirty fix, didn't like it, considered the complexity of properly augmenting the nested messages and reparsing again and again... didn't like it, and then ended up with this PR. So:
This PR refactors the Protobuf-YAML conversion, considerably simplifying its logic, and fixes the missing unknown fields on known nested messages.
The previous schema augmentation is ripped out completely. I hope there weren't any further plans for it, but for just displaying protobufs, this is much easier to manage (the reencode parts didn't rely on it). It's noble, but it was beyond me.
I've also added some new tests, though only one of them will fail without the PR. But it's the important one! I've split the tests to their own initial commit for easier checking, but merge however is neat.
All the tests pass, and I've happily used this with mitmproxy proper through
maturin develop.This was my first foray into Rust, it was interesting, educational and went smoothly. I've tried to walk the idiomatic path, but might not have the eye for that yet. Open to any nits, or reconsidering the schema augmentation, since this comes unnannounced.