Skip to content

Conversation

@gmalette
Copy link

Similarly to msgpack#387 please don't use this code!

In prior versions, I had implemented reference tracking by having a proper Reference type. Roughly speaking, something like:

Reference = Struct.new(:value)

factory.register_type(
  0x22,
  Reference,
  packer: -> (obj, packer) {
    if (id = packer.id(obj))
      packer.write(id)
    else
      packer.track(obj)
      packer.write(nil)
      packer.write(obj.value)
    end
  },
  unpacker: -> (unpacker) {
    if (id = unpacker.read)
      unpacker.references(id)
    else
      unpacker.track(unpacker.read)
    end
  },
)

The main issue with this approach was the amount of short-lived Reference objects required only for serialization.

To avoid these, I moved the handling of the references directly in how each type is encoded (as seen in the benchmark). This is a Light Abuse™️ of messagepack but saves so much space and can properly deserialize entire graphs of objects.

When registering an extension type with ref_tracking: true, repeated
objects are serialized as back-references instead of being re-encoded.
This reduces payload size when the same object appears multiple times.

Uses ext type 127 with a compact wire format:
- New reference: nil marker followed by the serialized object
- Back-reference: positive integer ref_id pointing to earlier object
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant