perf(onpair): speed up compression with onpair 0.2.1 and row training - #9806
Conversation
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_neon[16384] |
13.3 µs | 20.5 µs | -35.01% |
| ⚡ | WallTime | words_gather_dispatch_avx2[1024] |
46 ns | 17 ns | ×2.7 |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing fg/onpair-compression-speedup (5b05483) with develop (a13c246)2
Footnotes
-
218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
develop(9fdaa38) during the generation of this report, so a13c246 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
| #[inline] | ||
| fn row(&self, i: usize) -> &[u8] { | ||
| let view = &self.views[i]; | ||
| if self.lengths[i] == 0 { |
There was a problem hiding this comment.
if len is 0 the view should also be 0, I would personally skip this branch
There was a problem hiding this comment.
lengths[i] is also zero for nulls, but masking can leave their views nonempty
robert3005
left a comment
robert3005
left a comment
There was a problem hiding this comment.
just the single comment
| flat.extend_from_slice(bytes); | ||
| offsets.push(u64::try_from(flat.len()).vortex_expect("offset must fit in u64")); | ||
| uncompressed_lengths | ||
| .push(i32::try_from(view.len()).vortex_expect("must fit in i32")); |
There was a problem hiding this comment.
this cannot fail, use debug assert
| offsets.push(u64::try_from(flat.len()).vortex_expect("offset must fit in u64")); | ||
| uncompressed_lengths | ||
| .push(i32::try_from(view.len()).vortex_expect("must fit in i32")); | ||
| total_bytes += view.len() as usize; |
| struct ViewRows<'a> { | ||
| views: &'a [BinaryView], | ||
| buffers: &'a [&'a ByteBuffer], | ||
| lengths: &'a [i32], |
| @@ -61,32 +55,32 @@ pub fn onpair_compress( | |||
| match mask.bit_buffer() { | |||
| AllOr::All => { | |||
| for view in views { | |||
There was a problem hiding this comment.
zip views with uncompressed_lengths
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
3 participants
Summary
Speed up OnPair compression by upgrading to onpair 0.2.1 and training directly from Vortex string views, avoiding the full input copy and temporary offsets buffer.
Changes