Repository navigation
Improve the insert performance in the server #75
Description
We are performing way too many allocations in the StartWritingLogs function in the server log writer file. We need to optimize that so that high bursts of incoming log traffic could be ingested. Things we can try:
- We can move the code in the
BarkLogDao.InsertBatchfunction to theStartWritingLogsfunction - that should avoid the memory copy required for making the function call and the memory allocations happening inside theInsertBatchfunction. - We can allocate a variable for large, medium and small batches. These should be arrays instead of slices so that the loop does not have to reallocate the slice on append.
Activity
itishrishikesh commentedon Oct 26, 2023
working on this
bhagatvansh commentedon Oct 26, 2023
Great, do you wanna collaborate on this? Seems like a good issue to work on.
itishrishikesh commentedon Oct 27, 2023
Sure @bhagatvansh! Created a draft PR of the solution that I thought of, let me know what you think
bhagatvansh commentedon Oct 27, 2023
Sure, did you compare the memory performance?
itishrishikesh commentedon Oct 27, 2023
@bhagatvansh Hadn't done it before, but checked just now, here's what I found,
The right side figures are the code on the main branch, and the left one is what I have written.
One weird thing is, for small batch size when we were creating multiple slices with
appendthe difference between allocations are huge when compared with new code, but for medium and large size logs the difference is negligible, but still it's a little less.I have added a test files, you can run the tests on your end once to verify.