fm: make sp_ereport_ingester status output less gaslighty#10382
fm: make sp_ereport_ingester status output less gaslighty#10382
sp_ereport_ingester status output less gaslighty#10382Conversation
Originally, when implementing the `sp_ereport_ingester` background task, I had the foolish and bad idea to make the task's status output omit entries for SPs which we communicated with successfully but which had no new ereports, thinking that it was "not interesting" to list them. Unfortunately, this also means that the total count of SPs that the task talked to, and the count of total HTTP requests sent, will also not count any SPs which had no ereports. This behavior is *incredibly misleading* to anyone who isn't aware that they are being intentionally excluded, which, as it turns out, includes me (since I had forgotten that I did this). This commit fixes this by not going out of our way to exclude these SPs from the status output. It turns out that this actually makes the code somewhat simpler, which is another argument in favor of the idea that I never should have done this. Sigh. Fixes #10380
hawkw
added
Debugging
As far as I can tell, what happened here is that one of the simulated SP's ereport listeners in this test received a UDP packet from...something, and it was not the correct length to be an ereport request header, so the simulated SP task panicked. This is odd to me because the ereport ingester task isn't even enabled in this test, so I'm really not sure where that UDP packet came from. It was on localhost port 47491, but there is no mention of that port in any of the logs, and it could presumably have been anything. I think maybe the solution is just to have the simulated SPs not panic upon receipt of invalid ereport requests. but...it's really weird that this happened and I'd like to understand where it came from. I'd feel a bit gross about not having the test fail if MGS actually sends a malformed request, but I think that's something that other testing (such as, say, the ones where we actually try to collect ereports). I just want to know where the random UDP packet came from. |
|
Okay, so I am fairly confident that the test flake is not in any way related to this change, and we are just unlucky enough to have hit a vaguely ereport-related test flake in a branch that makes changes that are also vaguely ereport-related. I've opened #10387 for the test flake and will re-run CI. |
1 participant
Originally, when implementing the
sp_ereport_ingesterbackground task, I had the foolish and bad idea to make the task's status output omit entries for SPs which we communicated with successfully but which had no new ereports, thinking that it was "not interesting" to list them. Unfortunately, this also means that the total count of SPs that the task talked to, and the count of total HTTP requests sent, will also not count any SPs which had no ereports. This behavior is incredibly misleading to anyone who isn't aware that they are being intentionally excluded, which, as it turns out, includes me (since I had forgotten that I did this).This commit fixes this by not going out of our way to exclude these SPs from the status output. It turns out that this actually makes the code somewhat simpler, which is another argument in favor of the idea that I never should have done this. Sigh.
Fixes #10380