JIT: Let cold blocks participate in layout opts [experiment]

[BoyBaykiller]

#129275 (comment)

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[jakobbotsch]

The diffs show more or less what I would expect -- size improvements at the cost of moving 0 weight blocks inline and polluting the icache. It seems unlikely to be a good change.

[AndyAyersMS]

Copying what I wrote over on the discord...

There is an argument to be made that letting a little bit of cold code into the hot section could be good, if it actually makes the hot section more compact. EG a long branch around a simple one instruction cold block. But we don't have accurate metrics for what the JIT considers the hot and cold sections today: we put all the code into the hot section (no splitting). It might be worth introducing a metric for unsplit codegen to see if the "true" hot section is getting smaller or what we're mostly seeing is the cold section getting smaller.

I don't think this effect is easy to model with our current layout approach since we don't know the sizes of things or how span-dependent branch encodings work when we do the layouts. I know @amanasifkhalid and I discussed trying to model this but never did because we couldn't quite get a handle on it.

The long-range plan here is to move away from having so many blocks with zero profile weight. We allow a small amount of residual (probably synthetic) profile to wash over all blocks, and then the cold section possibly gets plausible layout, but we still would miss the opportunistic cold-in-hot cases without some kind of modelling.

Also someday, actual splitting... I think we had a plausible approach lined up where we put the cold code section at the far end of the code heap (odd things happen if the cold code can end up at a lower address). Perhaps worth dusting this off. I think we still stress test this setup, so maybe not even bit rotted.