Order of nominal feature levels change the results

[aswer-svg]

Hi,

I am building an EBM regressor with some nominal and continuous features with only main effects. I have tried ordering the nominal features by the mean of the response for better visualization. For example, if a feature has levels named ["cat", "dog", "fish"], the transformed levels would be ["2-cat", "0-dog", "1-fish"], where the number at the start indicate the order with respect to the mean of the response.

With this transformation, the scores of transformed feature levels change a lot. I had a categorical feature whose importance was quite low before the transformation, and it is the most important feature after this transformation.

I didn't specify the feature_types parameter, and the categorical features has type "nominal" when inspecting feature_types_in_.

As far as I know, EBM uses Fischer method for nominal categoricals, so the order shouldn't matter.

Can you please clarify the underlying cause of this discrepancy in the model results?

Take care,

[paulbkoch]

Thanks @aswer-svg -- This is very interesting and sounds like it might be a bug. Can you tell us what version of interpret you are using? Categorical handling is fairly new.

[aswer-svg]

I experienced this issue on 0.7.0, and I'm able to recreate this on 0.7.2.

[paulbkoch]

@aswer-svg -- The smoothing rounds seem to be causing this effect. Fischer is only applied to categoricals during the greedy boosting phase. At the start of fitting an EBM, the current default EBM algorithm executes a fixed number of smoothing rounds where the splitting points in the trees are randomly chosen. This initial smoothing prevents larger jumps in the graphs that would otherwise occur during the initial boosting steps when the shape functions are far away from their final shapes. During that phase the categories in the categorical features are placed in alphanumeric sorted order. I remember deciding to do it this way because we also use the equivalent of smoothing rounds for differentially private EBMs, and sorting the categories by the target would leak private information in those models. There are better things we could do for normal EBMs where privacy leakage isn't an issue, like for example dropping fitting the nominal features during the initial smoothing rounds, or possibly using Fischer on the categoricals while still randomizing the cut points. I'd want to benchmark those choices, so I'm not going to make this change immediately. I wouldn't call the current situation a bug since Fisher is an optimization, but I understand why this is unexpected behavior, so I'll plan to make some kind of change to rectify that. I'll leave this issue open until that is done.

If you'd like to verify that this is the same issue you are observing on your dataset, try setting smoothing_rounds=0 and retry your experiment. I tried this on a synthetic dataset, and the results were invariant to changes in the category order.

[aswer-svg]

Huge thank you for the detailed reply. Indeed, setting smoothing_rounds=0 makes the alphanumeric order irrelevant. I'm looking forward to any changes in this regard. Take care.