Working on caret based models #445

Activity

1. mattansb commented on May 4, 2022

   

   mattansb

   on May 4, 2022

   Member

   More actions

   Can you give more information, perhaps an example of what you are looking for?

   Non-parametric models (like xgboot) don't really lend themselves to predictor specific effect sizes. There are variable importance measures you might want to look at:
   https://topepo.github.io/caret/variable-importance.html
2. asheetal commented on May 4, 2022

   

   asheetal

   on May 4, 2022

   Author

   More actions

   yes variable importance is part of the solution. However conceptually if one could consider R-square of 100% to be a model that perfectly explains the dependent variable, then an R-square of 10% means 90% is unexplained effect. So hoping for a way to get model agnostic effect distribution of predictors
3. mattansb commented on May 4, 2022

   

   mattansb

   on May 4, 2022

   Member

   More actions

   You can get model-wise diagnostics.

   But predictor-wise is much harder, and in any case would not be model agnostic.

   As an example, here is a simple linear model - we can compare the unique contribution of each predictor by leaving it out and seeing the change in R square. And yet...

   m <- lm(mpg ~ cyl + am + hp,
           data = mtcars)
   
   m_drop_cyl <- update(m, formula. = . ~ . - cyl)
   m_drop_am <- update(m, formula. = . ~ . - am)
   m_drop_hp <- update(m, formula. = . ~ . - hp)
   
   R2_total <- performance::r2(m)[[1]]

   The difference should be the unique contribution of each model

   R2_delta_cyl <- R2_total - performance::r2(m_drop_cyl)[[1]]
   R2_delta_am <- R2_total - performance::r2(m_drop_am)[[1]]
   R2_delta_hp <- R2_total - performance::r2(m_drop_hp)[[1]]
   
   c(R2_total = R2_total,
     Sum_unique_R2 = R2_delta_cyl + R2_delta_am + R2_delta_hp)
   #>      R2_total.R2 Sum_unique_R2.R2 
   #>        0.8041352        0.1306490

   ^{Created on 2022-05-04 by the reprex package (v2.0.1)}

   In more complex models this becomes even more prominent - e.g., in tree based models or KNN, predictors interact in complex ways...
4. easystats

   locked and limited conversation to collaborators on May 4, 2022
5. mattansb

   converted this issue into a discussion #446 on May 4, 2022