Regularization

• In Machine Learning, to prevent overfitting in regression models, we can reduce the impact of each feature by using regularization techniques.

  • Ridge Regression is a type of L2 regularization that adjusts the model for linear models that are not responsive.
    • This is done by manipulating the Alpha value.
      • A higher Alpha value imposes more constraints on the coefficients, leading to generalization.
      • As Alpha approaches 0, the regularization effect weakens.
      • The Learning Curve shows that if there is a lot of data, regularization is not very relevant.
  • Lasso Regression (L1 regularization) can result in some coefficients becoming exactly 0, making it a more extreme form of regularization.

• Choosing between Ridge and Lasso:

  • Start with Ridge.
  • If it is expected that most of the features are not meaningful, try Lasso.
  • Also try Lasso when a simpler model is desired.

• There is also ElasticNet, which combines the benefits of both Ridge and Lasso.

  • It is practically useful, but requires tuning two parameters.

• In classification problems, the effect is as shown in the graph below:

  • The degree of regularization is determined by C.
    • A higher C value weakens the regularization (opposite of Alpha).
    • In other words, weaker regularization leads to overfitting, making the model more sensitive to small details (as shown in the third graph). #Getting Started with Machine Learning in Python