Sparse modeling is a powerful approach in machine learning and statistics where models are encouraged to use only a subset of the available features. This is particularly useful when dealing with high-dimensional data, where many features may be irrelevant or redundant. Two commonly used regularization techniques in sparse modeling are L1 norm and L2 norm, which penalize the size of the model's coefficients and encourage sparsity or smoothness, respectively.
What is a Norm?
A norm is a mathematical concept used to measure the size or magnitude of a vector. In the context of machine learning, norms are used to constrain the magnitude of model parameters to prevent overfitting and encourage sparsity.
L1 Norm (Manhattan Norm)
The L1 norm is the sum of the absolute values of the vector’s components. It is also known as the Manhattan norm because it measures distance by summing the absolute differences along each axis.
\|x\|_1 = \sum_{i=1}^{n} |x_i|
L1 Regularization (Lasso)
L1 regularization, also known as Lasso (Least Absolute Shrinkage and Selection Operator), adds a penalty term proportional to the L1 norm of the coefficients to the objective function.
\text{Objective: } \frac{1}{2n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^{p} |w_j|
Effect of L1 Norm
- Encourages sparsity by shrinking some coefficients to zero.
- Selects relevant features, making it useful for feature selection.
- Useful in high-dimensional datasets where many features may be irrelevant.
L2 Norm (Euclidean Norm)
The L2 norm is the square root of the sum of the squared components of a vector. It is also known as the Euclidean norm because it measures the straight-line distance between points.
\|x\|_2 = \sqrt{\sum_{i=1}^{n} x_i^2}
L2 Regularization (Ridge)
L2 regularization, also known as Ridge regression, adds a penalty term proportional to the square of the L2 norm of the coefficients to the objective function.
\text{Objective: } \frac{1}{2n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^{p} w_j^2
Effect of L2 Norm:
- Shrinks coefficients towards zero but does not set them to zero.
- Penalizes large coefficients more than smaller ones.
- Encourages smoothness and prevents overfitting by controlling model complexity.
Key Differences Between L1 and L2 Norms
Feature | L1 Norm (Lasso) | L2 Norm (Ridge) |
|---|---|---|
Effect on Coefficients | Produces sparse models | Shrinks coefficients uniformly |
Feature Selection | Selects relevant features | Keeps all features but reduces magnitude |
Sparsity | High sparsity (many zeros) | No sparsity |
Computational Efficiency | Faster with sparse solutions | Slower due to non-zero coefficients |
Usage | Ideal for feature selection | Ideal when all features contribute |
Combination: Elastic Net
To combine the benefits of both L1 and L2 norms, Elastic Net is often used. It adds both L1 and L2 penalties to the objective function:
\text{Objective: } \frac{1}{2n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 + \lambda_1 \sum_{j=1}^{p} |w_j| + \lambda_2 \sum_{j=1}^{p} w_j^2
Benefits of Elastic Net
- Handles correlated features better than Lasso.
- Provides feature selection and coefficient shrinkage.
When to Use L1 and L2 Regularization?
Use L1 (Lasso) when:
- Feature selection is desired.
- You expect many irrelevant features.
Use L2 (Ridge) when:
- You want to retain all features but control large coefficients.
- Multicollinearity is a concern.
Challenges and Considerations
- Lasso Limitation: Can randomly select one of the correlated features.
- Ridge Limitation: Does not perform feature selection.
- Elastic Net Trade-off: Requires careful tuning of hyperparameters to balance L1 and L2 penalties.
