Predictive Analytics

9.

Model Evaluation and Validation

9.1.

Data Splitting Strategies

9.1.1.

Hold-out Validation

9.1.1.1.

Training Set Size

9.1.1.2.

Validation Set Purpose

9.1.1.3.

Test Set Independence

9.1.2.

Time-based Splitting

9.1.2.1.

Temporal Validation

9.1.2.2.

Walk-forward Analysis

9.1.2.3.

Expanding Window

9.1.2.4.

9.1.3.

Stratified Sampling

9.1.3.1.

Class Balance Preservation

9.1.3.2.

Stratification Variables

9.2.

Cross-Validation Techniques

9.2.1.

K-Fold Cross-Validation

9.2.1.1.

9.2.1.2.

Variance Estimation

9.2.1.3.

Computational Considerations

9.2.2.

Stratified K-Fold

9.2.2.1.

Class Distribution Maintenance

9.2.2.2.

Imbalanced Data Handling

9.2.3.

Leave-One-Out Cross-Validation

9.2.3.1.

Bias-Variance Properties

9.2.3.2.

Computational Complexity

9.2.4.

Time Series Cross-Validation

9.2.4.1.

Temporal Dependencies

9.2.4.2.

Forecast Horizon Considerations

9.2.5.

Nested Cross-Validation

9.2.5.1.

Model Selection and Assessment

9.2.5.2.

Hyperparameter Tuning

9.3.

Regression Evaluation Metrics

9.3.1.

Error-based Metrics

9.3.1.1.

Mean Absolute Error

9.3.1.2.

Mean Squared Error

9.3.1.3.

Root Mean Squared Error

9.3.1.4.

Mean Absolute Percentage Error

9.3.2.

Relative Metrics

9.3.2.1.

9.3.2.2.

Adjusted R-squared

9.3.2.3.

Mean Absolute Scaled Error

9.3.3.

Distribution-based Metrics

9.3.3.1.

9.3.3.2.

9.4.

Classification Evaluation Metrics

9.4.1.

Confusion Matrix Analysis

9.4.1.1.

True Positives and Negatives

9.4.1.2.

False Positives and Negatives

9.4.1.3.

9.4.2.

Single-value Metrics

9.4.2.1.

9.4.2.2.

9.4.2.3.

9.4.2.4.

9.4.2.5.

9.4.3.

Threshold-dependent Analysis

9.4.3.1.

9.4.3.2.

Precision-Recall Curve

9.4.3.3.

Area Under Curve

9.4.4.

Multi-class Extensions

9.4.4.1.

Macro Averaging

9.4.4.2.

Micro Averaging

9.4.4.3.

Weighted Averaging

9.4.5.

Class Imbalance Considerations

9.4.5.1.

Balanced Accuracy

9.4.5.2.

Matthews Correlation Coefficient

9.4.5.3.

9.5.

Model Comparison and Selection

9.5.1.

Statistical Significance Testing

9.5.1.1.

9.5.1.2.

9.5.1.3.

Wilcoxon Signed-rank Test

9.5.2.

Information Criteria

9.5.2.1.

Akaike Information Criterion

9.5.2.2.

Bayesian Information Criterion

9.5.2.3.

Cross-validation Information Criterion

9.5.3.

Hyperparameter Optimization

9.5.3.1.

9.5.3.2.

9.5.3.3.

Bayesian Optimization

9.5.3.4.

Evolutionary Algorithms

9.6.

Bias-Variance Analysis

9.6.1.

Bias-Variance Decomposition

9.6.2.

Overfitting Detection

9.6.2.1.

Learning Curves

9.6.2.2.

Validation Curves

9.6.2.3.

Complexity Analysis

9.6.3.

Underfitting Identification

9.6.3.1.

Model Capacity Assessment

9.6.3.2.

Feature Adequacy

9.6.4.

Regularization Effects

9.6.4.1.

Regularization Paths

9.6.4.2.

Cross-validation for Regularization

Previous

8. Unsupervised Learning

Go to top

Next

10. Model Interpretability and Explainability