GradientBoosting / Regressor Layer

Loss function selection:

Mathematical forms:

1. Squared Error: $$L(y,F)=\frac{1}{2}(y-F{)}^2$$
2. Absolute Error: $$L(y,F)=|y-F|$$
3. Huber: Combines L1 and L2
4. Quantile: $$L(y,F)=\tau (y-F{)}_{+}+(1-\tau )(F-y{)}_{+}$$

Impact:

• Optimization behavior
• Robustness properties
• Prediction characteristics
• Training dynamics

The number of boosting stages to perform. Gradient boosting is fairly robust to over-fitting so a large number usually results in better performance.

Impact: $$F_m=F_{m-1}+\nu h_m$$ where ν is learning rate

Trade-off:

• Small ν: More trees needed
• Large ν: Fewer trees

Typical ranges:

• Small: 0.01-0.1
• Large: 0.1-0.3

Number of boosting stages:

Selection guide:

1. With small learning rate:

   • More estimators needed
   • Better generalization

2. With large learning rate:

   • Fewer estimators
   • Faster training

Common ranges:

• Basic: 100-500
• Advanced: 500-2000

The fraction of samples to be used for fitting the individual base learners.

Effects:

• <1.0: Stochastic boosting
• 1.0: Traditional boosting

Benefits of subsampling:

• Reduces variance
• Prevents overfitting
• Faster training

Typical values:

• 0.5-0.8: More randomization
• 0.8-1.0: More stability

Split quality measures:

Options:

1. Friedman MSE:

   • Enhanced measurement
   • Improvement scoring
   • Better splits

2. Standard MSE:

   • Simple variance reduction
   • Direct measurement
   • Basic approach

Impact:

• Tree structure
• Split selection
• Learning quality

Minimum samples for split:

Effects:

• Controls tree growth
• Prevents overfitting
• Ensures stability

Typical values:

• 2-10: Detailed trees
• 10-50: More stable

Minimum samples per leaf:

Impact:

• Prediction stability
• Smoothing effect
• Overfitting control

Common settings:

• 1-5: Fine detail
• 5-20: Smoother predictions

Minimum weighted leaf fraction:

Purpose:

• Weight-based control
• Balanced trees
• Handling importance

Range: [0.0, 0.5]

• 0.0: No constraint
• >0.0: Enforces balance

Maximum tree depth:

Control options:

• 0: Unlimited growth
• >0: Fixed depth limit

Guidelines:

• Shallow (3-5): Fast, robust
• Medium (5-8): Balanced
• Deep (>8): Complex patterns

Note: Key parameter for boosting

A node will be split if this split induces a decrease of the impurity greater than or equal to this value.

Usage:

• Pre-pruning method
• Quality control
• Prevents weak splits

Values:

• 0.0: All splits allowed
• >0.0: Quality threshold

Controls the random seed given to each Tree estimator at each boosting iteration. In addition, it controls the random permutation of the features at each split.

Affects:

• Subsampling
• Feature selection
• Tree structure

Settings:

• 0: System random
• Fixed: Reproducible
• Different: Variations

Feature subset selection:

Strategies:

1. All features (Auto)
2. Square root scaling
3. Logarithmic scaling
4. Custom selection

Effects:

• Split randomization
• Training speed
• Model variance
• Feature exploration

Custom feature count:

Usage:

• With MaxFeatures=Custom
• Must be ≤ total features

Selection:

• Small: More randomization
• Large: Better splits
• Balance performance

The alpha-quantile of the huber loss function and the quantile loss function. Only if loss is huber or quantile.

Maximum leaf node count:

Control:

• 0: Unlimited leaves
• >0: Best-first growth

Benefits:

• Memory control
• Tree size limit
• Quality-based growth

Alternative to max_depth

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

When True:

• Keep existing trees
• Add more stages
• Continue training

Applications:

• Model updating
• Iterative fitting
• Parameter tuning

Early stopping validation size:

Usage:

• Monitors convergence
• Prevents overfitting
• Automatic stopping

Typical values:

• Small (0.1): More training
• Large (0.2): Better validation

Early stopping patience:

Behavior:

• 0: No early stopping
• >0: Stop after n stagnant rounds

Settings:

• Small (5-10): Aggressive
• Large (10-20): Patient

Used with validation_fraction

Early stopping tolerance:

Criterion: $$|scor{e}_t-scor{e}_{t-1}|<tol$$

Values:

• Strict (1e-4): Precise
• Loose (1e-3): Faster

Used with n_iter_no_change

Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ccp_alpha will be chosen.

Loss function selection:

Mathematical forms:

1. Squared Error: $$L(y,F)=\frac{1}{2}(y-F{)}^2$$
2. Absolute Error: $$L(y,F)=|y-F|$$
3. Huber: Combines L1 and L2
4. Quantile: $$L(y,F)=\tau (y-F{)}_{+}+(1-\tau )(F-y{)}_{+}$$

Impact:

• Optimization behavior
• Robustness properties
• Prediction characteristics
• Training dynamics

Learning rate search:

Search spaces:

1. Log scale:

   • [0.001, 0.01, 0.1]
   • Wide exploration

2. Fine-tuning:

   • [0.05, 0.1, 0.15]
   • Around optimal

Note: Pair with n_estimators

Number of trees search:

Search ranges:

1. Quick exploration:

   • [50, 100, 200]
   • Initial testing

2. Production:

   • [200, 500, 1000]
   • Full performance

3. Large scale:

   • [500, 1000, 2000]
   • High accuracy

Subsample ratio search:

Patterns:

1. Conservative:

   • [0.8, 0.9, 1.0]
   • Stable learning

2. Aggressive:

   • [0.5, 0.7, 0.9]
   • More randomization

3. Comprehensive:

   • [0.6, 0.8, 1.0]
   • Full range

Split quality measures:

Options:

1. Friedman MSE:

   • Enhanced measurement
   • Improvement scoring
   • Better splits

2. Standard MSE:

   • Simple variance reduction
   • Direct measurement
   • Basic approach

Impact:

• Tree structure
• Split selection
• Learning quality

Split threshold search:

Search ranges:

1. Fine grain:

   • [2, 5, 10]
   • Detailed splits

2. Conservative:

   • [10, 20, 50]
   • Stable trees

3. Mixed: [2, 10, 30]

   • Range comparison

Leaf size search:

Patterns:

1. Detailed:

   • [1, 3, 5]
   • Fine predictions

2. Stable:

   • [5, 10, 20]
   • Robust leaves

3. Study: [1, 5, 15]

   • Effect analysis

Weight constraint search:

Search spaces:

1. No constraint: [0.0]

   • Default behavior

2. Light balance:

   • [0.0, 0.1, 0.2]
   • Gentle weighting

3. Strong balance:

   • [0.2, 0.3, 0.4]
   • Heavy constraints

Tree depth search:

Ranges:

1. Shallow trees:

   • [3, 4, 5]
   • Fast, robust

2. Deep trees:

   • [6, 8, 10]
   • Complex patterns

3. Full range:

   • [3, 6, 9]
   • Depth impact

Split quality threshold search:

Search patterns:

1. All splits: [0.0]

   • No filtering

2. Light pruning:

   • [0.0, 1e-4, 1e-3]
   • Gentle filtering

3. Heavy pruning:

   • [1e-3, 1e-2, 1e-1]
   • Strong filtering

Feature subset selection:

Strategies:

1. All features (Auto)
2. Square root scaling
3. Logarithmic scaling
4. Custom selection

Effects:

• Split randomization
• Training speed
• Model variance
• Feature exploration

Custom feature count search:

Ranges:

1. Small sets:

   • [2, 4, 6]
   • Fast splits

2. Large sets:

   • [8, 12, 16]
   • More features

Note: Used with Custom max_features

Quantile/Huber parameter search:

Search spaces:

1. Quantile loss:

   • [0.1, 0.5, 0.9]
   • Different quantiles

2. Huber loss:

   • [0.7, 0.9, 0.95]
   • Robustness levels

Note: Loss-specific parameter

Maximum leaves search:

Search patterns:

1. Small trees:

   • [10, 20, 30]
   • Simple models

2. Large trees:

   • [50, 100, 200]
   • Complex models

3. Mixed:

   • [0, 31, 63]
   • Compare unlimited

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

Options:

1. Standard: [false]

   • Fresh training
   • Independent models

2. Compare: [false, true]

   • Study reuse impact
   • Training efficiency

Early stopping validation size:

Guidelines:

1. Small datasets:

   • 0.2 fraction
   • More validation

2. Large datasets:

   • 0.1 fraction
   • More training

Note: Used with early stopping

Early stopping patience:

Settings:

1. No early stop: 0

   • Full iterations
   • Complete training

2. With early stop: >0

   • Monitors improvement
   • Prevents overfitting

Used with validation_fraction

Improvement tolerance:

Usage:

• Stops if score gain < tol
• Over n_iter_no_change rounds

Values:

• Strict: 1e-4 (default)
• Relaxed: 1e-3
• Tight: 1e-5

Cost-complexity search:

Search spaces:

1. No pruning: [0.0]

   • Full trees
   • Maximum detail

2. Light pruning:

   • [0.001, 0.01, 0.1]
   • Gentle simplification

3. Heavy pruning:

   • [0.1, 0.2, 0.3]
   • Strong simplification

Regression model evaluation metrics:

Purpose:

• Model performance evaluation
• Error measurement
• Quality assessment
• Model comparison

Selection criteria:

• Error distribution
• Scale sensitivity
• Domain requirements
• Business objectives

Random seed for reproducible splits. Ensures:

• Consistent train/test sets
• Reproducible experiments
• Comparable model evaluations

Same seed guarantees identical splits across runs.

Data shuffling before splitting. Effects:

• true: Randomizes order, better for i.i.d. data
• false: Maintains order, important for time series

When to disable:

• Time dependent data
• Sequential patterns
• Grouped observations

Proportion of data for training. Considerations:

• Larger (e.g., 0.8-0.9): Better model learning
• Smaller (e.g., 0.5-0.7): Better validation

Common splits:

• 0.8: Standard (80/20 split)
• 0.7: More validation emphasis
• 0.9: More training emphasis

Maintain class distribution in splits. Important when:

• Classes are imbalanced
• Small classes present
• Representative splits needed

Requirements:

• Classification tasks only
• Cannot use with shuffle=false
• Sufficient samples per class

Number of cross-validation folds. Guidelines:

• 3-5: Large datasets, faster training
• 5-10: Standard choice, good balance
• 10+: Small datasets, thorough evaluation

Trade-offs:

• More folds: Better evaluation, slower training
• Fewer folds: Faster training, higher variance

Must be at least 2.

Number of folds for cross-validation. Selection guide: Recommended values:

• 5: Standard choice (default)
• 3: Large datasets/quick evaluation
• 10: Thorough evaluation/smaller datasets

Trade-offs:

• Higher values: More thorough, computationally expensive
• Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

Random seed for fold generation when shuffling. Important for:

• Reproducible results
• Consistent fold assignments
• Benchmark comparisons
• Debugging and validation

Set specific value for reproducibility across runs.

Whether to shuffle data before splitting into folds. Effects:

• true: Randomized fold composition (recommended)
• false: Sequential splitting

Enable when:

• Data may have ordering
• Better fold independence needed

Disable for:

• Time series data
• Ordered observations

Number of stratified folds. Guidelines: Typical values:

• 5: Standard for most cases
• 3: Quick evaluation/large datasets
• 10: Detailed evaluation/smaller datasets

Considerations:

• Must allow sufficient samples per class per fold
• Balance between stability and computation time
• Consider smallest class size when choosing

Seed for reproducible stratified splits. Ensures:

• Consistent fold assignments
• Reproducible results
• Comparable experiments
• Systematic validation

Fixed seed guarantees identical stratified splits.

Data shuffling before stratified splitting. Impact:

• true: Randomizes while maintaining stratification
• false: Maintains data order within strata

Use cases:

• true: Independent observations
• false: Grouped or sequential data

Class proportions maintained regardless of setting.

Number of random splits to perform. Consider: Common values:

• 5: Standard evaluation
• 10: More thorough assessment
• 3: Quick estimates

Trade-offs:

• More splits: Better estimation, longer runtime
• Fewer splits: Faster, less stable estimates

Balance between computation and stability.

Random seed for reproducible shuffling. Controls:

• Split randomization
• Sample selection
• Result reproducibility

Important for:

• Debugging
• Comparative studies
• Result verification

Proportion of samples for test set. Guidelines: Common ratios:

• 0.2: Standard (80/20 split)
• 0.25: More validation emphasis
• 0.1: More training data

Considerations:

• Dataset size
• Model complexity
• Validation requirements

It must be between 0.0 and 1.0.

Number of stratified random splits. Guidelines: Recommended values:

• 5: Standard evaluation
• 10: Detailed analysis
• 3: Quick assessment

Consider:

• Sample size per class
• Computational resources
• Stability requirements

Seed for reproducible stratified sampling. Ensures:

• Consistent class proportions
• Reproducible splits
• Comparable experiments

Critical for:

• Benchmarking
• Research studies
• Quality assurance

Fraction of samples for stratified test set. Best practices: Common splits:

• 0.2: Balanced evaluation
• 0.3: More thorough testing
• 0.15: Preserve training size

Consider:

• Minority class size
• Overall dataset size
• Validation objectives

It must be between 0.0 and 1.0.

Number of temporal splits. Considerations: Typical values:

• 5: Standard forward chaining
• 3: Limited historical data
• 10: Long time series

Impact:

• Affects training window growth
• Determines validation points
• Influences computational load

Maximum size of training set. Should be strictly less than the number of samples. Applications:

• 0: Use all available past data
• >0: Rolling window of fixed size

Use cases:

• Limit historical relevance
• Control computational cost
• Handle concept drift
• Memory constraints

Number of samples in each test set. When 0:

• Auto-calculated as n_samples/(n_splits+1)
• Ensures equal-sized test sets

Considerations:

• Forecast horizon
• Validation requirements
• Available future data

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

• Avoid data leakage
• Model forecast lag
• Buffer periods

Common scenarios:

• 0: Continuous prediction
• >0: Forward gap for realistic evaluation
• Match business forecasting needs