HistGradientBoosting / Regressor Layer

Loss functions for gradient boosting:

Optimization objective: $$\underset{F}{\min }\sum _{i=1}^{n}L(y_i,F(x_i))$$

Key properties:

1. Squared Error: L2 norm
2. Absolute Error: L1 norm
3. Gamma: Right-skewed data
4. Poisson: Count data
5. Quantile: Percentile estimation

Selection impact:

• Prediction behavior
• Outlier sensitivity
• Distribution assumptions
• Optimization dynamics

If loss is “quantile”, this parameter specifies which quantile to be estimated and must be between 0 and 1.

Parameter meaning:

• 0.5: Median (P50)
• 0.1: Lower decile (P10)
• 0.9: Upper decile (P90)

Common values:

• Central: 0.5
• Lower tail: [0.1, 0.25]
• Upper tail: [0.75, 0.9]

Applications:

• Risk assessment
• Prediction intervals
• Extreme values
• Distribution analysis

Note: Only used with Quantile loss

Gradient descent step size:

Formula: $$F_m=F_{m-1}+\nu h_m$$ where ν is learning rate

Guidelines:

• Small (0.01-0.05): Precise
• Medium (0.1): Standard
• Large (0.3-0.5): Fast

Trade-offs:

• Smaller: Better accuracy
• Larger: Faster training
• Balance with iterations
• Monitor convergence

Maximum boosting iterations:

Iteration process: $$F_m=\sum _{i=1}^M\nu h_i$$ where M is max_iter

Guidelines:

• Small (50-100): Simple
• Medium (100-500): Standard
• Large (500+): Complex

Consider with:

• Learning rate
• Early stopping
• Dataset size
• Model complexity

Maximum leaf nodes per tree:

Tree capacity:

• 0: Unlimited leaves
• Powers of 2 minus 1: Balanced
• Other values: Custom structure

Common settings:

• Small (7-15): Simple trees
• Medium (31): Default
• Large (63-127): Complex

Impact:

• Model complexity
• Memory usage
• Training speed
• Prediction detail

Maximum tree depth limit:

Depth control:

• 0: No limit
• log2(n): Balanced
• Fixed value: Controlled

Guidelines:

• Shallow (3-5): Fast, simple
• Medium (6-10): Balanced
• Deep (10+): Complex

Trade-offs:

• Deeper: More detail
• Shallower: Less overfitting
• Memory impact
• Training speed

Minimum samples per leaf:

Guidelines:

• Small data: 5-10
• Medium: 20 (default)
• Large data: 50-100

Benefits:

• Prevents overfitting
• Stabilizes trees
• Reduces variance
• Controls complexity

L2 regularization strength:

Penalty term: $$L_{reg}=L+\lambda ||w|{|}_2^2$$

Values:

• 0.0: No regularization
• 0.1-1.0: Moderate
• 1.0+: Strong

Effects:

• Reduces overfitting
• Smooths predictions
• Shrinks weights
• Improves stability

Feature fraction for splits:

Selection process: $$n_{features}=max\_features\ast N_{total}$$

Common values:

• 1.0: Use all features
• 0.7-0.9: Slight reduction
• 0.5-0.7: Significant reduction

Benefits:

• Reduces overfitting
• Increases randomization
• Speeds up training
• Memory efficient

The maximum number of bins to use for non-missing values. Before training, each feature of the input array X is binned into integer-valued bins, which allows for a much faster training stage.

Binning impact:

• Memory usage ∝ bins
• Training speed ∝ 1/bins
• Precision ∝ bins

Guidelines:

• 255: Maximum precision
• 100-200: Balanced
• 50-100: Memory efficient

Trade-offs:

• More bins: Better splits
• Fewer bins: Faster training
• Memory constraints
• Numerical precision

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

When True:

• Reuse existing trees
• Add more iterations
• Continue training
• Preserve learning

Applications:

• Online learning
• Model updating
• Iterative fitting
• Performance tuning

Trade-offs:

• Memory persistence
• Training flexibility
• Parameter consistency
• State management

Validation-based stopping:

Benefits:

• Prevents overfitting
• Optimizes iterations
• Saves computation
• Automatic tuning

Requirements:

• Validation data
• Patience setting
• Tolerance level
• Score monitoring

Early stopping validation size:

Data split: $$n_{val}=validation\_fraction\ast n_{samples}$$

Guidelines:

• Small data: 0.2
• Medium: 0.1 (default)
• Large data: 0.05

Considerations:

• Dataset size
• Validation stability
• Training efficiency
• Stopping reliability

Early stopping patience:

Stopping logic:

• Monitor n consecutive iterations
• Check score improvement > tol
• Stop if no improvement

Settings:

• Short (5): Quick stopping
• Medium (10): Default
• Long (20+): Patient

Impact:

• Training duration
• Convergence certainty
• Resource usage
• Model optimization

Early stopping tolerance:

Improvement threshold: $$|scor{e}_t-scor{e}_{t-1}|<tol$$

Typical values:

• Strict (1e-7): Default
• Medium (1e-5): Standard
• Loose (1e-3): Quick

Trade-offs:

• Smaller: More precise
• Larger: Faster stopping
• Balance with patience
• Consider noise level

Pseudo-random number generator to control the subsampling in the binning process, and the train/validation data split if early stopping is enabled.

Controls randomization in:

• Feature selection
• Validation splits
• Histogram binning
• Tree building

Usage:

• 0: System random
• Fixed: Reproducible
• Different: Ensemble study

Important for:

• Reproducibility
• Debugging
• Research
• Model comparison

Loss functions for gradient boosting:

Optimization objective: $$\underset{F}{\min }\sum _{i=1}^{n}L(y_i,F(x_i))$$

Key properties:

1. Squared Error: L2 norm
2. Absolute Error: L1 norm
3. Gamma: Right-skewed data
4. Poisson: Count data
5. Quantile: Percentile estimation

Selection impact:

• Prediction behavior
• Outlier sensitivity
• Distribution assumptions
• Optimization dynamics

Quantile level search space:

Search patterns:

1. Central tendency:

   • [0.5]
   • Median prediction
   • Standard reference

2. Prediction intervals:

   • [0.1, 0.5, 0.9]
   • 80% interval
   • Risk assessment

3. Fine-grained:

   • [0.25, 0.5, 0.75]
   • Quartile analysis
   • Distribution study

Note: Only for quantile loss

Learning rate search space:

Search strategies:

1. Logarithmic scale:

   • [0.01, 0.1, 0.3]
   • Wide exploration
   • Standard range

2. Fine-tuning:

   • [0.08, 0.1, 0.12]
   • Around optimal
   • Precise control

3. Combined with iterations:

   • Smaller rate → more iterations
   • Larger rate → fewer iterations
   • Balance accuracy/speed

Maximum iterations search:

Search ranges:

1. Standard scale:

   • [50, 100, 200]
   • Basic problems
   • Quick exploration

2. Extended range:

   • [100, 300, 500]
   • Complex problems
   • Higher accuracy

3. With early stopping:

   • Set higher limits
   • Let stopping decide
   • Monitor convergence

Leaf nodes search space:

Search patterns:

1. Binary tree levels:

   • [15, 31, 63]
   • Powers of 2 minus 1
   • Balanced trees

2. Memory constrained:

   • [7, 15, 31]
   • Smaller trees
   • Faster training

3. Complex patterns:

   • [31, 63, 127]
   • Detailed trees
   • More capacity

Tree depth search space:

Search ranges:

1. Controlled depth:

   • [3, 5, 7]
   • Simple trees
   • Fast prediction

2. Standard range:

   • [6, 8, 10]
   • Balanced complexity
   • Medium datasets

3. Deep trees:

   • [10, 15, 0]
   • Complex patterns
   • Large datasets
   • 0 for unlimited

Minimum leaf samples search:

Search spaces:

1. Small datasets:

   • [5, 10, 20]
   • Detailed trees
   • Fine patterns

2. Medium scale:

   • [20, 50, 100]
   • Balanced smoothing
   • Standard choice

3. Large datasets:

   • [100, 200, 500]
   • Strong smoothing
   • Stable predictions

L2 regularization search:

Search patterns:

1. Light regularization:

   • [0.0, 0.01, 0.1]
   • Gentle control
   • Initial study

2. Standard range:

   • [0.1, 1.0, 10.0]
   • Log-scale search
   • Common values

3. Strong regularization:

   • [1.0, 10.0, 100.0]
   • Heavy smoothing
   • Overfitting control

Feature fraction search:

Search spaces:

1. Minor reduction:

   • [0.8, 0.9, 1.0]
   • Gentle randomization
   • Speed improvement

2. Moderate reduction:

   • [0.6, 0.7, 0.8]
   • Balance speed/accuracy
   • Standard range

3. Significant reduction:

   • [0.4, 0.5, 0.6]
   • Strong randomization
   • Fast training

Histogram bins search:

Search ranges:

1. Memory efficient:

   • [32, 64, 128]
   • Fast training
   • Low memory

2. Balanced:

   • [128, 200, 255]
   • Good precision
   • Standard choice

3. High precision:

   • [200, 225, 255]
   • Best splits
   • More memory

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

Options:

1. Standard: [false]

   • Fresh training
   • Independent runs

2. Compare: [true, false]

   • Study reuse impact
   • Performance effect

Consider with:

• Training strategy
• Memory usage
• Iteration control

Early stopping control:

When enabled:

• Monitors validation score
• Prevents overfitting
• Optimizes iterations
• Saves computation

Requirements:

• Validation data
• Patience setting
• Score monitoring

Validation split size:

Considerations:

• Dataset size impact
• Stopping reliability
• Training efficiency
• Validation stability

Guidelines:

• Small data: 0.2
• Standard: 0.1
• Large data: 0.05

Stopping patience:

Impact:

• Training duration
• Convergence check
• Resource usage
• Optimization level

Settings:

• Quick: 5 iterations
• Standard: 10 iterations
• Patient: 20+ iterations

Improvement tolerance:

Usage guide:

• Strict (1e-7): Default
• Medium (1e-5): Common
• Loose (1e-3): Quick

Trade-offs:

• Convergence precision
• Training duration
• Resource efficiency

Random seed control:

Usage:

• 0: System random
• Fixed: Reproducible
• None: Random behavior

Important for:

• Result stability
• Cross-validation
• Research studies

Regression model evaluation metrics:

Purpose:

• Model performance evaluation
• Error measurement
• Quality assessment
• Model comparison

Selection criteria:

• Error distribution
• Scale sensitivity
• Domain requirements
• Business objectives

Random seed for reproducible splits. Ensures:

• Consistent train/test sets
• Reproducible experiments
• Comparable model evaluations

Same seed guarantees identical splits across runs.

Data shuffling before splitting. Effects:

• true: Randomizes order, better for i.i.d. data
• false: Maintains order, important for time series

When to disable:

• Time dependent data
• Sequential patterns
• Grouped observations

Proportion of data for training. Considerations:

• Larger (e.g., 0.8-0.9): Better model learning
• Smaller (e.g., 0.5-0.7): Better validation

Common splits:

• 0.8: Standard (80/20 split)
• 0.7: More validation emphasis
• 0.9: More training emphasis

Maintain class distribution in splits. Important when:

• Classes are imbalanced
• Small classes present
• Representative splits needed

Requirements:

• Classification tasks only
• Cannot use with shuffle=false
• Sufficient samples per class

Number of cross-validation folds. Guidelines:

• 3-5: Large datasets, faster training
• 5-10: Standard choice, good balance
• 10+: Small datasets, thorough evaluation

Trade-offs:

• More folds: Better evaluation, slower training
• Fewer folds: Faster training, higher variance

Must be at least 2.

Number of folds for cross-validation. Selection guide: Recommended values:

• 5: Standard choice (default)
• 3: Large datasets/quick evaluation
• 10: Thorough evaluation/smaller datasets

Trade-offs:

• Higher values: More thorough, computationally expensive
• Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

Random seed for fold generation when shuffling. Important for:

• Reproducible results
• Consistent fold assignments
• Benchmark comparisons
• Debugging and validation

Set specific value for reproducibility across runs.

Whether to shuffle data before splitting into folds. Effects:

• true: Randomized fold composition (recommended)
• false: Sequential splitting

Enable when:

• Data may have ordering
• Better fold independence needed

Disable for:

• Time series data
• Ordered observations

Number of stratified folds. Guidelines: Typical values:

• 5: Standard for most cases
• 3: Quick evaluation/large datasets
• 10: Detailed evaluation/smaller datasets

Considerations:

• Must allow sufficient samples per class per fold
• Balance between stability and computation time
• Consider smallest class size when choosing

Seed for reproducible stratified splits. Ensures:

• Consistent fold assignments
• Reproducible results
• Comparable experiments
• Systematic validation

Fixed seed guarantees identical stratified splits.

Data shuffling before stratified splitting. Impact:

• true: Randomizes while maintaining stratification
• false: Maintains data order within strata

Use cases:

• true: Independent observations
• false: Grouped or sequential data

Class proportions maintained regardless of setting.

Number of random splits to perform. Consider: Common values:

• 5: Standard evaluation
• 10: More thorough assessment
• 3: Quick estimates

Trade-offs:

• More splits: Better estimation, longer runtime
• Fewer splits: Faster, less stable estimates

Balance between computation and stability.

Random seed for reproducible shuffling. Controls:

• Split randomization
• Sample selection
• Result reproducibility

Important for:

• Debugging
• Comparative studies
• Result verification

Proportion of samples for test set. Guidelines: Common ratios:

• 0.2: Standard (80/20 split)
• 0.25: More validation emphasis
• 0.1: More training data

Considerations:

• Dataset size
• Model complexity
• Validation requirements

It must be between 0.0 and 1.0.

Number of stratified random splits. Guidelines: Recommended values:

• 5: Standard evaluation
• 10: Detailed analysis
• 3: Quick assessment

Consider:

• Sample size per class
• Computational resources
• Stability requirements

Seed for reproducible stratified sampling. Ensures:

• Consistent class proportions
• Reproducible splits
• Comparable experiments

Critical for:

• Benchmarking
• Research studies
• Quality assurance

Fraction of samples for stratified test set. Best practices: Common splits:

• 0.2: Balanced evaluation
• 0.3: More thorough testing
• 0.15: Preserve training size

Consider:

• Minority class size
• Overall dataset size
• Validation objectives

It must be between 0.0 and 1.0.

Number of temporal splits. Considerations: Typical values:

• 5: Standard forward chaining
• 3: Limited historical data
• 10: Long time series

Impact:

• Affects training window growth
• Determines validation points
• Influences computational load

Maximum size of training set. Should be strictly less than the number of samples. Applications:

• 0: Use all available past data
• >0: Rolling window of fixed size

Use cases:

• Limit historical relevance
• Control computational cost
• Handle concept drift
• Memory constraints

Number of samples in each test set. When 0:

• Auto-calculated as n_samples/(n_splits+1)
• Ensures equal-sized test sets

Considerations:

• Forecast horizon
• Validation requirements
• Available future data

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

• Avoid data leakage
• Model forecast lag
• Buffer periods

Common scenarios:

• 0: Continuous prediction
• >0: Forward gap for realistic evaluation
• Match business forecasting needs