HistGradientBoosting / Classifier Layer

The learning rate, also known as shrinkage. This is used as a multiplicative factor for the leaves values. Use 1 for no shrinkage.

Mathematical impact: $$F_m(x)=F_{m-1}(x)+\nu h_m(x)$$ where $\nu$ is the learning rate

Trade-offs:

• Lower values: Better accuracy, slower training
• Higher values: Faster training, potential instability

Typical ranges:

• Conservative: 0.01-0.05
• Standard: 0.1 (default)
• Aggressive: 0.2-0.3

Note: Smaller learning rates require more iterations

The maximum number of iterations of the boosting process, i.e. the maximum number of trees for binary classification. For multiclass classification, n_classes trees per iteration are built.

Impact:

• Model complexity
• Training time
• Convergence guarantee
• Final ensemble size

Selection guidelines:

• Simple problems: 50-100
• Medium complexity: 100-500
• Complex problems: 500-2000

Interaction notes:

• Inversely related to learning rate
• Early stopping may reduce actual iterations
• Higher values needed for smaller learning rates

The maximum number of leaves for each tree. Cannot be equal to 1. If 0, there is no maximum limit.

Controls:

• Tree complexity
• Model capacity
• Memory usage
• Training speed

Guidelines:

• Small (15-31): Simple problems
• Medium (32-127): Standard datasets
• Large (128+): Complex relationships

Trade-offs:

• More leaves: Better accuracy, higher complexity
• Fewer leaves: Faster training, better generalization

Note: Set to 0 for unlimited nodes

The maximum depth of each tree. The depth of a tree is the number of edges to go from the root to the deepest leaf.

Effects:

• Controls tree hierarchy levels
• Limits feature interactions
• Manages model complexity
• Influences training time

Typical values:

• Shallow (3-5): Simple relationships
• Medium (6-10): Standard problems
• Deep (11+): Complex interactions

Considerations:

• Memory usage grows exponentially with depth
• Deeper trees risk overfitting
• Set to 0 for unlimited depth

Minimum samples required at leaf nodes:

Purpose:

• Prevents overfitting
• Ensures statistical significance
• Controls tree granularity
• Stabilizes predictions

Selection guide:

• Small datasets (<1000): 5-10 samples
• Medium datasets: 20-50 samples
• Large datasets: 50-100+ samples

Impact:

• Higher values: More stable, less detailed
• Lower values: More detailed, risk of overfitting

L2 regularization strength:

Mathematical form: $$L=Loss+\lambda \sum w^2$$

Effects:

• Controls leaf weight smoothing
• Reduces overfitting
• Improves generalization

Typical values:

• None: 0.0
• Weak: 0.1-1.0
• Strong: 5.0-10.0

Note: Higher values mean stronger regularization

Fraction of features to consider for splits:

Purpose:

• Introduces randomness
• Reduces correlation between trees
• Prevents overfitting
• Speeds up training

Common settings:

• 1.0: Use all features (default)
• 0.7-0.8: Standard random selection
• sqrt(n)/n: Traditional random forest style

Note: Lower values increase randomization but may miss important features

The maximum number of bins to use for non-missing values.

Impact:

• Memory efficiency
• Training speed
• Feature resolution

Trade-offs:

• Lower values: Faster, less precise
• Higher values: More precise, more memory

Guidelines:

• Minimum: 2 bins
• Default: 255 bins (optimal)
• Maximum: 255 bins (uint8 limit)

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble.

Functionality:

• Reuses previous model
• Adds more estimators
• Continues training
• Preserves learned patterns

Use cases:

• Online learning
• Iterative training
• Model updating
• Experimental tuning

Note: May affect convergence behavior

Enable validation-based training stop:

Operation:

• Monitors validation score
• Prevents overfitting
• Optimizes iterations
• Saves computation

Requirements:

• Validation fraction > 0
• n_iter_no_change > 0
• Sufficient training data

Best practices:

• Enable for large datasets
• Use with validation_fraction
• Monitor convergence pattern

Proportion (or absolute size) of training data to set aside as validation data for early stopping. If None, early stopping is done on the training data. Only used if early stopping is performed.

Usage:

• Early stopping validation
• Model monitoring
• Convergence checking

Guidelines:

• Small datasets: 0.1-0.15
• Medium datasets: 0.1 (default)
• Large datasets: 0.05-0.1

Considerations:

• Dataset size
• Validation stability
• Training data needs

Early stopping patience parameter:

Function:

• Defines stopping patience
• Controls training length
• Balances optimization
• Prevents premature stopping

Setting guidelines:

• Conservative: 15-20 iterations
• Standard: 10 iterations (default)
• Aggressive: 5-8 iterations

Note: Larger values ensure better convergence

Score improvement tolerance for early stopping:

Purpose:

• Defines meaningful improvement
• Controls convergence precision
• Affects training duration

Typical values:

• Strict: 1e-8 to 1e-7
• Standard: 1e-7 (default)
• Relaxed: 1e-6 to 1e-5

Impact:

• Smaller values: More precise, longer training
• Larger values: Faster convergence, less precise

Random number generator seed:

Controls randomization in:

• Feature selection
• Sample bootstrapping
• Split point selection
• Early stopping splits

Importance:

• Reproducibility
• Debugging
• Result verification
• Experimental control

Class weighting schemes:

Note: Affects all trees in ensemble

Learning rate search space:

Search strategies:

1. Logarithmic scale (recommended):
   • Coarse: [0.01, 0.1, 1.0]
   • Fine: [0.01, 0.03, 0.1, 0.3]

Optimization tips:

• Pair with max_iter
• Smaller values need more iterations
• Consider training time budget

Example ranges:

• Fast training: [0.1, 0.3]
• Balanced: [0.05, 0.1, 0.15]
• Precise: [0.01, 0.03, 0.05]

Number of boosting iterations search space:

Search patterns:

1. Basic range:
   • [50, 100, 200] iterations
2. Extended search:
   • [100, 300, 500, 1000] iterations

Considerations:

• Dataset size and complexity
• Learning rate interaction
• Early stopping impact
• Computational resources

Formula guide:

• Base iterations ≈ 1/(learning_rate)
• Adjust for problem complexity

Tree size exploration space:

Search strategies:

1. Powers of 2 (common):
   • [15, 31, 63, 127] nodes
2. Linear increments:
   • [20, 50, 100, 200] nodes

Performance impact:

• Memory usage scales linearly
• Training time correlation
• Model complexity control

Problem-based ranges:

• Simple: [15-31] nodes
• Medium: [31-127] nodes
• Complex: [127-511] nodes

Tree depth search range:

Search patterns:

1. Conservative:
   • [3, 5, 7] levels
2. Extended:
   • [4, 6, 8, 10] levels

Resource impact:

• Memory: O(2^depth)
• Training time: O(depth)
• Feature interactions: O(depth)

Guidelines:

• Start shallow (3-5)
• Increment gradually
• Monitor overfitting
• Consider max_leaf_nodes interaction

Minimum leaf size search space:

Search strategies:

1. Small datasets:
   • [5, 10, 20] samples
2. Large datasets:
   • [20, 50, 100] samples

Scaling considerations:

• Dataset size correlation
• Noise sensitivity
• Overfitting prevention

Selection guide:

• Base value ≈ sqrt(n_samples)
• Adjust for noise level
• Consider class distribution

L2 regularization strength search space:

Search patterns:

1. Logarithmic scale:
   • [0.0, 0.1, 1.0, 10.0]
2. Fine-tuning:
   • [0.01, 0.1, 0.3, 1.0]

Optimization focus:

• Overfitting control
• Model stability
• Prediction smoothness

Dataset considerations:

• Noise level
• Feature count
• Sample size
• Model complexity

Feature fraction search space:

Search strategies:

1. Standard range:
   • [0.6, 0.8, 1.0]
2. Extended search:
   • [0.4, 0.6, 0.8, 1.0]

Performance impact:

• Training speed
• Tree diversity
• Feature utilization

Selection guide:

• High features: Lower fractions
• Few features: Higher fractions
• Consider feature importance spread

Binning resolution search space:

Search patterns:

1. Memory-conscious:
   • [32, 64, 128, 255] bins
2. Precision-focused:
   • [128, 192, 255] bins

Trade-off analysis:

• Memory usage vs precision
• Training speed vs accuracy
• Data resolution needs

Guidelines:

• Start with lower values
• Increase if underfitting
• Monitor memory usage

Warm start strategy evaluation:

Search options:

• [false]: Standard training
• [true, false]: Compare approaches

Use cases:

• Incremental learning
• Model extension
• Parameter sensitivity analysis

Evaluation metrics:

• Training stability
• Convergence behavior
• Resource utilization

Class weighting schemes:

Note: Affects all trees in ensemble

Early stopping strategy control:

Configuration purpose:

• Training efficiency
• Overfitting prevention
• Resource optimization
• Performance monitoring

Best practices:

• Enable for large datasets
• Use with validation_fraction
• Pair with n_iter_no_change
• Monitor convergence patterns

Validation set size control:

Selection guide:

• Small data: 0.15-0.20
• Medium data: 0.10 (default)
• Large data: 0.05-0.10

Considerations:

• Dataset size
• Validation stability
• Training data needs
• Early stopping requirements

Early stopping patience configuration:

Common ranges:

• Aggressive: 5-8 iterations
• Standard: 10 iterations
• Conservative: 15-20 iterations

Selection factors:

• Learning rate
• Dataset size
• Model complexity
• Convergence patterns

Improvement tolerance threshold:

Typical ranges:

• Strict: 1e-8 to 1e-7
• Standard: 1e-7 (default)
• Relaxed: 1e-6 to 1e-5

Impact factors:

• Convergence speed
• Training duration
• Score precision
• Resource usage

Random seed for reproducibility:

Applications:

• Result reproducibility
• Parameter tuning
• Model comparison
• Debug scenarios

Best practices:

• Fix seed for development
• Vary for robustness checks
• Document for replication

Performance evaluation metrics for classification:

Purpose:

• Model evaluation
• Ensemble selection
• Early stopping
• Performance tracking

Selection criteria:

• Problem objectives
• Class distribution
• Ensemble size
• Computation resources

Random seed for reproducible splits. Ensures:

• Consistent train/test sets
• Reproducible experiments
• Comparable model evaluations

Same seed guarantees identical splits across runs.

Data shuffling before splitting. Effects:

• true: Randomizes order, better for i.i.d. data
• false: Maintains order, important for time series

When to disable:

• Time dependent data
• Sequential patterns
• Grouped observations

Proportion of data for training. Considerations:

• Larger (e.g., 0.8-0.9): Better model learning
• Smaller (e.g., 0.5-0.7): Better validation

Common splits:

• 0.8: Standard (80/20 split)
• 0.7: More validation emphasis
• 0.9: More training emphasis

Maintain class distribution in splits. Important when:

• Classes are imbalanced
• Small classes present
• Representative splits needed

Requirements:

• Classification tasks only
• Cannot use with shuffle=false
• Sufficient samples per class

Number of cross-validation folds. Guidelines:

• 3-5: Large datasets, faster training
• 5-10: Standard choice, good balance
• 10+: Small datasets, thorough evaluation

Trade-offs:

• More folds: Better evaluation, slower training
• Fewer folds: Faster training, higher variance

Must be at least 2.

Number of folds for cross-validation. Selection guide: Recommended values:

• 5: Standard choice (default)
• 3: Large datasets/quick evaluation
• 10: Thorough evaluation/smaller datasets

Trade-offs:

• Higher values: More thorough, computationally expensive
• Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

Random seed for fold generation when shuffling. Important for:

• Reproducible results
• Consistent fold assignments
• Benchmark comparisons
• Debugging and validation

Set specific value for reproducibility across runs.

Whether to shuffle data before splitting into folds. Effects:

• true: Randomized fold composition (recommended)
• false: Sequential splitting

Enable when:

• Data may have ordering
• Better fold independence needed

Disable for:

• Time series data
• Ordered observations

Number of stratified folds. Guidelines: Typical values:

• 5: Standard for most cases
• 3: Quick evaluation/large datasets
• 10: Detailed evaluation/smaller datasets

Considerations:

• Must allow sufficient samples per class per fold
• Balance between stability and computation time
• Consider smallest class size when choosing

Seed for reproducible stratified splits. Ensures:

• Consistent fold assignments
• Reproducible results
• Comparable experiments
• Systematic validation

Fixed seed guarantees identical stratified splits.

Data shuffling before stratified splitting. Impact:

• true: Randomizes while maintaining stratification
• false: Maintains data order within strata

Use cases:

• true: Independent observations
• false: Grouped or sequential data

Class proportions maintained regardless of setting.

Number of random splits to perform. Consider: Common values:

• 5: Standard evaluation
• 10: More thorough assessment
• 3: Quick estimates

Trade-offs:

• More splits: Better estimation, longer runtime
• Fewer splits: Faster, less stable estimates

Balance between computation and stability.

Random seed for reproducible shuffling. Controls:

• Split randomization
• Sample selection
• Result reproducibility

Important for:

• Debugging
• Comparative studies
• Result verification

Proportion of samples for test set. Guidelines: Common ratios:

• 0.2: Standard (80/20 split)
• 0.25: More validation emphasis
• 0.1: More training data

Considerations:

• Dataset size
• Model complexity
• Validation requirements

It must be between 0.0 and 1.0.

Number of stratified random splits. Guidelines: Recommended values:

• 5: Standard evaluation
• 10: Detailed analysis
• 3: Quick assessment

Consider:

• Sample size per class
• Computational resources
• Stability requirements

Seed for reproducible stratified sampling. Ensures:

• Consistent class proportions
• Reproducible splits
• Comparable experiments

Critical for:

• Benchmarking
• Research studies
• Quality assurance

Fraction of samples for stratified test set. Best practices: Common splits:

• 0.2: Balanced evaluation
• 0.3: More thorough testing
• 0.15: Preserve training size

Consider:

• Minority class size
• Overall dataset size
• Validation objectives

It must be between 0.0 and 1.0.

Number of temporal splits. Considerations: Typical values:

• 5: Standard forward chaining
• 3: Limited historical data
• 10: Long time series

Impact:

• Affects training window growth
• Determines validation points
• Influences computational load

Maximum size of training set. Should be strictly less than the number of samples. Applications:

• 0: Use all available past data
• >0: Rolling window of fixed size

Use cases:

• Limit historical relevance
• Control computational cost
• Handle concept drift
• Memory constraints

Number of samples in each test set. When 0:

• Auto-calculated as n_samples/(n_splits+1)
• Ensures equal-sized test sets

Considerations:

• Forecast horizon
• Validation requirements
• Available future data

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

• Avoid data leakage
• Model forecast lag
• Buffer periods

Common scenarios:

• 0: Continuous prediction
• >0: Forward gap for realistic evaluation
• Match business forecasting needs