GradientBoosting / Classifier Layer

Gradient Boosting Classification - This classifier builds an additive model in a forward stage-wise fashion; it allows for the optimization of arbitrary differentiable loss functions. In each stage n_classes_ regression trees are fit on the negative gradient of the loss function, e.g. binary or multiclass log loss. Binary classification is a special case where only a single regression tree is induced.

Mathematical form: $F (x) = F_{0} (x) + m = 1 \sum M γ_{m} h_{m} (x)$ where:

$F_{0} (x)$ is initial prediction
$h_{m} (x)$ are weak learners
$γ_{m}$ are step sizes
M is number of boosting stages

Key characteristics:

Forward stagewise additive modeling
Gradient-based optimization
Strong predictive power
Handles different loss functions
Automatic feature selection

Common applications:

High-performance classification
Robust predictions
Complex non-linear patterns
Feature importance ranking
Production systems

Computational notes:

Training: O(n_estimators * n_samples * log(n_samples))
Memory: O(n_estimators * n_nodes)
Sequential nature (less parallel)
Early stopping available

Note: Consider HistGradientBoosting for large datasets (n_samples >= 10,000)

Table

Predicted Table

Validation Results

Test Metric

ROC Curve Data

Confusion Matrix

Feature Importances

SelectFeatures

[column, ...]

Feature columns for Gradient Boosting Classification:

Data requirements:

General format:
- Numeric features
- Encoded categoricals
- No missing values
- Clean data
Preprocessing notes:
- No scaling needed (tree-based)
- Handle missing values first
- Encode categorical features
- Remove unnecessary features
Feature quality:
- Check distributions
- Remove constants
- Handle outliers (robust)
- Monitor importance scores
Performance tips:
- Use feature selection
- Consider interactions
- Monitor training time
- Watch memory usage

Note: If empty, uses all numeric columns except target

SelectTarget

column

Target column for Gradient Boosting Classification:

Requirements:

Data format:
- Categorical labels
- At least two classes
- No missing values
- Clean encoding
Class characteristics:
- Handles imbalance well
- Works with multi-class
- Supports probabilities
- Preserves label order
Quality checks:
- Validate encodings
- Check distributions
- Monitor rare classes
- Verify consistency
Performance notes:
- Different loss per class
- Multi-class overhead
- Memory scales with classes
- Consider class balance

Params

oneof

DefaultParams

Default configuration for Gradient Boosting Classification:

Core parameters:
- Loss: Deviance (logistic)
- Learning rate: 0.1
- N estimators: 100
- Subsample: 1.0
Tree parameters:
- Max depth: 3
- Min samples split: 2
- Min samples leaf: 1
- Criterion: Friedman MSE
Early stopping:
- Validation fraction: 0.1
- Tolerance: 1e-4
- N iter no change: None
Other settings:
- No warm start
- No pruning (ccp_alpha: 0.0)
- Full feature set

Best suited for:

Initial modeling
Medium-sized datasets
General classification
Balanced trade-offs

CustomParams

Fine-tuned configuration for Gradient Boosting Classification:

Parameter categories:

Boosting control
Tree structure
Optimization settings
Early stopping criteria

Loss

enum

Deviance

Loss function for gradient optimization:

Purpose:

Defines optimization objective
Guides boosting process
Affects model behavior
Determines gradients

Deviance ~

Logistic regression deviance:

Formula: $- \sum_{i = 1}^{n} [y_{i} lo g (p_{i}) + (1 - y_{i}) lo g (1 - p_{i})]$

Properties:

Standard choice
Probability outputs
Stable training
Well-calibrated

Best for:

Most classification tasks
Probability needs
Balanced performance

Exponential ~

AdaBoost-style exponential loss:

Formula: $\sum_{i = 1}^{n} exp (- y_{i} F (x_{i}))$

Properties:

More aggressive
Focus on hard cases
AdaBoost equivalent
Sensitive to noise

Best for:

Clean datasets
Binary classification
AdaBoost comparison

LearningRate

f64

0.1

Gradient descent step size. Shrinks the contribution of each tree by learning_rate.

Typical ranges:

Small: 0.01-0.1 (more trees needed)
Medium: 0.1-0.3 (default range)
Large: >0.3 (fewer trees needed)

Note: Trade-off with n_estimators

NEstimators

u32

100

Number of boosting stages:

Guidelines:

Small: 50-100 (quick models)
Medium: 100-500 (standard)
Large: >500 (complex problems)

Note: More robust to overfitting with small learning_rate

Subsample

f64

The fraction of samples to be used for fitting the individual base learners:

Effects:

1.0: Standard gradient boosting
<1.0: Stochastic gradient boosting
Typical range: [0.5, 1.0]

Note: Reduces variance and overfitting

Criterion

enum

FriedmanMse

Split quality measures for base learners:

Purpose:

Tree construction
Split evaluation
Gradient fitting
Node optimization

FriedmanMse ~

Friedman's improvement score:

Formula: $MSE - γ^{2} / n$

Properties:

Variance reduction
Gradient awareness
Default choice
Better suited for boosting

Mse ~

Mean squared error:

Formula: $\frac{1}{n} \sum (y_{i} - \overset{y}{^}_{i})^{2}$

Properties:

Simple metric
Fast computation
Standard regression
Less boosting-specific

MinSamplesSplit

u32

The minimum number of samples required to split an internal node:

Controls:

Tree growth
Overfitting prevention
Model complexity

Typical range: [2, 20]

MinSamplesLeaf

u32

The minimum number of samples required to be at a leaf node.:

Purpose:

Ensures prediction stability
Controls overfitting
Smooths predictions

Typical range: [1, 10]

MinWeightFractionLeaf

f64

The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node:

Range: [0.0, 0.5]
Used with weighted samples
Alternative to min_samples_leaf

MaxDepth

u32

The maximum depth limits the number of nodes in the tree.

Guidelines:

Shallow (1-3): Fast, simple trees
Medium (3-7): Standard choice
Deep (>7): Complex patterns

Note: Small depth preferred in boosting

MinImpurityDecrease

f64

A node will be split if this split induces a decrease of the impurity greater than or equal to this value.

Controls:

Split quality
Tree complexity
Stopping criterion

Typical range: [0.0, 0.1]

RandomState

u64

Controls the random seed given to each Tree estimator at each boosting iteration. In addition, it controls the random permutation of the features at each split.

Affects:

Data subsampling
Feature selection
Tree building

Set for reproducibility

MaxFeatures

enum

Auto

Feature subset size for tree splits:

Purpose:

Control tree complexity
Reduce overfitting
Speed up training
Add randomization

Auto ~

Use all features:

Formula: $F_{ma x} = F$

Properties:

Maximum information
Slower training
Full optimization
No randomization

Sqrt ~

Square root scaling:

Formula: $F_{ma x} = F$

Properties:

Balanced choice
Moderate speed
Some randomization
Good default

Log2 ~

Logarithmic scaling:

Formula: $F_{ma x} = lo g_{2} (F)$

Properties:

Aggressive reduction
Fastest training
More randomization
High dimensions

Custom ~

User-defined count:

Properties:

Manual control
Flexible tuning
Problem-specific
Optimization needs

MaxFeaturesF

u32

Custom value of max features. Only used when max_features is custom.

MaxLeafNodes

u32

Maximum leaf node count. Best nodes are defined as relative reduction in impurity:

Values:

0: Unlimited
Value can not be '1'- >1: Best-first growth

Alternative to max_depth

WarmStart

bool

false

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just erase the previous solution.

ValidationFraction

f64

0.1

The proportion of training data to set aside as validation set for early stopping.

Range: (0.0, 1.0)
Used with early stopping
Typical: 0.1-0.2

NIterNoChange

u32

Early stopping patience. Used to decide if early stopping will be used to terminate training when validation score is not improving.

0: No early stopping
>0: Stop after N non-improving iterations
Prevents unnecessary training

Tol

f64

0.0001

Tolerance for the early stopping. When the loss is not improving by at least tol for n_iter_no_change iterations (if set to a number), the training stops.

Smaller: More precise
Larger: Earlier stopping

CcpAlpha

f64

Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ccp_alpha will be chosen.

0.0: No pruning
>0.0: Prune subtrees
Controls tree complexity

GridSearch

Hyperparameter optimization for Gradient Boosting Classification:

Search dimensions:

Core boosting:
- Loss function
- Learning rate
- Number of estimators
- Subsampling
Tree structure:
- Depth/nodes
- Split criteria
- Sample thresholds
Early stopping:
- Validation settings
- Stopping criteria
- Tolerance levels

Computational impact:

Time: O(n_params * n_estimators * n_samples * log(n_samples))
Memory: O(n_params * n_estimators)

Best practices:

Start with learning rate and n_estimators
Then tune tree parameters
Finally adjust early stopping

Loss

[enum, ...]

Deviance

Loss function for gradient optimization:

Purpose:

Defines optimization objective
Guides boosting process
Affects model behavior
Determines gradients

Deviance ~

Logistic regression deviance:

Formula: $- \sum_{i = 1}^{n} [y_{i} lo g (p_{i}) + (1 - y_{i}) lo g (1 - p_{i})]$

Properties:

Standard choice
Probability outputs
Stable training
Well-calibrated

Best for:

Most classification tasks
Probability needs
Balanced performance

Exponential ~

AdaBoost-style exponential loss:

Formula: $\sum_{i = 1}^{n} exp (- y_{i} F (x_{i}))$

Properties:

More aggressive
Focus on hard cases
AdaBoost equivalent
Sensitive to noise

Best for:

Clean datasets
Binary classification
AdaBoost comparison

LearningRate

[f64, ...]

0.1

Learning rates to search:

Common patterns:

Coarse: [0.01, 0.1, 1.0]
Fine: [0.01, 0.05, 0.1, 0.3]
Focused: [0.05, 0.075, 0.1, 0.15]

Note: Inversely related to n_estimators

NEstimators

[u32, ...]

100

Number of boosting stages to try:

Search ranges:

Quick: [50, 100, 200]
Standard: [100, 300, 500]
Extensive: [200, 500, 1000]

Note: More stages needed with smaller learning rates

Subsample

[f64, ...]

Subsampling rates to evaluate:

Ranges:

Conservative: [0.8, 1.0]
Standard: [0.6, 0.8, 1.0]
Aggressive: [0.5, 0.7, 0.9, 1.0]

Criterion

[enum, ...]

FriedmanMse

Split quality measures for base learners:

Purpose:

Tree construction
Split evaluation
Gradient fitting
Node optimization

FriedmanMse ~

Friedman's improvement score:

Formula: $MSE - γ^{2} / n$

Properties:

Variance reduction
Gradient awareness
Default choice
Better suited for boosting

Mse ~

Mean squared error:

Formula: $\frac{1}{n} \sum (y_{i} - \overset{y}{^}_{i})^{2}$

Properties:

Simple metric
Fast computation
Standard regression
Less boosting-specific

MinSamplesSplit

[u32, ...]

Minimum split samples to try:

Ranges:

Fine: [2, 5, 10]
Medium: [5, 15, 30]
Coarse: [10, 30, 50]

MinSamplesLeaf

[u32, ...]

Minimum leaf samples to evaluate:

Ranges:

Fine: [1, 3, 5]
Medium: [3, 7, 11]
Coarse: [5, 10, 20]

MinWeightFractionLeaf

[f64, ...]

Minimum weighted fractions to try:

Ranges:

Light: [0.0, 0.1]
Medium: [0.0, 0.1, 0.2]
Heavy: [0.1, 0.2, 0.3]

MaxDepth

[u32, ...]

Tree depth values to evaluate:

Ranges:

Shallow: [2, 3, 4]
Medium: [3, 5, 7]
Deep: [5, 7, 9]

MinImpurityDecrease

[f64, ...]

Impurity thresholds to try:

Ranges:

Fine: [0.0, 0.0001, 0.001]
Medium: [0.0, 0.001, 0.01]
Coarse: [0.01, 0.05, 0.1]

RandomState

u64

MaxFeatures

[enum, ...]

Auto

Feature subset size for tree splits:

Purpose:

Control tree complexity
Reduce overfitting
Speed up training
Add randomization

Auto ~

Use all features:

Formula: $F_{ma x} = F$

Properties:

Maximum information
Slower training
Full optimization
No randomization

Sqrt ~

Square root scaling:

Formula: $F_{ma x} = F$

Properties:

Balanced choice
Moderate speed
Some randomization
Good default

Log2 ~

Logarithmic scaling:

Formula: $F_{ma x} = lo g_{2} (F)$

Properties:

Aggressive reduction
Fastest training
More randomization
High dimensions

Custom ~

User-defined count:

Properties:

Manual control
Flexible tuning
Problem-specific
Optimization needs

MaxFeaturesF

[u32, ...]

MaxLeafNodes

[u32, ...]

Maximum leaf counts to evaluate:

Ranges:

Limited: [8, 16, 32]
Medium: [16, 32, 64]
Many/None: [0, 32, 64, 128]

WarmStart

[bool, ...]

false

Warm start options:

Choices:

[false]: Fresh ensembles
[true]: Incremental building
[false, true]: Compare both

ValidationFraction

f64

0.1

Validation set size for early stopping:

Range: (0.0, 1.0) Typical values: 0.1-0.2 Affects early stopping reliability

NIterNoChange

u32

Early stopping iterations:

Values: 0: No early stopping >0: Stop after N non-improving rounds Typical range: 5-20

Tol

f64

0.0001

Improvement tolerance for early stopping:

Smaller: More precise training Larger: Earlier stopping Typical range: 1e-4 to 1e-2

CcpAlpha

[f64, ...]

Cost-complexity pruning alphas:

Ranges:

Light: [0.0, 0.001, 0.01]
Medium: [0.0, 0.01, 0.05]
Heavy: [0.05, 0.1, 0.2]

RefitScore

enum

Accuracy

Performance evaluation metrics for classification:

Purpose:

Model evaluation
Ensemble selection
Early stopping
Performance tracking

Selection criteria:

Problem objectives
Class distribution
Ensemble size
Computation resources

Default ~

Uses ensemble's built-in scoring:

Properties:

Weighted accuracy metric
Ensemble-aware scoring
Fast computation
Boosting-compatible

Best for:

Standard problems
Quick evaluation
Initial testing
Performance tracking

Accuracy ~

Standard classification accuracy:

Formula: $Accuracy = \frac{Correct Predictions}{Total Samples}$

Properties:

Range: [0, 1]
Ensemble consensus
Intuitive metric
Equal error weights

Best for:

Balanced datasets
Equal error costs
Simple evaluation
Quick benchmarking

BalancedAccuracy ~

Class-normalized accuracy score:

Formula: $\frac{1}{K} \sum_{i = 1}^{K} \frac{T P _{i}}{T P _{i} + F N _{i}}$

Properties:

Range: [0, 1]
Class-weighted
Imbalance-robust
Fair evaluation

Best for:

Imbalanced data
Varied class sizes
Minority focus
Fair assessment

LogLoss ~

Logarithmic loss (Cross-entropy):

Formula: $- \frac{1}{N} \sum_{i = 1}^{N} \sum_{j = 1}^{K} y_{ij} lo g (p_{ij})$

Properties:

Range: [0, ∞)
Probability-sensitive
Boosting-optimal
Confidence-aware

Best for:

Probability estimation
Boosting optimization
Risk assessment
Model calibration

RocAuc ~

Area Under ROC Curve:

Properties:

Range: [0, 1]
Ranking quality
Threshold-invariant
Ensemble-appropriate

Best for:

Binary problems
Ranking tasks
Score calibration
Model comparison

Note: Extended to multi-class via averaging

Split

oneof

DefaultSplit

Standard train-test split configuration optimized for general classification tasks.

Configuration:

Test size: 20% (0.2)
Random seed: 98
Shuffling: Enabled
Stratification: Based on target distribution

Advantages:

Preserves class distribution
Provides reliable validation
Suitable for most datasets

Best for:

Medium to large datasets
Independent observations
Initial model evaluation

Splitting uses the ShuffleSplit strategy or StratifiedShuffleSplit strategy depending on the field stratified. Note: If shuffle is false then stratified must be false.

CustomSplit

Configurable train-test split parameters for specialized requirements. Allows fine-tuning of data division strategy for specific use cases or constraints.

Use cases:

Time series data
Grouped observations
Specific train/test ratios
Custom validation schemes

RandomState

u64

Random seed for reproducible splits. Ensures:

Consistent train/test sets
Reproducible experiments
Comparable model evaluations

Same seed guarantees identical splits across runs.

Shuffle

bool

true

Data shuffling before splitting. Effects:

true: Randomizes order, better for i.i.d. data
false: Maintains order, important for time series

When to disable:

Time dependent data
Sequential patterns
Grouped observations

TrainSize

f64

0.8

Proportion of data for training. Considerations:

Larger (e.g., 0.8-0.9): Better model learning
Smaller (e.g., 0.5-0.7): Better validation

Common splits:

0.8: Standard (80/20 split)
0.7: More validation emphasis
0.9: More training emphasis

Stratified

bool

false

Maintain class distribution in splits. Important when:

Classes are imbalanced
Small classes present
Representative splits needed

Requirements:

Classification tasks only
Cannot use with shuffle=false
Sufficient samples per class

Cv

oneof

DefaultCv

Standard cross-validation configuration using stratified 3-fold splitting.

Configuration:

Folds: 3
Method: StratifiedKFold
Stratification: Preserves class proportions

Advantages:

Balanced evaluation
Reasonable computation time
Good for medium-sized datasets

Limitations:

May be insufficient for small datasets
Higher variance than larger fold counts
May miss some data patterns

CustomCv

Configurable stratified k-fold cross-validation for specific validation requirements.

Features:

Adjustable fold count with NFolds determining the number of splits.
Stratified sampling
Preserved class distributions

Use cases:

Small datasets (more folds)
Large datasets (fewer folds)
Detailed model evaluation
Robust performance estimation

NFolds

u32

Number of cross-validation folds. Guidelines:

3-5: Large datasets, faster training
5-10: Standard choice, good balance
10+: Small datasets, thorough evaluation

Trade-offs:

More folds: Better evaluation, slower training
Fewer folds: Faster training, higher variance

Must be at least 2.

KfoldCv

K-fold cross-validation without stratification. Divides data into k consecutive folds for iterative validation.

Process:

Splits data into k equal parts
Each fold serves as validation once
Remaining k-1 folds form training set

Use cases:

Regression problems
Large, balanced datasets
When stratification unnecessary
Continuous target variables

Limitations:

May not preserve class distributions
Less suitable for imbalanced data
Can create biased splits with ordered data

NSplits

u32

Number of folds for cross-validation. Selection guide: Recommended values:

5: Standard choice (default)
3: Large datasets/quick evaluation
10: Thorough evaluation/smaller datasets

Trade-offs:

Higher values: More thorough, computationally expensive
Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

RandomState

u64

Random seed for fold generation when shuffling. Important for:

Reproducible results
Consistent fold assignments
Benchmark comparisons
Debugging and validation

Set specific value for reproducibility across runs.

Shuffle

bool

true

Whether to shuffle data before splitting into folds. Effects:

true: Randomized fold composition (recommended)
false: Sequential splitting

Enable when:

Data may have ordering
Better fold independence needed

Disable for:

Time series data
Ordered observations

StratifiedKfoldCv

Stratified K-fold cross-validation maintaining class proportions across folds.

Key features:

Preserves class distribution in each fold
Handles imbalanced datasets
Ensures representative splits

Best for:

Classification problems
Imbalanced class distributions
When class proportions matter

Requirements:

Classification tasks only
Sufficient samples per class
Categorical target variable

NSplits

u32

Number of stratified folds. Guidelines: Typical values:

5: Standard for most cases
3: Quick evaluation/large datasets
10: Detailed evaluation/smaller datasets

Considerations:

Must allow sufficient samples per class per fold
Balance between stability and computation time
Consider smallest class size when choosing

RandomState

u64

Seed for reproducible stratified splits. Ensures:

Consistent fold assignments
Reproducible results
Comparable experiments
Systematic validation

Fixed seed guarantees identical stratified splits.

Shuffle

bool

false

Data shuffling before stratified splitting. Impact:

true: Randomizes while maintaining stratification
false: Maintains data order within strata

Use cases:

true: Independent observations
false: Grouped or sequential data

Class proportions maintained regardless of setting.

ShuffleSplitCv

Random permutation cross-validator with independent sampling.

Characteristics:

Random sampling for each split
Independent train/test sets
More flexible than K-fold
Can have overlapping test sets

Advantages:

Control over test size
Fresh splits each iteration
Good for large datasets

Limitations:

Some samples might never be tested
Others might be tested multiple times
No guarantee of complete coverage

NSplits

u32

Number of random splits to perform. Consider: Common values:

5: Standard evaluation
10: More thorough assessment
3: Quick estimates

Trade-offs:

More splits: Better estimation, longer runtime
Fewer splits: Faster, less stable estimates

Balance between computation and stability.

RandomState

u64

Random seed for reproducible shuffling. Controls:

Split randomization
Sample selection
Result reproducibility

Important for:

Debugging
Comparative studies
Result verification

TestSize

f64

0.2

Proportion of samples for test set. Guidelines: Common ratios:

0.2: Standard (80/20 split)
0.25: More validation emphasis
0.1: More training data

Considerations:

Dataset size
Model complexity
Validation requirements

It must be between 0.0 and 1.0.

StratifiedShuffleSplitCv

Stratified random permutation cross-validator combining shuffle-split with stratification.

Features:

Maintains class proportions
Random sampling within strata
Independent splits
Flexible test size

Ideal for:

Imbalanced datasets
Large-scale problems
When class distributions matter
Flexible validation schemes

NSplits

u32

Number of stratified random splits. Guidelines: Recommended values:

5: Standard evaluation
10: Detailed analysis
3: Quick assessment

Consider:

Sample size per class
Computational resources
Stability requirements

RandomState

u64

Seed for reproducible stratified sampling. Ensures:

Consistent class proportions
Reproducible splits
Comparable experiments

Critical for:

Benchmarking
Research studies
Quality assurance

TestSize

f64

0.2

Fraction of samples for stratified test set. Best practices: Common splits:

0.2: Balanced evaluation
0.3: More thorough testing
0.15: Preserve training size

Consider:

Minority class size
Overall dataset size
Validation objectives

It must be between 0.0 and 1.0.

TimeSeriesSplitCv

Time Series cross-validator. Provides train/test indices to split time series data samples that are observed at fixed time intervals, in train/test sets. It is a variation of k-fold which returns first k folds as train set and the k + 1th fold as test set. Note that unlike standard cross-validation methods, successive training sets are supersets of those that come before them. Also, it adds all surplus data to the first training partition, which is always used to train the model. Key features:

Maintains temporal dependence
Expanding window approach
Forward-chaining splits
No future data leakage

Use cases:

Sequential data
Financial forecasting
Temporal predictions
Time-dependent patterns

Note: Training sets are supersets of previous iterations.

NSplits

u32

Number of temporal splits. Considerations: Typical values:

5: Standard forward chaining
3: Limited historical data
10: Long time series

Impact:

Affects training window growth
Determines validation points
Influences computational load

MaxTrainSize

u64

Maximum size of training set. Should be strictly less than the number of samples. Applications:

0: Use all available past data
>0: Rolling window of fixed size

Use cases:

Limit historical relevance
Control computational cost
Handle concept drift
Memory constraints

TestSize

u64

Number of samples in each test set. When 0:

Auto-calculated as n_samples/(n_splits+1)
Ensures equal-sized test sets

Considerations:

Forecast horizon
Validation requirements
Available future data

Gap

u64

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

Avoid data leakage
Model forecast lag
Buffer periods

Common scenarios:

0: Continuous prediction
>0: Forward gap for realistic evaluation
Match business forecasting needs