Bagging / Classifier Layer

Bagging (Bootstrap Aggregating) Classifier Ensemble: A Bagging classifier is an ensemble meta-estimator that fits base classifiers each on random subsets of the original dataset and then aggregate their individual predictions (either by voting or by averaging) to form a final prediction. Such a meta-estimator can typically be used as a way to reduce the variance of a black-box estimator (e.g., a decision tree), by introducing randomization into its construction procedure and then making an ensemble out of it.

Mathematical form: $f (x) = mode {f_{1} (x), f_{2} (x), ..., f_{M} (x)}$ where:

$f_{i} (x)$ are base classifiers
M is number of estimators
Each classifier trained on bootstrap sample

Key characteristics:

Parallel ensemble learning
Random bootstrap sampling
Independent base estimators
Variance reduction
Majority voting

Common applications:

Reducing overfitting
Stable predictions
Noisy data handling
Model averaging

Computational notes:

Parallel training possible
Memory: O(n_estimators * model_size)
Training: O(n_estimators * base_training)
Prediction: O(n_estimators)

Outputs:

Predicted Table: Input data with predictions
Validation Results: Cross-validation metrics
Test Metric: Test set performance
ROC Curve Data: ROC analysis information
Confusion Matrix: Classification breakdown
Feature Importances: Aggregated importances

Table

Predicted Table

Validation Results

Test Metric

ROC Curve Data

Confusion Matrix

Feature Importances

SelectFeatures

[column, ...]

Feature columns for Bagging Classification:

Data requirements:

General format:
- Numeric features
- Encoded categoricals
- No missing values
- Clean data
Preprocessing needs:
- Depends on base estimator
- Trees: No scaling needed
- Linear models: Standardize
- SVMs: Scale to [-1, 1]
Feature quality:
- Handle missing data
- Remove constants
- Check correlations
- Consider feature_subset_size

Note: If empty, uses all numeric columns except target

SelectTarget

column

Target column for Bagging Classification:

Requirements:

Data format:
- Categorical labels
- At least two classes
- No missing values
- Consistent encoding
Class characteristics:
- Any distribution acceptable
- Bagging preserves ratios
- Bootstrap maintains balance
- Supports multi-class
Quality checks:
- Verify encodings
- Check distributions
- Validate categories
- Monitor rare classes

Params

oneof

DefaultParams

Default configuration for Bagging Classifier:

Default settings:

Base estimator:
- DecisionTree classifier
- Default tree parameters
Ensemble parameters:
- N estimators: 10
- Max samples: 1.0 (100%)
- Max features: 1.0 (100%)
Bootstrap settings:
- Bootstrap: True (sample with replacement)
- Bootstrap features: False
- OOB score: False
Other settings:
- Warm start: False
- N jobs: None
- Random state: None

Best suited for:

Initial modeling
Quick prototyping
Standard datasets

CustomParams

Fine-tuned configuration for Bagging Classifier:

Parameter categories:

Ensemble structure
Sampling strategy
Feature selection
Training control

BaseEstimator

enum

DecisionTree

Base estimator options for Bagging ensemble:

Selection criteria:

Model stability
Training efficiency
Memory requirements
Parallelization capability

Note: Unstable learners often benefit most from bagging

DecisionTree ~

Decision Tree classifier:

Properties:

High variance (ideal for bagging)
Fast training
No preprocessing needed
Native multi-class

Best for:

Standard bagging
Random Forest-like ensembles
Most applications

RandomForest ~

Random Forest classifier:

Properties:

Already an ensemble
Built-in feature sampling
Double randomization
Parallel processing

Best for:

High-dimensional data
Complex problems
Extra randomization

ExtraTrees ~

Extremely Randomized Trees classifier:

Properties:

Additional randomization
Faster than standard trees
Lower variance
Parallel capable

Best for:

Faster training
Very large datasets
Maximum randomization

AdaBoost ~

AdaBoost classifier:

Properties:

Sequential boosting
Adaptive weights
Focus on hard cases
Complex ensembles

Best for:

Hybrid ensembles
Combining boost and bag
Difficult problems

GradientBoosting ~

Gradient Boosting classifier:

Properties:

Sequential optimization
High accuracy
Gradient-based
Strong learner

Best for:

Complex ensembles
High accuracy needs
Hybrid models

Knn ~

K-Nearest Neighbors classifier:

Properties:

Instance-based
No training phase
Memory-intensive
Local decisions

Best for:

Small datasets
Local patterns
Feature spaces

Mlp ~

Multi-Layer Perceptron classifier:

Properties:

Neural network
Non-linear mapping
Requires scaling
Complex patterns

Best for:

Deep patterns
Feature learning
Complex boundaries

LogisiticReg ~

Logistic Regression classifier:

Properties:

Linear model
Probabilistic output
Fast training
Requires scaling

Best for:

Linear problems
Large datasets
Probability needs

Svc ~

Support Vector classifier:

Properties:

Kernel methods
Margin optimization
Requires scaling
Memory intensive

Best for:

Medium datasets
Complex boundaries
Non-linear patterns

LinearSvc ~

Linear Support Vector classifier:

Properties:

Linear boundaries
Faster than SVC
Requires scaling
Memory efficient

Best for:

Large datasets
Linear problems
High dimensions

GaussianNb ~

Gaussian Naive Bayes classifier:

Properties:

Probabilistic model
Fast computation
Continuous features
Feature independence

Best for:

Real-valued features
Quick baselines
Independent features

BernoulliNb ~

Bernoulli Naive Bayes classifier:

Properties:

Binary features
Fast computation
Probabilistic model
Feature independence

Best for:

Binary data
Text classification
Sparse features

MultinomialNb ~

Multinomial Naive Bayes classifier:

Properties:

Count features
Fast computation
Probabilistic model
Feature independence

Best for:

Text data
Count features
Document classification

Lda ~

Linear Discriminant Analysis classifier:

Properties:

Linear boundaries
Dimensionality reduction
Requires scaling
Fast computation

Best for:

Multi-class
Dimensionality reduction
Linear problems

Qda ~

Quadratic Discriminant Analysis classifier:

Properties:

Quadratic boundaries
Class covariances
Requires scaling
More flexible than LDA

Best for:

Non-linear problems
Class-specific variance
Normal distributions

Sgd ~

Stochastic Gradient Descent classifier:

Properties:

Online learning
Linear model
Requires scaling
Memory efficient

Best for:

Large datasets
Online learning
Linear problems

PassiveAggressive ~

Passive Aggressive classifier:

Properties:

Online learning
Linear model
Requires scaling
Fast updates

Best for:

Online learning
Streaming data
Linear problems

Ridge ~

Ridge classifier:

Properties:

Linear model
L2 regularization
Requires scaling
Stable solutions

Best for:

Linear problems
Correlated features
Stable predictions

NEstimators

u32

Number of base estimators:

Trade-offs:

More estimators: Better stability
Parallel training possible
Memory scales linearly
Diminishing returns

MaxSamples

f64

Sample size for training each estimator: the number of samples to draw from X to train each base estimator.

Range: (0.0, 1.0]

1.0: Same size as original
<1.0: Smaller bootstrap samples. Draws max_samples * X.shape[0] samples

Note: Affects diversity and computation

MaxFeatures

f64

Feature subset size for each estimator: the number of features to draw from X to train each base estimator.

Range: (0.0, 1.0]

1.0: All features
<1.0: Feature subsampling. Draws max(1, int(max_features * n_features_in_)) features.

Note: Adds feature diversity

Bootstrap

bool

true

Whether samples are drawn with replacement.

True:

Bootstrap sampling
~63.2% unique samples
Traditional bagging

False:

Sample without replacement
All samples unique
Random subsampling

BootstrapFeatures

bool

false

Whether features are drawn with replacement.

True:

Random feature sets
May include duplicates

False:

Unique feature subsets
Standard approach

OobScore

bool

false

Whether to use out-of-bag samples to estimate the generalization error.

True:

Enables OOB estimation
Unbiased performance estimate
Additional computation

Note: Requires bootstrap=True

WarmStart

bool

false

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

True:

Add more estimators
Continue training

False:

Fresh ensemble
Start from scratch

RandomState

u64

Controls the random resampling of the original dataset (sample wise and feature wise). If the base estimator accepts a random_state attribute, a different seed is generated for each instance in the ensemble.

Controls:

Bootstrap sampling
Feature selection
Base estimator randomness

Set for reproducible results

GridSearch

Hyperparameter optimization for Bagging Classifier:

Search dimensions:

Model selection:
- Base estimators
- Ensemble size
Sampling strategy:
- Sample sizes
- Feature subsets
- Bootstrap options

Computational impact:

Time: O(n_params * n_estimators * base_training)
Memory: O(n_params * n_estimators)

Note: Parallel training possible

BaseEstimator

[enum, ...]

DecisionTree

Base estimator options for Bagging ensemble:

Selection criteria:

Model stability
Training efficiency
Memory requirements
Parallelization capability

Note: Unstable learners often benefit most from bagging

DecisionTree ~

Decision Tree classifier:

Properties:

High variance (ideal for bagging)
Fast training
No preprocessing needed
Native multi-class

Best for:

Standard bagging
Random Forest-like ensembles
Most applications

RandomForest ~

Random Forest classifier:

Properties:

Already an ensemble
Built-in feature sampling
Double randomization
Parallel processing

Best for:

High-dimensional data
Complex problems
Extra randomization

ExtraTrees ~

Extremely Randomized Trees classifier:

Properties:

Additional randomization
Faster than standard trees
Lower variance
Parallel capable

Best for:

Faster training
Very large datasets
Maximum randomization

AdaBoost ~

AdaBoost classifier:

Properties:

Sequential boosting
Adaptive weights
Focus on hard cases
Complex ensembles

Best for:

Hybrid ensembles
Combining boost and bag
Difficult problems

GradientBoosting ~

Gradient Boosting classifier:

Properties:

Sequential optimization
High accuracy
Gradient-based
Strong learner

Best for:

Complex ensembles
High accuracy needs
Hybrid models

Knn ~

K-Nearest Neighbors classifier:

Properties:

Instance-based
No training phase
Memory-intensive
Local decisions

Best for:

Small datasets
Local patterns
Feature spaces

Mlp ~

Multi-Layer Perceptron classifier:

Properties:

Neural network
Non-linear mapping
Requires scaling
Complex patterns

Best for:

Deep patterns
Feature learning
Complex boundaries

LogisiticReg ~

Logistic Regression classifier:

Properties:

Linear model
Probabilistic output
Fast training
Requires scaling

Best for:

Linear problems
Large datasets
Probability needs

Svc ~

Support Vector classifier:

Properties:

Kernel methods
Margin optimization
Requires scaling
Memory intensive

Best for:

Medium datasets
Complex boundaries
Non-linear patterns

LinearSvc ~

Linear Support Vector classifier:

Properties:

Linear boundaries
Faster than SVC
Requires scaling
Memory efficient

Best for:

Large datasets
Linear problems
High dimensions

GaussianNb ~

Gaussian Naive Bayes classifier:

Properties:

Probabilistic model
Fast computation
Continuous features
Feature independence

Best for:

Real-valued features
Quick baselines
Independent features

BernoulliNb ~

Bernoulli Naive Bayes classifier:

Properties:

Binary features
Fast computation
Probabilistic model
Feature independence

Best for:

Binary data
Text classification
Sparse features

MultinomialNb ~

Multinomial Naive Bayes classifier:

Properties:

Count features
Fast computation
Probabilistic model
Feature independence

Best for:

Text data
Count features
Document classification

Lda ~

Linear Discriminant Analysis classifier:

Properties:

Linear boundaries
Dimensionality reduction
Requires scaling
Fast computation

Best for:

Multi-class
Dimensionality reduction
Linear problems

Qda ~

Quadratic Discriminant Analysis classifier:

Properties:

Quadratic boundaries
Class covariances
Requires scaling
More flexible than LDA

Best for:

Non-linear problems
Class-specific variance
Normal distributions

Sgd ~

Stochastic Gradient Descent classifier:

Properties:

Online learning
Linear model
Requires scaling
Memory efficient

Best for:

Large datasets
Online learning
Linear problems

PassiveAggressive ~

Passive Aggressive classifier:

Properties:

Online learning
Linear model
Requires scaling
Fast updates

Best for:

Online learning
Streaming data
Linear problems

Ridge ~

Ridge classifier:

Properties:

Linear model
L2 regularization
Requires scaling
Stable solutions

Best for:

Linear problems
Correlated features
Stable predictions

NEstimators

[u32, ...]

Number of estimators to evaluate:

Typical ranges:

Quick: [5, 10, 20]
Standard: [10, 30, 50]
Large: [50, 100, 200]

Note: More estimators increase stability

MaxSamples

[f64, ...]

Sample size fractions to try:

Ranges:

Conservative: [0.8, 1.0]
Standard: [0.5, 0.7, 1.0]
Aggressive: [0.3, 0.5, 0.7, 1.0]

Note: Affects ensemble diversity

MaxFeatures

[f64, ...]

Feature subset sizes to evaluate:

Common ranges:

Full: [1.0]
Partial: [0.5, 0.7, 1.0]
Random forest-like: [0.3, 0.5, 0.7]

Note: Smaller values increase diversity

Bootstrap

[bool, ...]

true

Bootstrap sampling options:

Choices:

[true]: Traditional bagging
[false]: Random subsampling
[true, false]: Compare both

Note: Affects OOB score availability

BootstrapFeatures

[bool, ...]

false

Feature bootstrap options:

Choices:

[false]: Standard feature subsets
[true]: Random feature sets
[true, false]: Compare approaches

Note: Additional randomization level

OobScore

[bool, ...]

false

Out-of-bag scoring options:

Choices:

[false]: No OOB estimation
[true]: Use OOB samples
[true, false]: Compare both

Note: Requires bootstrap=True

WarmStart

[bool, ...]

false

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

Choices:

[false]: Fresh ensembles
[true]: Reuse estimators
[true, false]: Compare methods

Note: Affects training behavior

RandomState

u64

Random seed for reproducibility:

Controls:

Sampling randomization
Feature selection
Cross-validation splits

Fixed value ensures reproducible search

RefitScore

enum

Accuracy

Performance evaluation metrics for Bagging classification:

Purpose:

Model evaluation
Ensemble selection
Early stopping
Performance tracking

Selection criteria:

Problem objectives
Class distribution
Ensemble size
Computation resources

Default ~

Uses ensemble's built-in scoring:

Properties:

Weighted accuracy metric
Ensemble-aware scoring
Fast computation
Boosting-compatible

Best for:

Standard problems
Quick evaluation
Initial testing
Performance tracking

Accuracy ~

Standard classification accuracy:

Formula: $Accuracy = \frac{Correct Predictions}{Total Samples}$

Properties:

Range: [0, 1]
Ensemble consensus
Intuitive metric
Equal error weights

Best for:

Balanced datasets
Equal error costs
Simple evaluation
Quick benchmarking

BalancedAccuracy ~

Class-normalized accuracy score:

Formula: $\frac{1}{K} \sum_{i = 1}^{K} \frac{T P _{i}}{T P _{i} + F N _{i}}$

Properties:

Range: [0, 1]
Class-weighted
Imbalance-robust
Fair evaluation

Best for:

Imbalanced data
Varied class sizes
Minority focus
Fair assessment

LogLoss ~

Logarithmic loss (Cross-entropy):

Formula: $- \frac{1}{N} \sum_{i = 1}^{N} \sum_{j = 1}^{K} y_{ij} lo g (p_{ij})$

Properties:

Range: [0, ∞)
Probability-sensitive
Boosting-optimal
Confidence-aware

Best for:

Probability estimation
Boosting optimization
Risk assessment
Model calibration

RocAuc ~

Area Under ROC Curve:

Properties:

Range: [0, 1]
Ranking quality
Threshold-invariant
Ensemble-appropriate

Best for:

Binary problems
Ranking tasks
Score calibration
Model comparison

Note: Extended to multi-class via averaging

Split

oneof

DefaultSplit

Standard train-test split configuration optimized for general classification tasks.

Configuration:

Test size: 20% (0.2)
Random seed: 98
Shuffling: Enabled
Stratification: Based on target distribution

Advantages:

Preserves class distribution
Provides reliable validation
Suitable for most datasets

Best for:

Medium to large datasets
Independent observations
Initial model evaluation

Splitting uses the ShuffleSplit strategy or StratifiedShuffleSplit strategy depending on the field stratified. Note: If shuffle is false then stratified must be false.

CustomSplit

Configurable train-test split parameters for specialized requirements. Allows fine-tuning of data division strategy for specific use cases or constraints.

Use cases:

Time series data
Grouped observations
Specific train/test ratios
Custom validation schemes

RandomState

u64

Random seed for reproducible splits. Ensures:

Consistent train/test sets
Reproducible experiments
Comparable model evaluations

Same seed guarantees identical splits across runs.

Shuffle

bool

true

Data shuffling before splitting. Effects:

true: Randomizes order, better for i.i.d. data
false: Maintains order, important for time series

When to disable:

Time dependent data
Sequential patterns
Grouped observations

TrainSize

f64

0.8

Proportion of data for training. Considerations:

Larger (e.g., 0.8-0.9): Better model learning
Smaller (e.g., 0.5-0.7): Better validation

Common splits:

0.8: Standard (80/20 split)
0.7: More validation emphasis
0.9: More training emphasis

Stratified

bool

false

Maintain class distribution in splits. Important when:

Classes are imbalanced
Small classes present
Representative splits needed

Requirements:

Classification tasks only
Cannot use with shuffle=false
Sufficient samples per class

Cv

oneof

DefaultCv

Standard cross-validation configuration using stratified 3-fold splitting.

Configuration:

Folds: 3
Method: StratifiedKFold
Stratification: Preserves class proportions

Advantages:

Balanced evaluation
Reasonable computation time
Good for medium-sized datasets

Limitations:

May be insufficient for small datasets
Higher variance than larger fold counts
May miss some data patterns

CustomCv

Configurable stratified k-fold cross-validation for specific validation requirements.

Features:

Adjustable fold count with NFolds determining the number of splits.
Stratified sampling
Preserved class distributions

Use cases:

Small datasets (more folds)
Large datasets (fewer folds)
Detailed model evaluation
Robust performance estimation

NFolds

u32

Number of cross-validation folds. Guidelines:

3-5: Large datasets, faster training
5-10: Standard choice, good balance
10+: Small datasets, thorough evaluation

Trade-offs:

More folds: Better evaluation, slower training
Fewer folds: Faster training, higher variance

Must be at least 2.

KfoldCv

K-fold cross-validation without stratification. Divides data into k consecutive folds for iterative validation.

Process:

Splits data into k equal parts
Each fold serves as validation once
Remaining k-1 folds form training set

Use cases:

Regression problems
Large, balanced datasets
When stratification unnecessary
Continuous target variables

Limitations:

May not preserve class distributions
Less suitable for imbalanced data
Can create biased splits with ordered data

NSplits

u32

Number of folds for cross-validation. Selection guide: Recommended values:

5: Standard choice (default)
3: Large datasets/quick evaluation
10: Thorough evaluation/smaller datasets

Trade-offs:

Higher values: More thorough, computationally expensive
Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

RandomState

u64

Random seed for fold generation when shuffling. Important for:

Reproducible results
Consistent fold assignments
Benchmark comparisons
Debugging and validation

Set specific value for reproducibility across runs.

Shuffle

bool

true

Whether to shuffle data before splitting into folds. Effects:

true: Randomized fold composition (recommended)
false: Sequential splitting

Enable when:

Data may have ordering
Better fold independence needed

Disable for:

Time series data
Ordered observations

StratifiedKfoldCv

Stratified K-fold cross-validation maintaining class proportions across folds.

Key features:

Preserves class distribution in each fold
Handles imbalanced datasets
Ensures representative splits

Best for:

Classification problems
Imbalanced class distributions
When class proportions matter

Requirements:

Classification tasks only
Sufficient samples per class
Categorical target variable

NSplits

u32

Number of stratified folds. Guidelines: Typical values:

5: Standard for most cases
3: Quick evaluation/large datasets
10: Detailed evaluation/smaller datasets

Considerations:

Must allow sufficient samples per class per fold
Balance between stability and computation time
Consider smallest class size when choosing

RandomState

u64

Seed for reproducible stratified splits. Ensures:

Consistent fold assignments
Reproducible results
Comparable experiments
Systematic validation

Fixed seed guarantees identical stratified splits.

Shuffle

bool

false

Data shuffling before stratified splitting. Impact:

true: Randomizes while maintaining stratification
false: Maintains data order within strata

Use cases:

true: Independent observations
false: Grouped or sequential data

Class proportions maintained regardless of setting.

ShuffleSplitCv

Random permutation cross-validator with independent sampling.

Characteristics:

Random sampling for each split
Independent train/test sets
More flexible than K-fold
Can have overlapping test sets

Advantages:

Control over test size
Fresh splits each iteration
Good for large datasets

Limitations:

Some samples might never be tested
Others might be tested multiple times
No guarantee of complete coverage

NSplits

u32

Number of random splits to perform. Consider: Common values:

5: Standard evaluation
10: More thorough assessment
3: Quick estimates

Trade-offs:

More splits: Better estimation, longer runtime
Fewer splits: Faster, less stable estimates

Balance between computation and stability.

RandomState

u64

Random seed for reproducible shuffling. Controls:

Split randomization
Sample selection
Result reproducibility

Important for:

Debugging
Comparative studies
Result verification

TestSize

f64

0.2

Proportion of samples for test set. Guidelines: Common ratios:

0.2: Standard (80/20 split)
0.25: More validation emphasis
0.1: More training data

Considerations:

Dataset size
Model complexity
Validation requirements

It must be between 0.0 and 1.0.

StratifiedShuffleSplitCv

Stratified random permutation cross-validator combining shuffle-split with stratification.

Features:

Maintains class proportions
Random sampling within strata
Independent splits
Flexible test size

Ideal for:

Imbalanced datasets
Large-scale problems
When class distributions matter
Flexible validation schemes

NSplits

u32

Number of stratified random splits. Guidelines: Recommended values:

5: Standard evaluation
10: Detailed analysis
3: Quick assessment

Consider:

Sample size per class
Computational resources
Stability requirements

RandomState

u64

Seed for reproducible stratified sampling. Ensures:

Consistent class proportions
Reproducible splits
Comparable experiments

Critical for:

Benchmarking
Research studies
Quality assurance

TestSize

f64

0.2

Fraction of samples for stratified test set. Best practices: Common splits:

0.2: Balanced evaluation
0.3: More thorough testing
0.15: Preserve training size

Consider:

Minority class size
Overall dataset size
Validation objectives

It must be between 0.0 and 1.0.

TimeSeriesSplitCv

Time Series cross-validator. Provides train/test indices to split time series data samples that are observed at fixed time intervals, in train/test sets. It is a variation of k-fold which returns first k folds as train set and the k + 1th fold as test set. Note that unlike standard cross-validation methods, successive training sets are supersets of those that come before them. Also, it adds all surplus data to the first training partition, which is always used to train the model. Key features:

Maintains temporal dependence
Expanding window approach
Forward-chaining splits
No future data leakage

Use cases:

Sequential data
Financial forecasting
Temporal predictions
Time-dependent patterns

Note: Training sets are supersets of previous iterations.

NSplits

u32

Number of temporal splits. Considerations: Typical values:

5: Standard forward chaining
3: Limited historical data
10: Long time series

Impact:

Affects training window growth
Determines validation points
Influences computational load

MaxTrainSize

u64

Maximum size of training set. Should be strictly less than the number of samples. Applications:

0: Use all available past data
>0: Rolling window of fixed size

Use cases:

Limit historical relevance
Control computational cost
Handle concept drift
Memory constraints

TestSize

u64

Number of samples in each test set. When 0:

Auto-calculated as n_samples/(n_splits+1)
Ensures equal-sized test sets

Considerations:

Forecast horizon
Validation requirements
Available future data

Gap

u64

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

Avoid data leakage
Model forecast lag
Buffer periods

Common scenarios:

0: Continuous prediction
>0: Forward gap for realistic evaluation
Match business forecasting needs