AdaBoost / Classifier Layer

Number of boosting stages:

Properties:

• Controls ensemble size
• Affects model complexity
• Impacts training time
• Memory usage scales linearly

Guidelines:

• Small (10-50): Quick models
• Medium (50-200): Standard use
• Large (>200): Complex problems

Trade-offs:

• More estimators: Better performance but slower
• Interacts with learning rate
• Diminishing returns after convergence

Contribution weight for each classifier:

Formula: $F_m(x)=F_{m-1}(x)+\alpha h_m(x)$ where:

• $\alpha$ is learning rate
• $h_m(x)$ is weak learner

Properties:

• Controls step size
• Shrinkage parameter
• Regularization effect

Typical ranges:

• Strong (0.5-1.0): Fast learning
• Moderate (0.1-0.5): Balanced
• Weak (0.01-0.1): Careful steps

Random number generator seed. Controls the random seed given at each estimator at each boosting iteration. Thus, it is only used when estimator exposes a random_state. Pass an int for reproducible output across multiple function calls.

Important for:

• Reproducibility
• Debugging
• Result validation
• Experimental control

Split quality criteria for base estimators:

Purpose:

• Weak learner optimization
• Local decision quality
• Split point selection
• Tree construction

Impact:

• Base model accuracy
• Ensemble diversity
• Learning speed
• Model complexity

Split strategy for base estimators:

Purpose:

• Tree construction
• Ensemble diversity
• Computation control
• Memory management

Impact:

• Training speed
• Model variance
• Ensemble quality
• Resource usage

Maximum depth of base estimators:

Values:

• 1: Decision stumps (recommended)
• >1: Deeper trees
• 0: Unlimited depth

Impact:

• Weak learner complexity
• Training time
• Memory usage
• Overfitting risk

Note: Simple trees often work best in AdaBoost

Minimum samples for node splitting:

Constraint: $N\ge N_{min}$ where N is node samples

Effects:

• Controls tree growth
• Prevents overfitting
• Ensures stability
• Affects tree size

Minimum samples in leaf nodes:

Constraint: $L\ge L_{min}$ where L is leaf samples

Purpose:

• Ensures prediction stability
• Controls overfitting
• Manages leaf size
• Affects tree structure

Minimum weighted fraction at leaves:

Constraint: $W_L\ge W_{min}$ where W is sample weight sum

Usage:

• Weighted sample control
• Adapts to boosting weights
• Alternative size control
• Handles imbalance

Feature subset size for base estimators:

Purpose:

• Feature randomization
• Ensemble diversity
• Computation control
• Overfitting prevention

Impact:

• Model variance
• Training speed
• Memory usage
• Ensemble strength

Custom value of max features. Only used when max_features is custom.

Maximum number of leaf nodes:

Values:

• 0: Unlimited leaves
• >0: Leaf count limit

Controls:

• Tree size
• Model complexity
• Memory usage
• Training speed

Minimum impurity decrease for splits:

Constraint: $I_{parent}-{\sum }_c\frac{n_c}{n}{I}_c\ge I_{min}$

Purpose:

• Quality threshold
• Prevents weak splits
• Controls growth
• Optimizes structure

Class weighting schemes for base estimators:

Purpose:

• Sample importance
• Class balance
• Error costs
• Learning focus

Note: Interacts with AdaBoost's own weighting

Cost-Complexity Pruning alpha:

Formula: $R_{\alpha }(T)=R(T)+\alpha |T|$

Effects:

• Tree simplification
• Complexity control
• Overfitting prevention
• Size optimization

Note: Less critical for decision stumps

Number of boosting stages to search:

Common patterns:

1. Quick search: [10, 50, 100]
2. Standard: [50, 100, 200, 500]
3. Extensive: [100, 250, 500, 1000]

Trade-offs:

• Model performance
• Training time
• Memory usage
• Convergence rate

Note: Consider learning_rate interaction - lower rates need more estimators

Contribution weight of each classifier to search:

Common ranges:

1. Coarse: [0.01, 0.1, 1.0]
2. Fine: [0.01, 0.05, 0.1, 0.5]
3. Detailed: [0.001, 0.01, 0.05, 0.1]

Guidelines:

• Lower rates need more estimators
• Higher rates risk overfitting
• Balance with n_estimators
• Consider convergence speed

Random seed for reproducibility:

Controls randomization in:

• Sample weight initialization
• Base estimator splits
• Feature selection

Pass an int for reproducible output across multiple function calls

Split quality criteria for base estimators:

Purpose:

• Weak learner optimization
• Local decision quality
• Split point selection
• Tree construction

Impact:

• Base model accuracy
• Ensemble diversity
• Learning speed
• Model complexity

Split strategy for base estimators:

Purpose:

• Tree construction
• Ensemble diversity
• Computation control
• Memory management

Impact:

• Training speed
• Model variance
• Ensemble quality
• Resource usage

Maximum tree depths to evaluate:

Values:

• 0: Unlimited depth
• 1: Decision stumps (common for AdaBoost)
• >1: Deeper trees (use with caution)

Note: Nodes expand until pure or below min_samples_split

Minimum samples for split thresholds to search:

Common ranges:

• Small trees: [2, 5, 10]
• Larger trees: [10, 20, 50]

Controls complexity and prevents overfitting

Minimum samples in leaf thresholds to evaluate:

Common ranges:

• Decision stumps: [1, 2]
• Deeper trees: [1, 5, 10]

Ensures statistical significance in predictions

Minimum weighted fraction of samples in leaves to search:

Typical ranges:

• Conservative: [0.0, 0.01, 0.05]
• Aggressive: [0.1, 0.2, 0.3]

Important when using sample weights or class weights

Feature subset size for base estimators:

Purpose:

• Feature randomization
• Ensemble diversity
• Computation control
• Overfitting prevention

Impact:

• Model variance
• Training speed
• Memory usage
• Ensemble strength

Feature count values to try when max_features is Custom:

Common ranges:

• Conservative: [1, 2, 3]
• Moderate: [2, 4, 6] Must be ≥ 1 and ≤ total features

Maximum leaf node counts to evaluate:

Values:

• 0: Unlimited leaves
• >0: Best-first growth with specified limit

Alternative to max_depth for controlling tree size

Minimum impurity decrease thresholds to search:

Common ranges:

• Fine: [0.0, 0.0001, 0.001]
• Coarse: [0.0, 0.001, 0.01]

Controls split quality and tree growth

Class weighting schemes for base estimators:

Purpose:

• Sample importance
• Class balance
• Error costs
• Learning focus

Note: Interacts with AdaBoost's own weighting

Cost-complexity pruning alphas to evaluate:

Typical ranges:

• Light pruning: [0.0, 0.001, 0.01]
• Heavy pruning: [0.01, 0.05, 0.1]

Controls tree complexity post-training

Performance evaluation metrics for AdaBoost classification:

Purpose:

• Model evaluation
• Ensemble selection
• Early stopping
• Performance tracking

Selection criteria:

• Problem objectives
• Class distribution
• Ensemble size
• Computation resources

Random seed for reproducible splits. Ensures:

• Consistent train/test sets
• Reproducible experiments
• Comparable model evaluations

Same seed guarantees identical splits across runs.

Data shuffling before splitting. Effects:

• true: Randomizes order, better for i.i.d. data
• false: Maintains order, important for time series

When to disable:

• Time dependent data
• Sequential patterns
• Grouped observations

Proportion of data for training. Considerations:

• Larger (e.g., 0.8-0.9): Better model learning
• Smaller (e.g., 0.5-0.7): Better validation

Common splits:

• 0.8: Standard (80/20 split)
• 0.7: More validation emphasis
• 0.9: More training emphasis

Maintain class distribution in splits. Important when:

• Classes are imbalanced
• Small classes present
• Representative splits needed

Requirements:

• Classification tasks only
• Cannot use with shuffle=false
• Sufficient samples per class

Number of cross-validation folds. Guidelines:

• 3-5: Large datasets, faster training
• 5-10: Standard choice, good balance
• 10+: Small datasets, thorough evaluation

Trade-offs:

• More folds: Better evaluation, slower training
• Fewer folds: Faster training, higher variance

Must be at least 2.

Number of folds for cross-validation. Selection guide: Recommended values:

• 5: Standard choice (default)
• 3: Large datasets/quick evaluation
• 10: Thorough evaluation/smaller datasets

Trade-offs:

• Higher values: More thorough, computationally expensive
• Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

Random seed for fold generation when shuffling. Important for:

• Reproducible results
• Consistent fold assignments
• Benchmark comparisons
• Debugging and validation

Set specific value for reproducibility across runs.

Whether to shuffle data before splitting into folds. Effects:

• true: Randomized fold composition (recommended)
• false: Sequential splitting

Enable when:

• Data may have ordering
• Better fold independence needed

Disable for:

• Time series data
• Ordered observations

Number of stratified folds. Guidelines: Typical values:

• 5: Standard for most cases
• 3: Quick evaluation/large datasets
• 10: Detailed evaluation/smaller datasets

Considerations:

• Must allow sufficient samples per class per fold
• Balance between stability and computation time
• Consider smallest class size when choosing

Seed for reproducible stratified splits. Ensures:

• Consistent fold assignments
• Reproducible results
• Comparable experiments
• Systematic validation

Fixed seed guarantees identical stratified splits.

Data shuffling before stratified splitting. Impact:

• true: Randomizes while maintaining stratification
• false: Maintains data order within strata

Use cases:

• true: Independent observations
• false: Grouped or sequential data

Class proportions maintained regardless of setting.

Number of random splits to perform. Consider: Common values:

• 5: Standard evaluation
• 10: More thorough assessment
• 3: Quick estimates

Trade-offs:

• More splits: Better estimation, longer runtime
• Fewer splits: Faster, less stable estimates

Balance between computation and stability.

Random seed for reproducible shuffling. Controls:

• Split randomization
• Sample selection
• Result reproducibility

Important for:

• Debugging
• Comparative studies
• Result verification

Proportion of samples for test set. Guidelines: Common ratios:

• 0.2: Standard (80/20 split)
• 0.25: More validation emphasis
• 0.1: More training data

Considerations:

• Dataset size
• Model complexity
• Validation requirements

It must be between 0.0 and 1.0.

Number of stratified random splits. Guidelines: Recommended values:

• 5: Standard evaluation
• 10: Detailed analysis
• 3: Quick assessment

Consider:

• Sample size per class
• Computational resources
• Stability requirements

Seed for reproducible stratified sampling. Ensures:

• Consistent class proportions
• Reproducible splits
• Comparable experiments

Critical for:

• Benchmarking
• Research studies
• Quality assurance

Fraction of samples for stratified test set. Best practices: Common splits:

• 0.2: Balanced evaluation
• 0.3: More thorough testing
• 0.15: Preserve training size

Consider:

• Minority class size
• Overall dataset size
• Validation objectives

It must be between 0.0 and 1.0.

Number of temporal splits. Considerations: Typical values:

• 5: Standard forward chaining
• 3: Limited historical data
• 10: Long time series

Impact:

• Affects training window growth
• Determines validation points
• Influences computational load

Maximum size of training set. Should be strictly less than the number of samples. Applications:

• 0: Use all available past data
• >0: Rolling window of fixed size

Use cases:

• Limit historical relevance
• Control computational cost
• Handle concept drift
• Memory constraints

Number of samples in each test set. When 0:

• Auto-calculated as n_samples/(n_splits+1)
• Ensures equal-sized test sets

Considerations:

• Forecast horizon
• Validation requirements
• Available future data

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

• Avoid data leakage
• Model forecast lag
• Buffer periods

Common scenarios:

• 0: Continuous prediction
• >0: Forward gap for realistic evaluation
• Match business forecasting needs