ExtraTrees / Regressor Layer

Number of trees in forest:

Impact:

• Variance reduction: $$Var(\bar{f})=\frac{\rho {\sigma }^2}{M}$$ where M is n_estimators

Guidelines:

• Small: 50-100 (fast)
• Medium: 100-300 (balanced)
• Large: 300+ (stable)

Trade-off: Accuracy vs speed

Split quality measures:

Mathematical forms:

1. Squared Error: $$\sum (y-\bar{y}{)}^2$$
2. Friedman MSE: Enhanced variance reduction
3. Absolute Error: $$\sum |y-median(y)|$$
4. Poisson: $$2\sum (y\log (\frac{y}{\hat{y}})-(y-\hat{y}))$$

Impact on splits:

• Error measurement
• Node homogeneity
• Prediction type
• Training dynamics

Maximum tree depth:

Control options:

• 0: Unlimited growth
• >0: Fixed depth limit

Effects:

• Model complexity
• Memory usage
• Training speed

Typical values:

• Shallow: 3-5
• Medium: 5-10
• Deep: 10+

Minimum samples for split:

Purpose:

• Controls granularity
• Prevents overfitting
• Ensures stability

Common values:

• 2: Maximum splits
• 5-10: Balanced
• >10: Conservative

Minimum samples in leaves:

Benefits:

• Stable predictions
• Prevents overfitting
• Controls variance

Settings:

• 1: Maximum detail
• 5-10: Stable
• >10: Smooth

Minimum weighted leaf fraction:

Usage:

• 0.0: No constraint
• >0.0: Weight control
• Range: [0.0, 0.5]

For weighted samples

Feature subset size control:

Options:

1. Auto: All features
2. Sqrt: √n_features
3. Log2: log₂(n_features)
4. Custom: User-defined

Impact:

• Split randomization
• Tree diversity
• Computation speed
• Feature exploration

Custom feature count:

Usage:

• Active with MaxFeatures=Custom
• Must be ≤ total features

Selection guide:

• Small: More randomization
• Large: Better splits
• Balance: Speed vs quality

Maximum leaf node count:

Control:

• 0: Unlimited leaves
• >0: Best-first growth

Effects:

• Tree size control
• Memory bounds
• Model complexity

Alternative to max_depth

Minimum impurity decrease:

Purpose:

• Prevents weak splits
• Quality control
• Pre-pruning method

Higher values: More conservative

Whether samples are drawn with replacement.

When True:

• Random sampling with replacement
• ~63.2% unique samples per tree
• Increased diversity

When False:

• Use full dataset
• Default for Extra Trees
• Pure random splits

Whether to use out-of-bag samples to estimate the generalization score.

Requirements:

• bootstrap = True
• Sufficient samples

Benefits:

• Unbiased estimation
• No validation set needed
• Performance monitoring

Random number control:

Affects:

• Feature selection
• Split points
• Bootstrap samples

Settings:

• 0: System random
• Fixed: Reproducible
• Different: Variations

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

When True:

• Keep existing trees
• Add more estimators
• Continue training

When False:

• Fresh forest
• Complete rebuild
• Independent training

Cost-complexity pruning:

Criterion: $$R_{\alpha }(T)=R(T)+\alpha |T|$$

Impact:

• Larger α: More pruning
• Smaller α: Less pruning
• 0: No pruning

Post-training optimization

If bootstrap is True, the fraction of samples to draw from X to train each base estimator.

When bootstrap=True:

• Fraction of samples
• Range: [0, 1.0]
• 0: Use full size

Effects:

• Tree diversity
• Training speed
• Memory usage

Forest size search:

Search patterns:

1. Basic range:

   • [50, 100, 200]
   • Initial testing
   • Quick evaluation

2. Production range:

   • [100, 300, 500]
   • Full performance
   • Stable results

3. Large-scale:

   • [500, 1000]
   • Maximum stability
   • Resource intensive

Split quality measures:

Mathematical forms:

1. Squared Error: $$\sum (y-\bar{y}{)}^2$$
2. Friedman MSE: Enhanced variance reduction
3. Absolute Error: $$\sum |y-median(y)|$$
4. Poisson: $$2\sum (y\log (\frac{y}{\hat{y}})-(y-\hat{y}))$$

Impact on splits:

• Error measurement
• Node homogeneity
• Prediction type
• Training dynamics

Tree depth search:

Search spaces:

1. Shallow trees:

   • [3, 5, 7]
   • Fast training
   • Simple models

2. Deep trees:

   • [10, 15, 20]
   • Complex patterns
   • More capacity

3. Mixed range:

   • [5, 10, None]
   • Compare depths

Split threshold search:

Search patterns:

1. Detailed splits:

   • [2, 5, 10]
   • Fine granularity

2. Conservative:

   • [10, 20, 50]
   • Stable splits

3. Mixed approach:

   • [2, 10, 30]
   • Range comparison

Leaf size search:

Search ranges:

1. Fine-grained:

   • [1, 2, 4]
   • Detailed predictions
   • Maximum splits

2. Stable leaves:

   • [5, 10, 20]
   • Robust predictions
   • Noise reduction

3. Balanced:

   • [1, 5, 10]
   • Compare effects

Weighted leaf fraction search:

Search spaces:

1. No constraint:

   • [0.0]
   • Default behavior

2. Light weights:

   • [0.0, 0.1, 0.2]
   • Gentle control

3. Strong weights:

   • [0.2, 0.3, 0.4]
   • Heavy balancing

Feature subset size control:

Options:

1. Auto: All features
2. Sqrt: √n_features
3. Log2: log₂(n_features)
4. Custom: User-defined

Impact:

• Split randomization
• Tree diversity
• Computation speed
• Feature exploration

Custom feature count search:

Patterns:

1. Small subsets:

   • [2, 4, 6]
   • High randomization

2. Large subsets:

   • [8, 12, 16]
   • More information

Note: Used with MaxFeatures=Custom

Leaf count search:

Search ranges:

1. Limited trees:

   • [10, 20, 30]
   • Controlled size

2. Large trees:

   • [50, 100, None]
   • Full growth

3. Mixed:

   • [20, 50, 100]
   • Size comparison

Impurity threshold search:

Search patterns:

1. Fine control:

   • [0.0, 1e-4, 1e-3]
   • Subtle pruning

2. Strong control:

   • [1e-3, 1e-2, 1e-1]
   • Heavy pruning

3. Range study:

   • Multiple scales
   • Effect analysis

Bootstrap sampling search:

Options:

1. Extra Trees style:

   • [false]
   • Pure random splits

2. Random Forest style:

   • [true]
   • With replacement

3. Compare both:

   • [true, false]
   • Method study

OOB scoring search:

Options:

1. No OOB: [false]

   • Faster training
   • Standard CV

2. With OOB: [true]

   • Built-in validation
   • Extra metrics

Note: Requires bootstrap=true

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

Options:

1. Fresh start: [false]

   • New ensemble
   • Independent runs

2. Incremental: [true]

   • Add estimators
   • Continue training

3. Compare: [true, false]

   • Study effect

Random seed control:

Applications:

1. Development:

   • Fixed seed
   • Reproducible
   • Debugging

2. Production:

   • Multiple seeds
   • Stability check
   • Model variation

Cost-complexity search:

Search ranges:

1. No pruning:

   • [0.0]
   • Full trees

2. Light pruning:

   • [0.001, 0.01, 0.1]
   • Gentle effect

3. Heavy pruning:

   • [0.1, 0.2, 0.3]
   • Strong simplification

Sample fraction search:

Search spaces:

1. Large samples:

   • [0.7, 0.8, 0.9]
   • More stable

2. Small samples:

   • [0.4, 0.5, 0.6]
   • More diversity

3. Full range:

   • [0.3, 0.6, 0.9]
   • Compare effects

Regression model evaluation metrics:

Purpose:

• Model performance evaluation
• Error measurement
• Quality assessment
• Model comparison

Selection criteria:

• Error distribution
• Scale sensitivity
• Domain requirements
• Business objectives

Random seed for reproducible splits. Ensures:

• Consistent train/test sets
• Reproducible experiments
• Comparable model evaluations

Same seed guarantees identical splits across runs.

Data shuffling before splitting. Effects:

• true: Randomizes order, better for i.i.d. data
• false: Maintains order, important for time series

When to disable:

• Time dependent data
• Sequential patterns
• Grouped observations

Proportion of data for training. Considerations:

• Larger (e.g., 0.8-0.9): Better model learning
• Smaller (e.g., 0.5-0.7): Better validation

Common splits:

• 0.8: Standard (80/20 split)
• 0.7: More validation emphasis
• 0.9: More training emphasis

Maintain class distribution in splits. Important when:

• Classes are imbalanced
• Small classes present
• Representative splits needed

Requirements:

• Classification tasks only
• Cannot use with shuffle=false
• Sufficient samples per class

Number of cross-validation folds. Guidelines:

• 3-5: Large datasets, faster training
• 5-10: Standard choice, good balance
• 10+: Small datasets, thorough evaluation

Trade-offs:

• More folds: Better evaluation, slower training
• Fewer folds: Faster training, higher variance

Must be at least 2.

Number of folds for cross-validation. Selection guide: Recommended values:

• 5: Standard choice (default)
• 3: Large datasets/quick evaluation
• 10: Thorough evaluation/smaller datasets

Trade-offs:

• Higher values: More thorough, computationally expensive
• Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

Random seed for fold generation when shuffling. Important for:

• Reproducible results
• Consistent fold assignments
• Benchmark comparisons
• Debugging and validation

Set specific value for reproducibility across runs.

Whether to shuffle data before splitting into folds. Effects:

• true: Randomized fold composition (recommended)
• false: Sequential splitting

Enable when:

• Data may have ordering
• Better fold independence needed

Disable for:

• Time series data
• Ordered observations

Number of stratified folds. Guidelines: Typical values:

• 5: Standard for most cases
• 3: Quick evaluation/large datasets
• 10: Detailed evaluation/smaller datasets

Considerations:

• Must allow sufficient samples per class per fold
• Balance between stability and computation time
• Consider smallest class size when choosing

Seed for reproducible stratified splits. Ensures:

• Consistent fold assignments
• Reproducible results
• Comparable experiments
• Systematic validation

Fixed seed guarantees identical stratified splits.

Data shuffling before stratified splitting. Impact:

• true: Randomizes while maintaining stratification
• false: Maintains data order within strata

Use cases:

• true: Independent observations
• false: Grouped or sequential data

Class proportions maintained regardless of setting.

Number of random splits to perform. Consider: Common values:

• 5: Standard evaluation
• 10: More thorough assessment
• 3: Quick estimates

Trade-offs:

• More splits: Better estimation, longer runtime
• Fewer splits: Faster, less stable estimates

Balance between computation and stability.

Random seed for reproducible shuffling. Controls:

• Split randomization
• Sample selection
• Result reproducibility

Important for:

• Debugging
• Comparative studies
• Result verification

Proportion of samples for test set. Guidelines: Common ratios:

• 0.2: Standard (80/20 split)
• 0.25: More validation emphasis
• 0.1: More training data

Considerations:

• Dataset size
• Model complexity
• Validation requirements

It must be between 0.0 and 1.0.

Number of stratified random splits. Guidelines: Recommended values:

• 5: Standard evaluation
• 10: Detailed analysis
• 3: Quick assessment

Consider:

• Sample size per class
• Computational resources
• Stability requirements

Seed for reproducible stratified sampling. Ensures:

• Consistent class proportions
• Reproducible splits
• Comparable experiments

Critical for:

• Benchmarking
• Research studies
• Quality assurance

Fraction of samples for stratified test set. Best practices: Common splits:

• 0.2: Balanced evaluation
• 0.3: More thorough testing
• 0.15: Preserve training size

Consider:

• Minority class size
• Overall dataset size
• Validation objectives

It must be between 0.0 and 1.0.

Number of temporal splits. Considerations: Typical values:

• 5: Standard forward chaining
• 3: Limited historical data
• 10: Long time series

Impact:

• Affects training window growth
• Determines validation points
• Influences computational load

Maximum size of training set. Should be strictly less than the number of samples. Applications:

• 0: Use all available past data
• >0: Rolling window of fixed size

Use cases:

• Limit historical relevance
• Control computational cost
• Handle concept drift
• Memory constraints

Number of samples in each test set. When 0:

• Auto-calculated as n_samples/(n_splits+1)
• Ensures equal-sized test sets

Considerations:

• Forecast horizon
• Validation requirements
• Available future data

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

• Avoid data leakage
• Model forecast lag
• Buffer periods

Common scenarios:

• 0: Continuous prediction
• >0: Forward gap for realistic evaluation
• Match business forecasting needs