RandomForest / Regressor Layer

Number of trees in the forest:

Impact on model: $$Var(\hat{f})\propto \frac{\rho }{B}{\sigma }^2$$ where:

• B is number of trees
• ρ is tree correlation
• σ² is tree variance

Guidelines:

• Small (10-50): Fast training
• Medium (100-300): Standard use
• Large (300+): High accuracy

Trade-offs:

• More trees = Better stability
• Diminishing returns after threshold
• Linear memory scaling
• Parallel training capable

Split quality measurement criteria:

Selection impact:

• Prediction accuracy
• Training dynamics
• Outlier sensitivity
• Computational efficiency

Use cases:

1. General regression: MSE
2. Numeric stability: Friedman MSE
3. Robust modeling: MAE
4. Count prediction: Poisson

Trade-offs:

• Accuracy vs speed
• Robustness vs sensitivity
• Bias vs variance
• Memory vs precision

Maximum tree depth limit:

Complexity control:

• 0: Unlimited depth
• log₂(n): Balanced trees
• Small (3-7): Simple models
• Large (8+): Complex patterns

Impact:

• Memory usage: O(2^depth)
• Training time
• Model complexity
• Overfitting risk

Guidelines:

• Start shallow
• Monitor validation
• Consider sample size
• Balance with min_samples

Minimum samples for node split:

Guidelines:

• Small (2-5): Detailed splits
• Medium (5-20): Balanced
• Large (20+): Conservative

Effects:

• Prevents overfitting
• Controls tree size
• Affects leaf purity
• Smooths predictions

Scale with dataset:

• Small data: Lower values
• Large data: Higher values
• Consider noise level

Minimum samples in leaf nodes:

Guidelines:

• 1: Maximum detail
• 2-10: Standard range
• 10+: Smooth predictions

Benefits:

• Prevents single-sample leaves
• Reduces variance
• Stabilizes predictions
• Controls overfitting

Consider with:

• Dataset size
• Noise level
• Prediction stability needs

Minimum weighted fraction at leaf:

Usage:

• 0.0: No constraint
• 0.1: 10% minimum weight
• 0.5: Balanced trees

Applications:

• Imbalanced data
• Weighted samples
• Cost-sensitive learning
• Stratified modeling

Note: Requires sample_weights

Feature subset selection strategy for splits:

Tree building process:

• At each split, sample features
• Select best feature from subset
• Increases tree diversity
• Controls randomization

Trade-offs:

• More features = Better local splits
• Fewer features = More diverse trees
• Speed vs randomization
• Memory vs computation

Selection guide:

• High dim data: Use sqrt/log2
• Few features: Try all
• Specific needs: Use custom
• Unsure: Start with sqrt

Custom value of max features. Only used when max_features is custom.

Maximum leaf node limit:

Tree size control:

• 0: Unlimited leaves
• log₂(n): Balanced trees
• n/10: Conservative size

Properties:

• Controls tree complexity
• Alternative to max_depth
• Memory efficient
• Best-first growth

Usage scenarios:

• Memory constraints
• Speed requirements
• Size control needs
• Model simplification

Minimum impurity decrease for split:

Usage:

• 0.0: All valid splits
• Small (1e-5 to 1e-3): Fine control
• Large (> 1e-2): Aggressive pruning

Benefits:

• Prevents weak splits
• Reduces complexity
• Improves efficiency
• Natural pruning

Whether bootstrap samples are used when building trees.

When True:

• Sample with replacement
• Enables OOB estimation
• Reduces variance

When False:

• Use all samples
• No sample duplication
• Lower bias
• No OOB scores

Trade-offs:

• Bias vs variance
• Training stability
• Model diversity
• Evaluation options

Whether to use out-of-bag samples to estimate the generalization sxore.

Controls both the randomness of the bootstrapping of the samples used when building trees (if bootstrap=True) and the sampling of the features to consider when looking for the best split at each node (if max_features < n_features).

Controls randomization in:

• Bootstrap sampling
• Feature selection
• Split selection
• Tree building

Usage patterns:

• 0: System random source
• Fixed value: Reproducible results
• None: Non-deterministic

Important for:

• Reproducibility
• Debugging
• Cross-validation
• Research studies

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

When True:

• Keep existing trees
• Add more estimators
• Incremental training
• Continue fitting

Applications:

• Online learning
• Model updating
• Iterative fitting
• Performance tuning

Considerations:

• Memory usage
• Training history
• Parameter consistency
• State management

Cost-Complexity Pruning alpha:

Pruning criterion: $$R_{\alpha }(T)=R(T)+\alpha |T|$$ where:

• R(T) is tree error
• |T| is leaf count
• α is complexity parameter

Values:

• 0.0: No pruning
• Small (1e-4 to 1e-2): Light pruning
• Large (> 1e-2): Heavy pruning

Benefits:

• Reduces overfitting
• Optimizes size
• Improves generalization
• Controls complexity

If bootstrap is True, the fraction of samples to draw from X to train each base estimator.

Settings:

• 0.0: Auto (n_samples)
• < 1.0: Fraction of samples
• 1.0: Full bootstrap

Impact:

• Tree diversity
• Training speed
• Memory usage
• Model variance

Note: Only used when bootstrap=True

Number of trees search space:

Search strategies:

1. Logarithmic scale:

   • [10, 50, 100, 200, 500]
   • Broad exploration
   • Resource efficient

2. Fine-tuning:

   • [80, 100, 120, 150]
   • Around optimal value
   • Precision focus

3. Large-scale:

   • [200, 500, 1000]
   • High accuracy needs
   • Resource intensive

Consider:

• Training time
• Memory limits
• Accuracy needs
• Diminishing returns

Split quality measurement criteria:

Selection impact:

• Prediction accuracy
• Training dynamics
• Outlier sensitivity
• Computational efficiency

Use cases:

1. General regression: MSE
2. Numeric stability: Friedman MSE
3. Robust modeling: MAE
4. Count prediction: Poisson

Trade-offs:

• Accuracy vs speed
• Robustness vs sensitivity
• Bias vs variance
• Memory vs precision

Tree depth search space:

Search patterns:

1. Basic range:

   • [5, 10, 15, None]
   • Standard problems
   • Balanced approach

2. Shallow trees:

   • [3, 5, 7, 9]
   • Simple patterns
   • Fast training

3. Deep trees:

   • [10, 15, 20, None]
   • Complex patterns
   • Large datasets

Monitor:

• Overfitting risk
• Memory usage
• Training time
• Model complexity

Minimum split samples search:

Search ranges:

1. Fine-grained:

   • [2, 5, 10]
   • Detailed splits
   • Small datasets

2. Standard:

   • [10, 20, 50]
   • Medium datasets
   • Balanced control

3. Conservative:

   • [50, 100, 200]
   • Large datasets
   • Noise control

Scale with:

• Dataset size
• Noise level
• Model stability
• Memory constraints

Minimum leaf samples search:

Search spaces:

1. Detailed:

   • [1, 2, 4]
   • Fine patterns
   • Clean data

2. Standard:

   • [5, 10, 20]
   • Normal noise
   • Stable predictions

3. Robust:

   • [20, 50, 100]
   • High noise
   • Smooth predictions

Consider with:

• min_samples_split
• Data noise level
• Prediction stability
• Sample size

Leaf weight fraction search:

Search ranges:

1. No constraint:

   • [0.0]
   • Default behavior
   • Unweighted cases

2. Light weighting:

   • [0.0, 0.1, 0.2]
   • Mild balancing
   • Gentle constraint

3. Heavy weighting:

   • [0.2, 0.3, 0.4]
   • Strong balancing
   • Strict constraint

Use with:

• Sample weights
• Class imbalance
• Cost sensitivity
• Data distribution

Feature subset selection strategy for splits:

Tree building process:

• At each split, sample features
• Select best feature from subset
• Increases tree diversity
• Controls randomization

Trade-offs:

• More features = Better local splits
• Fewer features = More diverse trees
• Speed vs randomization
• Memory vs computation

Selection guide:

• High dim data: Use sqrt/log2
• Few features: Try all
• Specific needs: Use custom
• Unsure: Start with sqrt

Custom feature count search:

Search patterns:

1. Conservative:

   • [n_features/3]
   • Safe selection
   • Fast training

2. Standard:

   • [n_features/2]
   • Balanced choice
   • Medium range

3. Aggressive:

   • [2n_features/3]
   • More features
   • Better splits

Note: Used only with MaxFeatures = Custom

Maximum leaf nodes search:

Search spaces:

1. Unlimited: [0]

   • No restriction
   • Full growth

2. Controlled:

   • [10, 50, 100]
   • Size limiting
   • Memory efficient

3. Large-scale:

   • [100, 500, 1000]
   • Complex trees
   • Big datasets

Balance with:

• max_depth
• Memory limits
• Model complexity
• Training speed

Impurity decrease threshold search:

Search ranges:

1. No pruning: [0.0]

   • All splits allowed
   • Maximum detail

2. Light pruning:

   • [1e-5, 1e-4, 1e-3]
   • Noise reduction
   • Gentle control

3. Heavy pruning:

   • [1e-3, 1e-2, 1e-1]
   • Strong control
   • Significant reduction

Impact:

• Tree complexity
• Training speed
• Model size
• Generalization

Bootstrap sampling evaluation:

Options:

1. Standard: [true]

   • With replacement
   • OOB available

2. Alternative: [false]

   • Full samples
   • Lower variance

3. Compare: [true, false]

   • Complete study
   • Method impact

Consider:

• Dataset size
• Model stability
• OOB estimation needs
• Training dynamics

Out-of-bag scoring options:

Choices:

1. Disabled: [false]

   • No OOB estimate
   • Faster training

2. Enabled: [true]

   • With estimation
   • Extra validation

3. Compare: [true, false]

   • Full evaluation
   • Cost-benefit study

Requirements:

• bootstrap = true
• Sufficient samples
• Memory overhead
• Computation cost

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

Options:

1. Enable: [true]

   • Reuse trees
   • Incremental fits

2. Disable: [false]

   • Fresh starts
   • Independent fits

3. Compare: [true, false]

   • Method study
   • Performance impact

Use cases:

• Online learning
• Parameter tuning
• Memory efficiency
• Iterative training

Cost-complexity pruning search:

Search ranges:

1. No pruning: [0.0]

   • Full trees
   • Maximum detail

2. Light pruning:

   • [0.001, 0.01, 0.1]
   • Size optimization
   • Gentle reduction

3. Heavy pruning:

   • [0.1, 0.2, 0.3]
   • Strong reduction
   • Simple trees

Monitor:

• Tree complexity
• Performance impact
• Model size
• Generalization

Bootstrap sample size search:

Search spaces:

1. Default: [0.0]

   • Auto selection
   • Full bootstrapping

2. Reduced:

   • [0.5, 0.7, 0.9]
   • Faster training
   • Memory efficient

3. Fine-tuning:

   • [0.8, 0.9, 1.0]
   • Performance study
   • Optimal size

Consider:

• Dataset size
• Memory limits
• Training speed
• Model diversity

Random seed control:

Usage:

• 0: System random
• Fixed: Reproducible
• None: Random behavior

Important for:

• Result stability
• Cross-validation
• Parameter tuning
• Research studies

Regression model evaluation metrics:

Purpose:

• Model performance evaluation
• Error measurement
• Quality assessment
• Model comparison

Selection criteria:

• Error distribution
• Scale sensitivity
• Domain requirements
• Business objectives

Random seed for reproducible splits. Ensures:

• Consistent train/test sets
• Reproducible experiments
• Comparable model evaluations

Same seed guarantees identical splits across runs.

Data shuffling before splitting. Effects:

• true: Randomizes order, better for i.i.d. data
• false: Maintains order, important for time series

When to disable:

• Time dependent data
• Sequential patterns
• Grouped observations

Proportion of data for training. Considerations:

• Larger (e.g., 0.8-0.9): Better model learning
• Smaller (e.g., 0.5-0.7): Better validation

Common splits:

• 0.8: Standard (80/20 split)
• 0.7: More validation emphasis
• 0.9: More training emphasis

Maintain class distribution in splits. Important when:

• Classes are imbalanced
• Small classes present
• Representative splits needed

Requirements:

• Classification tasks only
• Cannot use with shuffle=false
• Sufficient samples per class

Number of cross-validation folds. Guidelines:

• 3-5: Large datasets, faster training
• 5-10: Standard choice, good balance
• 10+: Small datasets, thorough evaluation

Trade-offs:

• More folds: Better evaluation, slower training
• Fewer folds: Faster training, higher variance

Must be at least 2.

Number of folds for cross-validation. Selection guide: Recommended values:

• 5: Standard choice (default)
• 3: Large datasets/quick evaluation
• 10: Thorough evaluation/smaller datasets

Trade-offs:

• Higher values: More thorough, computationally expensive
• Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

Random seed for fold generation when shuffling. Important for:

• Reproducible results
• Consistent fold assignments
• Benchmark comparisons
• Debugging and validation

Set specific value for reproducibility across runs.

Whether to shuffle data before splitting into folds. Effects:

• true: Randomized fold composition (recommended)
• false: Sequential splitting

Enable when:

• Data may have ordering
• Better fold independence needed

Disable for:

• Time series data
• Ordered observations

Number of stratified folds. Guidelines: Typical values:

• 5: Standard for most cases
• 3: Quick evaluation/large datasets
• 10: Detailed evaluation/smaller datasets

Considerations:

• Must allow sufficient samples per class per fold
• Balance between stability and computation time
• Consider smallest class size when choosing

Seed for reproducible stratified splits. Ensures:

• Consistent fold assignments
• Reproducible results
• Comparable experiments
• Systematic validation

Fixed seed guarantees identical stratified splits.

Data shuffling before stratified splitting. Impact:

• true: Randomizes while maintaining stratification
• false: Maintains data order within strata

Use cases:

• true: Independent observations
• false: Grouped or sequential data

Class proportions maintained regardless of setting.

Number of random splits to perform. Consider: Common values:

• 5: Standard evaluation
• 10: More thorough assessment
• 3: Quick estimates

Trade-offs:

• More splits: Better estimation, longer runtime
• Fewer splits: Faster, less stable estimates

Balance between computation and stability.

Random seed for reproducible shuffling. Controls:

• Split randomization
• Sample selection
• Result reproducibility

Important for:

• Debugging
• Comparative studies
• Result verification

Proportion of samples for test set. Guidelines: Common ratios:

• 0.2: Standard (80/20 split)
• 0.25: More validation emphasis
• 0.1: More training data

Considerations:

• Dataset size
• Model complexity
• Validation requirements

It must be between 0.0 and 1.0.

Number of stratified random splits. Guidelines: Recommended values:

• 5: Standard evaluation
• 10: Detailed analysis
• 3: Quick assessment

Consider:

• Sample size per class
• Computational resources
• Stability requirements

Seed for reproducible stratified sampling. Ensures:

• Consistent class proportions
• Reproducible splits
• Comparable experiments

Critical for:

• Benchmarking
• Research studies
• Quality assurance

Fraction of samples for stratified test set. Best practices: Common splits:

• 0.2: Balanced evaluation
• 0.3: More thorough testing
• 0.15: Preserve training size

Consider:

• Minority class size
• Overall dataset size
• Validation objectives

It must be between 0.0 and 1.0.

Number of temporal splits. Considerations: Typical values:

• 5: Standard forward chaining
• 3: Limited historical data
• 10: Long time series

Impact:

• Affects training window growth
• Determines validation points
• Influences computational load

Maximum size of training set. Should be strictly less than the number of samples. Applications:

• 0: Use all available past data
• >0: Rolling window of fixed size

Use cases:

• Limit historical relevance
• Control computational cost
• Handle concept drift
• Memory constraints

Number of samples in each test set. When 0:

• Auto-calculated as n_samples/(n_splits+1)
• Ensures equal-sized test sets

Considerations:

• Forecast horizon
• Validation requirements
• Available future data

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

• Avoid data leakage
• Model forecast lag
• Buffer periods

Common scenarios:

• 0: Continuous prediction
• >0: Forward gap for realistic evaluation
• Match business forecasting needs