MultiLayerPerceptron / Regressor Layer

Neural network architecture:

Layer configuration:

• Number of hidden units
• Single layer default: 100
• Multiple layers possible

Guidelines:

• Start with: √(n_in * n_out)
• Pyramid structure: Decreasing
• Consider data complexity

Impact:

• Model capacity
• Learning ability
• Computational cost
• Memory usage

Neural network activation functions:

Mathematical forms:

1. Identity: f(x) = x
2. Logistic: f(x) = 1/(1 + e⁻ˣ)
3. Tanh: f(x) = tanh(x)
4. ReLU: f(x) = max(0,x)

Properties:

• Non-linearity
• Differentiability
• Range bounds
• Gradient flow

Selection impact:

• Training dynamics
• Feature extraction
• Convergence speed
• Model capacity

Neural network optimization algorithms:

Algorithm characteristics:

1. LBFGS: Quasi-Newton method

   • Second-order optimization
   • Full batch updates
   • Memory intensive

2. SGD: Stochastic Gradient Descent

   • First-order optimization
   • Minibatch updates
   • Momentum options

3. Adam: Adaptive Moment Estimation

   • Adaptive learning rates
   • Momentum correction
   • Robust convergence

Selection criteria:

• Dataset size
• Memory constraints
• Convergence needs
• Training stability

L2 regularization strength:

Penalty term: $$\frac{\alpha }{2}||w|{|}_2^2$$

Effects:

• Weight decay
• Overfitting control
• Solution stability

Selection guide:

• Small: 1e-5 to 1e-4
• Medium: 1e-4 to 1e-3
• Large: 1e-3 to 1e-2

Mini-batch size for gradient updates:

Impact on training:

• Gradient estimation
• Memory usage
• Update frequency
• Training speed

Selection guidelines:

• Small (16-32):

  • Better generalization
  • Noisy updates
  • Less memory

• Large (128-256):

  • Stable gradients
  • Faster training
  • GPU efficiency

Note: Ignored for LBFGS solver

Learning rate scheduling strategies:

Schedule types:

1. Constant: Fixed rate
2. Inverse scaling: Time decay
3. Adaptive: Performance-based

Impact on training:

• Convergence speed
• Solution quality
• Training stability
• Optimization dynamics

Initial learning rate:

Importance:

• Controls step size
• Training stability
• Convergence speed

Typical ranges:

• Conservative: 1e-4
• Standard: 1e-3
• Aggressive: 1e-2

Note: Critical for SGD/Adam

Inverse scaling exponent:

Schedule formula: $${\eta }_t={\eta }_0/t^{power\_t}$$

Common values:

• 0.5: Standard decay
• 0.25: Slower decay
• 1.0: Faster decay

Note: Used with invscaling

Maximum training iterations:

Training control:

• Complete dataset passes
• Convergence limit
• Resource control

Guidelines:

• Simple: 100-200
• Complex: 500-1000
• Deep: 1000+

Consider with early stopping

Whether to shuffle samples in each iteration.

Benefits:

• Prevents cycles
• Better generalization
• Reduces bias
• Improves convergence

When False:

• Deterministic order
• Debugging easier
• Time series cases

Note: For SGD and Adam only

Random number generation seed:

Controls:

• Weight initialization
• Data shuffling
• Batch sampling
• Reproducibility

Usage:

• Fixed: Reproducible runs
• Different: Robustness check
• 0: System random

Optimization tolerance:

Convergence criterion: $$|los{s}_{t+1}-los{s}_t|<tol$$

Values:

• Strict: 1e-5
• Standard: 1e-4
• Relaxed: 1e-3

Impact:

• Solution precision
• Training duration
• Stopping condition

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

When True:

• Reuses weights
• Continues training
• Transfer learning
• Incremental learning

Best for:

• Fine-tuning
• Progressive training
• Online learning

Gradient descent momentum:

Update rule: $$v_t=\mu v_{t-1}+\nabla L(w_t)$$ $$w_{t+1}=w_t-\eta v_t$$

Effects:

• Accelerates training
• Reduces oscillations
• Better local minima
• Smoother convergence

Typical values:

• None: 0.0
• Standard: 0.9
• High: 0.99

Note: SGD solver only

Nesterov momentum control:

Update rule: $$v_t=\mu v_{t-1}+\nabla L(w_t-\mu v_{t-1})$$ $$w_{t+1}=w_t-\eta v_t$$

Advantages:

• Faster convergence
• Better theoretical properties
• Look-ahead gradient
• Enhanced momentum

Use when:

• momentum > 0
• SGD solver
• Speed critical

Early stopping control:

Mechanism:

• Monitors validation score
• Stops on plateau
• Prevents overfitting
• Saves computation

Requirements:

• Validation fraction
• Patience setting
• Tolerance level
• Score monitoring

Best practices:

• Use with sufficient data
• Monitor validation curve
• Balance patience/tolerance

Early stopping validation size:

Data split:

• Training: 1 - fraction
• Validation: fraction

Guidelines:

• Small data: 0.2
• Medium data: 0.1
• Large data: 0.05

Trade-offs:

• Validation quality
• Training data size
• Stopping reliability

Adam first moment decay:

Role in Adam: $$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t$$

Properties:

• Momentum control
• Gradient smoothing
• Historical influence

Typical values:

• Standard: 0.9
• Range: [0.85, 0.95]
• Paper default: 0.9

Adam second moment decay:

Role in Adam: $$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2$$

Properties:

• Variance control
• Adaptive learning
• Scale adjustment

Typical values:

• Standard: 0.999
• Range: [0.99, 0.9999]
• Paper default: 0.999

Adam numerical stability:

Used in update: $$w_t=w_{t-1}-\eta \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}$$

Purpose:

• Prevents division by zero
• Numerical stability
• Update conditioning

Typical values:

• Default: 1e-8
• Range: [1e-10, 1e-7]
• Stability vs precision

Early stopping patience:

Stopping logic:

• Monitor n consecutive epochs
• Check improvement > tol
• Stop if no progress

Settings:

• Short: 5-10 epochs
• Medium: 10-20 epochs
• Long: 20+ epochs

Trade-offs:

• Training time
• Convergence certainty
• Resource usage

LBFGS function evaluation limit:

Controls:

• Optimization depth
• Computation budget
• Solution precision

Guidelines:

• Standard: 15000
• Simple: 5000-10000
• Complex: 20000+

Note: LBFGS solver only

Network architecture search:

Search patterns:

1. Single layer:

   • [50, 100, 200]
   • Width exploration

2. Multiple layers:

   • [32-16, 64-32, 128-64]
   • Depth vs width

3. Systematic:

   • Based on input size
   • Geometric progression
   • Capacity scaling

Neural network activation functions:

Mathematical forms:

1. Identity: f(x) = x
2. Logistic: f(x) = 1/(1 + e⁻ˣ)
3. Tanh: f(x) = tanh(x)
4. ReLU: f(x) = max(0,x)

Properties:

• Non-linearity
• Differentiability
• Range bounds
• Gradient flow

Selection impact:

• Training dynamics
• Feature extraction
• Convergence speed
• Model capacity

Neural network optimization algorithms:

Algorithm characteristics:

1. LBFGS: Quasi-Newton method

   • Second-order optimization
   • Full batch updates
   • Memory intensive

2. SGD: Stochastic Gradient Descent

   • First-order optimization
   • Minibatch updates
   • Momentum options

3. Adam: Adaptive Moment Estimation

   • Adaptive learning rates
   • Momentum correction
   • Robust convergence

Selection criteria:

• Dataset size
• Memory constraints
• Convergence needs
• Training stability

L2 regularization search:

Search spaces:

1. Standard range:

   • [1e-5, 1e-4, 1e-3]
   • Common values

2. Wide range:

   • [1e-6, 1e-4, 1e-2]
   • Regularization study

3. Fine-tuning:

   • Around best alpha
   • Narrow search

Mini-batch size search:

Search patterns:

1. Power of 2:

   • [32, 64, 128, 256]
   • GPU efficient

2. Memory-based:

   • Based on resources
   • System constraints

3. Learning-focused:

   • Small to large
   • Noise vs stability

Learning rate scheduling strategies:

Schedule types:

1. Constant: Fixed rate
2. Inverse scaling: Time decay
3. Adaptive: Performance-based

Impact on training:

• Convergence speed
• Solution quality
• Training stability
• Optimization dynamics

Initial learning rate search:

Search spaces:

1. Standard range:

   • [0.0001, 0.001, 0.01]
   • Log-scale search
   • Common values

2. Fine-grained:

   • [0.0003, 0.001, 0.003]
   • Around promising value
   • Precise tuning

3. Solver-specific:

   • Adam: [0.0001, 0.001]
   • SGD: [0.01, 0.1]
   • Algorithm-matched

Learning rate decay search:

Search ranges:

1. Standard decay:

   • [0.3, 0.5, 0.7]
   • Around classical 0.5

2. Wide range:

   • [0.1, 0.5, 1.0]
   • Decay impact study

3. Fine control:

   • [0.4, 0.5, 0.6]
   • Precise adjustment

Note: For invscaling only

Maximum iterations search:

Search patterns:

1. Standard range:

   • [100, 200, 500]
   • Basic problems

2. Deep learning:

   • [500, 1000, 2000]
   • Complex networks

3. Convergence study:

   • Log-scale spacing
   • Resource impact
   • Training dynamics

Training order evaluation:

Options:

1. Standard: [true]

   • Random order
   • Better generalization

2. Sequential: [false]

   • Fixed order
   • Time series data

3. Compare: [true, false]

   • Order sensitivity
   • Learning stability

Random seed configuration:

Usage patterns:

1. Development:

   • Fixed seed
   • Reproducible results
   • Debug capability

2. Production:

   • Multiple seeds
   • Robustness check
   • Model stability

Best practices:

• Document seeds
• Verify consistency
• Multiple trials

Convergence tolerance search:

Search spaces:

1. Standard range:

   • [1e-4, 1e-3, 1e-2]
   • Common values

2. High precision:

   • [1e-5, 1e-4, 1e-3]
   • Exact solutions

3. Quick convergence:

   • [1e-3, 1e-2]
   • Faster training

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

Options:

1. Fresh start: [false]

   • New weights
   • Independent trials

2. Continue: [true]

   • Weight reuse
   • Transfer learning

3. Compare: [true, false]

   • Initialization impact
   • Learning transfer

Momentum parameter search:

Search ranges:

1. Classical:

   • [0.0, 0.9, 0.99]
   • Standard values
   • Acceleration study

2. Fine-tuning:

   • [0.85, 0.9, 0.95]
   • Around 0.9
   • Precise control

3. Comprehensive:

   • [0.5, 0.7, 0.9, 0.99]
   • Full range impact
   • Training dynamics

Note: SGD solver only

Nesterov momentum evaluation:

Options:

1. Standard: [true]

   • Advanced momentum
   • Better convergence

2. Classical: [false]

   • Regular momentum
   • Simpler updates

3. Compare: [true, false]

   • Acceleration impact
   • Convergence speed

Use with: momentum > 0

Early stopping evaluation:

Configuration impact:

1. Without (false):

   • Full training
   • Maximum iterations
   • Complete convergence

2. With (true):

   • Validation-based
   • Automatic stopping
   • Overfitting prevention

Consider with:

• validation_fraction
• n_iter_no_change
• tol settings

Validation split size:

Selection guide:

1. Small datasets:

   • 0.2 validation
   • Stable evaluation

2. Large datasets:

   • 0.1 or less
   • More training data

3. Considerations:

   • Data availability
   • Validation stability
   • Training needs

Adam first moment search:

Search spaces:

1. Standard range:

   • [0.85, 0.9, 0.95]
   • Around default 0.9

2. Wide range:

   • [0.8, 0.9, 0.99]
   • Momentum impact

3. Fine-tuning:

   • [0.88, 0.9, 0.92]
   • Precise control

Note: Adam solver only

Adam second moment search:

Search spaces:

1. Conservative:

   • [0.995, 0.999, 0.9999]
   • High stability

2. Exploratory:

   • [0.99, 0.999, 0.9999]
   • Variance control

3. Fine-tuning:

   • [0.998, 0.999, 0.9995]
   • Around optimal

Note: Adam solver only

Adam numerical stability search:

Search ranges:

1. Standard:

   • [1e-8, 1e-7, 1e-6]
   • Common values

2. Stability focus:

   • [1e-9, 1e-8, 1e-7]
   • Numerical robustness

3. Precision study:

   • Multiple scales
   • Stability impact

Note: Adam solver only

Early stopping patience:

Configuration guide:

1. Quick stopping:

   • 5-10 epochs
   • Fast adaptation

2. Patient training:

   • 10-20 epochs
   • Better convergence

3. Deep learning:

   • 20+ epochs
   • Complex patterns

LBFGS function evaluation search:

Search patterns:

1. Quick solutions:

   • [5000, 10000]
   • Fast convergence

2. Standard range:

   • [10000, 15000, 20000]
   • Balanced approach

3. Thorough search:

   • [15000, 25000, 35000]
   • High precision

Note: LBFGS solver only

Regression model evaluation metrics:

Purpose:

• Model performance evaluation
• Error measurement
• Quality assessment
• Model comparison

Selection criteria:

• Error distribution
• Scale sensitivity
• Domain requirements
• Business objectives

Random seed for reproducible splits. Ensures:

• Consistent train/test sets
• Reproducible experiments
• Comparable model evaluations

Same seed guarantees identical splits across runs.

Data shuffling before splitting. Effects:

• true: Randomizes order, better for i.i.d. data
• false: Maintains order, important for time series

When to disable:

• Time dependent data
• Sequential patterns
• Grouped observations

Proportion of data for training. Considerations:

• Larger (e.g., 0.8-0.9): Better model learning
• Smaller (e.g., 0.5-0.7): Better validation

Common splits:

• 0.8: Standard (80/20 split)
• 0.7: More validation emphasis
• 0.9: More training emphasis

Maintain class distribution in splits. Important when:

• Classes are imbalanced
• Small classes present
• Representative splits needed

Requirements:

• Classification tasks only
• Cannot use with shuffle=false
• Sufficient samples per class

Number of cross-validation folds. Guidelines:

• 3-5: Large datasets, faster training
• 5-10: Standard choice, good balance
• 10+: Small datasets, thorough evaluation

Trade-offs:

• More folds: Better evaluation, slower training
• Fewer folds: Faster training, higher variance

Must be at least 2.

Number of folds for cross-validation. Selection guide: Recommended values:

• 5: Standard choice (default)
• 3: Large datasets/quick evaluation
• 10: Thorough evaluation/smaller datasets

Trade-offs:

• Higher values: More thorough, computationally expensive
• Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

Random seed for fold generation when shuffling. Important for:

• Reproducible results
• Consistent fold assignments
• Benchmark comparisons
• Debugging and validation

Set specific value for reproducibility across runs.

Whether to shuffle data before splitting into folds. Effects:

• true: Randomized fold composition (recommended)
• false: Sequential splitting

Enable when:

• Data may have ordering
• Better fold independence needed

Disable for:

• Time series data
• Ordered observations

Number of stratified folds. Guidelines: Typical values:

• 5: Standard for most cases
• 3: Quick evaluation/large datasets
• 10: Detailed evaluation/smaller datasets

Considerations:

• Must allow sufficient samples per class per fold
• Balance between stability and computation time
• Consider smallest class size when choosing

Seed for reproducible stratified splits. Ensures:

• Consistent fold assignments
• Reproducible results
• Comparable experiments
• Systematic validation

Fixed seed guarantees identical stratified splits.

Data shuffling before stratified splitting. Impact:

• true: Randomizes while maintaining stratification
• false: Maintains data order within strata

Use cases:

• true: Independent observations
• false: Grouped or sequential data

Class proportions maintained regardless of setting.

Number of random splits to perform. Consider: Common values:

• 5: Standard evaluation
• 10: More thorough assessment
• 3: Quick estimates

Trade-offs:

• More splits: Better estimation, longer runtime
• Fewer splits: Faster, less stable estimates

Balance between computation and stability.

Random seed for reproducible shuffling. Controls:

• Split randomization
• Sample selection
• Result reproducibility

Important for:

• Debugging
• Comparative studies
• Result verification

Proportion of samples for test set. Guidelines: Common ratios:

• 0.2: Standard (80/20 split)
• 0.25: More validation emphasis
• 0.1: More training data

Considerations:

• Dataset size
• Model complexity
• Validation requirements

It must be between 0.0 and 1.0.

Number of stratified random splits. Guidelines: Recommended values:

• 5: Standard evaluation
• 10: Detailed analysis
• 3: Quick assessment

Consider:

• Sample size per class
• Computational resources
• Stability requirements

Seed for reproducible stratified sampling. Ensures:

• Consistent class proportions
• Reproducible splits
• Comparable experiments

Critical for:

• Benchmarking
• Research studies
• Quality assurance

Fraction of samples for stratified test set. Best practices: Common splits:

• 0.2: Balanced evaluation
• 0.3: More thorough testing
• 0.15: Preserve training size

Consider:

• Minority class size
• Overall dataset size
• Validation objectives

It must be between 0.0 and 1.0.

Number of temporal splits. Considerations: Typical values:

• 5: Standard forward chaining
• 3: Limited historical data
• 10: Long time series

Impact:

• Affects training window growth
• Determines validation points
• Influences computational load

Maximum size of training set. Should be strictly less than the number of samples. Applications:

• 0: Use all available past data
• >0: Rolling window of fixed size

Use cases:

• Limit historical relevance
• Control computational cost
• Handle concept drift
• Memory constraints

Number of samples in each test set. When 0:

• Auto-calculated as n_samples/(n_splits+1)
• Ensures equal-sized test sets

Considerations:

• Forecast horizon
• Validation requirements
• Available future data

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

• Avoid data leakage
• Model forecast lag
• Buffer periods

Common scenarios:

• 0: Continuous prediction
• >0: Forward gap for realistic evaluation
• Match business forecasting needs