PassiveAggressive / Regressor Layer

Aggressiveness parameter:

Role in update: $$w_{t+1}=w_t+{\tau }_t{y}_t{x}_t$$ where τₜ depends on C

Effects:

• Controls update size
• Balances stability
• Adaptation speed

Selection guide:

• Small: More passive
• Large: More aggressive
• Balance needed

Intercept calculation control:

Model forms: With intercept: $$y=Xw+b+ϵ$$ Without intercept: $$y=Xw+ϵ$$

Online impact:

• Updated per sample
• Adapts to shifts
• Tracks bias
• Stream handling

Effects when True:

• Centers predictions
• Handles drift
• Better adaptation
• Standard modeling

Maximum passes over training data:

Impact on learning:

• Multiple passes allowed
• Affects fit method only
• Not partial_fit
• Convergence control

Typical ranges:

• Simple: 100-500
• Standard: 1000
• Complex: 2000+

Note: Online learning may need fewer

Convergence tolerance threshold:

Stopping criterion: $$loss>best\_loss-tol$$

Usage:

• 0: No early stopping
• >0: Enables stopping
• Checks n_iter_no_change

Trade-offs:

• Stability vs speed
• Accuracy vs time
• Stream adaptation

If the difference between the current prediction and the correct label is below this threshold, the model is not updated.

Update condition: $$|y-\hat{y}|>ϵ$$

Effects:

• Controls sensitivity
• Update frequency
• Noise tolerance
• Solution stability

Selection guide:

• Small: More updates
• Large: More passive
• Based on noise level

Whether or not the training data should be shuffled after each epoch.

When True:

• Randomizes order
• Better convergence
• Prevents cycles
• Improves generalization

When False:

• Sequential updates
• Order-sensitive
• Stream simulation
• Time series cases

Loss functions for PA updates:

1. Epsilon-insensitive:

   • Linear penalty
   • Robust behavior
   • SVR-like loss

2. Squared epsilon-insensitive:

   • Quadratic penalty
   • Smoother updates
   • More aggressive

Selection impact:

• Update magnitude
• Convergence behavior
• Outlier sensitivity
• Solution stability

Early stopping control:

When enabled:

• Monitors validation
• Prevents overfitting
• Saves computation
• Adaptive stopping

Requirements:

• validation_fraction > 0
• Sufficient data
• n_iter_no_change set
• tol > 0

Validation set proportion:

Usage:

• Early stopping only
• Hold-out validation
• Progress monitoring

Typical values:

• Small data: 0.2
• Large data: 0.1
• Streaming: Consider less

Iteration patience:

Early stopping when:

• No improvement for n iterations
• Based on validation score
• Checks tolerance threshold

Selection:

• Small: Quick stopping
• Large: More patience
• Balance needed

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

When True:

• Reuses weights
• Continues learning
• Stream handling
• Incremental updates

Best for:

• Online learning
• Stream processing
• Continuous adaptation
• Transfer learning

Random number generation seed:

Controls:

• Shuffling order
• Reproducibility
• Validation split

Usage:

• Fixed: Reproducible
• Different: Robustness
• 0: System random

Aggressiveness parameter search:

Search patterns:

1. Conservative:

   • [0.1, 0.5, 1.0]
   • Stable updates

2. Aggressive:

   • [1.0, 10.0, 100.0]
   • Fast adaptation

3. Comprehensive:

   • [0.1, 1.0, 10.0, 100.0]
   • Full range study

Intercept adaptation search:

Options:

1. With bias: [true]

   • Stream adaptation
   • Drift handling

2. No bias: [false]

   • Fixed origin
   • Theory-driven

3. Compare: [true, false]

   • Adaptation study
   • Bias impact

Maximum iterations search:

Search ranges:

1. Quick learning:

   • [100, 300, 500]
   • Fast adaptation

2. Standard:

   • [500, 1000, 2000]
   • Better stability

3. Thorough:

   • [1000, 3000, 5000]
   • Complex problems

Tolerance threshold search:

Search spaces:

1. Tight convergence:

   • [1e-4, 1e-3, 1e-2]
   • Precise updates

2. Relaxed:

   • [1e-3, 1e-2, 1e-1]
   • Faster adaptation

3. Online-focused:

   • [1e-2, 1e-1]
   • Stream processing

Margin threshold search:

Search patterns:

1. Sensitive:

   • [0.01, 0.05, 0.1]
   • Frequent updates

2. Conservative:

   • [0.1, 0.2, 0.5]
   • Stable behavior

3. Mixed:

   • Multiple ranges
   • Update study

Order randomization search:

Options:

1. Random: [true]

   • Better convergence
   • General cases

2. Sequential: [false]

   • Stream simulation
   • Time series

3. Compare: [true, false]

   • Order sensitivity

Loss functions for PA updates:

1. Epsilon-insensitive:

   • Linear penalty
   • Robust behavior
   • SVR-like loss

2. Squared epsilon-insensitive:

   • Quadratic penalty
   • Smoother updates
   • More aggressive

Selection impact:

• Update magnitude
• Convergence behavior
• Outlier sensitivity
• Solution stability

Early stopping evaluation:

Search options:

1. Full training: [false]

   • Complete passes
   • Stream processing

2. With stopping: [true]

   • Resource saving
   • Convergence check

3. Compare: [true, false]

   • Efficiency study

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble.

Options:

1. Fresh start: [false]

   • Independent runs
   • Clean training

2. Continuous: [true]

   • Stream handling
   • Online learning

3. Compare: [true, false]

   • Adaptation study

Validation size control:

Considerations:

• Early stopping needs
• Stream processing
• Data availability
• Online scenarios

Typical values:

• Small data: 0.2
• Large data: 0.1
• Streaming: Minimal

Patience parameter:

Impact:

• Stopping timing
• Adaptation period
• Resource usage

Selection:

• Short: 3-5 iterations
• Long: 10+ iterations
• Stream-appropriate

Random seed control:

Usage:

• Fixed: Reproducible
• Varied: Robustness
• Multiple: Stability

Best practices:

• Document seeds
• Verify consistency
• Test stability

Regression model evaluation metrics:

Purpose:

• Model performance evaluation
• Error measurement
• Quality assessment
• Model comparison

Selection criteria:

• Error distribution
• Scale sensitivity
• Domain requirements
• Business objectives

Random seed for reproducible splits. Ensures:

• Consistent train/test sets
• Reproducible experiments
• Comparable model evaluations

Same seed guarantees identical splits across runs.

Data shuffling before splitting. Effects:

• true: Randomizes order, better for i.i.d. data
• false: Maintains order, important for time series

When to disable:

• Time dependent data
• Sequential patterns
• Grouped observations

Proportion of data for training. Considerations:

• Larger (e.g., 0.8-0.9): Better model learning
• Smaller (e.g., 0.5-0.7): Better validation

Common splits:

• 0.8: Standard (80/20 split)
• 0.7: More validation emphasis
• 0.9: More training emphasis

Maintain class distribution in splits. Important when:

• Classes are imbalanced
• Small classes present
• Representative splits needed

Requirements:

• Classification tasks only
• Cannot use with shuffle=false
• Sufficient samples per class

Number of cross-validation folds. Guidelines:

• 3-5: Large datasets, faster training
• 5-10: Standard choice, good balance
• 10+: Small datasets, thorough evaluation

Trade-offs:

• More folds: Better evaluation, slower training
• Fewer folds: Faster training, higher variance

Must be at least 2.

Number of folds for cross-validation. Selection guide: Recommended values:

• 5: Standard choice (default)
• 3: Large datasets/quick evaluation
• 10: Thorough evaluation/smaller datasets

Trade-offs:

• Higher values: More thorough, computationally expensive
• Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

Random seed for fold generation when shuffling. Important for:

• Reproducible results
• Consistent fold assignments
• Benchmark comparisons
• Debugging and validation

Set specific value for reproducibility across runs.

Whether to shuffle data before splitting into folds. Effects:

• true: Randomized fold composition (recommended)
• false: Sequential splitting

Enable when:

• Data may have ordering
• Better fold independence needed

Disable for:

• Time series data
• Ordered observations

Number of stratified folds. Guidelines: Typical values:

• 5: Standard for most cases
• 3: Quick evaluation/large datasets
• 10: Detailed evaluation/smaller datasets

Considerations:

• Must allow sufficient samples per class per fold
• Balance between stability and computation time
• Consider smallest class size when choosing

Seed for reproducible stratified splits. Ensures:

• Consistent fold assignments
• Reproducible results
• Comparable experiments
• Systematic validation

Fixed seed guarantees identical stratified splits.

Data shuffling before stratified splitting. Impact:

• true: Randomizes while maintaining stratification
• false: Maintains data order within strata

Use cases:

• true: Independent observations
• false: Grouped or sequential data

Class proportions maintained regardless of setting.

Number of random splits to perform. Consider: Common values:

• 5: Standard evaluation
• 10: More thorough assessment
• 3: Quick estimates

Trade-offs:

• More splits: Better estimation, longer runtime
• Fewer splits: Faster, less stable estimates

Balance between computation and stability.

Random seed for reproducible shuffling. Controls:

• Split randomization
• Sample selection
• Result reproducibility

Important for:

• Debugging
• Comparative studies
• Result verification

Proportion of samples for test set. Guidelines: Common ratios:

• 0.2: Standard (80/20 split)
• 0.25: More validation emphasis
• 0.1: More training data

Considerations:

• Dataset size
• Model complexity
• Validation requirements

It must be between 0.0 and 1.0.

Number of stratified random splits. Guidelines: Recommended values:

• 5: Standard evaluation
• 10: Detailed analysis
• 3: Quick assessment

Consider:

• Sample size per class
• Computational resources
• Stability requirements

Seed for reproducible stratified sampling. Ensures:

• Consistent class proportions
• Reproducible splits
• Comparable experiments

Critical for:

• Benchmarking
• Research studies
• Quality assurance

Fraction of samples for stratified test set. Best practices: Common splits:

• 0.2: Balanced evaluation
• 0.3: More thorough testing
• 0.15: Preserve training size

Consider:

• Minority class size
• Overall dataset size
• Validation objectives

It must be between 0.0 and 1.0.

Number of temporal splits. Considerations: Typical values:

• 5: Standard forward chaining
• 3: Limited historical data
• 10: Long time series

Impact:

• Affects training window growth
• Determines validation points
• Influences computational load

Maximum size of training set. Should be strictly less than the number of samples. Applications:

• 0: Use all available past data
• >0: Rolling window of fixed size

Use cases:

• Limit historical relevance
• Control computational cost
• Handle concept drift
• Memory constraints

Number of samples in each test set. When 0:

• Auto-calculated as n_samples/(n_splits+1)
• Ensures equal-sized test sets

Considerations:

• Forecast horizon
• Validation requirements
• Available future data

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

• Avoid data leakage
• Model forecast lag
• Buffer periods

Common scenarios:

• 0: Continuous prediction
• >0: Forward gap for realistic evaluation
• Match business forecasting needs