LogisticRegression / Classifier Layer

The combination of solver and penalty to use. Solver-penalty combinations for optimization. Each combination offers different trade-offs:

Performance characteristics:

• Small datasets (n < 10k): Choose liblinear
• Large datasets: Prefer sag or saga
• Many features: newton-cg or lbfgs
• Sparse data: saga with L1

Memory usage:

• Low memory: liblinear, saga
• Medium: lbfgs, newton-cg
• High: newton-cholesky (quadratic in features)

Multiclass support:

• Full support: newton-cg, sag, saga, lbfgs
• Binary only: newton-cholesky
• One-vs-rest only: liblinear

Optimization formulation selection. Can be used only for liblinear solver with L2 penalty. Dual formulation is advantageous when n_samples < n_features. Primal (false) is preferred when n_samples > n_features. Automatically ignored if incompatible with solver.

Convergence criterion threshold. Controls optimization precision:

• Smaller values: More precise but slower convergence
• Larger values: Faster but potentially less optimal solution

Typical range: 1e-6 to 1e-3. Adjust based on precision needs.

Inverse regularization strength (C parameter). Controls model complexity:

• Small values (<1): Stronger regularization, simpler model
• Large values (>1): Weaker regularization, more complex model

Common ranges:

• 0.001 to 0.1: High regularization (noisy data)
• 0.1 to 1: Moderate regularization (typical)
• 1 to 10: Low regularization (clean data)

Must be positive. Scale often requires adjustment with dataset size.

Whether to include bias term (intercept). Effects:

• true: Model can learn offset from origin (recommended)
• false: Decision boundary through origin

Set false only when data is pre-centered or for theoretical analysis.

Scaling factor for synthetic intercept feature. Only used with liblinear solver and fit_intercept=true. Effects:

• Larger values: More emphasis on intercept fitting
• Smaller values: Less emphasis on intercept

Useful for handling numerical stability in specific cases.

Class weight adjustment strategy for handling imbalanced datasets:

Mathematical form:

• Balanced: $w_i=\frac{n_{samples}}{n_{classes}\times n_{sample{s}_i}}$
• Uniform: $w_i=1$ for all classes

Impact on model:

• Affects class importance during training
• Influences decision boundary placement
• Controls misclassification penalties
• Balances precision vs recall trade-off

Selection criteria:

• Class distribution in data
• Cost of different error types
• Business/domain requirements
• Performance metrics priorities

Random number generator seed. Used for:

• Reproducible results with stochastic solvers (sag, saga, liblinear)
• Consistent data shuffling
• Benchmark comparisons

Same seed guarantees identical results across runs.

Maximum number of solver iterations. Consider:

• Increase if model warns about non-convergence
• Typical ranges: • 100-500: Simple problems • 500-1000: Complex or large datasets • 1000+: Difficult convergence cases

Balance between convergence quality and computation time.

Strategy for handling multiple classes. Affects both model architecture and training:

• Auto: Automatically choose based on data and solver
• Ovr (One-vs-Rest):
  • Trains N binary classifiers
  • Memory efficient
  • Works with all solvers
  • Good for imbalanced classes
• Multinomial:
  • Single model for all classes
  • More accurate probability estimates
  • Only with newton-cg, sag, saga, lbfgs
  • Better when classes are balanced

ElasticNet mixing parameter [0, 1]. Controls L1 vs L2 penalty mix:

• 0.0: Pure L2 regularization
• 1.0: Pure L1 regularization
• 0.0-1.0: Mix of both

Only used with elasticnet penalty. Any value other than [0-1] disables elasticnet.

Regularization penalty type controlling model complexity and feature selection:

Selection impact:

• Controls overfitting prevention
• Influences feature selection
• Affects model sparsity
• Determines solution stability

Common use cases:

• L2: General purpose, dense features
• L1: Feature selection, sparse solutions
• ElasticNet: Combined benefits of L1 and L2
• None: When regularization not needed

Mathematical form: $$Loss(w)+Penalty(w)$$

Optimization algorithm for minimizing the logistic regression cost function:

Mathematical objective: $$\underset{w,c}{\min }\frac{1}{n_{samples}}\sum _{i=1}^{n_{samples}}\log (1+e^{-y_i(X_i^{T}w+c)})+R(w)$$

Selection criteria:

1. Dataset characteristics:

   • Sample size (n_samples)
   • Feature count (n_features)
   • Sparsity level

2. Memory constraints:

   • Low: liblinear, saga
   • Medium: lbfgs, newton-cg
   • High: newton-cholesky

3. Problem type:

   • Binary classification
   • Multiclass problems
   • Different regularizations

Performance considerations:

• Convergence speed
• Memory usage
• Numerical stability
• Scalability

Regularization strength values to test. Recommended ranges:

• Logarithmic scale: [0.001, 0.01, 0.1, 1, 10, 100]
• Fine-tuning: [0.1, 0.2, 0.5, 1.0]

Smaller values = stronger regularization. Tips:

• Start with broad range
• Refine around best values
• Consider dataset size in selection

Dual formulation options to test. Considerations:

• [false]: Standard for n_samples > n_features
• [true, false]: When optimal formulation unclear

Only relevant for liblinear solver with L2 penalty.

Convergence tolerance values to test. Typical ranges:

• Coarse: [1e-2, 1e-3, 1e-4]
• Fine: [1e-4, 1e-5, 1e-6]

Trade-off:

• Lower values: Better precision, slower convergence
• Higher values: Faster training, less precise

Synthetic feature scaling for intercept. Relevant when:

• Using liblinear solver
• fit_intercept is true

Typical values: 1.0 (default) to 100.0 Increase if model has trouble learning intercept.

Class weight adjustment strategy for handling imbalanced datasets:

Mathematical form:

• Balanced: $w_i=\frac{n_{samples}}{n_{classes}\times n_{sample{s}_i}}$
• Uniform: $w_i=1$ for all classes

Impact on model:

• Affects class importance during training
• Influences decision boundary placement
• Controls misclassification penalties
• Balances precision vs recall trade-off

Selection criteria:

• Class distribution in data
• Cost of different error types
• Business/domain requirements
• Performance metrics priorities

When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new ensemble. Effects:

• true: Faster for multiple similar fits
• false: Fresh start each time

Useful for:

• Path algorithms
• Incremental learning

Random seed for reproducibility. Ensures:

• Consistent cross-validation splits
• Reproducible solver behavior
• Comparable grid search results

Essential for research and debugging.Used when solver == sag, saga or liblinear to shuffle the data.

Maximum iterations for each fit. Common ranges:

• Standard: [100, 200, 500]
• Extended: [100, 500, 1000, 2000]

Tips:

• Include larger values if seeing non-convergence
• Consider computation budget
• Monitor convergence warnings

Strategy for handling multiple classes. Affects both model architecture and training:

• Auto: Automatically choose based on data and solver
• Ovr (One-vs-Rest):
  • Trains N binary classifiers
  • Memory efficient
  • Works with all solvers
  • Good for imbalanced classes
• Multinomial:
  • Single model for all classes
  • More accurate probability estimates
  • Only with newton-cg, sag, saga, lbfgs
  • Better when classes are balanced

ElasticNet mixing parameter values. Common patterns:

• Broad search: [0.0, 0.25, 0.5, 0.75, 1.0]
• Fine-tuning: [0.1, 0.2, 0.3, 0.4]

Only relevant when using elasticnet penalty. Any value other than [0-1] disables elasticnet.

Metric for evaluating model performance during training and validation:

• Default: Uses estimator's built-in scoring
• Accuracy: Proportion of correct predictions
• BalancedAccuracy: Arithmetic mean of recall for each class
• LogLoss: Negative log-likelihood of true labels
• RocAuc: Area under ROC curve (threshold-independent)

Choose based on:

• Class balance (balanced_accuracy for imbalanced)
• Need for probability calibration (log_loss)
• Binary vs multiclass (roc_auc for binary)

Random seed for reproducible splits. Ensures:

• Consistent train/test sets
• Reproducible experiments
• Comparable model evaluations

Same seed guarantees identical splits across runs.

Data shuffling before splitting. Effects:

• true: Randomizes order, better for i.i.d. data
• false: Maintains order, important for time series

When to disable:

• Time dependent data
• Sequential patterns
• Grouped observations

Proportion of data for training. Considerations:

• Larger (e.g., 0.8-0.9): Better model learning
• Smaller (e.g., 0.5-0.7): Better validation

Common splits:

• 0.8: Standard (80/20 split)
• 0.7: More validation emphasis
• 0.9: More training emphasis

Maintain class distribution in splits. Important when:

• Classes are imbalanced
• Small classes present
• Representative splits needed

Requirements:

• Classification tasks only
• Cannot use with shuffle=false
• Sufficient samples per class

Number of cross-validation folds. Guidelines:

• 3-5: Large datasets, faster training
• 5-10: Standard choice, good balance
• 10+: Small datasets, thorough evaluation

Trade-offs:

• More folds: Better evaluation, slower training
• Fewer folds: Faster training, higher variance

Must be at least 2.

Number of folds for cross-validation. Selection guide: Recommended values:

• 5: Standard choice (default)
• 3: Large datasets/quick evaluation
• 10: Thorough evaluation/smaller datasets

Trade-offs:

• Higher values: More thorough, computationally expensive
• Lower values: Faster, potentially higher variance

Must be at least 2 for valid cross-validation.

Random seed for fold generation when shuffling. Important for:

• Reproducible results
• Consistent fold assignments
• Benchmark comparisons
• Debugging and validation

Set specific value for reproducibility across runs.

Whether to shuffle data before splitting into folds. Effects:

• true: Randomized fold composition (recommended)
• false: Sequential splitting

Enable when:

• Data may have ordering
• Better fold independence needed

Disable for:

• Time series data
• Ordered observations

Number of stratified folds. Guidelines: Typical values:

• 5: Standard for most cases
• 3: Quick evaluation/large datasets
• 10: Detailed evaluation/smaller datasets

Considerations:

• Must allow sufficient samples per class per fold
• Balance between stability and computation time
• Consider smallest class size when choosing

Seed for reproducible stratified splits. Ensures:

• Consistent fold assignments
• Reproducible results
• Comparable experiments
• Systematic validation

Fixed seed guarantees identical stratified splits.

Data shuffling before stratified splitting. Impact:

• true: Randomizes while maintaining stratification
• false: Maintains data order within strata

Use cases:

• true: Independent observations
• false: Grouped or sequential data

Class proportions maintained regardless of setting.

Number of random splits to perform. Consider: Common values:

• 5: Standard evaluation
• 10: More thorough assessment
• 3: Quick estimates

Trade-offs:

• More splits: Better estimation, longer runtime
• Fewer splits: Faster, less stable estimates

Balance between computation and stability.

Random seed for reproducible shuffling. Controls:

• Split randomization
• Sample selection
• Result reproducibility

Important for:

• Debugging
• Comparative studies
• Result verification

Proportion of samples for test set. Guidelines: Common ratios:

• 0.2: Standard (80/20 split)
• 0.25: More validation emphasis
• 0.1: More training data

Considerations:

• Dataset size
• Model complexity
• Validation requirements

It must be between 0.0 and 1.0.

Number of stratified random splits. Guidelines: Recommended values:

• 5: Standard evaluation
• 10: Detailed analysis
• 3: Quick assessment

Consider:

• Sample size per class
• Computational resources
• Stability requirements

Seed for reproducible stratified sampling. Ensures:

• Consistent class proportions
• Reproducible splits
• Comparable experiments

Critical for:

• Benchmarking
• Research studies
• Quality assurance

Fraction of samples for stratified test set. Best practices: Common splits:

• 0.2: Balanced evaluation
• 0.3: More thorough testing
• 0.15: Preserve training size

Consider:

• Minority class size
• Overall dataset size
• Validation objectives

It must be between 0.0 and 1.0.

Number of temporal splits. Considerations: Typical values:

• 5: Standard forward chaining
• 3: Limited historical data
• 10: Long time series

Impact:

• Affects training window growth
• Determines validation points
• Influences computational load

Maximum size of training set. Should be strictly less than the number of samples. Applications:

• 0: Use all available past data
• >0: Rolling window of fixed size

Use cases:

• Limit historical relevance
• Control computational cost
• Handle concept drift
• Memory constraints

Number of samples in each test set. When 0:

• Auto-calculated as n_samples/(n_splits+1)
• Ensures equal-sized test sets

Considerations:

• Forecast horizon
• Validation requirements
• Available future data

Number of samples to exclude from the end of each train set before the test set.Gap between train and test sets. Uses:

• Avoid data leakage
• Model forecast lag
• Buffer periods

Common scenarios:

• 0: Continuous prediction
• >0: Forward gap for realistic evaluation
• Match business forecasting needs