Overview · How it works · Installation · Quick start · Usage guide · Examples · Citation
The Atlantic project constitutes a comprehensive and objective approach to simplify and automate data processing through the integration and validated application of various preprocessing mechanisms, ranging from feature engineering, automated feature selection, multiple encoding versions and null imputation methods. The optimization methodology of this framework follows an evaluation structured in tree-based model ensembles.
- Works on any tabular dataset. Regression, binary and multiclass classification, across domains, with numeric, categorical, boolean and datetime columns.
- Decisions backed by evidence. Encoding, imputation and multicollinearity filtering are selected by validation score, not by rule of thumb.
- Automated feature selection. H2O AutoML variable importance keeps the features that carry the share of signal you ask for.
- Leak-free by design. Final transformers are fitted on training rows only, so validation rows never influence them.
- Built for reuse. A fitted pipeline replays the exact same steps on new data and can be saved and reloaded.
- Transparent. Every decision, including every dropped column and the reason it was dropped, is exposed for inspection.
The pipeline runs in six stages:
| # | Stage | What happens |
|---|---|---|
| 1 | Validate & prepare | Infers the task, drops rows without a target, encodes booleans as 0/1, expands datetimes and removes unusable columns |
| 2 | Split & select features | Splits into train and validation (stratified for classification) and, when relevance < 1.0, keeps the most important features found by H2O AutoML |
| 3 | Select by validation score | Chooses the encoding, imputation and VIF filtering steps one at a time, keeping each only if it scores best |
| 4 | Fit final transformers | Refits the chosen scaler, encoder and imputer on the training rows only |
| 5 | Transform new data | data_processing replays the same steps on any future frame |
| 6 | Inspect & persist | Exposes every decision and saves the fitted pipeline for reuse |
Stage-by-stage details
1. Validate & prepare
The task is inferred from the target column:
| Target type | Task | Evaluation metric |
|---|---|---|
| Numeric | Regression | MAE |
| Other, with 2 classes | Binary classification | Precision |
| Other, with more than 2 classes | Multiclass classification | F1 |
- Rows with no target value are dropped.
- Boolean columns become 0/1 features.
- Each datetime column is expanded into nine calendar features.
- High-null, constant and identifier-like columns are removed. Every removal is recorded in
dropped_columns.
2. Split & select features
- The data is divided into a train and a validation split. Classification splits are stratified.
- When
relevance < 1.0, only the features carrying the requested share of H2O AutoML variable importance are kept.
3. Select by validation score
| Step | Candidates | Applied when |
|---|---|---|
| Encoding | 4 combinations: Standard or MinMax scaler × IFrequency or Label encoder | Always |
| Imputation | Simple (mean), KNN, Iterative | The data has missing values |
| VIF filtering | Iterative removal of features above vif_ratio |
It improves the validation score |
- Each candidate is fitted on the train split and scored on the validation split, using tuned RandomForest, ExtraTrees and XGBoost models.
- The number of tuning trials adapts to the data size and to
optimization_level.
4. Fit final transformers
The chosen scaler, encoder and imputer are refitted on the training rows only, so validation rows never influence them.
5. Transform new data
- Frames missing a fitted column are rejected with a clear error.
- Unseen categories and missing values are handled.
6. Inspect & persist
- The pipeline's decisions are exposed as
enc_method,imp_method,colsanddropped_columns. - The fitted pipeline can be saved with
saveand reloaded withAtlantic.load.
Atlantic is published on PyPI. One command installs the full framework, H2O AutoML included:
pip install atlanticfrom sklearn.model_selection import train_test_split
from atlantic import Atlantic
from atlantic.data import DatasetGenerator
# Any pandas DataFrame with a target column works; here we generate one.
data, target = DatasetGenerator.generate_classification(
n_samples=2000, n_features=15, n_classes=2, n_categorical=4, null_percentage=0.05
)
train, test = train_test_split(data, train_size=0.8, random_state=42)
atl = Atlantic(X=train, target=target)
atl.fit_processing() # H2O feature selection + balanced search
train_ready = atl.data_processing(X=train)
test_ready = atl.data_processing(X=test)
print(atl.enc_method, atl.imp_method) # what Atlantic decidedThe Atlantic class runs the full pipeline. Fit it once on training data, then use data_processing to transform any future frame with the same structure.
fit_processing parameters
| Parameter | Description | Default |
|---|---|---|
split_ratio |
Train/validation split ratio used to evaluate preprocessing candidates (0.5 – 0.98) | 0.75 |
relevance |
Share of H2O AutoML feature importance to keep (0.4 – 1.0). 1.0 skips H2O feature selection |
0.99 |
h2o_fs_models |
Number of models trained by H2O AutoML for feature selection | 7 |
encoding_fs |
Encode categorical features before H2O selection | True |
vif_ratio |
Variance Inflation Factor threshold (3 – 30) | 10.0 |
optimization_level |
Search intensity: "fast", "balanced" or "thorough" |
"balanced" |
Atlantic constructor parameters
| Parameter | Description | Default |
|---|---|---|
random_state |
Seed for the internal train/validation split and every model evaluation | 42 |
n_jobs |
Parallelism of the evaluation models. None keeps the tree ensembles single-threaded; -1 uses all cores (same results, faster on mid-sized data) |
None |
verbosity |
0 silent, 1 selection summary, 2 detailed |
1 |
Task type. A numeric target is treated as regression; anything else (strings, categories, booleans) as classification. Integer-coded class labels such as
0/1are therefore treated as regression and a warning is raised. Cast them first:data["Target_Column"] = data["Target_Column"].astype(str).
import pandas as pd
from sklearn.model_selection import train_test_split
from atlantic.pipeline import Atlantic
data = pd.read_csv("csv_directory_path", encoding="latin", delimiter=",")
train, test = train_test_split(data, train_size=0.8)
test, future_data = train_test_split(test, train_size=0.6)
train = train.reset_index(drop=True)
test = test.reset_index(drop=True)
future_data = future_data.reset_index(drop=True).drop(columns=["Target_Column"])
# Fit
atl = Atlantic(X=train, target="Target_Column")
atl.fit_processing(
split_ratio=0.75,
relevance=0.99, # H2O AutoML feature selection; 1.0 skips it
h2o_fs_models=7,
vif_ratio=10.0,
optimization_level="balanced",
)
# Transform
train = atl.data_processing(X=train)
test = atl.data_processing(X=test)
future_data = atl.data_processing(X=future_data)
# Inspect what the pipeline decided
print(atl.enc_method, atl.imp_method) # selected encoding version / imputation method
print(atl.cols) # feature columns kept
print(atl.dropped_columns) # columns removed, grouped by reason
# Persist (pickle-based; the training data is not stored in the file)
atl.save("fit_atl.pkl")
loaded_atl = Atlantic.load("fit_atl.pkl")Security note. Pipeline files are pickles. Only load files you trust.
For fine-grained control over each step, compose a pipeline with the fluent AtlanticBuilder interface:
from sklearn.model_selection import train_test_split
from atlantic.pipeline import AtlanticBuilder
train, test = train_test_split(data, train_size=0.8)
train = train.reset_index(drop=True)
test = test.reset_index(drop=True)
pipeline = (
AtlanticBuilder()
.with_date_engineering(enabled=True, drop=True)
.with_null_removal(threshold=0.90)
.with_feature_selection(method="h2o", relevance=0.95, h2o_models=10, encoding_fs=True)
.with_encoding(scaler="standard", encoder="ifrequency", auto_select=True)
.with_imputation(method="knn", auto_select=True)
.with_vif_filtering(threshold=10.0)
.with_optimization(optimization_level="balanced")
.build()
)
train_processed = pipeline.fit_transform(train, target="Target_Column")
test_processed = pipeline.transform(test)
print(pipeline.atlantic.enc_method, pipeline.atlantic.dropped_columns)
pipeline.save("fit_pipeline.pkl")
# from atlantic.pipeline import AtlanticPipeline
# pipeline = AtlanticPipeline.load("fit_pipeline.pkl")| Builder method | Behaviour |
|---|---|
with_date_engineering(enabled, drop) |
enabled=False leaves datetime columns out of the features; drop=False keeps the original datetime columns in the output next to the generated features |
with_encoding(..., auto_select=True) |
Selects the best scaler and encoder by validation score and ignores the explicit scaler / encoder. With auto_select=False they are applied as given |
with_imputation(..., auto_select=True) |
Selects the best imputer by validation score and ignores the explicit method. With auto_select=False it is applied as given |
with_optimization(optimization_level, random_state, split_ratio, n_jobs, verbosity) |
Controls the evaluation budget, reproducibility and console output |
| Preset | Use case | Key settings |
|---|---|---|
| Fast | Quick prototyping | h2o_models=3, method="simple", optimization_level="fast" |
| Balanced | General purpose | Default settings |
| Thorough | Best results | h2o_models=15, method="iterative", optimization_level="thorough" |
| High-null | More than 20% missing data | threshold=0.80, scaler="robust", method="iterative" |
| No-H2O | No Java runtime available | method="none", VIF filtering only |
# Fast prototyping
fast_pipeline = (
AtlanticBuilder()
.with_feature_selection(method="h2o", relevance=0.85, h2o_models=3)
.with_encoding(scaler="minmax", encoder="label", auto_select=False)
.with_imputation(method="simple", auto_select=False)
.with_optimization(optimization_level="fast")
.build()
)
# Thorough optimization
thorough_pipeline = (
AtlanticBuilder()
.with_feature_selection(method="h2o", relevance=0.98, h2o_models=15)
.with_encoding(auto_select=True)
.with_imputation(method="iterative", auto_select=True)
.with_vif_filtering(threshold=8.0)
.with_optimization(optimization_level="thorough")
.build()
)
# High-null data
high_null_pipeline = (
AtlanticBuilder()
.with_null_removal(threshold=0.80)
.with_encoding(scaler="robust")
.with_imputation(method="iterative", auto_select=True)
.build()
)Every building block of the pipeline is also available on its own, with a scikit-learn style fit / transform interface.
| Component | Classes |
|---|---|
| Encoders | AutoLabelEncoder, AutoIFrequencyEncoder, AutoOneHotEncoder |
| Scalers | AutoStandardScaler, AutoMinMaxScaler, AutoRobustScaler |
| Imputers | AutoSimpleImputer, AutoKNNImputer, AutoIterativeImputer |
| Feature selectors | VIFFeatureSelector, H2OFeatureSelector |
Label encoding (ordinal mapping) and inverse-frequency encoding (IDF-based weights) both preserve the column layout, so the result can be assigned back in place. AutoOneHotEncoder expands each categorical column into several binary columns and returns a wider frame.
from sklearn.model_selection import train_test_split
from atlantic.preprocessing import AutoLabelEncoder, AutoIFrequencyEncoder, AutoOneHotEncoder
from atlantic.utils import get_categorical_columns
train, test = train_test_split(data, train_size=0.8)
train = train.reset_index(drop=True)
test = test.reset_index(drop=True)
target = "Target_Column"
cat_cols = get_categorical_columns(data, exclude=[target]) # object, category and string dtypes
# Same-shape output: label or inverse-frequency
encoder = AutoLabelEncoder()
# encoder = AutoIFrequencyEncoder()
encoder.fit(train[cat_cols])
train[cat_cols] = encoder.transform(train[cat_cols])
test[cat_cols] = encoder.transform(test[cat_cols])
train[cat_cols] = encoder.inverse_transform(train[cat_cols]) # if needed
# One-hot returns a WIDER frame: do not assign it back into cat_cols
# ohe = AutoOneHotEncoder()
# ohe.fit(train[cat_cols])
# train = ohe.transform(train) # original cat_cols are replaced by binary columns
# test = ohe.transform(test)Standard scaling for roughly normal distributions, MinMax for bounded ranges, and Robust scaling for data with outliers.
from atlantic.preprocessing import AutoStandardScaler, AutoMinMaxScaler, AutoRobustScaler
from atlantic.utils import get_numeric_columns
target = "Target_Column"
num_cols = get_numeric_columns(train, exclude=[target]) # never scale the target
scaler = AutoStandardScaler() # zero mean, unit variance
# scaler = AutoMinMaxScaler() # scale to [0, 1]
# scaler = AutoRobustScaler() # median/IQR based, outlier-resistant
scaler.fit(train[num_cols])
train[num_cols] = scaler.transform(train[num_cols])
test[num_cols] = scaler.transform(test[num_cols])Simple imputation for speed, KNN for local patterns, or iterative modelling for complex dependencies between features.
from atlantic.preprocessing import AutoSimpleImputer, AutoKNNImputer, AutoIterativeImputer
imputer = AutoSimpleImputer(strategy="mean", target=target)
# imputer = AutoKNNImputer(n_neighbors=5, target=target)
# imputer = AutoIterativeImputer(max_iter=10, target=target)
imputer.fit(train)
train = imputer.transform(train)
test = imputer.transform(test)VIF filtering removes multicollinear features; H2O AutoML keeps the most predictive ones.
from atlantic.feature_selection import VIFFeatureSelector, H2OFeatureSelector
# VIF: expects encoded, imputed, numeric features
vif_selector = VIFFeatureSelector(target=target, vif_threshold=10.0)
vif_selector.fit(train)
train = vif_selector.transform(train)
test = vif_selector.transform(test)
# H2O AutoML: starts a local H2O cluster (needs Java)
h2o_selector = H2OFeatureSelector(target=target, relevance=0.95, max_models=7)
h2o_selector.fit(train)
train = h2o_selector.transform(train)
test = h2o_selector.transform(test)Note.
VIFFeatureSelectorevaluates numeric, null-free columns, so impute nulls first (it raisesValidationErroron nulls). Non-numeric columns are passed through untouched, andvif_thresholdmust be within[3.0, 30.0].
| Resource | What it covers |
|---|---|
automated_pipeline.py |
The automated pipeline on regression, binary and multiclass tasks |
builder_complete.py |
Every builder configuration, plus an end-to-end fit |
custom_preprocessing.py |
Each preprocessing component used on its own |
| Jupyter notebooks | Interactive, step-by-step walkthroughs of all of the above |
If Atlantic supports your work, please cite the accompanying paper:
@article{SANTOS2023100532,
author = {Luis Fernando Santos and Luis Ferreira},
title = {Atlantic - Automated data preprocessing framework for supervised machine learning},
journal = {Software Impacts},
volume = {17},
year = {2023},
issn = {2665-9638},
doi = {http://dx.doi.org/10.1016/j.simpa.2023.100532},
url = {https://www.sciencedirect.com/science/article/pii/S2665963823000696}
}Distributed under the MIT License. See LICENSE for details.
Luís Santos · LinkedIn