ADARUL¶
In [1]:
Copied!
import rul_datasets
import rul_adapt
import pytorch_lightning as pl
import omegaconf
import rul_datasets
import rul_adapt
import pytorch_lightning as pl
import omegaconf
Reproduce original configurations¶
You can reproduce the original experiments by Ragab et al. by using the get_adarul constructor function.
Additional kwargs for the trainer, e.g. accelerator="gpu" for training on a GPU, can be passed to the function as a dictionary. The first dictionary is used for the pre-training trainer and the second one for the main trainer.
In [2]:
Copied!
pl.seed_everything(42, workers=True) # makes is reproducible
pre_training, main_training = rul_adapt.construct.get_adarul(
3, 1, {"max_epochs": 1}, {"max_epochs": 1}
)
pl.seed_everything(42, workers=True) # makes is reproducible
pre_training, main_training = rul_adapt.construct.get_adarul(
3, 1, {"max_epochs": 1}, {"max_epochs": 1}
)
Global seed set to 42 GPU available: False, used: False TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs GPU available: False, used: False TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs
The function returns two tuples. The first contains everything needed for pre-training, the second everything needed for the main training.
In [3]:
Copied!
pre_dm, pre_approach, pre_trainer = pre_training
pre_trainer.fit(pre_approach, pre_dm)
pre_dm, pre_approach, pre_trainer = pre_training
pre_trainer.fit(pre_approach, pre_dm)
| Name | Type | Params ---------------------------------------------------------------- 0 | train_loss | MeanSquaredError | 0 1 | val_loss | MeanSquaredError | 0 2 | test_loss | MeanSquaredError | 0 3 | evaluator | AdaptionEvaluator | 0 4 | _feature_extractor | ActivationDropoutWrapper | 62.5 K 5 | _regressor | FullyConnectedHead | 6.3 K ---------------------------------------------------------------- 68.7 K Trainable params 0 Non-trainable params 68.7 K Total params 0.275 Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]
Training: 0it [00:00, ?it/s]
Validation: 0it [00:00, ?it/s]
`Trainer.fit` stopped: `max_epochs=1` reached.
After pre-training, we can use the pre-trained networks to initialize the main training.
The networks of the pre-training approach, i.e. feature_extractor and regressor, can be accessed as properties.
In [4]:
Copied!
dm, approach, domain_disc, trainer = main_training
approach.set_model(pre_approach.feature_extractor, pre_approach.regressor, domain_disc)
trainer.fit(approach, dm)
trainer.test(approach, dm)
dm, approach, domain_disc, trainer = main_training
approach.set_model(pre_approach.feature_extractor, pre_approach.regressor, domain_disc)
trainer.fit(approach, dm)
trainer.test(approach, dm)
| Name | Type | Params ---------------------------------------------------------------------- 0 | gan_loss | BCEWithLogitsLoss | 0 1 | evaluator | AdaptionEvaluator | 0 2 | _feature_extractor | ActivationDropoutWrapper | 62.5 K 3 | _regressor | FullyConnectedHead | 6.3 K 4 | _domain_disc | FullyConnectedHead | 6.3 K 5 | frozen_feature_extractor | ActivationDropoutWrapper | 62.5 K ---------------------------------------------------------------------- 75.0 K Trainable params 62.5 K Non-trainable params 137 K Total params 0.550 Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]
/home/tilman/Programming/rul-adapt/.venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1609: PossibleUserWarning: The number of training batches (36) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch. rank_zero_warn(
Training: 0it [00:00, ?it/s]
Validation: 0it [00:00, ?it/s]
`Trainer.fit` stopped: `max_epochs=1` reached.
Testing: 0it [00:00, ?it/s]
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Test metric DataLoader 0 DataLoader 1
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test/source/rmse 24.635229110717773
test/source/score 1310.8724365234375
test/target/rmse 31.754472732543945
test/target/score 2988.716064453125
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Out[4]:
[{'test/source/rmse/dataloader_idx_0': 24.635229110717773,
'test/source/score/dataloader_idx_0': 1310.8724365234375},
{'test/target/rmse/dataloader_idx_1': 31.754472732543945,
'test/target/score/dataloader_idx_1': 2988.716064453125}]
If you only want to see the hyperparameters, you can use the get_adarul_config function.
This returns an omegaconf.DictConfig which you can modify.
Afterwards, you can pass the config to adarul_from_config to receive the training-ready approach.
In [5]:
Copied!
three2one_config = rul_adapt.construct.get_adarul_config(3, 1)
print(omegaconf.OmegaConf.to_yaml(three2one_config, resolve=True))
three2one_config = rul_adapt.construct.get_adarul_config(3, 1)
print(omegaconf.OmegaConf.to_yaml(three2one_config, resolve=True))
dm:
source:
_target_: rul_datasets.CmapssReader
fd: 3
window_size: 30
max_rul: 130
operation_condition_aware_scaling: true
target:
fd: 1
percent_broken: 1.0
batch_size: 10
feature_extractor:
_convert_: all
_target_: rul_adapt.model.ActivationDropoutWrapper
wrapped:
_target_: rul_adapt.model.LstmExtractor
input_channels: 14
units:
- 32
- 32
- 32
bidirectional: true
dropout: 0.5
regressor:
_convert_: all
_target_: rul_adapt.model.FullyConnectedHead
input_channels: 64
act_func_on_last_layer: false
units:
- 64
- 32
- 1
dropout: 0.5
domain_disc:
_convert_: all
_target_: rul_adapt.model.FullyConnectedHead
input_channels: 64
act_func_on_last_layer: false
units:
- 64
- 32
- 1
adarul_pre:
_target_: rul_adapt.approach.SupervisedApproach
lr: 0.0001
loss_type: mse
optim_type: adam
rul_scale: 130
adarul:
_target_: rul_adapt.approach.AdaRulApproach
lr: 0.0001
max_rul: 130
num_disc_updates: 35
num_gen_updates: 1
trainer_pre:
_target_: pytorch_lightning.Trainer
max_epochs: 5
gradient_clip_val: 1.0
trainer:
_target_: pytorch_lightning.Trainer
max_epochs: 20
limit_train_batches: 36
Run your own experiments¶
You can use the ADARUL implementation to run your own experiments with different hyperparameters or on different datasets. Here we build an approach with an CNN feature extractor.
In [7]:
Copied!
source = rul_datasets.CmapssReader(3)
target = source.get_compatible(1, percent_broken=0.8)
pre_dm = rul_datasets.RulDataModule(source, batch_size=32)
dm = rul_datasets.DomainAdaptionDataModule(
pre_dm, rul_datasets.RulDataModule(target, batch_size=32),
)
feature_extractor = rul_adapt.model.CnnExtractor(
input_channels=14,
units=[16, 16, 16],
seq_len=30,
fc_units=8,
)
regressor = rul_adapt.model.FullyConnectedHead(
input_channels=8,
units=[8, 1],
act_func_on_last_layer=False,
)
domain_disc = rul_adapt.model.FullyConnectedHead(
input_channels=8,
units=[8, 1],
act_func_on_last_layer=False,
)
pre_approach = rul_adapt.approach.SupervisedApproach(
lr=0.001, loss_type="mse", optim_type="adam", rul_scale=source.max_rul
)
pre_approach.set_model(feature_extractor, regressor)
pre_trainer = pl.Trainer(max_epochs=1)
trainer.fit(pre_approach, pre_dm)
approach = rul_adapt.approach.AdaRulApproach(
lr=0.001,
max_rul=source.max_rul,
num_disc_updates=35,
num_gen_updates=1,
)
approach.set_model(
pre_approach.feature_extractor, pre_approach.regressor, domain_disc
)
trainer = pl.Trainer(max_epochs=1)
trainer.fit(approach, dm)
trainer.test(approach, dm)
source = rul_datasets.CmapssReader(3)
target = source.get_compatible(1, percent_broken=0.8)
pre_dm = rul_datasets.RulDataModule(source, batch_size=32)
dm = rul_datasets.DomainAdaptionDataModule(
pre_dm, rul_datasets.RulDataModule(target, batch_size=32),
)
feature_extractor = rul_adapt.model.CnnExtractor(
input_channels=14,
units=[16, 16, 16],
seq_len=30,
fc_units=8,
)
regressor = rul_adapt.model.FullyConnectedHead(
input_channels=8,
units=[8, 1],
act_func_on_last_layer=False,
)
domain_disc = rul_adapt.model.FullyConnectedHead(
input_channels=8,
units=[8, 1],
act_func_on_last_layer=False,
)
pre_approach = rul_adapt.approach.SupervisedApproach(
lr=0.001, loss_type="mse", optim_type="adam", rul_scale=source.max_rul
)
pre_approach.set_model(feature_extractor, regressor)
pre_trainer = pl.Trainer(max_epochs=1)
trainer.fit(pre_approach, pre_dm)
approach = rul_adapt.approach.AdaRulApproach(
lr=0.001,
max_rul=source.max_rul,
num_disc_updates=35,
num_gen_updates=1,
)
approach.set_model(
pre_approach.feature_extractor, pre_approach.regressor, domain_disc
)
trainer = pl.Trainer(max_epochs=1)
trainer.fit(approach, dm)
trainer.test(approach, dm)
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/home/tilman/Programming/rul-adapt/.venv/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:613: UserWarning: Checkpoint directory /home/tilman/Programming/rul-adapt/docs/examples/lightning_logs/version_24/checkpoints exists and is not empty.
rank_zero_warn(f"Checkpoint directory {dirpath} exists and is not empty.")
| Name | Type | Params
----------------------------------------------------------
0 | train_loss | MeanSquaredError | 0
1 | val_loss | MeanSquaredError | 0
2 | test_loss | MeanSquaredError | 0
3 | evaluator | AdaptionEvaluator | 0
4 | _feature_extractor | CnnExtractor | 5.3 K
5 | _regressor | FullyConnectedHead | 81
----------------------------------------------------------
5.4 K Trainable params
0 Non-trainable params
5.4 K Total params
0.022 Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]
`Trainer.fit` stopped: `max_epochs=1` reached. GPU available: False, used: False TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs | Name | Type | Params ---------------------------------------------------------------- 0 | gan_loss | BCEWithLogitsLoss | 0 1 | evaluator | AdaptionEvaluator | 0 2 | _feature_extractor | CnnExtractor | 5.3 K 3 | _regressor | FullyConnectedHead | 81 4 | _domain_disc | FullyConnectedHead | 81 5 | frozen_feature_extractor | CnnExtractor | 5.3 K ---------------------------------------------------------------- 5.5 K Trainable params 5.3 K Non-trainable params 10.8 K Total params 0.043 Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]
Training: 0it [00:00, ?it/s]
Validation: 0it [00:00, ?it/s]
`Trainer.fit` stopped: `max_epochs=1` reached.
Testing: 0it [00:00, ?it/s]
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Test metric DataLoader 0 DataLoader 1
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test/source/rmse 65.23387908935547
test/source/score 70147.4921875
test/target/rmse 66.29417419433594
test/target/score 64386.578125
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Out[7]:
[{'test/source/rmse/dataloader_idx_0': 65.23387908935547,
'test/source/score/dataloader_idx_0': 70147.4921875},
{'test/target/rmse/dataloader_idx_1': 66.29417419433594,
'test/target/score/dataloader_idx_1': 64386.578125}]