I am working on a binary classification task with class imbalance using H2O AutoML with Python.

After training models with H2O AutoML, I obtain serious differences when I compute model_performance(train=True) and model_performance(test_data=data_train). To my understanding they should output the same results, but with the first I obtain a AUC of ~0.7 and with the later a AUC of ~0.98. balance_classes option is set to False and I use fold_column for proper stratification and weights_column option to balance each fold.