I am using IterativeImputer from sklearn.impute to fill missing values in my dataset. One of my columns, Education_Level, is a categorical feature, so I first applied LabelEncoder to convert it into numerical form before imputing. However, after inverse transforming the encoded values back to their original categories, I am getting NaN values in some rows.

Code I Am Using:

import numpy as np
import pandas as pd
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
from sklearn.preprocessing import StandardScaler, LabelEncoder

# Copy the original dataset
df_iter = df.copy()

# Encode categorical column
encoder = LabelEncoder()
df_iter['Education_Level'] = encoder.fit_transform(df_iter['Education_Level'])

# Apply StandardScaler
scaler = StandardScaler()
data_scaled = scaler.fit_transform(df_iter)

# Apply IterativeImputer
imputer = IterativeImputer(max_iter=10, random_state=42)
imputed_data = imputer.fit_transform(data_scaled)

# Convert back to original scale
df_iter = pd.DataFrame(scaler.inverse_transform(imputed_data), columns=df_iter.columns)

# Convert Education_Level back to integer values
df_iter['Education_Level'] = np.round(df_iter['Education_Level']).astype(int)

# Inverse transform the encoded labels
df_iter['Education_Level'] = encoder.inverse_transform(df_iter['Education_Level'])

Issue Faced: Some rows in Education_Level still contain NaN values after inverse_transform.

I suspect IterativeImputer is generating values that do not match the original encoded categories, leading to the error when trying to map them back.

Questions: Why is IterativeImputer generating values that are not exactly matching the original encoded categories?

What is the best way to ensure that inverse_transform does not result in NaN values?

Should I use a different imputation method for categorical data instead of IterativeImputer?

Would appreciate any insights or recommendations on how to handle this issue properly.

enter image description here enter image description here

What I Tried Used LabelEncoder to convert the Education_Level categorical column into numerical values before imputation.

Applied StandardScaler to normalize the data before feeding it into IterativeImputer.

Used IterativeImputer to fill missing values.

Inverse transformed the data back using StandardScaler.inverse_transform.

Rounded the Education_Level values to the nearest integer before applying LabelEncoder.inverse_transform.

What I Expected I expected IterativeImputer to fill missing values without modifying the structure of categorical variables.

After inverse transforming LabelEncoder, I expected all rows to have valid category labels instead of NaN.

What Actually Happened Some rows in Education_Level ended up as NaN after encoder.inverse_transform.

This likely happened because IterativeImputer generated intermediate values (e.g., 1.75, 2.3), which did not map back correctly to the original label categories.

Additional Attempts to Fix It Tried rounding values before inverse transforming, but still got NaN because some values were slightly outside the valid label range.

Tried clipping values to match the valid encoded labels, but NaN values persisted.