{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Regression with an Insurance Dataset","metadata":{}},{"cell_type":"markdown","source":"# Introduction\n\n#### **Kaggle Playground Series - Season 4, Episode 12**\n\n**Project Overview**\n\nThis project focuses on developing a machine learning model to predict insurance premium amounts based on various customer and policy-related features. As part of the Kaggle Playground Series (Season 4, Episode 12), the challenge involves building a regression model that can accurately estimate insurance premiums while minimizing the Root Mean Squared Logarithmic Error (RMSLE).\n\n**Dataset Description**\n\nThe dataset consists of 1.2 million training samples and 800,000 test samples, with 20 features covering diverse aspects of insurance customers:\nDemographic Information\n\n* Age\n* Gender\n* Annual Income\n* Marital Status\n* Number of Dependents\n* Education Level\n* Occupation\n\nHealth and Lifestyle Factors\n\n* Health Score\n* Smoking Status\n* Exercise Frequency\n\nInsurance-Related Features\n\n* Location\n* Policy Type\n* Previous Claims\n* Vehicle Age\n* Insurance Duration\n* Policy Start Date\n* Customer Feedback\n* Property Type\n* Credit Score\n\n**Project Challenges**\n\nMissing Data Handling: Several key features have significant missing values:\n\n* Occupation (~30% missing)\n* Previous Claims (~30% missing)\n* Credit Score (~11.5% missing)\n* Number of Dependents (~9% missing)\n\n\n**Feature Engineering Opportunities:**\n\n* Temporal features from Policy Start Date\n* Categorical encoding strategies\n* Interaction features between related variables\n\n\n**Model Optimization:**\n\n* RMSLE optimization requires careful handling of predictions\n* Balance between model complexity and performance\n* Proper cross-validation strategy\n\n\n\n**Project Goals**\n\n* Develop a robust preprocessing pipeline to handle missing values and feature engineering\n* Build and optimize machine learning models focusing on RMSLE minimization\n* Create an ensemble of best-performing models\n* Generate reliable premium predictions for the test dataset","metadata":{}},{"cell_type":"markdown","source":"# Preparation","metadata":{}},{"cell_type":"markdown","source":"## Intialization","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom scipy import stats\nfrom datetime import datetime\nfrom sklearn.preprocessing import LabelEncoder, PolynomialFeatures\nfrom sklearn.cluster import KMeans\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_log_error\nimport xgboost as xgb\nimport lightgbm as lgb\nimport catboost as cb","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-12-18T21:41:19.411562Z","iopub.execute_input":"2024-12-18T21:41:19.413352Z","iopub.status.idle":"2024-12-18T21:41:21.232999Z","shell.execute_reply.started":"2024-12-18T21:41:19.413307Z","shell.execute_reply":"2024-12-18T21:41:21.231739Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Load Data","metadata":{}},{"cell_type":"code","source":"# File paths\nTRAIN_PATH = '/kaggle/input/playground-series-s4e12/train.csv'\nTEST_PATH = '/kaggle/input/playground-series-s4e12/test.csv'\nSUBMISSION_PATH = '/kaggle/input/playground-series-s4e12/sample_submission.csv'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-18T21:21:13.270797Z","iopub.execute_input":"2024-12-18T21:21:13.271082Z","iopub.status.idle":"2024-12-18T21:21:13.283059Z","shell.execute_reply.started":"2024-12-18T21:21:13.271057Z","shell.execute_reply":"2024-12-18T21:21:13.282037Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## General Information","metadata":{}},{"cell_type":"code","source":"def load_and_examine_data():\n    # Read training data\n    train_df = pd.read_csv(TRAIN_PATH)\n    test_df = pd.read_csv(TEST_PATH)\n    \n    print(\"\\n=== Dataset Shape ===\")\n    print(f\"Training set shape: {train_df.shape}\")\n    print(f\"Test set shape: {test_df.shape}\")\n    \n    print(\"\\n=== First Few Rows ===\")\n    print(train_df.head())\n    \n    print(\"\\n=== Data Info ===\")\n    print(train_df.info())\n    \n    return train_df, test_df\n\ntrain_df, test_df = load_and_examine_data()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-18T21:21:13.28458Z","iopub.execute_input":"2024-12-18T21:21:13.284884Z","iopub.status.idle":"2024-12-18T21:21:17.548756Z","shell.execute_reply.started":"2024-12-18T21:21:13.284853Z","shell.execute_reply":"2024-12-18T21:21:17.547601Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Fix Data","metadata":{}},{"cell_type":"code","source":"class DataQualityAnalyzer:\n    def __init__(self, train_df, test_df):\n        self.train_df = train_df.copy()\n        self.test_df = test_df.copy()\n        \n    def check_duplicates(self):\n        \"\"\"Check for duplicate rows and values in key columns\"\"\"\n        print(\"\\n=== Duplicate Analysis ===\")\n        \n        # Check for completely duplicate rows\n        train_duplicates = self.train_df.duplicated().sum()\n        test_duplicates = self.test_df.duplicated().sum()\n        \n        print(f\"Complete duplicate rows in training set: {train_duplicates}\")\n        print(f\"Complete duplicate rows in test set: {test_duplicates}\")\n        \n        # Check for potential near-duplicates by excluding id and target\n        train_features = self.train_df.drop(['id', 'Premium Amount'], axis=1)\n        test_features = self.test_df.drop(['id'], axis=1)\n        \n        train_feature_duplicates = train_features.duplicated().sum()\n        test_feature_duplicates = test_features.duplicated().sum()\n        \n        print(f\"\\nFeature-only duplicate rows in training set: {train_feature_duplicates}\")\n        print(f\"Feature-only duplicate rows in test set: {test_feature_duplicates}\")\n        \n        return {\n            'train_duplicates': train_duplicates,\n            'test_duplicates': test_duplicates,\n            'train_feature_duplicates': train_feature_duplicates,\n            'test_feature_duplicates': test_feature_duplicates\n        }\n    \n    def check_data_consistency(self):\n        \"\"\"Check for data consistency issues\"\"\"\n        print(\"\\n=== Data Consistency Analysis ===\")\n        \n        # Check value ranges\n        numeric_columns = self.train_df.select_dtypes(include=['int64', 'float64']).columns\n        \n        for col in numeric_columns:\n            if col not in ['id', 'Premium Amount']:\n                train_range = self.train_df[col].agg(['min', 'max'])\n                test_range = self.test_df[col].agg(['min', 'max'])\n                \n                print(f\"\\nColumn: {col}\")\n                print(f\"Train range: {train_range['min']:.2f} to {train_range['max']:.2f}\")\n                print(f\"Test range: {test_range['min']:.2f} to {test_range['max']:.2f}\")\n                \n                # Check for potential outliers using IQR method\n                Q1 = self.train_df[col].quantile(0.25)\n                Q3 = self.train_df[col].quantile(0.75)\n                IQR = Q3 - Q1\n                outlier_count = ((self.train_df[col] < (Q1 - 1.5 * IQR)) | \n                               (self.train_df[col] > (Q3 + 1.5 * IQR))).sum()\n                print(f\"Potential outliers: {outlier_count}\")\n    \n    def check_missing_values(self):\n        \"\"\"Analyze missing values in both datasets\"\"\"\n        print(\"\\n=== Missing Values Analysis ===\")\n        \n        # Calculate missing values for both datasets\n        train_missing = self.train_df.isnull().sum()\n        test_missing = self.test_df.isnull().sum()\n        \n        # Calculate missing value percentages\n        train_missing_pct = (train_missing / len(self.train_df)) * 100\n        test_missing_pct = (test_missing / len(self.test_df)) * 100\n        \n        missing_stats = pd.DataFrame({\n            'Train Missing Count': train_missing,\n            'Train Missing %': train_missing_pct,\n            'Test Missing Count': test_missing,\n            'Test Missing %': test_missing_pct\n        }).sort_values('Train Missing %', ascending=False)\n        \n        print(missing_stats[missing_stats['Train Missing %'] > 0])\n        \n        return missing_stats\n    \n    def check_categorical_consistency(self):\n        \"\"\"Check consistency of categorical variables between train and test\"\"\"\n        print(\"\\n=== Categorical Variables Analysis ===\")\n        \n        categorical_columns = self.train_df.select_dtypes(include=['object']).columns\n        \n        for col in categorical_columns:\n            train_categories = set(self.train_df[col].dropna().unique())\n            test_categories = set(self.test_df[col].dropna().unique())\n            \n            # Find categories that appear in test but not in train\n            test_only_categories = test_categories - train_categories\n            \n            print(f\"\\nColumn: {col}\")\n            print(f\"Unique values in train: {len(train_categories)}\")\n            print(f\"Unique values in test: {len(test_categories)}\")\n           # if test_only_categories:\n             #   print(f\"Categories in test but not in train: {test_only_categories}\")\n    \n    def analyze_target_distribution(self):\n        \"\"\"Analyze the distribution of the target variable\"\"\"\n        print(\"\\n=== Target Variable Analysis ===\")\n        \n        target = self.train_df['Premium Amount']\n        \n        # Basic statistics\n        stats = target.describe()\n        print(\"\\nTarget Variable Statistics:\")\n        print(stats)\n        \n        # Check skewness and kurtosis\n        print(f\"\\nSkewness: {target.skew():.2f}\")\n        print(f\"Kurtosis: {target.kurtosis():.2f}\")\n        \n        # Plot distribution\n        plt.figure(figsize=(10, 6))\n        plt.subplot(1, 2, 1)\n        sns.histplot(target, bins=50)\n        plt.title('Premium Amount Distribution')\n        \n        plt.subplot(1, 2, 2)\n        sns.histplot(np.log1p(target), bins=50)\n        plt.title('Log-transformed Premium Amount')\n        plt.tight_layout()\n        plt.show()\n    \n    def run_full_analysis(self):\n        \"\"\"Run all analysis functions\"\"\"\n        self.check_duplicates()\n        self.check_data_consistency()\n        self.check_missing_values()\n        self.check_categorical_consistency()\n        self.analyze_target_distribution()\n\n# Initialize the analyzer\nanalyzer = DataQualityAnalyzer(train_df, test_df)\n\n# Run the full analysis\nanalyzer.run_full_analysis()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-18T21:24:56.500594Z","iopub.execute_input":"2024-12-18T21:24:56.500899Z","iopub.status.idle":"2024-12-18T21:25:06.254201Z","shell.execute_reply.started":"2024-12-18T21:24:56.500874Z","shell.execute_reply":"2024-12-18T21:25:06.253358Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Data Cleaning Pipeline","metadata":{}},{"cell_type":"code","source":"class DataCleaner:\n    def __init__(self):\n        self.label_encoders = {}\n        \n    def clean_data(self, df, is_training=True):\n        \"\"\"Clean and preprocess the dataset\"\"\"\n        df = df.copy()\n        \n        # 1. Create missing value flags\n        missing_flags = self._create_missing_flags(df)\n        \n        # 2. Handle numeric features\n        df = self._clean_numeric_features(df)\n        \n        # 3. Handle categorical features\n        df = self._clean_categorical_features(df)\n        \n        # 4. Process dates\n        df = self._process_dates(df)\n        \n        # 5. Combine with missing flags\n        df = pd.concat([df, missing_flags], axis=1)\n        \n        return df\n    \n    def _create_missing_flags(self, df):\n        \"\"\"Create binary flags for missing values\"\"\"\n        cols_with_missing = ['Previous Claims', 'Occupation', 'Credit Score', \n                           'Number of Dependents', 'Customer Feedback', \n                           'Health Score', 'Annual Income']\n        \n        missing_flags = pd.DataFrame()\n        for col in cols_with_missing:\n            missing_flags[f'{col}_missing'] = df[col].isna().astype(int)\n            \n        return missing_flags\n    \n    def _clean_numeric_features(self, df):\n        \"\"\"Clean and impute numeric features\"\"\"\n        # Handle Annual Income outliers\n        df['Annual Income'] = df['Annual Income'].clip(\n            lower=df['Annual Income'].quantile(0.01),\n            upper=df['Annual Income'].quantile(0.99)\n        )\n        \n        # Handle Previous Claims outliers\n        df['Previous Claims'] = df['Previous Claims'].clip(0, 5)\n        \n        # Impute numeric features\n        numeric_features = {\n            'Age': 'median',\n            'Annual Income': 'median',\n            'Number of Dependents': 'median',\n            'Health Score': 'median',\n            'Previous Claims': 'median',\n            'Vehicle Age': 'median',\n            'Credit Score': 'median',\n            'Insurance Duration': 'median'\n        }\n        \n        for col, strategy in numeric_features.items():\n            if strategy == 'median':\n                df[col] = df[col].fillna(df[col].median())\n        \n        return df\n    \n    def _clean_categorical_features(self, df):\n        \"\"\"Clean and encode categorical features\"\"\"\n        # Mode imputation for categorical features\n        categorical_features = ['Occupation', 'Customer Feedback', 'Marital Status']\n        \n        for col in categorical_features:\n            df[col] = df[col].fillna(df[col].mode()[0])\n        \n        return df\n    \n    def _process_dates(self, df):\n        \"\"\"Process date features\"\"\"\n        df['Policy_Start_Date'] = pd.to_datetime(df['Policy Start Date'])\n        \n        # Extract date components\n        df['Policy_Year'] = df['Policy_Start_Date'].dt.year\n        df['Policy_Month'] = df['Policy_Start_Date'].dt.month\n        df['Policy_Day'] = df['Policy_Start_Date'].dt.day\n        df['Policy_DayOfWeek'] = df['Policy_Start_Date'].dt.dayofweek\n        df['Policy_DayOfYear'] = df['Policy_Start_Date'].dt.dayofyear\n        \n        # Drop original date column\n        df.drop(['Policy Start Date', 'Policy_Start_Date'], axis=1, inplace=True)\n        \n        return df\n\n# Function to prepare data for modeling\ndef prepare_data(train_df, test_df):\n    \"\"\"Prepare train and test data for modeling\"\"\"\n    cleaner = DataCleaner()\n    \n    # Clean data\n    train_cleaned = cleaner.clean_data(train_df, is_training=True)\n    test_cleaned = cleaner.clean_data(test_df, is_training=False)\n    \n    # Log transform the target\n    if 'Premium Amount' in train_cleaned.columns:\n        train_cleaned['Premium Amount'] = np.log1p(train_cleaned['Premium Amount'])\n    \n    return train_cleaned, test_cleaned\n\n# Clean the data\ntrain_cleaned, test_cleaned = prepare_data(train_df, test_df)\n\n# Check the results\nprint(\"Cleaned Training Data Shape:\", train_cleaned.shape)\nprint(\"Cleaned Test Data Shape:\", test_cleaned.shape)\nprint(\"\\nMissing Values After Cleaning:\")\nprint(train_cleaned.isnull().sum().sum())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-18T21:28:06.88619Z","iopub.execute_input":"2024-12-18T21:28:06.886542Z","iopub.status.idle":"2024-12-18T21:28:10.306741Z","shell.execute_reply.started":"2024-12-18T21:28:06.886518Z","shell.execute_reply":"2024-12-18T21:28:10.306003Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# EDA","metadata":{}},{"cell_type":"code","source":"# Display columns and basic info about the cleaned datasets\nprint(\"=== Training Dataset Columns ===\")\nprint(train_cleaned.columns.tolist())\n\nprint(\"\\n=== Basic Statistics of Numeric Features ===\")\nnumeric_cols = train_cleaned.select_dtypes(include=['float64', 'int64']).columns\nprint(train_cleaned[numeric_cols].describe())\n\n# Check correlation with target\ncorrelations = train_cleaned[numeric_cols].corr()['Premium Amount'].sort_values(ascending=False)\nprint(\"\\n=== Feature Correlations with Premium Amount ===\")\nprint(correlations)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-18T21:28:58.695883Z","iopub.execute_input":"2024-12-18T21:28:58.696199Z","iopub.status.idle":"2024-12-18T21:29:00.01065Z","shell.execute_reply.started":"2024-12-18T21:28:58.696175Z","shell.execute_reply":"2024-12-18T21:29:00.009753Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Looking at the data analysis, there are several interesting insights that can guide our feature engineering:\n\nWeak Direct Correlations:\n\n* Most features show surprisingly weak correlations with Premium Amount\n* Even Annual Income has a weak negative correlation (-0.08)\n* This suggests we need to create interaction features and transformations\n\n\nMissing Value Indicators:\n\n* Some missing value indicators (like Health Score_missing) show stronger correlations than their base features\n* This validates our decision to keep these as features","metadata":{}},{"cell_type":"markdown","source":"# Feature Engineering Pipeline","metadata":{}},{"cell_type":"code","source":"class FeatureEngineer:\n    def __init__(self):\n        self.label_encoders = {}\n        self.kmeans_models = {}\n        \n    def create_features(self, df, is_training=True):\n        \"\"\"Create all features\"\"\"\n        df = df.copy()\n        \n        # 1. Encode categorical variables\n        df = self._encode_categorical(df, is_training)\n        \n        # 2. Create ratio features\n        df = self._create_ratio_features(df)\n        \n        # 3. Create interaction features\n        df = self._create_interaction_features(df)\n        \n        # 4. Create polynomial features for key numeric variables\n        df = self._create_polynomial_features(df)\n        \n        # 5. Create cluster features\n        df = self._create_cluster_features(df, is_training)\n        \n        # 6. Create binned features\n        df = self._create_binned_features(df)\n        \n        # Drop unnecessary columns\n        df = self._drop_unused_features(df)\n        \n        return df\n    \n    def _encode_categorical(self, df, is_training):\n        \"\"\"Encode categorical variables\"\"\"\n        categorical_cols = ['Gender', 'Marital Status', 'Education Level', \n                          'Occupation', 'Location', 'Policy Type', \n                          'Customer Feedback', 'Smoking Status', \n                          'Exercise Frequency', 'Property Type']\n        \n        for col in categorical_cols:\n            if is_training:\n                self.label_encoders[col] = LabelEncoder()\n                df[col] = self.label_encoders[col].fit_transform(df[col])\n            else:\n                df[col] = self.label_encoders[col].transform(df[col])\n        \n        return df\n    \n    def _create_ratio_features(self, df):\n        \"\"\"Create ratio-based features\"\"\"\n        # Income related ratios\n        df['Income_per_Dependent'] = df['Annual Income'] / (df['Number of Dependents'] + 1)\n        df['Income_per_Age'] = df['Annual Income'] / df['Age']\n        \n        # Insurance related ratios\n        df['Claims_per_Duration'] = df['Previous Claims'] / df['Insurance Duration']\n        df['Health_Score_per_Age'] = df['Health Score'] / df['Age']\n        \n        # Credit related ratios\n        df['Credit_Score_per_Income'] = df['Credit Score'] / (df['Annual Income'] + 1)\n        df['Credit_Score_per_Age'] = df['Credit Score'] / df['Age']\n        \n        return df\n    \n    def _create_interaction_features(self, df):\n        \"\"\"Create interaction features\"\"\"\n        # Health and lifestyle interactions\n        df['Health_Smoking'] = df['Health Score'] * df['Smoking Status']\n        df['Health_Exercise'] = df['Health Score'] * df['Exercise Frequency']\n        \n        # Risk related interactions\n        df['Age_Claims'] = df['Age'] * df['Previous Claims']\n        df['Duration_Claims'] = df['Insurance Duration'] * df['Previous Claims']\n        \n        # Financial interactions\n        df['Income_Credit'] = df['Annual Income'] * df['Credit Score']\n        df['Income_Vehicle'] = df['Annual Income'] * df['Vehicle Age']\n        \n        return df\n    \n    def _create_polynomial_features(self, df):\n        \"\"\"Create polynomial features for key numeric variables\"\"\"\n        key_numeric = ['Age', 'Annual Income', 'Health Score', 'Credit Score']\n        for col in key_numeric:\n            df[f'{col}_squared'] = df[col] ** 2\n            df[f'{col}_cubed'] = df[col] ** 3\n        \n        return df\n    \n    def _create_cluster_features(self, df, is_training, n_clusters=5):\n        \"\"\"Create cluster-based features\"\"\"\n        # Financial clustering\n        financial_features = ['Annual Income', 'Credit Score', 'Previous Claims']\n        \n        # Health clustering\n        health_features = ['Age', 'Health Score', 'Exercise Frequency']\n        \n        for name, features in [('financial', financial_features), \n                             ('health', health_features)]:\n            feature_data = df[features].fillna(0)\n            \n            if is_training:\n                self.kmeans_models[name] = KMeans(n_clusters=n_clusters, random_state=42)\n                df[f'{name}_cluster'] = self.kmeans_models[name].fit_predict(feature_data)\n            else:\n                df[f'{name}_cluster'] = self.kmeans_models[name].predict(feature_data)\n        \n        return df\n    \n    def _create_binned_features(self, df):\n        \"\"\"Create binned versions of continuous features\"\"\"\n        # Age bins\n        df['Age_bin'] = pd.qcut(df['Age'], q=5, labels=False, duplicates='drop')\n        \n        # Income bins\n        df['Income_bin'] = pd.qcut(df['Annual Income'], q=5, labels=False, duplicates='drop')\n        \n        # Health score bins\n        df['Health_bin'] = pd.qcut(df['Health Score'], q=5, labels=False, duplicates='drop')\n        \n        return df\n    \n    def _drop_unused_features(self, df):\n        \"\"\"Drop features that won't be used in modeling\"\"\"\n        cols_to_drop = ['id']  # Add any other columns that shouldn't be used in modeling\n        return df.drop(cols_to_drop, axis=1)\n\ndef prepare_features(train_df, test_df):\n    \"\"\"Prepare features for both train and test sets\"\"\"\n    engineer = FeatureEngineer()\n    \n    # Create features\n    train_featured = engineer.create_features(train_df, is_training=True)\n    test_featured = engineer.create_features(test_df, is_training=False)\n    \n    return train_featured, test_featured\n\n# Create features\ntrain_featured, test_featured = prepare_features(train_cleaned, test_cleaned)\n\n# Display new features and their correlations with target\nprint(\"=== New Dataset Shapes ===\")\nprint(f\"Training set: {train_featured.shape}\")\nprint(f\"Test set: {test_featured.shape}\")\n\n# Calculate correlations with target for new features\ncorrelations = train_featured.corr()['Premium Amount'].sort_values(ascending=False)\nprint(\"\\n=== Top 15 Feature Correlations with Premium Amount ===\")\nprint(correlations.head(15))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-18T21:32:20.459074Z","iopub.execute_input":"2024-12-18T21:32:20.459461Z","iopub.status.idle":"2024-12-18T21:32:37.904098Z","shell.execute_reply.started":"2024-12-18T21:32:20.459433Z","shell.execute_reply":"2024-12-18T21:32:37.903233Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model Training Pipeline","metadata":{}},{"cell_type":"code","source":"class ModelTrainer:\n    def __init__(self, n_folds=5):\n        self.n_folds = n_folds\n        self.models = {}\n        self.feature_importance = pd.DataFrame()\n        \n    def train(self, X, y, categorical_features=None):\n        \"\"\"Train multiple models using k-fold cross validation\"\"\"\n        # Initialize KFold\n        kf = KFold(n_splits=self.n_folds, shuffle=True, random_state=42)\n        \n        # Initialize arrays for predictions\n        oof_predictions = np.zeros(len(X))\n        feature_importance = []\n        \n        # Initialize models\n        models = {\n            'xgb': self._get_xgb_model(),\n            'lgb': self._get_lgb_model(),\n            'cat': self._get_catboost_model()\n        }\n        \n        # Train models\n        for fold, (train_idx, val_idx) in enumerate(kf.split(X)):\n            print(f\"\\nTraining fold {fold + 1}/{self.n_folds}\")\n            \n            X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]\n            y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]\n            \n            fold_models = {}\n            fold_predictions = np.zeros(len(val_idx))\n            \n            # Train each model type\n            for model_name, model in models.items():\n                print(f\"Training {model_name}...\")\n                \n                if model_name == 'cat':\n                    model.fit(\n                        X_train, y_train,\n                        eval_set=[(X_val, y_val)],\n                        cat_features=categorical_features,\n                        early_stopping_rounds=50,\n                        verbose=False\n                    )\n                elif model_name == 'lgb':\n                    model.fit(\n                        X_train, y_train,\n                        eval_set=[(X_val, y_val)],\n                        callbacks=[lgb.early_stopping(50)]\n                    )\n                else:  # XGBoost\n                    model.fit(\n                        X_train, y_train,\n                        eval_set=[(X_val, y_val)],\n                        verbose=False\n                    )\n                \n                # Make predictions\n                fold_predictions += model.predict(X_val) / len(models)\n                fold_models[model_name] = model\n                \n                # Get feature importance\n                if model_name == 'xgb':\n                    importance = pd.DataFrame({\n                        'feature': X.columns,\n                        'importance': model.feature_importances_,\n                        'fold': fold,\n                        'model': model_name\n                    })\n                    feature_importance.append(importance)\n            \n            # Save fold predictions\n            oof_predictions[val_idx] = fold_predictions\n            self.models[fold] = fold_models\n        \n        # Calculate RMSLE\n        rmsle = np.sqrt(mean_squared_log_error(np.expm1(y), np.expm1(oof_predictions)))\n        print(f\"\\nOverall RMSLE: {rmsle}\")\n        \n        # Aggregate feature importance\n        self.feature_importance = pd.concat(feature_importance, ignore_index=True)\n        \n        return oof_predictions, rmsle\n    \n    def predict(self, X):\n        \"\"\"Make predictions using all trained models\"\"\"\n        predictions = np.zeros(len(X))\n        \n        for fold_models in self.models.values():\n            fold_pred = np.zeros(len(X))\n            for model in fold_models.values():\n                fold_pred += model.predict(X) / len(fold_models)\n            predictions += fold_pred / len(self.models)\n        \n        return predictions\n    \n    def _get_xgb_model(self):\n        \"\"\"Initialize XGBoost model with parameters\"\"\"\n        return xgb.XGBRegressor(\n            n_estimators=2000,\n            learning_rate=0.01,\n            max_depth=6,\n            subsample=0.8,\n            colsample_bytree=0.8,\n            tree_method='hist',\n            random_state=42,\n            early_stopping_rounds=50\n        )\n    \n    def _get_lgb_model(self):\n        \"\"\"Initialize LightGBM model with parameters\"\"\"\n        return lgb.LGBMRegressor(\n            n_estimators=2000,\n            learning_rate=0.01,\n            num_leaves=31,\n            subsample=0.8,\n            colsample_bytree=0.8,\n            random_state=42\n        )\n    \n    def _get_catboost_model(self):\n        \"\"\"Initialize CatBoost model with parameters\"\"\"\n        return cb.CatBoostRegressor(\n            iterations=2000,\n            learning_rate=0.01,\n            depth=6,\n            subsample=0.8,\n            colsample_bylevel=0.8,\n            random_state=42\n        )\n\ndef train_and_predict(train_df, test_df, target_col='Premium Amount'):\n    \"\"\"Train models and make predictions\"\"\"\n    # Prepare data\n    X = train_df.drop(target_col, axis=1)\n    y = train_df[target_col]\n    \n    # Get categorical features\n    categorical_features = X.select_dtypes(include=['object']).columns.tolist()\n    \n    # Initialize and train models\n    trainer = ModelTrainer(n_folds=5)\n    oof_predictions, rmsle = trainer.train(X, y, categorical_features)\n    \n    # Make predictions on test set\n    test_predictions = trainer.predict(test_df)\n    \n    # Get feature importance\n    importance = trainer.feature_importance.groupby('feature')['importance'].mean().sort_values(ascending=False)\n    \n    return np.expm1(test_predictions), importance, rmsle\n\n# Train models and make predictions\npredictions, feature_importance, rmsle = train_and_predict(train_featured, test_featured)\n\n# Display top 20 most important features\nprint(\"\\n=== Top 20 Most Important Features ===\")\nprint(feature_importance.head(20))\n\n# Create submission file\nsubmission = pd.DataFrame({\n    'id': test_df['id'],\n    'Premium Amount': predictions\n})\nsubmission.to_csv('submission.csv', index=False)\n\nprint(\"\\n=== Submission Preview ===\")\nprint(submission.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-18T22:12:35.544994Z","iopub.execute_input":"2024-12-18T22:12:35.545363Z","iopub.status.idle":"2024-12-18T23:12:17.961726Z","shell.execute_reply.started":"2024-12-18T22:12:35.545339Z","shell.execute_reply":"2024-12-18T23:12:17.960723Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}