{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30822,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#### Configs","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n\nif os.environ.get('KAGGLE_KERNEL_RUN_TYPE') is None:\n    PATH = 'D:\\\\dataset\\\\kaggle\\\\Regression-with-an-Insurance-Dataset\\\\'\nelse:\n    PATH = '/kaggle/input/playground-series-s4e12/'\n\nTRAIN_SET = 'train.csv'\nTEST_SET = 'test.csv'\nSUBMISSION = 'sample_submission.csv'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:20.015688Z","iopub.execute_input":"2024-12-20T18:19:20.016309Z","iopub.status.idle":"2024-12-20T18:19:20.024282Z","shell.execute_reply.started":"2024-12-20T18:19:20.016252Z","shell.execute_reply":"2024-12-20T18:19:20.022712Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Exploratory Data Analysis (EDA)","metadata":{}},{"cell_type":"markdown","source":"## 1. Introduction","metadata":{}},{"cell_type":"markdown","source":"### Objective of the Analysis","metadata":{}},{"cell_type":"markdown","source":"- The goal of this EDA is to gain a deeper understanding of the dataset provided for this competition.\n- Specifically, the analysis aims to:\n    - Explore the structure and key characteristics of the dataset.\n    - Identify patterns, trends, and potential relationships between variables.\n    - Understand the distribution and behavior of the target variable (`Premium Amount`).\n    - Detect and analyze missing values and their potential impact.\n    - Gather insights to guide preprocessing and feature engineering for model training.\n- The results from this analysis will help in designing an effective data preprocessing pipeline and building a robust regression model to minimize the competition's evaluation metric(RMSLE).","metadata":{}},{"cell_type":"markdown","source":"### Dataset Overview","metadata":{}},{"cell_type":"markdown","source":"- The dataset provided for this competition consists of training and testing datasets designed to solve the problem of predicting insurance premiums.\n- This overview provides a foundational understanding of the data structure and serves as a critical reference for subsequent analysis and preprocessing steps.\n- The key files are as follows:","metadata":{}},{"cell_type":"markdown","source":"#### 1. Train Dataset (`train.csv`)\n","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(os.path.join(PATH, TRAIN_SET))\nprint(\"Train Shape:\", train_df.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:20.025723Z","iopub.execute_input":"2024-12-20T18:19:20.026166Z","iopub.status.idle":"2024-12-20T18:19:27.089025Z","shell.execute_reply.started":"2024-12-20T18:19:20.026131Z","shell.execute_reply":"2024-12-20T18:19:27.088008Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:27.090569Z","iopub.execute_input":"2024-12-20T18:19:27.09092Z","iopub.status.idle":"2024-12-20T18:19:27.7414Z","shell.execute_reply.started":"2024-12-20T18:19:27.090892Z","shell.execute_reply":"2024-12-20T18:19:27.740534Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:27.742593Z","iopub.execute_input":"2024-12-20T18:19:27.74295Z","iopub.status.idle":"2024-12-20T18:19:27.774992Z","shell.execute_reply.started":"2024-12-20T18:19:27.742909Z","shell.execute_reply":"2024-12-20T18:19:27.774104Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- **Description:**\n    - The training dataset used for model development.\n    - Each sample contains multiple feature variables and the target variable (`Premium Amount`).\n- **Size:** A total of 1,200,000 rows with several columns.\n- **Target Variable:** `Premium Amount` (insurance premium).","metadata":{}},{"cell_type":"markdown","source":"#### 2. Test Dataset (`test.csv`)","metadata":{}},{"cell_type":"code","source":"test_df = pd.read_csv(os.path.join(PATH, TEST_SET))\nprint(\"Test Shape:\", test_df.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:27.77586Z","iopub.execute_input":"2024-12-20T18:19:27.776183Z","iopub.status.idle":"2024-12-20T18:19:31.975571Z","shell.execute_reply.started":"2024-12-20T18:19:27.776159Z","shell.execute_reply":"2024-12-20T18:19:31.974486Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:31.976704Z","iopub.execute_input":"2024-12-20T18:19:31.977049Z","iopub.status.idle":"2024-12-20T18:19:32.402302Z","shell.execute_reply.started":"2024-12-20T18:19:31.977023Z","shell.execute_reply":"2024-12-20T18:19:32.401372Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- **Description:** The test dataset does not include the target variable (`Premium Amount`) and is used to generate predictions.\n- **Size:** Structured similarly to the `train.csv` dataset with the same feature variables.","metadata":{}},{"cell_type":"markdown","source":"#### 3. Sample Submission (`sample_submission.csv`)","metadata":{}},{"cell_type":"code","source":"sample_submission = pd.read_csv(os.path.join(PATH, SUBMISSION))\nprint(\"Sample Submission Shape:\", sample_submission.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:32.403252Z","iopub.execute_input":"2024-12-20T18:19:32.403525Z","iopub.status.idle":"2024-12-20T18:19:32.694687Z","shell.execute_reply.started":"2024-12-20T18:19:32.403502Z","shell.execute_reply":"2024-12-20T18:19:32.693653Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sample_submission.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:32.697434Z","iopub.execute_input":"2024-12-20T18:19:32.697744Z","iopub.status.idle":"2024-12-20T18:19:32.709572Z","shell.execute_reply.started":"2024-12-20T18:19:32.697716Z","shell.execute_reply":"2024-12-20T18:19:32.708572Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- A sample file provided to guide the format for submission.\n- It consists of two columns:\n    - `id`: Unique identifier for each sample.\n    - `Premium Amount`: The target variable to be predicted.","metadata":{}},{"cell_type":"markdown","source":"#### 4. Dataset Characteristics","metadata":{}},{"cell_type":"markdown","source":"- The dataset was generated using a deep learning model trained on actual insurance data.\n- Feature distributions are similar to, but not exactly the same as, the original data.\n- The original dataset can be utilized to enhance analysis and improve model performance.","metadata":{}},{"cell_type":"markdown","source":"### Evaluation Metric: RMSLE","metadata":{}},{"cell_type":"markdown","source":"- The evaluation metric for this competition is the **Root Mean Squared Logarithmic Error (RMSLE)**.\n- RMSLE measures the logarithmic differences between predicted values and actual values, focusing on relative differences.\n\n#### RMSLE Formula\n$RMSLE = \\sqrt{\\frac{1}{n} \\sum_{i=1}^n (\\log(1 + \\hat{y}_i) - \\log(1 + y_i))^2}$\n- $\\hat{y}_i$: Predicted value\n- $y_i$: Actual value\n- n: Number of data points\n\n#### Key Characteristics\n- **Sensitive to small differences:** RMSLE penalizes small errors more heavily, encouraging accurate predictions for smaller values.\n- **Insensitive to large differences:** Logarithmic transformation reduces the impact of large prediction errors.\n- **Log transformation requirement:** To avoid calculation issues, target values must not include zeros or negative values.\n\nRMSLE is particularly suitable for problems like insurance premium prediction, where relative differences in predictions are more meaningful than absolute differences.","metadata":{}},{"cell_type":"markdown","source":"## 2. Target Variable Analysis","metadata":{}},{"cell_type":"markdown","source":"### Distribution of `Premium Amount`\n- The target variable, `Premium Amount`, represents the insurance premium to be predicted.\n- Understanding its distribution is crucial for effective modeling.","metadata":{}},{"cell_type":"code","source":"train_df['Premium Amount'].describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:32.711661Z","iopub.execute_input":"2024-12-20T18:19:32.711997Z","iopub.status.idle":"2024-12-20T18:19:32.777258Z","shell.execute_reply.started":"2024-12-20T18:19:32.711969Z","shell.execute_reply":"2024-12-20T18:19:32.776351Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.histplot(train_df['Premium Amount'], kde=True)\nplt.title('Target Variable Distribution')\nplt.xlabel('Premium Amount')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:32.778163Z","iopub.execute_input":"2024-12-20T18:19:32.778488Z","iopub.status.idle":"2024-12-20T18:19:38.768996Z","shell.execute_reply.started":"2024-12-20T18:19:32.778462Z","shell.execute_reply":"2024-12-20T18:19:38.767853Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Original Distribution:**\n- `Premium Amount` shows a **right-skewed distribution**, indicating a long tail of higher values.\n- The median is lower than the mean, suggesting the presence of extreme outliers that distort the overall distribution.\n- Outliers are likely present and may need special handling during preprocessing.","metadata":{}},{"cell_type":"code","source":"train_df['Log Premium Amount'] = np.log1p(train_df['Premium Amount'])\nsns.histplot(train_df['Log Premium Amount'], kde=True)\nplt.title('Log-Transformed Target Variable Distribution')\nplt.xlabel('Log Premium Amount')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:38.769924Z","iopub.execute_input":"2024-12-20T18:19:38.770272Z","iopub.status.idle":"2024-12-20T18:19:44.730368Z","shell.execute_reply.started":"2024-12-20T18:19:38.770234Z","shell.execute_reply":"2024-12-20T18:19:44.729116Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Log-Transformed Distribution:**\n- The `Premium Amount` variable originally exhibited a right-skewed distribution with a long tail, which can create issues during modeling.\n- After applying a log transformation (`log1p`), the distribution became closer to a normal distribution.\n- Benefits of Log Transformation:\n    - **Improved Prediction Stability**:\n        - Log transformation reduces the impact of extreme values (outliers), enhancing model stability.\n        - Example: Mitigating overfitting issues caused by skewed data.\n    - **Better Model Performance**:\n        - A distribution closer to normality often results in better performance in linear models like regression.\n    - **Consistency with RMSLE**:\n        - Since RMSLE is the competition’s evaluation metric, using log-transformed data ensures better alignment with the scale of the metric, leading to improved optimization.","metadata":{}},{"cell_type":"markdown","source":"## 3. Feature Analysis","metadata":{}},{"cell_type":"markdown","source":"### 3.1 Categorical Variables\n- Categorical variables represent distinct categories, and analyzing their impact on the target variable (`Premium Amount`) is critical.","metadata":{}},{"cell_type":"code","source":"# Identify categorical columns\ncategorical_cols = train_df.select_dtypes(include=['object']).columns\n\n# Display unique value counts for each categorical column\nfor col in categorical_cols:\n    print(f\"Value counts for {col}:\")\n    print(train_df[col].value_counts())\n    print(\"-\" * 50)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:44.731554Z","iopub.execute_input":"2024-12-20T18:19:44.732017Z","iopub.status.idle":"2024-12-20T18:19:46.300895Z","shell.execute_reply.started":"2024-12-20T18:19:44.731977Z","shell.execute_reply":"2024-12-20T18:19:46.299852Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Key Categorical Variables","metadata":{}},{"cell_type":"markdown","source":"1. **Gender**","metadata":{}},{"cell_type":"code","source":"category_means = train_df.groupby('Gender')['Premium Amount'].mean().sort_values()\nprint(category_means)\n\ncategory_means.plot(kind='bar', title='Gender vs Premium Amount', figsize=(8, 5))\nplt.ylabel('Average Premium Amount')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:46.30192Z","iopub.execute_input":"2024-12-20T18:19:46.302186Z","iopub.status.idle":"2024-12-20T18:19:46.56606Z","shell.execute_reply.started":"2024-12-20T18:19:46.302154Z","shell.execute_reply":"2024-12-20T18:19:46.564922Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.boxplot(x='Gender', y='Premium Amount', data=train_df)\nplt.title('Premium Amount by Gender')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:46.567272Z","iopub.execute_input":"2024-12-20T18:19:46.567665Z","iopub.status.idle":"2024-12-20T18:19:47.314682Z","shell.execute_reply.started":"2024-12-20T18:19:46.567605Z","shell.execute_reply":"2024-12-20T18:19:47.313573Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Composed of two categories: Male and Female.\n- The average premium for both groups is similar, suggesting `Gender` may not have a significant effect on the target variable.","metadata":{}},{"cell_type":"markdown","source":"2. **Marital Status**","metadata":{}},{"cell_type":"code","source":"category_means = train_df.groupby('Marital Status')['Premium Amount'].mean().sort_values()\nprint(category_means)\n\ncategory_means.plot(kind='bar', title='Marital Status vs Premium Amount', figsize=(8, 5))\nplt.ylabel('Average Premium Amount')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:47.315848Z","iopub.execute_input":"2024-12-20T18:19:47.316194Z","iopub.status.idle":"2024-12-20T18:19:47.586365Z","shell.execute_reply.started":"2024-12-20T18:19:47.316164Z","shell.execute_reply":"2024-12-20T18:19:47.585335Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.boxplot(x='Marital Status', y='Premium Amount', data=train_df)\nplt.title('Premium Amount by Marital Status')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:47.587358Z","iopub.execute_input":"2024-12-20T18:19:47.587636Z","iopub.status.idle":"2024-12-20T18:19:48.364968Z","shell.execute_reply.started":"2024-12-20T18:19:47.587612Z","shell.execute_reply":"2024-12-20T18:19:48.36394Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Includes three categories: Married, Single, Divorced.\n- Single individuals have the highest average premium, followed by Divorced, with Married individuals having the lowest average premium.","metadata":{}},{"cell_type":"markdown","source":"3. **Policy Type**","metadata":{}},{"cell_type":"code","source":"category_means = train_df.groupby('Policy Type')['Premium Amount'].mean().sort_values()\nprint(category_means)\n\ncategory_means.plot(kind='bar', title='Policy Type vs Premium Amount', figsize=(8, 5))\nplt.ylabel('Average Premium Amount')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:48.366056Z","iopub.execute_input":"2024-12-20T18:19:48.366375Z","iopub.status.idle":"2024-12-20T18:19:48.619773Z","shell.execute_reply.started":"2024-12-20T18:19:48.366336Z","shell.execute_reply":"2024-12-20T18:19:48.618793Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.boxplot(x='Policy Type', y='Premium Amount', data=train_df)\nplt.title('Premium Amount by Policy Type')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:48.620732Z","iopub.execute_input":"2024-12-20T18:19:48.62108Z","iopub.status.idle":"2024-12-20T18:19:49.572475Z","shell.execute_reply.started":"2024-12-20T18:19:48.621054Z","shell.execute_reply":"2024-12-20T18:19:49.571491Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Represents three types of insurance policies: `Basic`, `Comprehensive`, and `Premium`.\n- `Basic` type has the highest average premium.\n- `Comprehensive` type follows, while `Premium` type has the lowest average premium.\n- Although `Premium` type is expected to have the highest cost, the results suggest otherwise, potentially due to differences in policy calculation or insurance-specific factors.","metadata":{}},{"cell_type":"markdown","source":"4. **Education Level**","metadata":{}},{"cell_type":"code","source":"category_means = train_df.groupby('Education Level')['Premium Amount'].mean().sort_values()\nprint(category_means)\n\ncategory_means.plot(kind='bar', title='Education Level vs Premium Amount', figsize=(8, 5))\nplt.ylabel('Average Premium Amount')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:49.573546Z","iopub.execute_input":"2024-12-20T18:19:49.573903Z","iopub.status.idle":"2024-12-20T18:19:49.897118Z","shell.execute_reply.started":"2024-12-20T18:19:49.573863Z","shell.execute_reply":"2024-12-20T18:19:49.895904Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.boxplot(x='Education Level', y='Premium Amount', data=train_df)\nplt.title('Premium Amount by Education Level')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:49.898069Z","iopub.execute_input":"2024-12-20T18:19:49.898343Z","iopub.status.idle":"2024-12-20T18:19:50.687588Z","shell.execute_reply.started":"2024-12-20T18:19:49.898319Z","shell.execute_reply":"2024-12-20T18:19:50.686538Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Categorized by education levels.\n- The results contradict the expectation that higher education levels correlate with higher premiums, suggesting that other factors may have a stronger influence on premium calculation.","metadata":{}},{"cell_type":"markdown","source":"#### Relationship with Target Variable\n- The relationship between each categorical variable and the target variable (`Premium Amount`) was analyzed by comparing average premiums across categories.\n- Results were visualized using bar charts for clarity.\n- Variables with high variance across categories may require further analysis.","metadata":{}},{"cell_type":"markdown","source":"### 3.2 Continuous Variables","metadata":{}},{"cell_type":"markdown","source":"- Continuous variables are analyzed to identify their correlation with the target variable (`Premium Amount`) and their potential importance for the model.","metadata":{}},{"cell_type":"markdown","source":"#### Key Continuous Variables","metadata":{}},{"cell_type":"markdown","source":"1. **Annual Income**","metadata":{}},{"cell_type":"code","source":"correlation = train_df[['Annual Income', 'Premium Amount']].corr()\nprint(correlation)\n\nsns.lmplot(x='Annual Income', y='Premium Amount', data=train_df, line_kws={'color': 'red'})\nplt.title('Annual Income vs Premium Amount')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:19:50.688626Z","iopub.execute_input":"2024-12-20T18:19:50.68895Z","iopub.status.idle":"2024-12-20T18:22:28.274743Z","shell.execute_reply.started":"2024-12-20T18:19:50.688925Z","shell.execute_reply":"2024-12-20T18:22:28.273623Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.boxplot(x=train_df['Annual Income'])\nplt.title(f'Boxplot of Annual Income')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:22:28.275812Z","iopub.execute_input":"2024-12-20T18:22:28.276211Z","iopub.status.idle":"2024-12-20T18:22:28.657141Z","shell.execute_reply.started":"2024-12-20T18:22:28.27618Z","shell.execute_reply":"2024-12-20T18:22:28.656129Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Represents annual income.\n- Correlation coefficient: -0.01239, indicating almost no linear relationship with Premium Amount.\n- The lmplot also shows no meaningful pattern between the two variables.","metadata":{}},{"cell_type":"markdown","source":"2. **Credit Score**","metadata":{}},{"cell_type":"code","source":"correlation = train_df[['Credit Score', 'Premium Amount']].corr()\nprint(correlation)\n\nsns.lmplot(x='Credit Score', y='Premium Amount', data=train_df, line_kws={'color': 'red'})\nplt.title('Credit Score vs Premium Amount')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:22:28.660979Z","iopub.execute_input":"2024-12-20T18:22:28.661303Z","iopub.status.idle":"2024-12-20T18:24:52.114475Z","shell.execute_reply.started":"2024-12-20T18:22:28.661276Z","shell.execute_reply":"2024-12-20T18:24:52.113541Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.boxplot(x=train_df['Credit Score'])\nplt.title(f'Boxplot of Credit Score')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:24:52.115901Z","iopub.execute_input":"2024-12-20T18:24:52.116202Z","iopub.status.idle":"2024-12-20T18:24:52.347249Z","shell.execute_reply.started":"2024-12-20T18:24:52.116177Z","shell.execute_reply":"2024-12-20T18:24:52.346254Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Represents credit score.\n- Correlation coefficient: -0.026014, indicating almost no linear relationship with Premium Amount.\n- The lmplot also shows no meaningful pattern between the two variables.","metadata":{}},{"cell_type":"markdown","source":"3. **Number of Dependents**","metadata":{}},{"cell_type":"code","source":"correlation = train_df[['Number of Dependents', 'Premium Amount']].corr()\nprint(correlation)\n\nsns.lmplot(x='Number of Dependents', y='Premium Amount', data=train_df, line_kws={'color': 'red'})\nplt.title('Number of Dependents vs Premium Amount')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:24:52.34809Z","iopub.execute_input":"2024-12-20T18:24:52.348345Z","iopub.status.idle":"2024-12-20T18:27:16.98164Z","shell.execute_reply.started":"2024-12-20T18:24:52.348323Z","shell.execute_reply":"2024-12-20T18:27:16.980598Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.boxplot(x=train_df['Number of Dependents'])\nplt.title(f'Boxplot of Number of Dependents')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:27:16.982718Z","iopub.execute_input":"2024-12-20T18:27:16.983136Z","iopub.status.idle":"2024-12-20T18:27:17.162653Z","shell.execute_reply.started":"2024-12-20T18:27:16.983098Z","shell.execute_reply":"2024-12-20T18:27:17.161273Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Represents the number of dependents.\n- Correlation coefficient: -0.000976, indicating no linear relationship with Premium Amount.\n- The lmplot also shows no meaningful pattern between the two variables.","metadata":{}},{"cell_type":"markdown","source":"#### Correlation with Target Variable\n- Correlation analysis results:\n  - `Annual Income` and `Premium Amount`: No significant correlation (correlation coefficient: -0.01239).\n  - `Credit Score` and `Premium Amount`: No significant correlation (correlation coefficient: -0.026014).\n  - `Number of Dependents` and `Premium Amount`: No significant correlation (correlation coefficient: -0.000976).\n- Scatter plots and regression lines confirm the absence of meaningful linear relationships between these variables and the target.\n- None of the continuous variables (`Annual Income`, `Credit Score`, `Number of Dependents`) show meaningful linear relationships with the target variable.\n- These features may not significantly contribute to the model's predictive performance in their current form.\n\n#### Visual Analysis\n- The scatter plots and regression lines showed no meaningful patterns or trends between `Annual Income`, `Credit Score`, and `Number of Dependents` with the target variable.\n- All three variables appear to have negligible influence on `Premium Amount`.","metadata":{}},{"cell_type":"code","source":"corr = train_df.corr(numeric_only=True)\nsns.heatmap(corr, annot=False, cmap='coolwarm')\nplt.title('Correlation Matrix')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:28:25.633742Z","iopub.execute_input":"2024-12-20T18:28:25.634237Z","iopub.status.idle":"2024-12-20T18:28:26.603881Z","shell.execute_reply.started":"2024-12-20T18:28:25.634203Z","shell.execute_reply":"2024-12-20T18:28:26.60269Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Continuous variables show limited correlations:\n- Variables like `Annual Income` and `Credit Score` have weak positive correlations with the target variable.\n- No strong correlations observed.","metadata":{}},{"cell_type":"markdown","source":"### Missing Value Count","metadata":{}},{"cell_type":"code","source":"missing_values = train_df.isnull().sum()\nmissing_percentage = (missing_values / len(train_df)) * 100\n\nmissing_data = pd.DataFrame({\n    'Missing Values': missing_values,\n    'Percentage (%)': missing_percentage\n}).sort_values(by='Percentage (%)', ascending=False)\n\nprint(missing_data[missing_data['Missing Values'] > 0])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:28:54.151483Z","iopub.execute_input":"2024-12-20T18:28:54.152058Z","iopub.status.idle":"2024-12-20T18:28:54.801484Z","shell.execute_reply.started":"2024-12-20T18:28:54.152011Z","shell.execute_reply":"2024-12-20T18:28:54.800175Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The top 4 variables with missing values and their proportions are as follows:\n- `Previous Claims`: 30.34% (364,029 missing values)\n- `Occupation`: 29.84% (358,075 missing values)\n- `Credit Score`: 11.49% (137,882 missing values)\n- `Number of Dependents`: 9.14% (109,672 missing values)","metadata":{}},{"cell_type":"markdown","source":"### Relationship between Missingness and Target Variable","metadata":{}},{"cell_type":"markdown","source":"1. **Previous Claims**","metadata":{}},{"cell_type":"code","source":"train_df['Missing'] = train_df['Previous Claims'].isnull()\nmeans = train_df.groupby('Missing')['Premium Amount'].mean()\nprint(\"Previous Claims missing status vs Premium Amount:\\n\", means)\ntrain_df.drop('Missing', axis=1, inplace=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:29:02.725636Z","iopub.execute_input":"2024-12-20T18:29:02.726184Z","iopub.status.idle":"2024-12-20T18:29:02.991425Z","shell.execute_reply.started":"2024-12-20T18:29:02.726137Z","shell.execute_reply":"2024-12-20T18:29:02.990219Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Rows without missing values have a higher average premium (1113.69).\n- Rows with missing values have a lower average premium (1076.94).\n- This suggests that customers without prior claims may have lower premiums.","metadata":{}},{"cell_type":"markdown","source":"2. **Occupation**","metadata":{}},{"cell_type":"code","source":"train_df['Missing'] = train_df['Occupation'].isnull()\nmeans = train_df.groupby('Missing')['Premium Amount'].mean()\nprint(\"Occupation missing status vs Premium Amount:\\n\", means)\ntrain_df.drop('Missing', axis=1, inplace=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:29:02.992705Z","iopub.execute_input":"2024-12-20T18:29:02.992993Z","iopub.status.idle":"2024-12-20T18:29:03.288328Z","shell.execute_reply.started":"2024-12-20T18:29:02.99297Z","shell.execute_reply":"2024-12-20T18:29:03.287282Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Rows without missing values have a higher average premium (1106.47).\n- Rows with missing values have a lower average premium (1093.32).\n- This suggests that customers without occupation information may have slightly lower premiums.","metadata":{}},{"cell_type":"markdown","source":"3. **Credit Score**","metadata":{}},{"cell_type":"code","source":"train_df['Missing'] = train_df['Credit Score'].isnull()\nmeans = train_df.groupby('Missing')['Premium Amount'].mean()\nprint(\"Credit Score missing status vs Premium Amount:\\n\", means)\ntrain_df.drop('Missing', axis=1, inplace=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:29:03.290392Z","iopub.execute_input":"2024-12-20T18:29:03.290756Z","iopub.status.idle":"2024-12-20T18:29:03.539951Z","shell.execute_reply.started":"2024-12-20T18:29:03.290717Z","shell.execute_reply":"2024-12-20T18:29:03.538791Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Rows without missing values have a higher average premium (1104.74).\n- Rows with missing values have a lower average premium (1085.62).\n- Customers with missing credit scores tend to have slightly lower premiums.","metadata":{}},{"cell_type":"markdown","source":"4. **Number of Dependents**","metadata":{}},{"cell_type":"code","source":"train_df['Missing'] = train_df['Number of Dependents'].isnull()\nmeans = train_df.groupby('Missing')['Premium Amount'].mean()\nprint(\"Number of Dependents missing status vs Premium Amount:\\n\", means)\ntrain_df.drop('Missing', axis=1, inplace=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:29:03.541354Z","iopub.execute_input":"2024-12-20T18:29:03.541704Z","iopub.status.idle":"2024-12-20T18:29:03.790137Z","shell.execute_reply.started":"2024-12-20T18:29:03.541676Z","shell.execute_reply":"2024-12-20T18:29:03.788945Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Analysis shows that rows with missing `Number of Dependents` have a higher average premium (**1126.44**) compared to non-missing rows (**1100.14**).\n- This is an unusual pattern, as missingness is typically assumed to be random but may indicate a specific customer group in this case.\n- **Potential Link to Specific Customer Groups**:\n   - Customers without dependent information may belong to a group with higher premiums (e.g., higher-income customers or those enrolled in specific insurance plans).\n   - These customers could be premium customers who are willing to pay higher premiums.\n- **Suggestions for Modeling**:\n   - Instead of simply dropping missing values, create a binary derived feature (`is_dependents_missing`) to indicate missingness and include it in the model.\n   - Since missingness correlates directly with premium amounts, ignoring it may result in loss of valuable information.","metadata":{}},{"cell_type":"markdown","source":"### Conclusion\n- Missingness in `Previous Claims`, `Occupation`, `Credit Score`, and `Number of Dependents` could be important features for predicting premiums:\n  - `Previous Claims` and `Occupation`: Rows with missing values have lower average premiums.\n  - `Credit Score`: Rows with missing values also have lower average premiums.\n  - `Number of Dependents`: Rows with missing values have higher average premiums, showing an unusual pattern.\n- Consider generating a derived feature to indicate missingness or treating missing values as a separate category.\n- The unusual pattern observed with `Number of Dependents` missingness and higher premiums warrants further investigation.","metadata":{}},{"cell_type":"markdown","source":"## 5. Time-Based Analysis","metadata":{}},{"cell_type":"markdown","source":"### Policy Start Date\n- `Policy Start Date` represents the start date of the insurance policy.\n- It was analyzed on a yearly and monthly basis to explore its relationship with `Premium Amount`.","metadata":{}},{"cell_type":"markdown","source":"#### Yearly Trends","metadata":{}},{"cell_type":"code","source":"train_df['Policy Year'] = pd.to_datetime(train_df['Policy Start Date']).dt.year\nyearly_means = train_df.groupby('Policy Year')['Premium Amount'].mean()\nprint(yearly_means)\n\nyearly_means.plot(kind='bar', title='Yearly Average Premium Amount', figsize=(8, 5))\nsns.lineplot(x='Policy Year', y='Premium Amount', data=yearly_means.reset_index())\nplt.title('Year Average Premium Amount')\nplt.xlabel('Year')\nplt.ylabel('Average Premium Amount')\nplt.show()\n\ntrain_df.drop('Policy Year', axis=1, inplace=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:29:03.791298Z","iopub.execute_input":"2024-12-20T18:29:03.79159Z","iopub.status.idle":"2024-12-20T18:29:04.712955Z","shell.execute_reply.started":"2024-12-20T18:29:03.791564Z","shell.execute_reply":"2024-12-20T18:29:04.711875Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Analysis of yearly average premiums revealed:\n  - The average premium in 2019 (1189.56) is significantly higher than in other years.\n  - From 2020 to 2023, average premiums remained relatively stable between 1091.73 and 1096.65.\n  - In 2024, the average premium increased slightly to 1108.88.\n- **Conclusion:**\n  - 2019 and 2024 show some anomalies, suggesting potential external factors influencing premiums during these years.\n  - While year itself is generally stable, specific years could be considered as derived features.","metadata":{}},{"cell_type":"markdown","source":"#### Monthly Trends","metadata":{}},{"cell_type":"code","source":"train_df['Policy Month'] = pd.to_datetime(train_df['Policy Start Date']).dt.month\nmonthly_means = train_df.groupby('Policy Month')['Premium Amount'].mean()\nprint(monthly_means)\n\nmonthly_means.plot(kind='bar', title='Monthly Average Premium Amount', figsize=(8, 5))\n\nsns.lineplot(x='Policy Month', y='Premium Amount', data=monthly_means.reset_index())\nplt.title('Monthly Average Premium Amount')\nplt.xlabel('Month')\nplt.ylabel('Average Premium Amount')\nplt.show()\n\ntrain_df.drop('Policy Month', axis=1, inplace=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-20T18:29:04.714001Z","iopub.execute_input":"2024-12-20T18:29:04.714316Z","iopub.status.idle":"2024-12-20T18:29:05.673423Z","shell.execute_reply.started":"2024-12-20T18:29:04.714278Z","shell.execute_reply":"2024-12-20T18:29:05.672341Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Analysis of monthly average premiums revealed:\n  - January to July: Relatively stable premiums, ranging between **1093 and 1100**.\n  - August to December: Premiums increase, peaking in **September to November** with the highest values.\n- The rise in average premiums during September to November may reflect seasonal factors or promotional campaigns.\n- **Conclusion:** Monthly trends show a noticeable impact on premiums, suggesting that month could be used as a derived feature.","metadata":{}},{"cell_type":"markdown","source":"## 6. Summary of Insights","metadata":{}},{"cell_type":"markdown","source":"### Key Findings from the Analysis\n1. **Target Variable Analysis**:\n   - `Premium Amount` exhibits a right-skewed distribution, improved by log transformation to resemble a normal distribution.\n\n2. **Categorical Variable Analysis**:\n   - `Marital Status` and `Policy Type` show significant differences in average premiums.\n   - Contrary to expectations, `High School` graduates have the highest average premiums among education levels.\n\n3. **Continuous Variable Analysis**:\n   - `Annual Income`, `Credit Score`, and `Number of Dependents` show little to no correlation with the target variable.\n\n4. **Missing Value Analysis**:\n   - Missingness in `Previous Claims` and `Occupation` could be important features for predicting premiums.\n   - Missing `Number of Dependents` correlates with higher premiums, indicating an unusual pattern.\n\n5. **Time-Based Analysis**:\n   - Yearly trends are mostly stable, with minor anomalies in 2019 and 2024.\n   - Monthly trends show increased premiums in September to November.\n\n### Implications for Preprocessing and Feature Engineering\n1. **Log Transformation**:\n   - Apply log transformation to `Premium Amount` to improve model stability and performance.\n\n2. **Missing Value Handling**:\n   - Generate derived features to indicate missingness in `Previous Claims`, `Occupation`, and `Number of Dependents`.\n   - Alternatively, impute missing values with mean/median or treat as a new category.\n\n3. **Categorical Variable Utilization**:\n   - Include `Marital Status`, `Policy Type`, and `Education Level` as significant features.\n\n4. **Time-Based Features**:\n   - Create derived features such as `is_peak_season` for September to November.\n   - Convert year into a categorical variable to capture anomalies in specific years.\n\n5. **Nonlinear Relationships**:\n   - Explore nonlinear relationships in continuous variables or interactions between variables for additional insights.\n","metadata":{}},{"cell_type":"markdown","source":"## 7. Next Steps","metadata":{}},{"cell_type":"markdown","source":"### Plan for Preprocessing\n1. **Missing Value Handling**:\n   - Create derived features to indicate missingness for `Previous Claims`, `Occupation`, and `Number of Dependents`.\n   - Impute missing values in low-missing variables like `Credit Score` with mean/median.\n\n2. **Log Transformation**:\n   - Apply log transformation to `Premium Amount` to normalize the distribution and improve model performance.\n\n3. **Categorical Variable Encoding**:\n   - Convert categorical variables like `Marital Status`, `Policy Type`, and `Education Level` into one-hot encoded features.\n\n4. **Time-Based Feature Creation**:\n   - Generate derived features such as `is_peak_season` for September to November.\n   - Convert `Policy Year` into a categorical variable to capture anomalies in specific years.\n\n5. **Scaling**:\n   - Standardize or normalize continuous variables like `Annual Income` and `Credit Score`.","metadata":{}}]}