{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Inferences:\n\n1. Since uniform distribution, we can impute NaN places of Age with MEAN/MEDIAN or Education level can be used.\n2. Leave the Gender column as it is. No NaN, uniform.\n3. Annual Income is Right-Skewed. But imputing NaN values will require more description and thinking.\n4. Marital Status have similar number of counts. Thus, more information is required to fill its NaN values.\n5. Similar number of counts in Number of Dependents. Thus, require more information for imputing NaN values\n6. Nearly same number of counts in category of Education level. It will be also helpful for deciding the imputing. Also, no NaN in Education level.\n7. Nearly same amount of category counts in Occupation. Will use other columns to decide what to impute.\n8. Fill NaN values with median or according to Education Level in Health Score.\n9. Keep the Location column as it is. No need to change anything. No NaN values.\n10. Keep the Policy Type column as it is. No need to change anything. No NaN values.\n\n11. Its remaining to decide what to impute for Previous claims. I will research more.\n\n12. Simply fill NaN values of Vehicle Age with 10.0 (median).\n13. Use Education level column for imputing the Credit Score NaN values.\n14. Fill one NaN value of Insurance Duration with median (5).\n15. We can drop 'Policy Start Date' column.\n16. Fill NaN of Customer Feedback by looking at the other columns like Education Level etc.\n17. No need to touch Smoking Status column.\n18. No need to touch Exercise Frequency column.\n19. No need to touch Property Type column.\n\n\n**Explanation of this is given in the code step by step**\n1. Fill NaN of Age with 41 in 1st iter, and its respective categorical models in 2nd iter.\n2. Fill NaN of Annual Income with 23911.0 (median).\n3. Fill NaN Marital Status NaN with 'Unknown'. (Since 18k NaN).\n4. Fill -1 in NaN of Number of Dependents.\n5. Fill NaN of Occupation with 'Unknown'.\n6. Fill NaN of Health Score with median (25).\n7. Fill NaN of Previous Claims with -1 (Unknown category since NaN is huge), drop in 2nd iter.\n8. Fill NaN values of Vehicle Age with 10.0 (median).\n9. Fill NaN values of Credit Score with 595.0 (Median).\n10. Fill NaN values of Insurance Duration with 5 (median).\n11. Make year column and Drop Policy Start Date Column.\n12. Fill NaN values of Customer Feedback column with 'Unknown'","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nfrom sklearn.preprocessing import LabelEncoder, StandardScaler\nfrom sklearn.model_selection import train_test_split\n\nfrom sklearn.linear_model import LinearRegression, Ridge, Lasso\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\nfrom sklearn.svm import SVR\nfrom sklearn.neighbors import KNeighborsRegressor\nfrom sklearn.metrics import mean_squared_log_error\n\nfrom xgboost import XGBRegressor\nfrom lightgbm import LGBMRegressor\nfrom catboost import CatBoostRegressor","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-12-01T09:01:44.161649Z","iopub.execute_input":"2024-12-01T09:01:44.162111Z","iopub.status.idle":"2024-12-01T09:01:49.570435Z","shell.execute_reply.started":"2024-12-01T09:01:44.162059Z","shell.execute_reply":"2024-12-01T09:01:49.569304Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Reading the data","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv('/kaggle/input/playground-series-s4e12/train.csv', index_col='id')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:12:22.808169Z","iopub.execute_input":"2024-12-01T09:12:22.808595Z","iopub.status.idle":"2024-12-01T09:12:28.12052Z","shell.execute_reply.started":"2024-12-01T09:12:22.80856Z","shell.execute_reply":"2024-12-01T09:12:28.119062Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:12:28.123164Z","iopub.execute_input":"2024-12-01T09:12:28.123652Z","iopub.status.idle":"2024-12-01T09:12:28.15731Z","shell.execute_reply.started":"2024-12-01T09:12:28.1236Z","shell.execute_reply":"2024-12-01T09:12:28.156069Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:01:56.862622Z","iopub.execute_input":"2024-12-01T09:01:56.86296Z","iopub.status.idle":"2024-12-01T09:01:57.552648Z","shell.execute_reply.started":"2024-12-01T09:01:56.862929Z","shell.execute_reply":"2024-12-01T09:01:57.551592Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"NaN in % column wise :\")\nfor col in data.columns:\n    print(f\"{col} ====> {round((data[col].isna().sum()/len(data))*100, 5)} %\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:01:57.554108Z","iopub.execute_input":"2024-12-01T09:01:57.554527Z","iopub.status.idle":"2024-12-01T09:01:58.17251Z","shell.execute_reply.started":"2024-12-01T09:01:57.554493Z","shell.execute_reply":"2024-12-01T09:01:58.171417Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:01:58.174024Z","iopub.execute_input":"2024-12-01T09:01:58.17434Z","iopub.status.idle":"2024-12-01T09:01:58.832141Z","shell.execute_reply.started":"2024-12-01T09:01:58.17431Z","shell.execute_reply":"2024-12-01T09:01:58.830844Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"len(data)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:01:58.833575Z","iopub.execute_input":"2024-12-01T09:01:58.83405Z","iopub.status.idle":"2024-12-01T09:01:58.841946Z","shell.execute_reply.started":"2024-12-01T09:01:58.833979Z","shell.execute_reply":"2024-12-01T09:01:58.840818Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Exploratory Data Analysis - Univariate","metadata":{}},{"cell_type":"markdown","source":"## Age Column","metadata":{}},{"cell_type":"code","source":"data['Age'].value_counts(ascending=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:01:58.843214Z","iopub.execute_input":"2024-12-01T09:01:58.843507Z","iopub.status.idle":"2024-12-01T09:01:58.880905Z","shell.execute_reply.started":"2024-12-01T09:01:58.843479Z","shell.execute_reply":"2024-12-01T09:01:58.879824Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"min(data['Age']), max(data['Age'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:01:58.8824Z","iopub.execute_input":"2024-12-01T09:01:58.883173Z","iopub.status.idle":"2024-12-01T09:01:59.114061Z","shell.execute_reply.started":"2024-12-01T09:01:58.883121Z","shell.execute_reply":"2024-12-01T09:01:59.112738Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.histplot(data=data, x='Age', kde=True, bins=20);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:01:59.117972Z","iopub.execute_input":"2024-12-01T09:01:59.118432Z","iopub.status.idle":"2024-12-01T09:02:04.082933Z","shell.execute_reply.started":"2024-12-01T09:01:59.118397Z","shell.execute_reply":"2024-12-01T09:02:04.081663Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.boxplot(data=data, x='Age');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:04.08442Z","iopub.execute_input":"2024-12-01T09:02:04.084871Z","iopub.status.idle":"2024-12-01T09:02:04.323168Z","shell.execute_reply.started":"2024-12-01T09:02:04.084817Z","shell.execute_reply":"2024-12-01T09:02:04.321941Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Fill NaN values of Age column with Mean of age\nnp.median(data['Age'].dropna())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:04.324634Z","iopub.execute_input":"2024-12-01T09:02:04.325093Z","iopub.status.idle":"2024-12-01T09:02:04.357666Z","shell.execute_reply.started":"2024-12-01T09:02:04.32505Z","shell.execute_reply":"2024-12-01T09:02:04.356502Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Gender Column","metadata":{}},{"cell_type":"code","source":"data['Gender'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:04.358881Z","iopub.execute_input":"2024-12-01T09:02:04.359232Z","iopub.status.idle":"2024-12-01T09:02:04.462586Z","shell.execute_reply.started":"2024-12-01T09:02:04.359201Z","shell.execute_reply":"2024-12-01T09:02:04.460918Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Gender');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:04.464296Z","iopub.execute_input":"2024-12-01T09:02:04.464796Z","iopub.status.idle":"2024-12-01T09:02:05.313889Z","shell.execute_reply.started":"2024-12-01T09:02:04.464747Z","shell.execute_reply":"2024-12-01T09:02:05.312442Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Nearly same amount of females and males.","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:05.315616Z","iopub.execute_input":"2024-12-01T09:02:05.316106Z","iopub.status.idle":"2024-12-01T09:02:05.322443Z","shell.execute_reply.started":"2024-12-01T09:02:05.316055Z","shell.execute_reply":"2024-12-01T09:02:05.320691Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Gender'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:05.32378Z","iopub.execute_input":"2024-12-01T09:02:05.324256Z","iopub.status.idle":"2024-12-01T09:02:05.395157Z","shell.execute_reply.started":"2024-12-01T09:02:05.324212Z","shell.execute_reply":"2024-12-01T09:02:05.393915Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# No NaN values in Gender.","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:05.397096Z","iopub.execute_input":"2024-12-01T09:02:05.39761Z","iopub.status.idle":"2024-12-01T09:02:05.406093Z","shell.execute_reply.started":"2024-12-01T09:02:05.397571Z","shell.execute_reply":"2024-12-01T09:02:05.404253Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Annual Income column","metadata":{}},{"cell_type":"code","source":"max(data['Annual Income'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:05.407671Z","iopub.execute_input":"2024-12-01T09:02:05.408125Z","iopub.status.idle":"2024-12-01T09:02:05.531693Z","shell.execute_reply.started":"2024-12-01T09:02:05.408089Z","shell.execute_reply":"2024-12-01T09:02:05.530372Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.histplot(data=data, x='Annual Income', kde=True, bins=100)\nplt.xlim(0, 150000);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:05.533399Z","iopub.execute_input":"2024-12-01T09:02:05.534026Z","iopub.status.idle":"2024-12-01T09:02:10.879538Z","shell.execute_reply.started":"2024-12-01T09:02:05.533954Z","shell.execute_reply":"2024-12-01T09:02:10.878288Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.boxplot(data=data, x='Annual Income');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:10.881247Z","iopub.execute_input":"2024-12-01T09:02:10.881601Z","iopub.status.idle":"2024-12-01T09:02:11.415227Z","shell.execute_reply.started":"2024-12-01T09:02:10.881568Z","shell.execute_reply":"2024-12-01T09:02:11.413883Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"np.median(data['Annual Income'].dropna())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:11.417043Z","iopub.execute_input":"2024-12-01T09:02:11.41746Z","iopub.status.idle":"2024-12-01T09:02:11.450618Z","shell.execute_reply.started":"2024-12-01T09:02:11.417425Z","shell.execute_reply":"2024-12-01T09:02:11.449334Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Marital Status column","metadata":{}},{"cell_type":"code","source":"data['Marital Status'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:11.452356Z","iopub.execute_input":"2024-12-01T09:02:11.452867Z","iopub.status.idle":"2024-12-01T09:02:11.55785Z","shell.execute_reply.started":"2024-12-01T09:02:11.452817Z","shell.execute_reply":"2024-12-01T09:02:11.556258Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Marital Status');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:11.559522Z","iopub.execute_input":"2024-12-01T09:02:11.560028Z","iopub.status.idle":"2024-12-01T09:02:12.409304Z","shell.execute_reply.started":"2024-12-01T09:02:11.559956Z","shell.execute_reply":"2024-12-01T09:02:12.408074Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# All values have nearly same/similar count.","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:12.410968Z","iopub.execute_input":"2024-12-01T09:02:12.411322Z","iopub.status.idle":"2024-12-01T09:02:12.418595Z","shell.execute_reply.started":"2024-12-01T09:02:12.41129Z","shell.execute_reply":"2024-12-01T09:02:12.417338Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Marital Status'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:12.420099Z","iopub.execute_input":"2024-12-01T09:02:12.420452Z","iopub.status.idle":"2024-12-01T09:02:12.489069Z","shell.execute_reply.started":"2024-12-01T09:02:12.42042Z","shell.execute_reply":"2024-12-01T09:02:12.487836Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Number of Dependents column","metadata":{}},{"cell_type":"code","source":"data['Number of Dependents'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:12.49073Z","iopub.execute_input":"2024-12-01T09:02:12.491106Z","iopub.status.idle":"2024-12-01T09:02:12.521048Z","shell.execute_reply.started":"2024-12-01T09:02:12.491072Z","shell.execute_reply":"2024-12-01T09:02:12.519773Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Number of Dependents');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:12.522464Z","iopub.execute_input":"2024-12-01T09:02:12.522877Z","iopub.status.idle":"2024-12-01T09:02:12.843158Z","shell.execute_reply.started":"2024-12-01T09:02:12.522832Z","shell.execute_reply":"2024-12-01T09:02:12.841933Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Number of Dependents'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:12.849734Z","iopub.execute_input":"2024-12-01T09:02:12.850151Z","iopub.status.idle":"2024-12-01T09:02:12.858691Z","shell.execute_reply.started":"2024-12-01T09:02:12.850116Z","shell.execute_reply":"2024-12-01T09:02:12.857511Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Similar number of counts in Number of Dependents. \n# Thus, require more information for imputing NaN values","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:12.85993Z","iopub.execute_input":"2024-12-01T09:02:12.860276Z","iopub.status.idle":"2024-12-01T09:02:12.870347Z","shell.execute_reply.started":"2024-12-01T09:02:12.860246Z","shell.execute_reply":"2024-12-01T09:02:12.869181Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Education Level column","metadata":{}},{"cell_type":"code","source":"data['Education Level'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:12.871833Z","iopub.execute_input":"2024-12-01T09:02:12.872355Z","iopub.status.idle":"2024-12-01T09:02:12.981172Z","shell.execute_reply.started":"2024-12-01T09:02:12.872301Z","shell.execute_reply":"2024-12-01T09:02:12.979906Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Education Level')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:12.98262Z","iopub.execute_input":"2024-12-01T09:02:12.983024Z","iopub.status.idle":"2024-12-01T09:02:13.813547Z","shell.execute_reply.started":"2024-12-01T09:02:12.982962Z","shell.execute_reply":"2024-12-01T09:02:13.812087Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Education Level'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:13.81538Z","iopub.execute_input":"2024-12-01T09:02:13.816054Z","iopub.status.idle":"2024-12-01T09:02:13.879322Z","shell.execute_reply.started":"2024-12-01T09:02:13.815966Z","shell.execute_reply":"2024-12-01T09:02:13.877966Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Nearly same number of counts in category of Education level.\n# It will be also helpful for deciding the imputing","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:13.880645Z","iopub.execute_input":"2024-12-01T09:02:13.881147Z","iopub.status.idle":"2024-12-01T09:02:13.893057Z","shell.execute_reply.started":"2024-12-01T09:02:13.881092Z","shell.execute_reply":"2024-12-01T09:02:13.891857Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Occupation column","metadata":{}},{"cell_type":"code","source":"data['Occupation'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:13.894409Z","iopub.execute_input":"2024-12-01T09:02:13.894767Z","iopub.status.idle":"2024-12-01T09:02:13.987186Z","shell.execute_reply.started":"2024-12-01T09:02:13.894728Z","shell.execute_reply":"2024-12-01T09:02:13.986097Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Occupation');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:13.988662Z","iopub.execute_input":"2024-12-01T09:02:13.989161Z","iopub.status.idle":"2024-12-01T09:02:14.86587Z","shell.execute_reply.started":"2024-12-01T09:02:13.989111Z","shell.execute_reply":"2024-12-01T09:02:14.864647Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Occupation'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:14.867185Z","iopub.execute_input":"2024-12-01T09:02:14.867506Z","iopub.status.idle":"2024-12-01T09:02:14.93109Z","shell.execute_reply.started":"2024-12-01T09:02:14.867475Z","shell.execute_reply":"2024-12-01T09:02:14.929686Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Nearly same amount of category counts in Occupation. Will use other columns to decide what to impute.","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:14.932375Z","iopub.execute_input":"2024-12-01T09:02:14.932691Z","iopub.status.idle":"2024-12-01T09:02:14.943765Z","shell.execute_reply.started":"2024-12-01T09:02:14.932663Z","shell.execute_reply":"2024-12-01T09:02:14.942442Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# # To try : \n# 1. Age imputing with Education Level\n# 2. Annual Income imputing with Education level\n# 3. Marital Status imputing with Education level\n# 4. Occupation column imuting with Education level","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:14.945736Z","iopub.execute_input":"2024-12-01T09:02:14.946141Z","iopub.status.idle":"2024-12-01T09:02:14.956868Z","shell.execute_reply.started":"2024-12-01T09:02:14.946108Z","shell.execute_reply":"2024-12-01T09:02:14.955388Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Health Score column","metadata":{}},{"cell_type":"code","source":"min(data['Health Score']), max(data['Health Score'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:14.958931Z","iopub.execute_input":"2024-12-01T09:02:14.95931Z","iopub.status.idle":"2024-12-01T09:02:15.187585Z","shell.execute_reply.started":"2024-12-01T09:02:14.959276Z","shell.execute_reply":"2024-12-01T09:02:15.186142Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.histplot(data=data, x='Health Score', bins=100, kde=True)\nplt.xlim(2, 60);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:15.18906Z","iopub.execute_input":"2024-12-01T09:02:15.189391Z","iopub.status.idle":"2024-12-01T09:02:20.305372Z","shell.execute_reply.started":"2024-12-01T09:02:15.189358Z","shell.execute_reply":"2024-12-01T09:02:20.304103Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.boxplot(data=data, x='Health Score');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:20.306857Z","iopub.execute_input":"2024-12-01T09:02:20.307221Z","iopub.status.idle":"2024-12-01T09:02:20.564291Z","shell.execute_reply.started":"2024-12-01T09:02:20.307187Z","shell.execute_reply":"2024-12-01T09:02:20.562944Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Health Score'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:20.56561Z","iopub.execute_input":"2024-12-01T09:02:20.566106Z","iopub.status.idle":"2024-12-01T09:02:20.575387Z","shell.execute_reply.started":"2024-12-01T09:02:20.566049Z","shell.execute_reply":"2024-12-01T09:02:20.574229Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"np.median(data['Health Score'].dropna()), np.mean(data['Health Score'].dropna())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:20.576941Z","iopub.execute_input":"2024-12-01T09:02:20.577286Z","iopub.status.idle":"2024-12-01T09:02:20.653022Z","shell.execute_reply.started":"2024-12-01T09:02:20.577255Z","shell.execute_reply":"2024-12-01T09:02:20.651904Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Fill NaN values with median or according to Education Level","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:20.654258Z","iopub.execute_input":"2024-12-01T09:02:20.654585Z","iopub.status.idle":"2024-12-01T09:02:20.66013Z","shell.execute_reply.started":"2024-12-01T09:02:20.654554Z","shell.execute_reply":"2024-12-01T09:02:20.65889Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Location column","metadata":{}},{"cell_type":"code","source":"data['Location'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:20.661478Z","iopub.execute_input":"2024-12-01T09:02:20.661874Z","iopub.status.idle":"2024-12-01T09:02:20.772646Z","shell.execute_reply.started":"2024-12-01T09:02:20.661809Z","shell.execute_reply":"2024-12-01T09:02:20.771152Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Location');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:20.774246Z","iopub.execute_input":"2024-12-01T09:02:20.774612Z","iopub.status.idle":"2024-12-01T09:02:21.689752Z","shell.execute_reply.started":"2024-12-01T09:02:20.774574Z","shell.execute_reply":"2024-12-01T09:02:21.688673Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Location'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:21.690866Z","iopub.execute_input":"2024-12-01T09:02:21.691202Z","iopub.status.idle":"2024-12-01T09:02:21.753203Z","shell.execute_reply.started":"2024-12-01T09:02:21.69117Z","shell.execute_reply":"2024-12-01T09:02:21.751891Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Policy Type column","metadata":{"execution":{"iopub.status.busy":"2024-12-01T04:25:01.566901Z","iopub.execute_input":"2024-12-01T04:25:01.567299Z","iopub.status.idle":"2024-12-01T04:25:01.574887Z","shell.execute_reply.started":"2024-12-01T04:25:01.567258Z","shell.execute_reply":"2024-12-01T04:25:01.573473Z"}}},{"cell_type":"code","source":"data['Policy Type'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:21.754638Z","iopub.execute_input":"2024-12-01T09:02:21.754982Z","iopub.status.idle":"2024-12-01T09:02:21.86231Z","shell.execute_reply.started":"2024-12-01T09:02:21.754952Z","shell.execute_reply":"2024-12-01T09:02:21.861058Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Policy Type');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:21.863853Z","iopub.execute_input":"2024-12-01T09:02:21.86489Z","iopub.status.idle":"2024-12-01T09:02:22.746127Z","shell.execute_reply.started":"2024-12-01T09:02:21.864838Z","shell.execute_reply":"2024-12-01T09:02:22.745051Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Policy Type'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:22.747385Z","iopub.execute_input":"2024-12-01T09:02:22.747758Z","iopub.status.idle":"2024-12-01T09:02:22.808964Z","shell.execute_reply.started":"2024-12-01T09:02:22.747726Z","shell.execute_reply":"2024-12-01T09:02:22.807717Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Previous Claims column","metadata":{"execution":{"iopub.status.busy":"2024-12-01T04:26:37.657034Z","iopub.execute_input":"2024-12-01T04:26:37.657742Z","iopub.status.idle":"2024-12-01T04:26:37.668448Z","shell.execute_reply.started":"2024-12-01T04:26:37.657688Z","shell.execute_reply":"2024-12-01T04:26:37.666976Z"}}},{"cell_type":"code","source":"data['Previous Claims'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:22.810397Z","iopub.execute_input":"2024-12-01T09:02:22.810712Z","iopub.status.idle":"2024-12-01T09:02:22.840819Z","shell.execute_reply.started":"2024-12-01T09:02:22.810681Z","shell.execute_reply":"2024-12-01T09:02:22.839663Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, y='Previous Claims');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:22.84222Z","iopub.execute_input":"2024-12-01T09:02:22.842635Z","iopub.status.idle":"2024-12-01T09:02:23.208181Z","shell.execute_reply.started":"2024-12-01T09:02:22.842588Z","shell.execute_reply":"2024-12-01T09:02:23.207157Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Previous Claims'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:23.210069Z","iopub.execute_input":"2024-12-01T09:02:23.210393Z","iopub.status.idle":"2024-12-01T09:02:23.219139Z","shell.execute_reply.started":"2024-12-01T09:02:23.210363Z","shell.execute_reply":"2024-12-01T09:02:23.218054Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Its remaining to decide what to impute for Previous claims. I will research more.\n# I will use Education Level column","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:23.220745Z","iopub.execute_input":"2024-12-01T09:02:23.221149Z","iopub.status.idle":"2024-12-01T09:02:23.228884Z","shell.execute_reply.started":"2024-12-01T09:02:23.221115Z","shell.execute_reply":"2024-12-01T09:02:23.227668Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Vehicle Age column","metadata":{"execution":{"iopub.status.busy":"2024-12-01T04:32:37.30275Z","iopub.execute_input":"2024-12-01T04:32:37.303122Z","iopub.status.idle":"2024-12-01T04:32:37.310486Z","shell.execute_reply.started":"2024-12-01T04:32:37.30309Z","shell.execute_reply":"2024-12-01T04:32:37.309186Z"}}},{"cell_type":"code","source":"data['Vehicle Age'].value_counts(ascending=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:23.230863Z","iopub.execute_input":"2024-12-01T09:02:23.231234Z","iopub.status.idle":"2024-12-01T09:02:23.260142Z","shell.execute_reply.started":"2024-12-01T09:02:23.2312Z","shell.execute_reply":"2024-12-01T09:02:23.258902Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(10, 4))\nsns.histplot(data=data, x='Vehicle Age', bins=20, kde=True)\nplt.xlim(0, 20);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:23.261925Z","iopub.execute_input":"2024-12-01T09:02:23.262334Z","iopub.status.idle":"2024-12-01T09:02:28.333714Z","shell.execute_reply.started":"2024-12-01T09:02:23.262302Z","shell.execute_reply":"2024-12-01T09:02:28.332388Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Vehicle Age'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:28.335134Z","iopub.execute_input":"2024-12-01T09:02:28.335464Z","iopub.status.idle":"2024-12-01T09:02:28.345339Z","shell.execute_reply.started":"2024-12-01T09:02:28.335429Z","shell.execute_reply":"2024-12-01T09:02:28.344184Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"np.median(data['Vehicle Age'].dropna())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:28.346722Z","iopub.execute_input":"2024-12-01T09:02:28.347178Z","iopub.status.idle":"2024-12-01T09:02:28.395286Z","shell.execute_reply.started":"2024-12-01T09:02:28.347142Z","shell.execute_reply":"2024-12-01T09:02:28.394079Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Simply fill NaN values of Vehicle Age with 10.0 (median).","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:28.396541Z","iopub.execute_input":"2024-12-01T09:02:28.396884Z","iopub.status.idle":"2024-12-01T09:02:28.403024Z","shell.execute_reply.started":"2024-12-01T09:02:28.396854Z","shell.execute_reply":"2024-12-01T09:02:28.401813Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Credit Score column","metadata":{"execution":{"iopub.status.busy":"2024-12-01T04:37:18.599961Z","iopub.execute_input":"2024-12-01T04:37:18.600357Z","iopub.status.idle":"2024-12-01T04:37:18.607456Z","shell.execute_reply.started":"2024-12-01T04:37:18.600323Z","shell.execute_reply":"2024-12-01T04:37:18.606145Z"}}},{"cell_type":"code","source":"plt.figure(figsize=(10, 4))\nsns.histplot(data=data, x='Credit Score', kde=True, bins=20);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:28.404491Z","iopub.execute_input":"2024-12-01T09:02:28.404969Z","iopub.status.idle":"2024-12-01T09:02:32.906442Z","shell.execute_reply.started":"2024-12-01T09:02:28.404908Z","shell.execute_reply":"2024-12-01T09:02:32.905341Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Credit Score'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:32.908081Z","iopub.execute_input":"2024-12-01T09:02:32.908543Z","iopub.status.idle":"2024-12-01T09:02:32.919246Z","shell.execute_reply.started":"2024-12-01T09:02:32.908495Z","shell.execute_reply":"2024-12-01T09:02:32.918085Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Use Education level column for imputing the Credit Score NaN values.","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:32.92075Z","iopub.execute_input":"2024-12-01T09:02:32.921257Z","iopub.status.idle":"2024-12-01T09:02:32.931451Z","shell.execute_reply.started":"2024-12-01T09:02:32.921209Z","shell.execute_reply":"2024-12-01T09:02:32.930308Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Insurance Duration column","metadata":{}},{"cell_type":"code","source":"data['Insurance Duration'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:32.933283Z","iopub.execute_input":"2024-12-01T09:02:32.933699Z","iopub.status.idle":"2024-12-01T09:02:32.965492Z","shell.execute_reply.started":"2024-12-01T09:02:32.933663Z","shell.execute_reply":"2024-12-01T09:02:32.964233Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Insurance Duration');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:32.966882Z","iopub.execute_input":"2024-12-01T09:02:32.967248Z","iopub.status.idle":"2024-12-01T09:02:33.333916Z","shell.execute_reply.started":"2024-12-01T09:02:32.967216Z","shell.execute_reply":"2024-12-01T09:02:33.332721Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Insurance Duration'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:33.335552Z","iopub.execute_input":"2024-12-01T09:02:33.335916Z","iopub.status.idle":"2024-12-01T09:02:33.345972Z","shell.execute_reply.started":"2024-12-01T09:02:33.335883Z","shell.execute_reply":"2024-12-01T09:02:33.344808Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Fill one NaN value of Insurance Duration with median (5).","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:33.347551Z","iopub.execute_input":"2024-12-01T09:02:33.348353Z","iopub.status.idle":"2024-12-01T09:02:33.35427Z","shell.execute_reply.started":"2024-12-01T09:02:33.348306Z","shell.execute_reply":"2024-12-01T09:02:33.352684Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Policy Start Date column","metadata":{"execution":{"iopub.status.busy":"2024-12-01T04:45:30.619003Z","iopub.execute_input":"2024-12-01T04:45:30.619377Z","iopub.status.idle":"2024-12-01T04:45:30.626672Z","shell.execute_reply.started":"2024-12-01T04:45:30.619343Z","shell.execute_reply":"2024-12-01T04:45:30.625474Z"}}},{"cell_type":"code","source":"data['Policy Start Date'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:33.355387Z","iopub.execute_input":"2024-12-01T09:02:33.355905Z","iopub.status.idle":"2024-12-01T09:02:33.975876Z","shell.execute_reply.started":"2024-12-01T09:02:33.355858Z","shell.execute_reply":"2024-12-01T09:02:33.974708Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Policy Start Date'] = pd.to_datetime(data['Policy Start Date'])\ndata['Year'] = data['Policy Start Date'].dt.year\ndata.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:33.977611Z","iopub.execute_input":"2024-12-01T09:02:33.978121Z","iopub.status.idle":"2024-12-01T09:02:34.441365Z","shell.execute_reply.started":"2024-12-01T09:02:33.978071Z","shell.execute_reply":"2024-12-01T09:02:34.440168Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Year'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:34.442736Z","iopub.execute_input":"2024-12-01T09:02:34.443195Z","iopub.status.idle":"2024-12-01T09:02:34.463951Z","shell.execute_reply.started":"2024-12-01T09:02:34.443148Z","shell.execute_reply":"2024-12-01T09:02:34.462651Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Year');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:34.465603Z","iopub.execute_input":"2024-12-01T09:02:34.465957Z","iopub.status.idle":"2024-12-01T09:02:34.770665Z","shell.execute_reply.started":"2024-12-01T09:02:34.465924Z","shell.execute_reply":"2024-12-01T09:02:34.769541Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Policy Start Date'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:34.77192Z","iopub.execute_input":"2024-12-01T09:02:34.77227Z","iopub.status.idle":"2024-12-01T09:02:34.780887Z","shell.execute_reply.started":"2024-12-01T09:02:34.772237Z","shell.execute_reply":"2024-12-01T09:02:34.779788Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Year'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:34.782217Z","iopub.execute_input":"2024-12-01T09:02:34.782547Z","iopub.status.idle":"2024-12-01T09:02:34.796293Z","shell.execute_reply.started":"2024-12-01T09:02:34.782517Z","shell.execute_reply":"2024-12-01T09:02:34.795017Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# We can drop 'Policy Start Date' column","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:34.797733Z","iopub.execute_input":"2024-12-01T09:02:34.798135Z","iopub.status.idle":"2024-12-01T09:02:34.804658Z","shell.execute_reply.started":"2024-12-01T09:02:34.798091Z","shell.execute_reply":"2024-12-01T09:02:34.803508Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Customer Feedback column","metadata":{"execution":{"iopub.status.busy":"2024-12-01T05:02:35.980828Z","iopub.execute_input":"2024-12-01T05:02:35.981239Z","iopub.status.idle":"2024-12-01T05:02:35.98978Z","shell.execute_reply.started":"2024-12-01T05:02:35.9812Z","shell.execute_reply":"2024-12-01T05:02:35.988321Z"}}},{"cell_type":"code","source":"data['Customer Feedback'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:34.805915Z","iopub.execute_input":"2024-12-01T09:02:34.806308Z","iopub.status.idle":"2024-12-01T09:02:34.912889Z","shell.execute_reply.started":"2024-12-01T09:02:34.806247Z","shell.execute_reply":"2024-12-01T09:02:34.911686Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Customer Feedback');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:34.914632Z","iopub.execute_input":"2024-12-01T09:02:34.91514Z","iopub.status.idle":"2024-12-01T09:02:35.815316Z","shell.execute_reply.started":"2024-12-01T09:02:34.915085Z","shell.execute_reply":"2024-12-01T09:02:35.814231Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Customer Feedback'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:35.816889Z","iopub.execute_input":"2024-12-01T09:02:35.817348Z","iopub.status.idle":"2024-12-01T09:02:35.87779Z","shell.execute_reply.started":"2024-12-01T09:02:35.817301Z","shell.execute_reply":"2024-12-01T09:02:35.876527Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Fill NaN of Customer Feedback by looking at the other columns like Education Level etc.","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:35.879087Z","iopub.execute_input":"2024-12-01T09:02:35.879411Z","iopub.status.idle":"2024-12-01T09:02:35.891808Z","shell.execute_reply.started":"2024-12-01T09:02:35.879381Z","shell.execute_reply":"2024-12-01T09:02:35.890693Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Smoking Status column","metadata":{"execution":{"iopub.status.busy":"2024-12-01T05:06:21.960763Z","iopub.execute_input":"2024-12-01T05:06:21.961152Z","iopub.status.idle":"2024-12-01T05:06:21.97072Z","shell.execute_reply.started":"2024-12-01T05:06:21.961114Z","shell.execute_reply":"2024-12-01T05:06:21.969113Z"}}},{"cell_type":"code","source":"data['Smoking Status'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:35.893209Z","iopub.execute_input":"2024-12-01T09:02:35.893543Z","iopub.status.idle":"2024-12-01T09:02:36.002496Z","shell.execute_reply.started":"2024-12-01T09:02:35.893506Z","shell.execute_reply":"2024-12-01T09:02:36.001252Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Smoking Status');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:36.004177Z","iopub.execute_input":"2024-12-01T09:02:36.004651Z","iopub.status.idle":"2024-12-01T09:02:36.786005Z","shell.execute_reply.started":"2024-12-01T09:02:36.004601Z","shell.execute_reply":"2024-12-01T09:02:36.784807Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Smoking Status'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:36.787302Z","iopub.execute_input":"2024-12-01T09:02:36.787768Z","iopub.status.idle":"2024-12-01T09:02:36.851188Z","shell.execute_reply.started":"2024-12-01T09:02:36.787714Z","shell.execute_reply":"2024-12-01T09:02:36.849979Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Exercise Frequency column","metadata":{"execution":{"iopub.status.busy":"2024-12-01T05:08:01.600639Z","iopub.execute_input":"2024-12-01T05:08:01.601024Z","iopub.status.idle":"2024-12-01T05:08:01.608298Z","shell.execute_reply.started":"2024-12-01T05:08:01.600991Z","shell.execute_reply":"2024-12-01T05:08:01.606947Z"}}},{"cell_type":"code","source":"data['Exercise Frequency'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:36.852945Z","iopub.execute_input":"2024-12-01T09:02:36.85332Z","iopub.status.idle":"2024-12-01T09:02:36.957915Z","shell.execute_reply.started":"2024-12-01T09:02:36.853288Z","shell.execute_reply":"2024-12-01T09:02:36.956568Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Exercise Frequency');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:36.969598Z","iopub.execute_input":"2024-12-01T09:02:36.970065Z","iopub.status.idle":"2024-12-01T09:02:37.83971Z","shell.execute_reply.started":"2024-12-01T09:02:36.969987Z","shell.execute_reply":"2024-12-01T09:02:37.838239Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Exercise Frequency'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:37.841248Z","iopub.execute_input":"2024-12-01T09:02:37.841701Z","iopub.status.idle":"2024-12-01T09:02:37.903429Z","shell.execute_reply.started":"2024-12-01T09:02:37.841645Z","shell.execute_reply":"2024-12-01T09:02:37.902184Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Property Type column","metadata":{"execution":{"iopub.status.busy":"2024-12-01T05:09:15.956655Z","iopub.execute_input":"2024-12-01T05:09:15.95727Z","iopub.status.idle":"2024-12-01T05:09:15.964879Z","shell.execute_reply.started":"2024-12-01T05:09:15.957226Z","shell.execute_reply":"2024-12-01T05:09:15.963583Z"}}},{"cell_type":"code","source":"data['Property Type'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:37.905202Z","iopub.execute_input":"2024-12-01T09:02:37.905595Z","iopub.status.idle":"2024-12-01T09:02:38.018176Z","shell.execute_reply.started":"2024-12-01T09:02:37.905553Z","shell.execute_reply":"2024-12-01T09:02:38.015946Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=data, x='Property Type');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:38.019894Z","iopub.execute_input":"2024-12-01T09:02:38.020302Z","iopub.status.idle":"2024-12-01T09:02:38.830075Z","shell.execute_reply.started":"2024-12-01T09:02:38.020265Z","shell.execute_reply":"2024-12-01T09:02:38.828914Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Property Type'].isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:38.831632Z","iopub.execute_input":"2024-12-01T09:02:38.832125Z","iopub.status.idle":"2024-12-01T09:02:38.894047Z","shell.execute_reply.started":"2024-12-01T09:02:38.832079Z","shell.execute_reply":"2024-12-01T09:02:38.892644Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Premium Amount - The OG Target Column","metadata":{"execution":{"iopub.status.busy":"2024-12-01T05:10:49.216351Z","iopub.execute_input":"2024-12-01T05:10:49.216787Z","iopub.status.idle":"2024-12-01T05:10:49.224887Z","shell.execute_reply.started":"2024-12-01T05:10:49.216751Z","shell.execute_reply":"2024-12-01T05:10:49.223484Z"}}},{"cell_type":"code","source":"sns.histplot(data=data, x='Premium Amount');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:38.895399Z","iopub.execute_input":"2024-12-01T09:02:38.895848Z","iopub.status.idle":"2024-12-01T09:02:39.777356Z","shell.execute_reply.started":"2024-12-01T09:02:38.895814Z","shell.execute_reply":"2024-12-01T09:02:39.776193Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Exploratory Data Analysis - Bivariate","metadata":{}},{"cell_type":"code","source":"sns.pairplot(data.sample(frac=0.05, random_state=47))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:02:39.778906Z","iopub.execute_input":"2024-12-01T09:02:39.779283Z","iopub.status.idle":"2024-12-01T09:03:28.142971Z","shell.execute_reply.started":"2024-12-01T09:02:39.779251Z","shell.execute_reply":"2024-12-01T09:03:28.141422Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.columns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:28.144495Z","iopub.execute_input":"2024-12-01T09:03:28.144974Z","iopub.status.idle":"2024-12-01T09:03:28.154341Z","shell.execute_reply.started":"2024-12-01T09:03:28.14491Z","shell.execute_reply":"2024-12-01T09:03:28.153231Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Age vs Educational Level, Premium Amount","metadata":{}},{"cell_type":"code","source":"# Age vs Education Level\nsns.boxplot(data=data, x='Education Level', y='Age');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:28.155669Z","iopub.execute_input":"2024-12-01T09:03:28.15613Z","iopub.status.idle":"2024-12-01T09:03:29.0982Z","shell.execute_reply.started":"2024-12-01T09:03:28.156087Z","shell.execute_reply":"2024-12-01T09:03:29.096861Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data[data['Education Level'] == \"Bachelor's\"]['Age'].median()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:29.100563Z","iopub.execute_input":"2024-12-01T09:03:29.101056Z","iopub.status.idle":"2024-12-01T09:03:29.277467Z","shell.execute_reply.started":"2024-12-01T09:03:29.100986Z","shell.execute_reply":"2024-12-01T09:03:29.276205Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data[data['Education Level'] == \"Master's\"]['Age'].median()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:29.278903Z","iopub.execute_input":"2024-12-01T09:03:29.279348Z","iopub.status.idle":"2024-12-01T09:03:29.458735Z","shell.execute_reply.started":"2024-12-01T09:03:29.279301Z","shell.execute_reply":"2024-12-01T09:03:29.457459Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data[data['Education Level'] == \"High School\"]['Age'].median()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:29.460143Z","iopub.execute_input":"2024-12-01T09:03:29.460605Z","iopub.status.idle":"2024-12-01T09:03:29.632449Z","shell.execute_reply.started":"2024-12-01T09:03:29.460557Z","shell.execute_reply":"2024-12-01T09:03:29.631181Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data[data['Education Level'] == \"PhD\"]['Age'].median()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:29.63402Z","iopub.execute_input":"2024-12-01T09:03:29.634455Z","iopub.status.idle":"2024-12-01T09:03:29.800541Z","shell.execute_reply.started":"2024-12-01T09:03:29.634409Z","shell.execute_reply":"2024-12-01T09:03:29.799393Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data[data['Education Level'] == \"Bachelor's\"]['Age'].mode()[0]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:29.801859Z","iopub.execute_input":"2024-12-01T09:03:29.802203Z","iopub.status.idle":"2024-12-01T09:03:29.957216Z","shell.execute_reply.started":"2024-12-01T09:03:29.802173Z","shell.execute_reply":"2024-12-01T09:03:29.956081Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data[data['Education Level'] == \"Master's\"]['Age'].mode()[0]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:29.958633Z","iopub.execute_input":"2024-12-01T09:03:29.959105Z","iopub.status.idle":"2024-12-01T09:03:30.119033Z","shell.execute_reply.started":"2024-12-01T09:03:29.959054Z","shell.execute_reply":"2024-12-01T09:03:30.117705Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data[data['Education Level'] == \"High School\"]['Age'].mode()[0]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:30.120376Z","iopub.execute_input":"2024-12-01T09:03:30.120749Z","iopub.status.idle":"2024-12-01T09:03:30.276706Z","shell.execute_reply.started":"2024-12-01T09:03:30.120714Z","shell.execute_reply":"2024-12-01T09:03:30.275187Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data[data['Education Level'] == \"PhD\"]['Age'].mode()[0]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:30.278327Z","iopub.execute_input":"2024-12-01T09:03:30.278914Z","iopub.status.idle":"2024-12-01T09:03:30.436251Z","shell.execute_reply.started":"2024-12-01T09:03:30.278867Z","shell.execute_reply":"2024-12-01T09:03:30.43515Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.histplot(data[data['Education Level'] == \"PhD\"]['Age'], kde=True)\nplt.show()\nsns.histplot(data[data['Education Level'] == \"Bachelor's\"]['Age'], kde=True)\nplt.show()\nsns.histplot(data[data['Education Level'] == \"Master's\"]['Age'], kde=True)\nplt.show()\nsns.histplot(data[data['Education Level'] == \"High School\"]['Age'], kde=True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:30.437747Z","iopub.execute_input":"2024-12-01T09:03:30.438244Z","iopub.status.idle":"2024-12-01T09:03:37.15748Z","shell.execute_reply.started":"2024-12-01T09:03:30.438195Z","shell.execute_reply":"2024-12-01T09:03:37.156263Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(12, 10))\ndata_grouped = data.groupby('Age')['Premium Amount'].mean().reset_index()\nsns.barplot(data=data_grouped, y='Premium Amount', x='Age')\nplt.xticks(rotation=45);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:37.158783Z","iopub.execute_input":"2024-12-01T09:03:37.159093Z","iopub.status.idle":"2024-12-01T09:03:37.817298Z","shell.execute_reply.started":"2024-12-01T09:03:37.159064Z","shell.execute_reply":"2024-12-01T09:03:37.816082Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Gender vs Premium Amount","metadata":{}},{"cell_type":"code","source":"sns.kdeplot(data=data, hue='Gender', x='Premium Amount');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:37.819242Z","iopub.execute_input":"2024-12-01T09:03:37.819715Z","iopub.status.idle":"2024-12-01T09:03:44.372149Z","shell.execute_reply.started":"2024-12-01T09:03:37.819668Z","shell.execute_reply":"2024-12-01T09:03:44.370849Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Annual Income vs Education Level, Premium Amount","metadata":{}},{"cell_type":"code","source":"sns.kdeplot(data=data, x='Annual Income', hue='Education Level');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:44.373533Z","iopub.execute_input":"2024-12-01T09:03:44.373913Z","iopub.status.idle":"2024-12-01T09:03:50.832301Z","shell.execute_reply.started":"2024-12-01T09:03:44.373878Z","shell.execute_reply":"2024-12-01T09:03:50.831047Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.scatterplot(data=data.sample(frac=0.05), x='Annual Income', y='Premium Amount', hue='Education Level');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:50.833629Z","iopub.execute_input":"2024-12-01T09:03:50.833973Z","iopub.status.idle":"2024-12-01T09:03:55.842535Z","shell.execute_reply.started":"2024-12-01T09:03:50.833941Z","shell.execute_reply":"2024-12-01T09:03:55.841157Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Not Helpul But Beautiful tho :)","metadata":{}},{"cell_type":"code","source":"# Since all categories of the Education level and Annual Income looks similar, we can impute with the same data i.e. median\ndata['Annual Income'].median()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:55.843624Z","iopub.execute_input":"2024-12-01T09:03:55.843929Z","iopub.status.idle":"2024-12-01T09:03:55.881704Z","shell.execute_reply.started":"2024-12-01T09:03:55.843901Z","shell.execute_reply":"2024-12-01T09:03:55.880491Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Marital Status vs Premium Amount, Education Level","metadata":{}},{"cell_type":"code","source":"sns.kdeplot(data=data, x='Premium Amount', hue='Marital Status');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:03:55.882686Z","iopub.execute_input":"2024-12-01T09:03:55.883034Z","iopub.status.idle":"2024-12-01T09:04:02.508467Z","shell.execute_reply.started":"2024-12-01T09:03:55.882974Z","shell.execute_reply":"2024-12-01T09:04:02.507371Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Number of Dependents vs Premium Amount","metadata":{}},{"cell_type":"code","source":"sns.kdeplot(data=data, x='Premium Amount', hue='Number of Dependents');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:02.510108Z","iopub.execute_input":"2024-12-01T09:04:02.51044Z","iopub.status.idle":"2024-12-01T09:04:07.771058Z","shell.execute_reply.started":"2024-12-01T09:04:02.510409Z","shell.execute_reply":"2024-12-01T09:04:07.769651Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Health Score vs Premium Amount, Education level","metadata":{}},{"cell_type":"code","source":"sns.kdeplot(data=data, x='Health Score', hue='Education Level');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:07.77264Z","iopub.execute_input":"2024-12-01T09:04:07.772989Z","iopub.status.idle":"2024-12-01T09:04:13.770123Z","shell.execute_reply.started":"2024-12-01T09:04:07.772957Z","shell.execute_reply":"2024-12-01T09:04:13.768887Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Health Score'].mean(), data['Health Score'].median()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:13.7715Z","iopub.execute_input":"2024-12-01T09:04:13.771864Z","iopub.status.idle":"2024-12-01T09:04:13.813241Z","shell.execute_reply.started":"2024-12-01T09:04:13.771831Z","shell.execute_reply":"2024-12-01T09:04:13.812209Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Previous Claims vs Premium Amount, Education level","metadata":{}},{"cell_type":"code","source":"sns.kdeplot(data=data, x='Previous Claims', hue='Education Level');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:13.814908Z","iopub.execute_input":"2024-12-01T09:04:13.815369Z","iopub.status.idle":"2024-12-01T09:04:18.996414Z","shell.execute_reply.started":"2024-12-01T09:04:13.815324Z","shell.execute_reply":"2024-12-01T09:04:18.995114Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Credit Score vs Premium Amount, Education Level","metadata":{}},{"cell_type":"code","source":"sns.kdeplot(data=data, x='Credit Score', hue='Education Level');","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:18.997818Z","iopub.execute_input":"2024-12-01T09:04:18.998193Z","iopub.status.idle":"2024-12-01T09:04:24.475258Z","shell.execute_reply.started":"2024-12-01T09:04:18.998157Z","shell.execute_reply":"2024-12-01T09:04:24.474086Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Credit Score'].median()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:24.476599Z","iopub.execute_input":"2024-12-01T09:04:24.47693Z","iopub.status.idle":"2024-12-01T09:04:24.508199Z","shell.execute_reply.started":"2024-12-01T09:04:24.4769Z","shell.execute_reply":"2024-12-01T09:04:24.506977Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Lets take some actions based on our inferences","metadata":{}},{"cell_type":"code","source":"data.columns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:24.509584Z","iopub.execute_input":"2024-12-01T09:04:24.510138Z","iopub.status.idle":"2024-12-01T09:04:24.518293Z","shell.execute_reply.started":"2024-12-01T09:04:24.510094Z","shell.execute_reply":"2024-12-01T09:04:24.51706Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:24.519985Z","iopub.execute_input":"2024-12-01T09:04:24.520461Z","iopub.status.idle":"2024-12-01T09:04:24.552144Z","shell.execute_reply.started":"2024-12-01T09:04:24.520414Z","shell.execute_reply":"2024-12-01T09:04:24.550675Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# **Explanation of this is given in the code step by step**\n# 1. Fill NaN of Age with 41 in 1st iter, and its respective categorical modes in 2nd iter.\n# 2. Fill NaN of Annual Income with 23911.0 (median).\n# 3. Fill NaN Marital Status NaN with 'Unknown'. (Since 18k NaN).\n# 4. Fill -1 in NaN of Number of Dependents.\n# 5. Fill NaN of Occupation with 'Unknown'.\n# 6. Fill NaN of Health Score with median (25).\n# 7. Fill NaN of Previous Claims with -1 (Unknown category since NaN is huge), drop in 2nd iter.\n# 8. Fill NaN values of Vehicle Age with 10.0 (median).\n# 9. Fill NaN values of Credit Score with 595.0 (Median).\n# 10. Fill NaN values of Insurance Duration with 5 (median).\n# 11. Make year column and Drop Policy Start Date Column.\n# 12. Fill NaN values of Customer Feedback column with 'Unknown'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:24.553764Z","iopub.execute_input":"2024-12-01T09:04:24.554939Z","iopub.status.idle":"2024-12-01T09:04:24.559883Z","shell.execute_reply.started":"2024-12-01T09:04:24.554895Z","shell.execute_reply":"2024-12-01T09:04:24.558803Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def impute(data):\n    data['Age'].fillna(41.0, inplace=True)\n    data['Annual Income'].fillna(23911.0, inplace=True)\n    data['Marital Status'].fillna('Unknown', inplace=True)\n    data['Number of Dependents'].fillna(-1, inplace=True)\n    data['Occupation'].fillna('Unknown', inplace=True)\n    data['Health Score'].fillna(25.0, inplace=True)\n    data['Previous Claims'].fillna(-1, inplace=True)\n    data['Vehicle Age'].fillna(10.0, inplace=True)\n    data['Credit Score'].fillna(595.0, inplace=True)\n    data['Insurance Duration'].fillna(5.0, inplace=True)\n    data['Customer Feedback'].fillna('Unknown', inplace=True)\n    if 'Policy Start Date' in data.columns:\n        data['Policy Start Date'] = pd.to_datetime(data['Policy Start Date'])\n        data['Year'] = data['Policy Start Date'].dt.year\n        data.drop('Policy Start Date', axis=1, inplace=True)\n    print(\"Done with Imputation!\")\n    return data","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:24.561588Z","iopub.execute_input":"2024-12-01T09:04:24.561903Z","iopub.status.idle":"2024-12-01T09:04:24.578769Z","shell.execute_reply.started":"2024-12-01T09:04:24.561873Z","shell.execute_reply":"2024-12-01T09:04:24.577596Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def encode_and_scale(data, le_cols, scale_cols, target_col, test=None, training=True):\n    X_train, X_test, y_train, y_test = train_test_split(data.drop(target_col, axis=1),\n                                                        data[target_col], \n                                                        random_state=42,\n                                                        test_size=0.2)\n    if training==True and test is None:\n        for col in le_cols:\n            le = LabelEncoder()\n            X_train[col] = le.fit_transform(X_train[col])\n            X_test[col] = le.transform(X_test[col])\n            print(f\"Done {col}\")\n        scaler = StandardScaler()\n        X_train[scale_cols] = scaler.fit_transform(X_train[scale_cols])\n        X_test[scale_cols] = scaler.transform(X_test[scale_cols])\n        return X_train, X_test, y_train, y_test\n\n    if training==False and test is not None:\n        for col in le_cols:\n            le = LabelEncoder()\n            data[col] = le.fit_transform(data[col])\n            test[col] = le.transform(test[col])\n            print(f\"Done {col}\")\n        scaler = StandardScaler()\n        data[scale_cols] = scaler.fit_transform(data[scale_cols])\n        test[scale_cols] = scaler.transform(test[scale_cols])\n        return data, test\n\n            \n    if training==False and test is  None:\n        print(\"Test data is empty!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:14:30.960824Z","iopub.execute_input":"2024-12-01T09:14:30.961472Z","iopub.status.idle":"2024-12-01T09:14:30.974302Z","shell.execute_reply.started":"2024-12-01T09:14:30.961404Z","shell.execute_reply":"2024-12-01T09:14:30.973053Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"le_cols = ['Gender', 'Marital Status', 'Education Level', 'Occupation',\n           'Location', 'Policy Type', 'Customer Feedback', 'Smoking Status',\n           'Exercise Frequency', 'Property Type']\n\nscale_cols = ['Annual Income', 'Health Score', 'Credit Score']\n\ntarget_col = 'Premium Amount'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:24.59407Z","iopub.execute_input":"2024-12-01T09:04:24.59457Z","iopub.status.idle":"2024-12-01T09:04:24.610843Z","shell.execute_reply.started":"2024-12-01T09:04:24.594495Z","shell.execute_reply":"2024-12-01T09:04:24.609523Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"imputed_data = impute(data)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:24.612863Z","iopub.execute_input":"2024-12-01T09:04:24.613349Z","iopub.status.idle":"2024-12-01T09:04:25.393113Z","shell.execute_reply.started":"2024-12-01T09:04:24.613301Z","shell.execute_reply":"2024-12-01T09:04:25.391861Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = encode_and_scale(data,\n                                                    le_cols,\n                                                    scale_cols,\n                                                    target_col,\n                                                    test=None,\n                                                    training=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:25.394542Z","iopub.execute_input":"2024-12-01T09:04:25.395027Z","iopub.status.idle":"2024-12-01T09:04:28.858419Z","shell.execute_reply.started":"2024-12-01T09:04:25.394957Z","shell.execute_reply":"2024-12-01T09:04:28.857332Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_train.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:28.859922Z","iopub.execute_input":"2024-12-01T09:04:28.860303Z","iopub.status.idle":"2024-12-01T09:04:28.880677Z","shell.execute_reply.started":"2024-12-01T09:04:28.860272Z","shell.execute_reply":"2024-12-01T09:04:28.879361Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_test.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:28.882224Z","iopub.execute_input":"2024-12-01T09:04:28.882692Z","iopub.status.idle":"2024-12-01T09:04:28.903299Z","shell.execute_reply.started":"2024-12-01T09:04:28.882641Z","shell.execute_reply":"2024-12-01T09:04:28.901968Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"y_train[:5], y_test[:5]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:28.904733Z","iopub.execute_input":"2024-12-01T09:04:28.905207Z","iopub.status.idle":"2024-12-01T09:04:28.920706Z","shell.execute_reply.started":"2024-12-01T09:04:28.905155Z","shell.execute_reply":"2024-12-01T09:04:28.919268Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Modelling Baselines","metadata":{}},{"cell_type":"code","source":"models = {\n    'Linear Regression': LinearRegression(n_jobs=-1),\n    'Ridge Regression': Ridge(),\n    'Lasso Regression': Lasso(),\n    'Decision Tree': DecisionTreeRegressor(),\n    'XGBoost': XGBRegressor(verbosity=0, n_jobs=-1),\n    'LightGBM': LGBMRegressor(verbosity=-1, n_jobs=-1),\n    'CatBoost': CatBoostRegressor(verbose=0),\n    # 'Gradient Boosting': GradientBoostingRegressor(),\n    # 'Support Vector Regressor': SVR(),\n    # 'KNN Regressor': KNeighborsRegressor(n_jobs=-1),\n    # 'Random Forest': RandomForestRegressor(n_jobs=-1),\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:28.922218Z","iopub.execute_input":"2024-12-01T09:04:28.922666Z","iopub.status.idle":"2024-12-01T09:04:28.941867Z","shell.execute_reply.started":"2024-12-01T09:04:28.922618Z","shell.execute_reply":"2024-12-01T09:04:28.940279Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"results = {}\nfor name, model in models.items():\n    print(f\"Training {name}...\")\n    \n    model.fit(X_train, y_train)\n    \n    y_pred = model.predict(X_test)\n    \n    # Clip negative predictions (RMSLE requires non-negative values)\n    y_pred = np.maximum(y_pred, 0)\n    \n    # Calculate RMSLE\n    rmsle = np.sqrt(mean_squared_log_error(y_test, y_pred))\n    \n    # Store results\n    results[name] = {'RMSLE': rmsle}\n    \n    # Print results for the current model\n    print(f\"{name}: RMSLE = {rmsle:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:04:28.943336Z","iopub.execute_input":"2024-12-01T09:04:28.943795Z","iopub.status.idle":"2024-12-01T09:06:27.582091Z","shell.execute_reply.started":"2024-12-01T09:04:28.943695Z","shell.execute_reply":"2024-12-01T09:06:27.5808Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Test Set ","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv('/kaggle/input/playground-series-s4e12/train.csv', index_col='id')\ntest_data = pd.read_csv('/kaggle/input/playground-series-s4e12/test.csv', index_col='id')\ntest_data.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:14:36.321664Z","iopub.execute_input":"2024-12-01T09:14:36.322112Z","iopub.status.idle":"2024-12-01T09:14:44.152334Z","shell.execute_reply.started":"2024-12-01T09:14:36.322077Z","shell.execute_reply":"2024-12-01T09:14:44.151272Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data = impute(data)\ntest_data = impute(test_data)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:14:44.154268Z","iopub.execute_input":"2024-12-01T09:14:44.154623Z","iopub.status.idle":"2024-12-01T09:14:45.621555Z","shell.execute_reply.started":"2024-12-01T09:14:44.154585Z","shell.execute_reply":"2024-12-01T09:14:45.620348Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data, test_data = encode_and_scale( data,\n                                    le_cols,\n                                    scale_cols,\n                                    target_col,\n                                    test=test_data,\n                                    training=False,)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:14:45.624823Z","iopub.execute_input":"2024-12-01T09:14:45.625191Z","iopub.status.idle":"2024-12-01T09:14:50.502263Z","shell.execute_reply.started":"2024-12-01T09:14:45.625157Z","shell.execute_reply":"2024-12-01T09:14:50.500932Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_data.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:14:50.504182Z","iopub.execute_input":"2024-12-01T09:14:50.504507Z","iopub.status.idle":"2024-12-01T09:14:50.524707Z","shell.execute_reply.started":"2024-12-01T09:14:50.504477Z","shell.execute_reply":"2024-12-01T09:14:50.52343Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:14:50.88969Z","iopub.execute_input":"2024-12-01T09:14:50.890153Z","iopub.status.idle":"2024-12-01T09:14:50.913397Z","shell.execute_reply.started":"2024-12-01T09:14:50.890115Z","shell.execute_reply":"2024-12-01T09:14:50.912031Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Baseline LightGBM","metadata":{}},{"cell_type":"code","source":"model = LGBMRegressor(verbosity=-1, n_jobs=-1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:16:36.068276Z","iopub.execute_input":"2024-12-01T09:16:36.068709Z","iopub.status.idle":"2024-12-01T09:16:36.074166Z","shell.execute_reply.started":"2024-12-01T09:16:36.068675Z","shell.execute_reply":"2024-12-01T09:16:36.072864Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model.fit(data.drop(target_col, axis=1), data[target_col])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:17:26.174613Z","iopub.execute_input":"2024-12-01T09:17:26.175062Z","iopub.status.idle":"2024-12-01T09:17:38.765848Z","shell.execute_reply.started":"2024-12-01T09:17:26.175016Z","shell.execute_reply":"2024-12-01T09:17:38.764669Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"y_pred = model.predict(test_data)\ny_pred = np.maximum(y_pred, 0)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:18:50.448135Z","iopub.execute_input":"2024-12-01T09:18:50.448576Z","iopub.status.idle":"2024-12-01T09:18:54.945902Z","shell.execute_reply.started":"2024-12-01T09:18:50.448539Z","shell.execute_reply":"2024-12-01T09:18:54.944832Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Submission file","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/playground-series-s4e12/sample_submission.csv')\nsub['Premium Amount'] = y_pred\nsub.to_csv(\"Submission_lgb_baseline.csv\", index=False)\nsub.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-01T09:20:53.693612Z","iopub.execute_input":"2024-12-01T09:20:53.694058Z","iopub.status.idle":"2024-12-01T09:20:55.559497Z","shell.execute_reply.started":"2024-12-01T09:20:53.693987Z","shell.execute_reply":"2024-12-01T09:20:55.55826Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Please UPVOTE!!","metadata":{}}]}