{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30918,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"dc142213","cell_type":"markdown","source":"# Problem statement\n\n## data description\n1. Age: Age of the insured individual (Numerical)\n1. Gender: Gender of the insured individual (Categorical: Male, Female)\n1. Annual Income: Annual income of the insured individual (Numerical, skewed)\n1. Marital Status: Marital status of the insured individual (Categorical: Single, Married, Divorced)\n1. Number of Dependents: Number of dependents (Numerical, with missing values)\n1. Education Level: Highest education level attained (Categorical: High School, Bachelor's, Master's, PhD)\n1. Occupation: Occupation of the insured individual (Categorical: Employed, Self-Employed, Unemployed)\n1. Health Score: A score representing the health status (Numerical, skewed)\n1. Location: Type of location (Categorical: Urban, Suburban, Rural)\n1. Policy Type: Type of insurance policy (Categorical: Basic, Comprehensive, Premium)\n1. Previous Claims: Number of previous claims made (Numerical, with outliers)\n1. Vehicle Age: Age of the vehicle insured (Numerical)\n1. Credit Score: Credit score of the insured individual (Numerical, with missing values)\n1. Insurance Duration: Duration of the insurance policy (Numerical, in years)\n1. Premium Amount: Target variable representing the insurance premium amount (Numerical, skewed)\n1. Policy Start Date: Start date of the insurance policy (Text, improperly formatted)\n1. Customer Feedback: Short feedback comments from customers (Text)\n1. Smoking Status: Smoking status of the insured individual (Categorical: Yes, No)\n1. Exercise Frequency: Frequency of exercise (Categorical: Daily, Weekly, Monthly, Rarely)\n1. Property Type: Type of property owned (Categorical: House, Apartment, Condo)","metadata":{}},{"id":"cf4637dd","cell_type":"markdown","source":"# Quick overview of EDA results:\n\n* As our data is synthetic, all the missing values comes under MCAR.\n* 5% of population's annual income is lower than premium amount paid\n* Insurance duration feature has inconsistency\n* Annual income and premium amount feature are right skewed\n* In Categorical features, all the categories are evenly distributed","metadata":{}},{"id":"be584c03","cell_type":"markdown","source":"### Let's explore these results step by step with Exploratory Data Analysis","metadata":{}},{"id":"59f20f3e","cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:04:44.40524Z","iopub.execute_input":"2025-03-16T19:04:44.405691Z","iopub.status.idle":"2025-03-16T19:04:45.118693Z","shell.execute_reply.started":"2025-03-16T19:04:44.405645Z","shell.execute_reply":"2025-03-16T19:04:45.117136Z"}},"outputs":[],"execution_count":null},{"id":"b7f47a69","cell_type":"code","source":"#additional config\npd.set_option('display.max_columns',None)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:04:45.121719Z","iopub.execute_input":"2025-03-16T19:04:45.122315Z","iopub.status.idle":"2025-03-16T19:04:45.134023Z","shell.execute_reply.started":"2025-03-16T19:04:45.122266Z","shell.execute_reply":"2025-03-16T19:04:45.13159Z"}},"outputs":[],"execution_count":null},{"id":"d5295a36","cell_type":"code","source":"data = pd.read_csv('/kaggle/input/playground-series-s4e12/train.csv')\ndata.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:18.315047Z","iopub.execute_input":"2025-03-16T19:05:18.315458Z","iopub.status.idle":"2025-03-16T19:05:25.258832Z","shell.execute_reply.started":"2025-03-16T19:05:18.315416Z","shell.execute_reply":"2025-03-16T19:05:25.257581Z"}},"outputs":[],"execution_count":null},{"id":"bb1820ab","cell_type":"code","source":"data.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:32.82077Z","iopub.execute_input":"2025-03-16T19:05:32.82134Z","iopub.status.idle":"2025-03-16T19:05:32.828829Z","shell.execute_reply.started":"2025-03-16T19:05:32.821296Z","shell.execute_reply":"2025-03-16T19:05:32.827473Z"}},"outputs":[],"execution_count":null},{"id":"e5e9309d","cell_type":"code","source":"data.dtypes","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:33.221963Z","iopub.execute_input":"2025-03-16T19:05:33.222479Z","iopub.status.idle":"2025-03-16T19:05:33.233756Z","shell.execute_reply.started":"2025-03-16T19:05:33.222437Z","shell.execute_reply":"2025-03-16T19:05:33.23259Z"}},"outputs":[],"execution_count":null},{"id":"b3f222f0","cell_type":"code","source":"features = data.columns\nnumerical_features = data.select_dtypes(exclude='object').columns\ncategorical_features = data.select_dtypes(include='object').columns\nprint(f'numerical features: {len(numerical_features)} \\ncategorical features: {len(categorical_features)}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:33.465999Z","iopub.execute_input":"2025-03-16T19:05:33.466574Z","iopub.status.idle":"2025-03-16T19:05:33.588483Z","shell.execute_reply.started":"2025-03-16T19:05:33.466528Z","shell.execute_reply":"2025-03-16T19:05:33.587236Z"}},"outputs":[],"execution_count":null},{"id":"6799f3ed","cell_type":"markdown","source":"## checking missing values","metadata":{}},{"id":"d16dd33d","cell_type":"code","source":"data.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:33.811609Z","iopub.execute_input":"2025-03-16T19:05:33.812002Z","iopub.status.idle":"2025-03-16T19:05:34.534638Z","shell.execute_reply.started":"2025-03-16T19:05:33.811944Z","shell.execute_reply":"2025-03-16T19:05:34.533548Z"}},"outputs":[],"execution_count":null},{"id":"6b7a974e","cell_type":"code","source":"data['id'].nunique()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:34.535773Z","iopub.execute_input":"2025-03-16T19:05:34.536115Z","iopub.status.idle":"2025-03-16T19:05:34.584762Z","shell.execute_reply.started":"2025-03-16T19:05:34.536088Z","shell.execute_reply":"2025-03-16T19:05:34.583701Z"}},"outputs":[],"execution_count":null},{"id":"a06c49bb","cell_type":"markdown","source":"**Assumption 1:**\nas our id is unique, we assume that no two claims are overlap by an individual (i.e), we have 12 lakh indivial people who are insured in this data.","metadata":{}},{"id":"8eaca051","cell_type":"code","source":"na_features = []\nfor col in features:\n    if data[col].isnull().any():\n        na_features.append(col)\n        print(f'{col:<22} has {np.round(data[col].isnull().mean() * 100,2)} % of NA')\nprint(f'\\nNo of NA features: {len(na_features)}')\nprint(na_features)\n        ","metadata":{"scrolled":true,"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:34.586149Z","iopub.execute_input":"2025-03-16T19:05:34.586451Z","iopub.status.idle":"2025-03-16T19:05:35.498316Z","shell.execute_reply.started":"2025-03-16T19:05:34.586425Z","shell.execute_reply":"2025-03-16T19:05:35.497219Z"}},"outputs":[],"execution_count":null},{"id":"331056f6","cell_type":"markdown","source":"Result: 11 features have missing values","metadata":{}},{"id":"581e7ed5","cell_type":"markdown","source":"# Exploratory data analysis","metadata":{}},{"id":"c49da5a9","cell_type":"markdown","source":"## Analysis for feature with missing values","metadata":{}},{"id":"60a946b1","cell_type":"markdown","source":"**Age feature analysis**","metadata":{}},{"id":"94836435","cell_type":"code","source":"data['Age'].nunique()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:37.349718Z","iopub.execute_input":"2025-03-16T19:05:37.350218Z","iopub.status.idle":"2025-03-16T19:05:37.372722Z","shell.execute_reply.started":"2025-03-16T19:05:37.350164Z","shell.execute_reply":"2025-03-16T19:05:37.371422Z"}},"outputs":[],"execution_count":null},{"id":"b9c1d4b1","cell_type":"code","source":"data['Age'].value_counts().sort_index().plot(kind='bar',figsize=(12,3))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:37.572616Z","iopub.execute_input":"2025-03-16T19:05:37.573072Z","iopub.status.idle":"2025-03-16T19:05:38.254348Z","shell.execute_reply.started":"2025-03-16T19:05:37.573036Z","shell.execute_reply":"2025-03-16T19:05:38.253011Z"}},"outputs":[],"execution_count":null},{"id":"b0ef7a6e","cell_type":"code","source":"data[data['Age'].isna()]['Gender'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:38.255778Z","iopub.execute_input":"2025-03-16T19:05:38.2562Z","iopub.status.idle":"2025-03-16T19:05:38.294473Z","shell.execute_reply.started":"2025-03-16T19:05:38.256158Z","shell.execute_reply":"2025-03-16T19:05:38.292395Z"}},"outputs":[],"execution_count":null},{"id":"090460ab","cell_type":"markdown","source":"**Result:** Age missing is not dependend on Gender","metadata":{}},{"id":"7b2d7fb6","cell_type":"markdown","source":"**Analysing Occupation feature**","metadata":{}},{"id":"beab57d2","cell_type":"code","source":"data['Education Level'].value_counts().plot.bar()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:38.30684Z","iopub.execute_input":"2025-03-16T19:05:38.307272Z","iopub.status.idle":"2025-03-16T19:05:38.668272Z","shell.execute_reply.started":"2025-03-16T19:05:38.307238Z","shell.execute_reply":"2025-03-16T19:05:38.666463Z"}},"outputs":[],"execution_count":null},{"id":"5610a7d4","cell_type":"code","source":"#checking whether educational level has any relation\ndata[data['Occupation'].isna()]['Education Level'].value_counts().plot(kind='pie',autopct='%1.1f%%')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:38.670126Z","iopub.execute_input":"2025-03-16T19:05:38.670568Z","iopub.status.idle":"2025-03-16T19:05:39.053996Z","shell.execute_reply.started":"2025-03-16T19:05:38.670536Z","shell.execute_reply":"2025-03-16T19:05:39.052518Z"}},"outputs":[],"execution_count":null},{"id":"b1293120","cell_type":"markdown","source":"**Result:** Missing values of occupation is almost equally distributed among education level.\n**type**: MCAR","metadata":{}},{"id":"cbb1d14e","cell_type":"markdown","source":"**Analysing annual income feature**","metadata":{}},{"id":"fbbe989f","cell_type":"code","source":"data['Occupation'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:40.104844Z","iopub.execute_input":"2025-03-16T19:05:40.105314Z","iopub.status.idle":"2025-03-16T19:05:40.214878Z","shell.execute_reply.started":"2025-03-16T19:05:40.105277Z","shell.execute_reply":"2025-03-16T19:05:40.213835Z"}},"outputs":[],"execution_count":null},{"id":"98488778","cell_type":"code","source":"data[data['Annual Income'].isna()]['Occupation'].value_counts().plot(kind='pie',autopct=\"%1.1f%%\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:40.280697Z","iopub.execute_input":"2025-03-16T19:05:40.2811Z","iopub.status.idle":"2025-03-16T19:05:40.423663Z","shell.execute_reply.started":"2025-03-16T19:05:40.281069Z","shell.execute_reply":"2025-03-16T19:05:40.42256Z"}},"outputs":[],"execution_count":null},{"id":"c8de6c0b","cell_type":"code","source":"data[data['Annual Income'].isna()]['Education Level'].value_counts().plot(kind='pie',autopct=\"%1.1f%%\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:40.786924Z","iopub.execute_input":"2025-03-16T19:05:40.787311Z","iopub.status.idle":"2025-03-16T19:05:40.94569Z","shell.execute_reply.started":"2025-03-16T19:05:40.78728Z","shell.execute_reply":"2025-03-16T19:05:40.944385Z"}},"outputs":[],"execution_count":null},{"id":"40f98ed6","cell_type":"markdown","source":"**Result:** Missing values in annual income is equally distrubed among education level and occupation","metadata":{}},{"id":"1dfc6bc6","cell_type":"markdown","source":"**Analyzing maritial status**","metadata":{}},{"id":"ec100902","cell_type":"code","source":"data['Marital Status'].value_counts().plot(kind='pie',autopct='%1.1f%%')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:40.94723Z","iopub.execute_input":"2025-03-16T19:05:40.947584Z","iopub.status.idle":"2025-03-16T19:05:41.17306Z","shell.execute_reply.started":"2025-03-16T19:05:40.947556Z","shell.execute_reply":"2025-03-16T19:05:41.17125Z"}},"outputs":[],"execution_count":null},{"id":"70456a94","cell_type":"code","source":"data[data['Marital Status'].isna()]['Number of Dependents'].value_counts().plot(kind='pie',autopct='%1.1f%%')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:41.177443Z","iopub.execute_input":"2025-03-16T19:05:41.178074Z","iopub.status.idle":"2025-03-16T19:05:41.464703Z","shell.execute_reply.started":"2025-03-16T19:05:41.178037Z","shell.execute_reply":"2025-03-16T19:05:41.463614Z"}},"outputs":[],"execution_count":null},{"id":"88b58ef0","cell_type":"code","source":"data.groupby('Number of Dependents')['Marital Status'].value_counts().plot.bar(figsize=(6,3))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:41.466132Z","iopub.execute_input":"2025-03-16T19:05:41.466494Z","iopub.status.idle":"2025-03-16T19:05:41.900314Z","shell.execute_reply.started":"2025-03-16T19:05:41.466454Z","shell.execute_reply":"2025-03-16T19:05:41.898999Z"}},"outputs":[],"execution_count":null},{"id":"7d1ede37","cell_type":"markdown","source":"**Result**:\n* No of dependents has no relationship with marital status\n* No of dependents column is float type i.e need to change to int","metadata":{}},{"id":"ceed958f","cell_type":"markdown","source":"**Analyzing health score feature**","metadata":{}},{"id":"a1716bd1","cell_type":"code","source":"data.columns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:41.901334Z","iopub.execute_input":"2025-03-16T19:05:41.901689Z","iopub.status.idle":"2025-03-16T19:05:41.909959Z","shell.execute_reply.started":"2025-03-16T19:05:41.901659Z","shell.execute_reply":"2025-03-16T19:05:41.908067Z"}},"outputs":[],"execution_count":null},{"id":"361b932a","cell_type":"code","source":"data['Smoking Status'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:41.912458Z","iopub.execute_input":"2025-03-16T19:05:41.912847Z","iopub.status.idle":"2025-03-16T19:05:42.019124Z","shell.execute_reply.started":"2025-03-16T19:05:41.912808Z","shell.execute_reply":"2025-03-16T19:05:42.018002Z"}},"outputs":[],"execution_count":null},{"id":"01d8a16e","cell_type":"code","source":"data.groupby('Smoking Status')['Health Score'].median().plot.bar(figsize=(6,3))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:42.020649Z","iopub.execute_input":"2025-03-16T19:05:42.021082Z","iopub.status.idle":"2025-03-16T19:05:42.291493Z","shell.execute_reply.started":"2025-03-16T19:05:42.021039Z","shell.execute_reply":"2025-03-16T19:05:42.29039Z"}},"outputs":[],"execution_count":null},{"id":"1691059b","cell_type":"code","source":"#Analysing health score by excercise habit\ndata['Exercise Frequency'].value_counts()\ndata[data['Health Score'].isna()]['Exercise Frequency'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:42.292827Z","iopub.execute_input":"2025-03-16T19:05:42.293497Z","iopub.status.idle":"2025-03-16T19:05:42.437433Z","shell.execute_reply.started":"2025-03-16T19:05:42.293452Z","shell.execute_reply":"2025-03-16T19:05:42.436137Z"}},"outputs":[],"execution_count":null},{"id":"df89c610","cell_type":"code","source":"data.groupby('Exercise Frequency')['Health Score'].mean()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:42.438404Z","iopub.execute_input":"2025-03-16T19:05:42.438736Z","iopub.status.idle":"2025-03-16T19:05:42.534862Z","shell.execute_reply.started":"2025-03-16T19:05:42.438709Z","shell.execute_reply":"2025-03-16T19:05:42.533788Z"}},"outputs":[],"execution_count":null},{"id":"3d46a00f","cell_type":"markdown","source":"**Result:**\n    In this dataset, smoking status has no affect on health score.\n    Also, Excercising doesn't have any health improvements on avg","metadata":{}},{"id":"128ce0d8","cell_type":"markdown","source":"**Analysing for insurance duration**","metadata":{}},{"id":"bd7d870c","cell_type":"code","source":"data[data['Insurance Duration'].isna()]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:42.536288Z","iopub.execute_input":"2025-03-16T19:05:42.536841Z","iopub.status.idle":"2025-03-16T19:05:42.561404Z","shell.execute_reply.started":"2025-03-16T19:05:42.536787Z","shell.execute_reply":"2025-03-16T19:05:42.560415Z"}},"outputs":[],"execution_count":null},{"id":"78250d7e","cell_type":"code","source":"#checking insurance duration for similar policy start data\ndata[data['Policy Start Date'].str.contains('2022-04-06')].head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:42.56247Z","iopub.execute_input":"2025-03-16T19:05:42.562801Z","iopub.status.idle":"2025-03-16T19:05:43.042198Z","shell.execute_reply.started":"2025-03-16T19:05:42.562773Z","shell.execute_reply":"2025-03-16T19:05:43.04112Z"}},"outputs":[],"execution_count":null},{"id":"d67ccaef","cell_type":"markdown","source":"**Result:**\nFor same policy start date, there are different insurance duration (inconsistency).\n","metadata":{}},{"id":"a20a4ee9","cell_type":"markdown","source":"**Further analysis**","metadata":{}},{"id":"b11e4c9c","cell_type":"code","source":"data[data['Premium Amount'] > data['Annual Income']].shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:43.04317Z","iopub.execute_input":"2025-03-16T19:05:43.043432Z","iopub.status.idle":"2025-03-16T19:05:43.080953Z","shell.execute_reply.started":"2025-03-16T19:05:43.043408Z","shell.execute_reply":"2025-03-16T19:05:43.079909Z"}},"outputs":[],"execution_count":null},{"id":"df1c19dd","cell_type":"markdown","source":"**result:** around 4% people's annual income is lower than premium amount they paid","metadata":{}},{"id":"b5cc7c5a","cell_type":"markdown","source":"## Distribution of numerical features","metadata":{}},{"id":"1063fef0","cell_type":"code","source":"for col in numerical_features:\n    data[col].plot(kind='hist',bins=20,figsize=(6,3),edgecolor='black')\n    plt.title(col)\n    plt.show()","metadata":{"scrolled":true,"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:43.083319Z","iopub.execute_input":"2025-03-16T19:05:43.083633Z","iopub.status.idle":"2025-03-16T19:05:46.408038Z","shell.execute_reply.started":"2025-03-16T19:05:43.083608Z","shell.execute_reply":"2025-03-16T19:05:46.407015Z"}},"outputs":[],"execution_count":null},{"id":"0e6b59d9","cell_type":"markdown","source":"**Insights:**\n* Annual income and premium amount are right skewed","metadata":{}},{"id":"6cbb6e16","cell_type":"markdown","source":"## Distribution of categorical features:","metadata":{}},{"id":"b3b0953b","cell_type":"code","source":"for col in categorical_features:\n    if col != 'Policy Start Date':\n        data[col].value_counts().plot(kind='bar',figsize=(6,3))\n        plt.title(col)\n        plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:46.409274Z","iopub.execute_input":"2025-03-16T19:05:46.409621Z","iopub.status.idle":"2025-03-16T19:05:49.059669Z","shell.execute_reply.started":"2025-03-16T19:05:46.409589Z","shell.execute_reply":"2025-03-16T19:05:49.058497Z"}},"outputs":[],"execution_count":null},{"id":"2fb00298","cell_type":"markdown","source":"**No Insights**","metadata":{}},{"id":"61ead788","cell_type":"markdown","source":"# Outliers Detection Analysis","metadata":{}},{"id":"41461b6d","cell_type":"markdown","source":"**Starting with numerical features**","metadata":{}},{"id":"59243f47","cell_type":"code","source":"import seaborn as sns\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:49.060631Z","iopub.execute_input":"2025-03-16T19:05:49.060923Z","iopub.status.idle":"2025-03-16T19:05:49.919873Z","shell.execute_reply.started":"2025-03-16T19:05:49.060898Z","shell.execute_reply":"2025-03-16T19:05:49.918648Z"}},"outputs":[],"execution_count":null},{"id":"98c348c4","cell_type":"code","source":"for col in numerical_features:\n    if data[col].skew() > 0.5 or data[col].skew() < -0.5:\n        print(f'{col} {np.round(data[col].skew(),2)}')","metadata":{"scrolled":true,"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:49.921214Z","iopub.execute_input":"2025-03-16T19:05:49.921797Z","iopub.status.idle":"2025-03-16T19:05:50.288873Z","shell.execute_reply.started":"2025-03-16T19:05:49.921756Z","shell.execute_reply":"2025-03-16T19:05:50.287902Z"}},"outputs":[],"execution_count":null},{"id":"e3412d6a","cell_type":"markdown","source":"So, features with outliers:\n* Annual Income\n* Previous claims\n* Premium amount","metadata":{}},{"id":"c65a5e6a","cell_type":"markdown","source":"**Analysing Annual income:**","metadata":{}},{"id":"1451eb77","cell_type":"code","source":"sns.distplot(data['Annual Income'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:50.289997Z","iopub.execute_input":"2025-03-16T19:05:50.290361Z","iopub.status.idle":"2025-03-16T19:05:55.610139Z","shell.execute_reply.started":"2025-03-16T19:05:50.290323Z","shell.execute_reply":"2025-03-16T19:05:55.608464Z"}},"outputs":[],"execution_count":null},{"id":"05e91465","cell_type":"code","source":"sns.boxplot(data['Annual Income'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:55.613724Z","iopub.execute_input":"2025-03-16T19:05:55.61413Z","iopub.status.idle":"2025-03-16T19:05:56.111393Z","shell.execute_reply.started":"2025-03-16T19:05:55.614098Z","shell.execute_reply":"2025-03-16T19:05:56.110297Z"}},"outputs":[],"execution_count":null},{"id":"8388e829","cell_type":"markdown","source":"**Lets go with IQR to find extreme outliers and treat them**","metadata":{}},{"id":"294b67de","cell_type":"code","source":"IQR = data['Annual Income'].quantile(0.75) - data['Annual Income'].quantile(0.25)\nlower_bound = (data['Annual Income'].quantile(0.25)) - (IQR * 1.5)\nupper_bound = (data['Annual Income'].quantile(0.75)) + (IQR * 1.5)\nprint(lower_bound,upper_bound)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:56.113345Z","iopub.execute_input":"2025-03-16T19:05:56.113636Z","iopub.status.idle":"2025-03-16T19:05:56.234564Z","shell.execute_reply.started":"2025-03-16T19:05:56.113611Z","shell.execute_reply":"2025-03-16T19:05:56.233558Z"}},"outputs":[],"execution_count":null},{"id":"66416fcf","cell_type":"code","source":"data[data['Annual Income'] > 99583].shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:56.235582Z","iopub.execute_input":"2025-03-16T19:05:56.236029Z","iopub.status.idle":"2025-03-16T19:05:56.277401Z","shell.execute_reply.started":"2025-03-16T19:05:56.235989Z","shell.execute_reply":"2025-03-16T19:05:56.276339Z"}},"outputs":[],"execution_count":null},{"id":"c1395c7c","cell_type":"code","source":"#for extreme outliers\nlower_bound = (data['Annual Income'].quantile(0.25)) - (IQR * 3)\nupper_bound = (data['Annual Income'].quantile(0.75)) + (IQR * 3)\nprint(lower_bound,upper_bound)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:56.278341Z","iopub.execute_input":"2025-03-16T19:05:56.278642Z","iopub.status.idle":"2025-03-16T19:05:56.341247Z","shell.execute_reply.started":"2025-03-16T19:05:56.278615Z","shell.execute_reply":"2025-03-16T19:05:56.340124Z"}},"outputs":[],"execution_count":null},{"id":"9cb83ba2","cell_type":"code","source":"data[data['Annual Income'] > 99583].shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:56.342113Z","iopub.execute_input":"2025-03-16T19:05:56.342378Z","iopub.status.idle":"2025-03-16T19:05:56.382512Z","shell.execute_reply.started":"2025-03-16T19:05:56.342354Z","shell.execute_reply":"2025-03-16T19:05:56.381627Z"}},"outputs":[],"execution_count":null},{"id":"ee4191d0","cell_type":"markdown","source":"**Result:** Annual Income has 67132 outliers","metadata":{}},{"id":"0e3b4c7b","cell_type":"markdown","source":"**Analying previous claims**","metadata":{}},{"id":"848cce3a","cell_type":"code","source":"sns.distplot(data['Previous Claims'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:05:56.38345Z","iopub.execute_input":"2025-03-16T19:05:56.383831Z","iopub.status.idle":"2025-03-16T19:06:00.081053Z","shell.execute_reply.started":"2025-03-16T19:05:56.383794Z","shell.execute_reply":"2025-03-16T19:06:00.080034Z"}},"outputs":[],"execution_count":null},{"id":"a45950f1","cell_type":"code","source":"sns.boxplot(data['Previous Claims'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:06:00.081777Z","iopub.execute_input":"2025-03-16T19:06:00.082095Z","iopub.status.idle":"2025-03-16T19:06:00.25927Z","shell.execute_reply.started":"2025-03-16T19:06:00.082066Z","shell.execute_reply":"2025-03-16T19:06:00.258059Z"}},"outputs":[],"execution_count":null},{"id":"6e238b2f","cell_type":"code","source":"IQR = data['Previous Claims'].quantile(0.75) - data['Previous Claims'].quantile(0.25)\nlower_bound = (data['Previous Claims'].quantile(0.25)) - (IQR * 1.5)\nupper_bound = (data['Previous Claims'].quantile(0.75)) + (IQR * 1.5)\nprint(lower_bound,upper_bound)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:06:00.260108Z","iopub.execute_input":"2025-03-16T19:06:00.260528Z","iopub.status.idle":"2025-03-16T19:06:00.382408Z","shell.execute_reply.started":"2025-03-16T19:06:00.260495Z","shell.execute_reply":"2025-03-16T19:06:00.381371Z"}},"outputs":[],"execution_count":null},{"id":"b029eb77","cell_type":"code","source":"data[data['Previous Claims'] > 5].shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:06:00.383387Z","iopub.execute_input":"2025-03-16T19:06:00.383735Z","iopub.status.idle":"2025-03-16T19:06:00.392945Z","shell.execute_reply.started":"2025-03-16T19:06:00.383697Z","shell.execute_reply":"2025-03-16T19:06:00.39194Z"}},"outputs":[],"execution_count":null},{"id":"157b2660","cell_type":"markdown","source":"**Result:**\nPrevious claims has 369 outliers","metadata":{}},{"id":"03d12f45","cell_type":"markdown","source":"**Analyzing Premium amount**","metadata":{}},{"id":"f1d03218","cell_type":"code","source":"sns.distplot(data['Premium Amount'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:06:00.393929Z","iopub.execute_input":"2025-03-16T19:06:00.394273Z","iopub.status.idle":"2025-03-16T19:06:05.605982Z","shell.execute_reply.started":"2025-03-16T19:06:00.394239Z","shell.execute_reply":"2025-03-16T19:06:05.604767Z"}},"outputs":[],"execution_count":null},{"id":"14508a99","cell_type":"code","source":"sns.boxplot(data['Premium Amount'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:06:05.607081Z","iopub.execute_input":"2025-03-16T19:06:05.607465Z","iopub.status.idle":"2025-03-16T19:06:06.016173Z","shell.execute_reply.started":"2025-03-16T19:06:05.607435Z","shell.execute_reply":"2025-03-16T19:06:06.015095Z"}},"outputs":[],"execution_count":null},{"id":"4f6555a2","cell_type":"code","source":"IQR = data['Premium Amount'].quantile(0.75) - data['Premium Amount'].quantile(0.25)\nlower_bound = (data['Premium Amount'].quantile(0.25)) - (IQR * 1.5)\nupper_bound = (data['Premium Amount'].quantile(0.75)) + (IQR * 1.5)\nprint(lower_bound,upper_bound)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:06:06.017534Z","iopub.execute_input":"2025-03-16T19:06:06.017853Z","iopub.status.idle":"2025-03-16T19:06:06.117247Z","shell.execute_reply.started":"2025-03-16T19:06:06.017827Z","shell.execute_reply":"2025-03-16T19:06:06.115782Z"}},"outputs":[],"execution_count":null},{"id":"08a434c7","cell_type":"code","source":"data[data['Premium Amount'] > 3001.5].shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:06:06.118147Z","iopub.execute_input":"2025-03-16T19:06:06.11842Z","iopub.status.idle":"2025-03-16T19:06:06.160675Z","shell.execute_reply.started":"2025-03-16T19:06:06.118396Z","shell.execute_reply":"2025-03-16T19:06:06.159632Z"}},"outputs":[],"execution_count":null},{"id":"761a933b","cell_type":"markdown","source":"**Result:** Premium amount has 49320 outliers\n    ","metadata":{}},{"id":"3a47ce96","cell_type":"markdown","source":"**for Categorical features**","metadata":{}},{"id":"c7944b78","cell_type":"markdown","source":"we check for rare categories and mark it as others","metadata":{}},{"id":"c67ea305","cell_type":"code","source":"for col in categorical_features:\n    if col != 'Policy Start Date':\n        data[col].value_counts().plot(kind='pie',autopct=\"%1.1f%%\",figsize=(3,3))\n        plt.title(col)\n        plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-16T19:06:06.161575Z","iopub.execute_input":"2025-03-16T19:06:06.161869Z","iopub.status.idle":"2025-03-16T19:06:07.987916Z","shell.execute_reply.started":"2025-03-16T19:06:06.161842Z","shell.execute_reply":"2025-03-16T19:06:07.985747Z"}},"outputs":[],"execution_count":null},{"id":"60f3a8cc","cell_type":"markdown","source":"**Result:** In our categorical features, all the categories are evenly distributed. So let's leave it","metadata":{}}]}