{"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.19"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"51f9dbdc-d373-49b5-8642-697f14c2ce55","cell_type":"markdown","source":"### Introduction\nThis report provides a comprehensive exploratory data analysis (EDA) of an insurance dataset, focusing on understanding its structure, identifying trends, addressing missing values, and examining relationships among variables. The training dataset contains 1.2 million entries and 21 features, including both numerical and categorical data. The target variable, Premium Amount, represents the insurance premium to be predicted. By analyzing distributions, feature interactions, and potential challenges, the report aims to guide effective preprocessing, feature engineering, and predictive modeling.","metadata":{}},{"id":"3ebd642e-ea3d-4ec8-8778-ed9aa0e73d76","cell_type":"markdown","source":"### Key Findings\n\n#### Missing Data:\n- Significant missing data in features such as Annual Income, Occupation, and Credit Score suggests the need for imputation or removal of rows/columns with substantial missing values.\n- Categorical features like Marital Status also show missing values, requiring appropriate handling.\n\n#### Distribution of Target Variable (Premium Amount):\n- Most premiums are below 2,000, with a small proportion of high-value outliers.\n- A highly skewed distribution suggests the need for potential transformations during modeling.\n\n#### Feature Relationships:\n- A positive but variable correlation exists between Annual Income and Premium Amount.\n- Features such as Property Type, Health Score, and Vehicle Age also show meaningful relationships with the target variable.\n\n#### Categorical Trends:\n- Gender distribution is balanced.\n- Property Type is skewed towards House ownership.\n- Education Level is dominated by individuals with a Master's degree.","metadata":{}},{"id":"93dc2d80-a48f-4611-889c-8804521b88d4","cell_type":"code","source":"# Import necessary libraries\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Configure seaborn\nsns.set(style=\"whitegrid\")","metadata":{},"outputs":[],"execution_count":null},{"id":"9ebae0b4-1493-44c2-8145-53e596e813a4","cell_type":"code","source":"# Load datasets\ntrain_file_path = \"/kaggle/input/playground-series-s4e12/train.csv\"  \ntest_file_path = \"/kaggle/input/playground-series-s4e12/test.csv\"\n\ntrain_data = pd.read_csv(train_file_path)\ntest_data = pd.read_csv(test_file_path)\n\n# Display the first few rows\ntrain_data.head()","metadata":{},"outputs":[],"execution_count":null},{"id":"69319599-15b5-433f-8001-43face52c8af","cell_type":"markdown","source":"### 1. Basic Information and Overview","metadata":{}},{"id":"4acec21c-b339-4a47-9144-11a65e4f1dba","cell_type":"code","source":"test_data.head()","metadata":{},"outputs":[],"execution_count":null},{"id":"584bc26e-c0fe-4ed7-ac95-0671c12a3e3d","cell_type":"code","source":"# Display basic information about the dataset\ntrain_data.info()","metadata":{},"outputs":[],"execution_count":null},{"id":"4f1b8389-a4cd-4d42-9cea-1b5d82dd5ba1","cell_type":"code","source":"test_data.info()","metadata":{},"outputs":[],"execution_count":null},{"id":"b5097ed3-8c8d-44cd-9334-e3c175abcca8","cell_type":"code","source":"# Descriptive statistics\ntrain_data.describe(include = 'all')","metadata":{},"outputs":[],"execution_count":null},{"id":"bf11a3a2-e536-4131-afa0-2a094aeeefeb","cell_type":"code","source":"test_data.describe(include = 'all')","metadata":{},"outputs":[],"execution_count":null},{"id":"5f78f4ff-b817-4253-ad70-5ae0ead6baf9","cell_type":"code","source":"# Check for missing values\ntrain_missing_values = train_data.isnull().sum()\ntrain_missing_values","metadata":{},"outputs":[],"execution_count":null},{"id":"44ad1800-b4ea-4827-a919-e08e22aa1685","cell_type":"code","source":"# Check for missing values\ntest_missing_values = test_data.isnull().sum()\ntest_missing_values","metadata":{},"outputs":[],"execution_count":null},{"id":"868aad7b-562f-445d-b9e4-5c6e917e7e88","cell_type":"code","source":"train_data.nunique()","metadata":{},"outputs":[],"execution_count":null},{"id":"b7731d44-84b9-4c16-8251-8af82fc77285","cell_type":"code","source":"test_data.nunique()","metadata":{},"outputs":[],"execution_count":null},{"id":"e6e844bb-8a26-48fe-a9cc-b5a89de9eef0","cell_type":"code","source":"train_data['Policy Start Date']","metadata":{},"outputs":[],"execution_count":null},{"id":"676411fb-7ca3-43a8-a912-200ae5f0349d","cell_type":"code","source":"# Convert to datetime and extract the year\ntrain_data['Policy Year'] = pd.to_datetime(train_data['Policy Start Date'], errors='coerce').dt.year\ntrain_data['Policy Year']","metadata":{},"outputs":[],"execution_count":null},{"id":"20ec7d01-84be-4e06-bf1a-533ee413cfcd","cell_type":"code","source":"train_data['Policy Year'].nunique()","metadata":{},"outputs":[],"execution_count":null},{"id":"be4612eb-1de5-4ec4-8638-c00beb6551aa","cell_type":"code","source":"# Convert to datetime and extract the year\ntrain_data['Policy Month'] = pd.to_datetime(train_data['Policy Start Date'], errors='coerce').dt.month\ntrain_data['Policy Month']","metadata":{},"outputs":[],"execution_count":null},{"id":"f11822c3-a3c0-4cf3-88bb-dedd40861555","cell_type":"code","source":"train_data['Policy Month'].nunique()","metadata":{},"outputs":[],"execution_count":null},{"id":"d16e38ed-0bb2-49ca-94d7-a1a2855ae40a","cell_type":"code","source":"test_data['Policy Year'] = pd.to_datetime(test_data['Policy Start Date'], errors='coerce').dt.year\ntest_data['Policy Year']","metadata":{},"outputs":[],"execution_count":null},{"id":"75dea00e-3cd3-44ef-8d6a-a1ee259d2ca2","cell_type":"code","source":"train_data['Policy Year'].nunique()","metadata":{},"outputs":[],"execution_count":null},{"id":"c4208d35-ef4a-4297-b350-7f094eb23d84","cell_type":"code","source":"test_data['Policy Month'] = pd.to_datetime(test_data['Policy Start Date'], errors='coerce').dt.month\ntest_data['Policy Month']","metadata":{},"outputs":[],"execution_count":null},{"id":"592b1771-4aac-4ef7-a670-b428b4d7c26a","cell_type":"code","source":"train_data['Policy Month'].nunique()","metadata":{},"outputs":[],"execution_count":null},{"id":"5fbb90fd-a40a-4fb2-9bf3-c16621bb4e72","cell_type":"markdown","source":"##### Dataset Overview\n- The training dataset contains 1.2 million rows and 21 columns. Key columns include:\n    - Numerical features: Age, Annual Income, Health Score, Vehicle Age, Credit Score, and Premium Amount.\n    - Categorical features: Gender, Marital Status, Education Level, Occupation, Property Type.\n- The test dataset has the same structure but lacks the Premium Amount target variable.\n- The dataset has some data quality issues, including missing values and outliers.","metadata":{}},{"id":"27bcd88f-2e31-4715-a188-0dc2df71ad7b","cell_type":"markdown","source":"### 2. Target Variable Analysis","metadata":{}},{"id":"0df967b4-9c38-4c74-8f42-135393526dae","cell_type":"code","source":"# Analyze the target variable ('Premium Amount')\nplt.figure(figsize=(10, 6))\nsns.histplot(train_data['Premium Amount'], kde=True, bins=30, color='blue')\nplt.title('Distribution of Premium Amount')\nplt.xlabel('Premium Amount')\nplt.ylabel('Frequency')\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"f0a205ee-8727-4f45-9608-543e40d9f6c6","cell_type":"markdown","source":"##### Target Variable: Premium Amount\n###### Distribution Analysis:\n- Mean: ₹1,102, Standard Deviation: ₹865.\n- Minimum: ₹20, Maximum: ₹4,999.\n- The target variable is heavily skewed, with most values below ₹2,000 and a few high outliers.\n- This skewness might require normalization or transformation during modeling.\n###### Outlier Detection:\n- Identified extreme outliers in Premium Amount, which might affect model performance if not addressed.","metadata":{}},{"id":"5dcde329-8fdf-4d30-a112-0a4e58452606","cell_type":"markdown","source":"### 3. Numerical Feature Analysis","metadata":{}},{"id":"19d7ec58-03eb-4f3e-9a53-f26417360635","cell_type":"code","source":"# Select numerical features\nnumerical_features = train_data.select_dtypes(include=['float64', 'int64']).columns\n\n# Plot distributions of numerical features\nfor feature in numerical_features:\n    plt.figure(figsize=(5, 3))\n    sns.histplot(train_data[feature], kde=True, bins=30)\n    plt.title(f'Distribution of {feature} in Train Dataset')\n    plt.xlabel(feature)\n    plt.ylabel('Frequency')\n    plt.show()","metadata":{"scrolled":true},"outputs":[],"execution_count":null},{"id":"704cb529-8b60-4b74-8596-32e251644071","cell_type":"code","source":"# Select numerical features\nnumerical_features_test = test_data.select_dtypes(include=['float64', 'int64']).columns\n\n# Plot distributions of numerical features\nfor feature in numerical_features_test:\n    plt.figure(figsize=(5, 3))\n    sns.histplot(test_data[feature], kde=True, bins=30)\n    plt.title(f'Distribution of {feature} in Test Dataset')\n    plt.xlabel(feature)\n    plt.ylabel('Frequency')\n    plt.show()","metadata":{"scrolled":true},"outputs":[],"execution_count":null},{"id":"39c8a9de-cfeb-4364-8450-01325af30985","cell_type":"markdown","source":"##### Numerical Features\n###### Age:\n- Mean: ~41 years, Minimum: 18, Maximum: 64.\n- Uniformly distributed with a peak in the 30–40 age range.\n###### Annual Income:\n- Mean: 32,745, Standard Deviation: 32,179.\n- Wide income range indicates substantial variability among customers.\n- Positive correlation with Premium Amount but with significant noise.\n###### Health Score:\n- Ranges from ~2 to ~59, with a mean of ~25.\n- Shows potential impact on premium calculation, reflecting customer health conditions.","metadata":{}},{"id":"c3120e1f-7cdd-444f-9117-8c2d09e58346","cell_type":"markdown","source":"### 4. Categorical Feature Analysis","metadata":{}},{"id":"7742c5c4-d037-4059-847b-c8453ec79079","cell_type":"code","source":"# Select categorical features\ncategorical_features = train_data.select_dtypes(include=['object']).columns\ncategorical_features = categorical_features.drop('Policy Start Date', errors='ignore')\n\n# Plot count plots for categorical features\nfor feature in categorical_features:\n    plt.figure(figsize=(5, 3))\n    sns.countplot(data=train_data, x=feature, palette='viridis')\n    plt.title(f'Count Plot of {feature} in Train Dataset')\n    plt.xlabel(feature)\n    plt.ylabel('Count')\n    plt.xticks(rotation=45)\n    plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"1efd58bf-95d9-411a-b4ed-d0ae8951722e","cell_type":"code","source":"# Select categorical features\ncategorical_features_test = test_data.select_dtypes(include=['object']).columns\ncategorical_features_test = categorical_features_test.drop('Policy Start Date', errors='ignore')\n\n# Plot count plots for categorical features\nfor feature in categorical_features:\n    plt.figure(figsize=(5, 3))\n    sns.countplot(data=test_data, x=feature, palette='viridis')\n    plt.title(f'Count Plot of {feature} in Test Dataset')\n    plt.xlabel(feature)\n    plt.ylabel('Count')\n    plt.xticks(rotation=45)\n    plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"00f04a44-e846-483b-a171-9c17c5bd837f","cell_type":"markdown","source":"##### Categorical Features\n###### Gender:\n- Balanced distribution: ~50% Male, ~50% Female.\n###### Marital Status:\n- Most customers are Single, followed by Married and Divorced.\n###### Education Level:\n- Predominantly Master's degree holders, followed by Bachelor's, PhD, and High School.\n###### Property Type:\n- Most common type is House, followed by Apartment and Condo.","metadata":{}},{"id":"3ba1011e-0773-46b5-8176-d3695d696ccb","cell_type":"markdown","source":"### 5. Missing Values Visualisation","metadata":{}},{"id":"c31b0735-5293-422e-a0bd-562d583acf30","cell_type":"code","source":"# Visualize missing data\nplt.figure(figsize=(10, 6))\nsns.heatmap(train_data.isnull(), cbar=False, cmap='viridis')\nplt.title('Missing Values Heatmap (Train Dataset)')\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"247a619f-0cec-4282-8392-e90c41ca9c72","cell_type":"code","source":"# Visualize missing data\nplt.figure(figsize=(10, 6))\nsns.heatmap(test_data.isnull(), cbar=False, cmap='viridis')\nplt.title('Missing Values Heatmap (Test Dataset)')\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"0d77dd0e-5176-4b12-8ca6-f3fc5630c1b2","cell_type":"markdown","source":"##### Missing Data\n###### Columns with missing values:\n- Annual Income (~47,949 missing values).\n- Occupation (~358,075 missing values).\n- Credit Score (~137,882 missing values).\n- Marital Status and other categorical features also have missing data.\n- Missing data visualization (e.g., heatmaps) revealed patterns suggesting that some features may have systematic missingness.","metadata":{}},{"id":"83c8a749-b528-4edb-aaaa-5776ae37c479","cell_type":"markdown","source":"### 6. Outlier Detection","metadata":{}},{"id":"20f62c5b-db1b-4d14-90b7-c16b239ce63c","cell_type":"code","source":"# Boxplots for outlier detection in numerical features\nfor feature in numerical_features:\n    plt.figure(figsize=(5, 3))\n    sns.boxplot(data=train_data, y=feature)\n    plt.title(f'Boxplot of {feature} in Train Dataset')\n    plt.ylabel(feature)\n    plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"70b1b4e2-366b-4bce-aa9c-85f749f055a6","cell_type":"code","source":"# Boxplots for outlier detection in numerical features\nfor feature in numerical_features_test:\n    plt.figure(figsize=(5, 3))\n    sns.boxplot(data=test_data, y=feature)\n    plt.title(f'Boxplot of {feature} in Test Dataset')\n    plt.ylabel(feature)\n    plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"be8040a9-637a-4299-8396-adafde2acfba","cell_type":"markdown","source":"### 7. Correlation Heatmap","metadata":{}},{"id":"abcd91f0-0ee5-42ce-ba80-f4feb90726b4","cell_type":"code","source":"# Select numeric columns from the data\nnumerical_data = train_data[numerical_features]\n\n# Compute the correlation matrix\ncorr_matrix = numerical_data.corr()\n\n# Plot the heatmap\nplt.figure(figsize=(10, 6))\nsns.heatmap(corr_matrix, annot=True, fmt=\".2f\", cmap='coolwarm', cbar=True)\nplt.title('Correlation Heatmap Train Dataset')\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"d7c9c74c-d577-43f9-894d-5042c1f895c0","cell_type":"code","source":"# Select numerical features in test dataset (common columns with train data)\nnumerical_features_test = [col for col in numerical_features if col in test_data.columns]\n\n# Extract the numerical data from the test dataset\nnumerical_data_test = test_data[numerical_features_test]\n\n# Compute the correlation matrix\ncorr_matrix_test = numerical_data_test.corr()\n\n# Plot the heatmap\nplt.figure(figsize=(10, 6))\nsns.heatmap(corr_matrix_test, annot=True, fmt=\".2f\", cmap='coolwarm', cbar=True)\nplt.title('Correlation Heatmap (Test Data)')\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"a41d1110-1c05-44cc-8e53-fb755aa83fa3","cell_type":"markdown","source":"##### Correlation Heatmap\n###### High Correlations:\n- Annual Income and Credit Score exhibit some positive correlation with Premium Amount.\n- Health Score shows a moderate negative correlation with the target variable.\n###### Weak Correlations:\n- Several features like Vehicle Age and Number of Dependents have minimal correlation with the target variable.","metadata":{}},{"id":"bdad1eb8-8865-484e-b286-9da32283bbd9","cell_type":"markdown","source":"### 8. Paiplot for Numerical Features","metadata":{}},{"id":"dc350e05-b62e-4988-8aa9-947f6d8c86c2","cell_type":"code","source":"# Pairplot for numerical features\nsns.pairplot(train_data[numerical_features], diag_kind='kde', corner=True)\nplt.title('Pairplot for Train Dataset')\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"a7b0eaaf-d7bd-409c-b0ce-9d16b34677c9","cell_type":"code","source":"# Pairplot for numerical features\nsns.pairplot(test_data[numerical_features_test], diag_kind='kde', corner=True)\nplt.title('Pairplot for Test Dataset')\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"621d41e7-3dda-4640-8454-4ac25fe926a9","cell_type":"markdown","source":"### 9. Feature Relationship with Target","metadata":{}},{"id":"ab10d2d5-6b7e-4726-9e96-0f5cf9c85f7b","cell_type":"code","source":"# Analyze relationships between features and the target variable\nfor feature in numerical_features:\n    if feature != 'Premium Amount':  # Skip target\n        plt.figure(figsize=(5, 3))\n        sns.scatterplot(data=train_data, x=feature, y='Premium Amount', alpha=0.6)\n        plt.title(f'{feature} vs Premium Amount')\n        plt.xlabel(feature)\n        plt.ylabel('Premium Amount')\n        plt.show()\n\nfor feature in categorical_features:\n    plt.figure(figsize=(5, 3))\n    sns.boxplot(data=train_data, x=feature, y='Premium Amount')\n    plt.title(f'{feature} vs Premium Amount')\n    plt.xlabel(feature)\n    plt.ylabel('Premium Amount')\n    plt.xticks(rotation=45)\n    plt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"2ea74528-8345-4ec5-bd92-181ddaa48f44","cell_type":"markdown","source":"##### Relationships Between Features\n###### Annual Income vs. Premium Amount:\n- Positive relationship, indicating that higher income generally leads to higher premiums.\n- However, significant spread suggests the influence of additional factors like property type and health score.\n###### Property Type vs. Premium Amount:\n- Customers with House properties tend to pay higher premiums than those with Apartment or Condo.\n###### Vehicle Age vs. Premium Amount:\n- Older vehicles generally correspond to lower premiums, indicating possible discounts or less coverage.\n###### Health Score vs. Premium Amount:\n- Customers with higher health scores are associated with lower premiums, reflecting the risk assessment by insurers.","metadata":{}},{"id":"b1235bae-219b-4ab9-a0b9-ccb04d25c2a8","cell_type":"markdown","source":"#### Conclusion\nThe exploratory data analysis provides several actionable insights:\n\n##### Data Quality Issues:\n- Missing values in key features like Annual Income, Occupation, and Credit Score must be addressed to avoid biases.\n- Outliers in numerical features, especially Premium Amount, should be handled cautiously.\n##### Key Drivers of Premium Amount:\n- Features like Annual Income, Property Type, Health Score, and Marital Status significantly impact premium calculations.\n- Positive but noisy relationships indicate the potential need for feature interactions or engineered features.\n##### Next Steps:\n- Impute missing values using statistical methods or machine learning techniques.\n- Normalize or transform skewed features like Premium Amount for improved modeling performance.\n- Consider interaction terms and additional transformations to capture complex relationships.\n\nThis analysis establishes a solid foundation for data preprocessing and predictive modeling. These findings can be leveraged to design a robust machine learning pipeline to predict insurance premiums effectively.","metadata":{}},{"id":"ce558123-c30c-4d91-98ac-b312c385d07f","cell_type":"code","source":"","metadata":{},"outputs":[],"execution_count":null}]}