{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 💰 Regression with an Insurance Dataset","metadata":{}},{"cell_type":"markdown","source":"## 🎯 Objective\n\nThis notebook aims to analyze a dataset on insurance premiums with the goal of building a regression model to predict premium amounts. Insurance premiums, which depend on factors such as demographics, health indicators, lifestyle choices, and claims history, are critical for risk assessment and policy pricing. This study seeks to understand the relationships between these factors and premium amounts, providing insights to improve risk prediction and optimize insurance models.\n\n\"Why seek insurance when you can win big\"\n![](https://static01.nyt.com/images/2023/12/07/multimedia/07squid-game-highs-lows2-zvwf/07squid-game-highs-lows2-zvwf-superJumbo.jpg)\nSource: Squid Games | Netflix","metadata":{}},{"cell_type":"markdown","source":"## 1. 📝 Introduction\n\nInsurance premiums are a critical aspect of the insurance industry, directly impacting both individuals and companies. These premiums, which vary based on numerous factors such as demographics, health status, lifestyle, and previous claims, determine the cost of insurance policies. Understanding the factors that influence premium amounts can lead to more accurate pricing models and better risk management strategies.\n\nIn this project, we will leverage a dataset containing various features related to personal and health information to predict the target variable, Premium Amount. This problem will be approached as a regression task, where we aim to predict the continuous premium value based on these influencing factors.\n\nProblem Statement: The goal is to predict the Premium Amount, a continuous variable, based on the input features provided in the dataset. This will involve using machine learning models to understand the relationships between the input variables and the premium amounts, ultimately minimizing the Root Mean Squared Logarithmic Error (RMSLE) to improve prediction accuracy.","metadata":{}},{"cell_type":"code","source":"import pandas\nimport numpy\nimport os\n\nimport catboost\nimport lightgbm\nimport plotly.express\nimport plotly.graph_objects\nimport sklearn.preprocessing\nimport sklearn.ensemble\nimport sklearn.linear_model\nimport xgboost\n\nDATA_DIR = \"/kaggle/input/playground-series-s4e12\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:10.152249Z","iopub.execute_input":"2024-12-29T15:42:10.152562Z","iopub.status.idle":"2024-12-29T15:42:15.286931Z","shell.execute_reply.started":"2024-12-29T15:42:10.15252Z","shell.execute_reply":"2024-12-29T15:42:15.285795Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. 📊 Data Overview","metadata":{}},{"cell_type":"code","source":"TRAINING_DATAFRAME = pandas.read_csv(os.path.join(DATA_DIR, \"train.csv\"))\nTESTING_DATAFRAME  = pandas.read_csv(os.path.join(DATA_DIR, \"test.csv\"))\n\nTRAINING_DATAFRAME.columns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:15.289881Z","iopub.execute_input":"2024-12-29T15:42:15.290729Z","iopub.status.idle":"2024-12-29T15:42:26.368979Z","shell.execute_reply.started":"2024-12-29T15:42:15.290672Z","shell.execute_reply":"2024-12-29T15:42:26.367842Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The dataset includes the following features:\n\n🧍 Demographic Information: `Age`, `Gender`, `Annual Income`, `Marital Status`, `Number of Dependents`, `Education Level`, `Occupation`, `Location`, `Property Type`.\n\n💼 Financial and Risk Factors: `Credit Score`, `Previous Claims`, `Insurance Duration`.\n\n💪 Health and Lifestyle Information: `Health Score`, `Smoking Status`, `Exercise Frequency`.\n\n📑 Policy and Feedback Information: `Policy Type`, `Policy Start Date`, `Customer Feedback`.\n\nOur target variable is `Premium Amount`, which we will predict based on these features.","metadata":{}},{"cell_type":"code","source":"TRAINING_DATAFRAME.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:26.370096Z","iopub.execute_input":"2024-12-29T15:42:26.370441Z","iopub.status.idle":"2024-12-29T15:42:26.412217Z","shell.execute_reply.started":"2024-12-29T15:42:26.37041Z","shell.execute_reply":"2024-12-29T15:42:26.411017Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"TRAINING_DATAFRAME.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:26.413628Z","iopub.execute_input":"2024-12-29T15:42:26.41399Z","iopub.status.idle":"2024-12-29T15:42:27.134712Z","shell.execute_reply.started":"2024-12-29T15:42:26.413956Z","shell.execute_reply":"2024-12-29T15:42:27.133514Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"TRAINING_DATAFRAME.describe(include='object')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:27.136089Z","iopub.execute_input":"2024-12-29T15:42:27.136513Z","iopub.status.idle":"2024-12-29T15:42:29.362023Z","shell.execute_reply.started":"2024-12-29T15:42:27.136477Z","shell.execute_reply":"2024-12-29T15:42:29.360849Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. 🚀 Project Workflow\n\n### 3.1 🧹 Data Preprocessing: Clean and preprocess the data, handle missing values, encode categorical features, and scale numerical features as necessary.","metadata":{}},{"cell_type":"code","source":"TRAINING_DATAFRAME.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:29.36384Z","iopub.execute_input":"2024-12-29T15:42:29.364244Z","iopub.status.idle":"2024-12-29T15:42:30.018565Z","shell.execute_reply.started":"2024-12-29T15:42:29.364205Z","shell.execute_reply":"2024-12-29T15:42:30.017264Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"TRAINING_DATAFRAME.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:30.023139Z","iopub.execute_input":"2024-12-29T15:42:30.023555Z","iopub.status.idle":"2024-12-29T15:42:30.66302Z","shell.execute_reply.started":"2024-12-29T15:42:30.023518Z","shell.execute_reply":"2024-12-29T15:42:30.661822Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pipeline = [\n    # (\"Column\", function, \"Column Name Append\")\n]\n\ndef run_pipeline(pipeline, dataframe, drop_updated_columns = True, verbose = True):\n    columns_to_clean = set()\n    for column_name, operation, column_name_append in pipeline:\n        if verbose:\n            print(f\"Applying operation: {operation.__name__} to {column_name}\")\n        dataframe[f\"{column_name}_{column_name_append}\"] = operation(dataframe[column_name])\n        columns_to_clean.add(column_name)\n\n    if drop_updated_columns:\n        if verbose:\n            print(f\"Dropping original columns: {list(columns_to_clean)}\")\n        dataframe = dataframe.drop(columns=list(columns_to_clean))\n\n    return dataframe","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:30.6643Z","iopub.execute_input":"2024-12-29T15:42:30.664651Z","iopub.status.idle":"2024-12-29T15:42:30.671727Z","shell.execute_reply.started":"2024-12-29T15:42:30.664616Z","shell.execute_reply":"2024-12-29T15:42:30.670496Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let us fix `Age` first. But before deciding what to do, let's see the data distribution of `Age`.","metadata":{}},{"cell_type":"code","source":"plotly.express.bar(\n    TRAINING_DATAFRAME[\"Age\"].value_counts().reset_index().sort_values(by = \"Age\"),\n    x = \"Age\",\n    y = \"count\",\n    labels = {'index': 'Category', 'Category': 'Frequency'},\n    title = \"Distribution of Age\",\n    template = \"plotly_dark\").show(renderer = 'iframe')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:30.673002Z","iopub.execute_input":"2024-12-29T15:42:30.67341Z","iopub.status.idle":"2024-12-29T15:42:32.707603Z","shell.execute_reply.started":"2024-12-29T15:42:30.673374Z","shell.execute_reply":"2024-12-29T15:42:32.706435Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"An interesting problem involves addressing 18,000 missing age values in a dataset where the data is uniformly distributed. Three potential strategies to resolve this issue have been identified:\n\n- Replacing the missing age values with the median age.\n- Randomly sampling age values from the uniform distribution.\n- Imputing missing age values using predictions from a model.\n  \nThe focus will initially be on exploring the first strategy.","metadata":{}},{"cell_type":"code","source":"def __impute_column_with_median(column):\n    assert isinstance(column, pandas.Series)\n\n    median = column.median(skipna = True)\n    return column.fillna(median)\n\ndef __impute_column_with_random_sample_from_distribution(column):\n    assert isinstance(column, pandas.Series)\n    observed_values = column.dropna()\n    assert not observed_values.empty\n    return column.apply(lambda x: numpy.random.choice(observed_values) if pandas.isna(x) else x)\n\npipeline.append((\"Age\", __impute_column_with_median, \"MEDIAN\"))\npipeline.append((\"Age\", __impute_column_with_random_sample_from_distribution, \"RANDOM\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:32.709001Z","iopub.execute_input":"2024-12-29T15:42:32.709451Z","iopub.status.idle":"2024-12-29T15:42:32.716864Z","shell.execute_reply.started":"2024-12-29T15:42:32.709407Z","shell.execute_reply":"2024-12-29T15:42:32.715656Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's understand how to approach missing values of `Annual Income` by analyzing the trend.","metadata":{}},{"cell_type":"code","source":"plotly.express.histogram(\n    TRAINING_DATAFRAME,\n    x=\"Annual Income\",\n    nbins=50, \n    labels={'Annual_Salary': 'Annual Salary', 'count': 'Frequency'},\n    title=\"Histogram of Annual Salaries\",\n    template=\"plotly_dark\"\n).show(renderer = \"iframe\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:32.718526Z","iopub.execute_input":"2024-12-29T15:42:32.719136Z","iopub.status.idle":"2024-12-29T15:42:33.007288Z","shell.execute_reply.started":"2024-12-29T15:42:32.7191Z","shell.execute_reply":"2024-12-29T15:42:33.006129Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"There is a right skew to this data. Instead of using the `Median` to replace the missing values, it might be better to use `Q1` quartile. ","metadata":{}},{"cell_type":"code","source":"def __impute_column_with_q1(column):\n    assert isinstance(column, pandas.Series)\n\n    q1 = column.quantile(0.25)\n    return column.fillna(q1)\n\npipeline.append((\"Annual Income\", __impute_column_with_q1, \"Q1\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:33.008589Z","iopub.execute_input":"2024-12-29T15:42:33.008947Z","iopub.status.idle":"2024-12-29T15:42:33.015236Z","shell.execute_reply.started":"2024-12-29T15:42:33.008912Z","shell.execute_reply":"2024-12-29T15:42:33.013524Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's now analyze `Marital Status`.","metadata":{}},{"cell_type":"code","source":"plotly.express.bar(\n    TRAINING_DATAFRAME[\"Marital Status\"].value_counts().reset_index(),\n    x = \"Marital Status\", \n    y = \"count\", \n    labels={'index': 'Marital Status', 'Marital Status': 'Frequency'},\n    title=\"Distribution of Marital Status\",\n    template=\"plotly_dark\"\n).show(renderer = \"iframe\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:33.016735Z","iopub.execute_input":"2024-12-29T15:42:33.017204Z","iopub.status.idle":"2024-12-29T15:42:33.306667Z","shell.execute_reply.started":"2024-12-29T15:42:33.017134Z","shell.execute_reply":"2024-12-29T15:42:33.305485Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"And again, another uniform distribution. Let's replace the missing value with `Unknown` token.","metadata":{}},{"cell_type":"code","source":"def __impute_column_with_mode(column):\n    assert isinstance(column, pandas.Series)\n    mode_value = column.mode().iloc[0]\n    return column.fillna(mode_value)\n\ndef __impute_column_with_value(column, value = \"UNKNOWN\"):\n    assert isinstance(column, pandas.Series)\n    return column.fillna(value)\n\npipeline.append((\"Marital Status\", __impute_column_with_mode, \"MODE\"))\npipeline.append((\"Marital Status\", __impute_column_with_value, \"W_VALUE\"))\npipeline.append((\"Marital Status\", __impute_column_with_random_sample_from_distribution, \"RANDOM\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:33.308074Z","iopub.execute_input":"2024-12-29T15:42:33.309074Z","iopub.status.idle":"2024-12-29T15:42:33.315386Z","shell.execute_reply.started":"2024-12-29T15:42:33.309035Z","shell.execute_reply":"2024-12-29T15:42:33.314095Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's hope the next one isn't another one of uniform distribution.","metadata":{}},{"cell_type":"code","source":"plotly.express.bar(\n    TRAINING_DATAFRAME[\"Number of Dependents\"].value_counts().reset_index(),\n    x = \"Number of Dependents\", \n    y = \"count\", \n    labels={'index': 'Number of Dependents', 'Number of Dependents': 'Frequency'},\n    title=\"Distribution of Number of Dependents\",\n    template=\"plotly_dark\"\n).show(renderer = \"iframe\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:33.316747Z","iopub.execute_input":"2024-12-29T15:42:33.31712Z","iopub.status.idle":"2024-12-29T15:42:33.407108Z","shell.execute_reply.started":"2024-12-29T15:42:33.317082Z","shell.execute_reply":"2024-12-29T15:42:33.405963Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Oh well... 🤦","metadata":{}},{"cell_type":"code","source":"pipeline.append((\"Number of Dependents\", __impute_column_with_median, \"MEDIAN\"))\npipeline.append((\"Number of Dependents\", __impute_column_with_random_sample_from_distribution, \"RANDOM\"))\npipeline.append((\"Number of Dependents\", __impute_column_with_value, \"W_VALUE\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:33.409026Z","iopub.execute_input":"2024-12-29T15:42:33.409487Z","iopub.status.idle":"2024-12-29T15:42:33.415673Z","shell.execute_reply.started":"2024-12-29T15:42:33.409437Z","shell.execute_reply":"2024-12-29T15:42:33.414494Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Time to fix `Occupation`.","metadata":{}},{"cell_type":"code","source":"plotly.express.bar(\n    TRAINING_DATAFRAME[\"Occupation\"].value_counts().reset_index(),\n    x = \"Occupation\", \n    y = \"count\", \n    labels={'index': 'Occupation', 'Occupation': 'Frequency'},\n    title=\"Distribution of Occupation\",\n    template=\"plotly_dark\"\n).show(renderer = \"iframe\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:33.41724Z","iopub.execute_input":"2024-12-29T15:42:33.417694Z","iopub.status.idle":"2024-12-29T15:42:33.567552Z","shell.execute_reply.started":"2024-12-29T15:42:33.417647Z","shell.execute_reply":"2024-12-29T15:42:33.566386Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pipeline.append((\"Occupation\", __impute_column_with_mode, \"MODE\"))\npipeline.append((\"Occupation\", __impute_column_with_value, \"W_VALUE\"))\npipeline.append((\"Occupation\", __impute_column_with_random_sample_from_distribution, \"RANDOM\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:33.568854Z","iopub.execute_input":"2024-12-29T15:42:33.569201Z","iopub.status.idle":"2024-12-29T15:42:33.574371Z","shell.execute_reply.started":"2024-12-29T15:42:33.569139Z","shell.execute_reply":"2024-12-29T15:42:33.573129Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Analyzing `Health Score`","metadata":{}},{"cell_type":"code","source":"plotly.express.histogram(\n    TRAINING_DATAFRAME,\n    x=\"Health Score\",\n    nbins=50, \n    labels={'Health_Score': 'Health Score', 'count': 'Frequency'},\n    title=\"Histogram of Health Score\",\n    template=\"plotly_dark\"\n).show(renderer = \"iframe\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:33.575945Z","iopub.execute_input":"2024-12-29T15:42:33.576397Z","iopub.status.idle":"2024-12-29T15:42:33.902534Z","shell.execute_reply.started":"2024-12-29T15:42:33.576341Z","shell.execute_reply":"2024-12-29T15:42:33.901334Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"There is a little bit of right skew in the distribution.","metadata":{}},{"cell_type":"code","source":"pipeline.append((\"Health Score\", __impute_column_with_median, \"MEDIAN\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:33.903833Z","iopub.execute_input":"2024-12-29T15:42:33.904201Z","iopub.status.idle":"2024-12-29T15:42:33.909795Z","shell.execute_reply.started":"2024-12-29T15:42:33.904134Z","shell.execute_reply":"2024-12-29T15:42:33.908354Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's analyze `Previous Claims`.","metadata":{}},{"cell_type":"code","source":"plotly.express.bar(\n    TRAINING_DATAFRAME[\"Previous Claims\"].value_counts().reset_index(),\n    x = \"Previous Claims\", \n    y = \"count\", \n    labels={'index': 'Previous Claims', 'Previous Claims': 'Frequency'},\n    title=\"Distribution of Previous Claims\",\n    template=\"plotly_dark\"\n).show(renderer = \"iframe\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:33.911723Z","iopub.execute_input":"2024-12-29T15:42:33.912935Z","iopub.status.idle":"2024-12-29T15:42:34.00382Z","shell.execute_reply.started":"2024-12-29T15:42:33.912878Z","shell.execute_reply":"2024-12-29T15:42:34.002193Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pipeline.append((\"Previous Claims\", __impute_column_with_q1, \"Q1\"))\npipeline.append((\"Previous Claims\", __impute_column_with_random_sample_from_distribution, \"RANDOM\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:34.00898Z","iopub.execute_input":"2024-12-29T15:42:34.009388Z","iopub.status.idle":"2024-12-29T15:42:34.015539Z","shell.execute_reply.started":"2024-12-29T15:42:34.009355Z","shell.execute_reply":"2024-12-29T15:42:34.014262Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plotly.express.bar(\n    TRAINING_DATAFRAME[\"Vehicle Age\"].value_counts().reset_index(),\n    x = \"Vehicle Age\", \n    y = \"count\", \n    labels={'index': 'Vehicle Age', 'Vehicle Age': 'Frequency'},\n    title=\"Distribution of Vehicle Age\",\n    template=\"plotly_dark\"\n).show(renderer = \"iframe\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:34.017043Z","iopub.execute_input":"2024-12-29T15:42:34.018089Z","iopub.status.idle":"2024-12-29T15:42:34.121066Z","shell.execute_reply.started":"2024-12-29T15:42:34.01805Z","shell.execute_reply":"2024-12-29T15:42:34.119466Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pipeline.append((\"Vehicle Age\", __impute_column_with_median, \"MEDIAN\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:34.122677Z","iopub.execute_input":"2024-12-29T15:42:34.123045Z","iopub.status.idle":"2024-12-29T15:42:34.128972Z","shell.execute_reply.started":"2024-12-29T15:42:34.123011Z","shell.execute_reply":"2024-12-29T15:42:34.127223Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plotly.express.histogram(\n    TRAINING_DATAFRAME,\n    x=\"Credit Score\",\n    nbins=50, \n    labels={'Credit_Score': 'Credit Score', 'count': 'Frequency'},\n    title=\"Histogram of Credit Score\",\n    template=\"plotly_dark\"\n).show(renderer = \"iframe\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:34.130579Z","iopub.execute_input":"2024-12-29T15:42:34.131128Z","iopub.status.idle":"2024-12-29T15:42:34.347112Z","shell.execute_reply.started":"2024-12-29T15:42:34.131076Z","shell.execute_reply":"2024-12-29T15:42:34.34592Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pipeline.append((\"Credit Score\", __impute_column_with_median, \"MEDIAN\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:34.348783Z","iopub.execute_input":"2024-12-29T15:42:34.349177Z","iopub.status.idle":"2024-12-29T15:42:34.354511Z","shell.execute_reply.started":"2024-12-29T15:42:34.349121Z","shell.execute_reply":"2024-12-29T15:42:34.353229Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plotly.express.bar(\n    TRAINING_DATAFRAME[\"Insurance Duration\"].value_counts().reset_index(),\n    x = \"Insurance Duration\", \n    y = \"count\", \n    labels={'index': 'Insurance Duration', 'Insurance Duration': 'Frequency'},\n    title=\"Distribution of Insurance Duration\",\n    template=\"plotly_dark\"\n).show(renderer = \"iframe\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:34.355767Z","iopub.execute_input":"2024-12-29T15:42:34.356098Z","iopub.status.idle":"2024-12-29T15:42:34.44696Z","shell.execute_reply.started":"2024-12-29T15:42:34.356057Z","shell.execute_reply":"2024-12-29T15:42:34.445663Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pipeline.append((\"Insurance Duration\", __impute_column_with_median, \"MEDIAN\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:34.448266Z","iopub.execute_input":"2024-12-29T15:42:34.448718Z","iopub.status.idle":"2024-12-29T15:42:34.454117Z","shell.execute_reply.started":"2024-12-29T15:42:34.44867Z","shell.execute_reply":"2024-12-29T15:42:34.452904Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plotly.express.bar(\n    TRAINING_DATAFRAME[\"Customer Feedback\"].value_counts().reset_index(),\n    x = \"Customer Feedback\", \n    y = \"count\", \n    labels={'index': 'Customer Feedback', 'Customer Feedback': 'Frequency'},\n    title=\"Distribution of Customer Feedback\",\n    template=\"plotly_dark\"\n).show(renderer = \"iframe\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:34.455721Z","iopub.execute_input":"2024-12-29T15:42:34.45612Z","iopub.status.idle":"2024-12-29T15:42:34.62568Z","shell.execute_reply.started":"2024-12-29T15:42:34.456085Z","shell.execute_reply":"2024-12-29T15:42:34.624348Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pipeline.append((\"Customer Feedback\", __impute_column_with_value, \"W_VALUE\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:34.626866Z","iopub.execute_input":"2024-12-29T15:42:34.627198Z","iopub.status.idle":"2024-12-29T15:42:34.632212Z","shell.execute_reply.started":"2024-12-29T15:42:34.627147Z","shell.execute_reply":"2024-12-29T15:42:34.631087Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Applying the cleaning pipeline.\n\n![](https://helios-i.mashable.com/imagery/articles/03AQw1L3spvJG54ChmQP95o/images-1.fill.size_2000x1125.v1700592699.jpg)\nSource: Squid Games | Netflix","metadata":{}},{"cell_type":"code","source":"TRAINING_DATAFRAME = run_pipeline(pipeline, TRAINING_DATAFRAME)\nTESTING_DATAFRAME  = run_pipeline(pipeline, TESTING_DATAFRAME)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:42:34.633761Z","iopub.execute_input":"2024-12-29T15:42:34.634112Z","iopub.status.idle":"2024-12-29T15:43:20.610305Z","shell.execute_reply.started":"2024-12-29T15:42:34.634072Z","shell.execute_reply":"2024-12-29T15:43:20.609214Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Dropping Column:","metadata":{}},{"cell_type":"code","source":"TRAINING_DATAFRAME.columns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:43:20.611708Z","iopub.execute_input":"2024-12-29T15:43:20.612201Z","iopub.status.idle":"2024-12-29T15:43:20.620343Z","shell.execute_reply.started":"2024-12-29T15:43:20.612122Z","shell.execute_reply":"2024-12-29T15:43:20.6193Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"columns_to_drop = [\n    'Age_RANDOM',\n    'Number of Dependents_RANDOM',\n    'Number of Dependents_MEDIAN',\n    'Previous Claims_RANDOM',\n    'Marital Status_MODE',\n    'Marital Status_RANDOM',\n    'Occupation_MODE',\n    'Occupation_RANDOM'\n]\n\nfor column in columns_to_drop:\n    if column in TRAINING_DATAFRAME.columns:\n        TRAINING_DATAFRAME = TRAINING_DATAFRAME.drop(column, axis = \"columns\")\n\n    if column in TESTING_DATAFRAME.columns:\n        TESTING_DATAFRAME = TESTING_DATAFRAME.drop(column, axis = \"columns\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:43:20.62162Z","iopub.execute_input":"2024-12-29T15:43:20.621944Z","iopub.status.idle":"2024-12-29T15:43:23.54484Z","shell.execute_reply.started":"2024-12-29T15:43:20.621903Z","shell.execute_reply":"2024-12-29T15:43:23.543565Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🤖 3.2 Generate New Features","metadata":{}},{"cell_type":"code","source":"def __encode_date(dataframe):\n    dataframe['Policy Start Date'] = pandas.to_datetime(dataframe['Policy Start Date'])\n    dataframe['Year'] = dataframe['Policy Start Date'].dt.year.astype(str)\n    dataframe.drop('Policy Start Date', axis=1, inplace=True)\n    return dataframe\n\nTRAINING_DATAFRAME = __encode_date(TRAINING_DATAFRAME)\nTESTING_DATAFRAME = __encode_date(TESTING_DATAFRAME)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:43:23.546266Z","iopub.execute_input":"2024-12-29T15:43:23.546618Z","iopub.status.idle":"2024-12-29T15:43:25.281778Z","shell.execute_reply.started":"2024-12-29T15:43:23.546575Z","shell.execute_reply":"2024-12-29T15:43:25.280893Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🤖 3.3 One hot encode categorical columns and normalize quantitative data","metadata":{}},{"cell_type":"code","source":"training_id = TRAINING_DATAFRAME[\"id\"]\nTRAINING_DATAFRAME = TRAINING_DATAFRAME.drop([\"id\"], axis = \"columns\")\n\ntesting_id = TESTING_DATAFRAME[\"id\"]\nTESTING_DATAFRAME = TESTING_DATAFRAME.drop([\"id\"], axis = \"columns\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:43:25.283136Z","iopub.execute_input":"2024-12-29T15:43:25.283576Z","iopub.status.idle":"2024-12-29T15:43:25.573105Z","shell.execute_reply.started":"2024-12-29T15:43:25.283533Z","shell.execute_reply":"2024-12-29T15:43:25.571726Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for categorical_column in TRAINING_DATAFRAME.describe(include='object').columns:\n    encoder = sklearn.preprocessing.OrdinalEncoder()\n    TRAINING_DATAFRAME[categorical_column] = TRAINING_DATAFRAME[categorical_column].astype(str)\n    TRAINING_DATAFRAME[categorical_column] = encoder.fit_transform(TRAINING_DATAFRAME[[categorical_column]])\n    \n    TESTING_DATAFRAME[categorical_column] = TESTING_DATAFRAME[categorical_column].astype(str)\n    TESTING_DATAFRAME[categorical_column] = encoder.fit_transform(TESTING_DATAFRAME[[categorical_column]])\n    ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:43:25.57438Z","iopub.execute_input":"2024-12-29T15:43:25.574719Z","iopub.status.idle":"2024-12-29T15:43:34.772025Z","shell.execute_reply.started":"2024-12-29T15:43:25.574685Z","shell.execute_reply":"2024-12-29T15:43:34.770781Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"normalizer = sklearn.preprocessing.StandardScaler()\nquantitative_columns = TESTING_DATAFRAME.describe().columns\n\nTRAINING_DATAFRAME[quantitative_columns] = pandas.DataFrame(\n    normalizer.fit_transform(TRAINING_DATAFRAME[quantitative_columns])\n)\nTESTING_DATAFRAME[quantitative_columns] = pandas.DataFrame(\n    normalizer.transform(TESTING_DATAFRAME[quantitative_columns])\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:43:34.773439Z","iopub.execute_input":"2024-12-29T15:43:34.773798Z","iopub.status.idle":"2024-12-29T15:43:36.305815Z","shell.execute_reply.started":"2024-12-29T15:43:34.773763Z","shell.execute_reply":"2024-12-29T15:43:36.304264Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 3.3 EDA: Todo","metadata":{}},{"cell_type":"markdown","source":"### 3.4 🤖 Model Building: Train machine learning models, starting with basic classifiers and then experimenting with more complex models as needed.","metadata":{}},{"cell_type":"code","source":"def model_pipeline(model, parameters, training_dataframe, testing_dataframe, testing_id):\n    y = training_dataframe[\"Premium Amount\"]\n    X = training_dataframe.drop(\"Premium Amount\", axis = \"columns\")\n    \n    classifier_ = sklearn.model_selection.GridSearchCV(\n        model,\n        parameters,\n        scoring='neg_mean_squared_error',\n    )\n    \n    classifier_.fit(X, y)\n    \n    best_params = classifier_.best_params_\n    best_score = classifier_.best_score_\n    \n    print(\"Best Parameters:\", best_params)\n    print(\"Best Score:\", best_score)\n\n    return classifier_.best_estimator_, best_params","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:43:36.307199Z","iopub.execute_input":"2024-12-29T15:43:36.307509Z","iopub.status.idle":"2024-12-29T15:43:36.313609Z","shell.execute_reply.started":"2024-12-29T15:43:36.307477Z","shell.execute_reply":"2024-12-29T15:43:36.312461Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"cat_features = TRAINING_DATAFRAME.select_dtypes(include=['object', 'bool']).columns.tolist()\nlgbm_cat_features = [f\"name:{col}\" for col in cat_features]\n\nmodels = [\n    {\n        \"name\": \"RandomForestRegressor_1\",\n        \"model\": sklearn.ensemble.RandomForestRegressor(\n            n_estimators=100,\n            max_depth=7,\n            min_samples_split=10,\n            min_samples_leaf=4,\n            max_features='sqrt',\n            bootstrap=True,\n            random_state=42,\n            n_jobs=-1 \n        ),\n        \"train\": False\n    },\n    {\n        \"name\": \"RandomForestRegressor_2\",\n        \"model\": sklearn.ensemble.RandomForestRegressor(\n            n_estimators=100,\n            max_depth=10,\n            min_samples_split=20,\n            min_samples_leaf=5,\n            max_features='log2',\n            bootstrap=False,\n            random_state=42,\n            n_jobs=-1 \n        ),\n        \"train\": False\n    },\n    {\n        \"name\": \"XGBRegressor_1\",\n        \"model\": xgboost.XGBRegressor(\n            random_state=42,\n            n_estimators=100,\n            max_depth=6,\n            learning_rate=0.01,\n            subsample=0.8,\n            colsample_bytree=0.8,\n            gamma=0.1,\n            n_jobs=-1 \n        ),\n        \"train\": False\n    },\n    {\n        \"name\": \"XGBRegressor_2\",\n        \"model\": xgboost.XGBRegressor(\n            random_state=42,\n            n_estimators=100,\n            max_depth=7,\n            learning_rate=0.05,\n            subsample=0.9,\n            colsample_bytree=1.0,\n            gamma=0.2,\n            n_jobs=-1 \n        ),\n        \"train\": False\n    },\n    {\n        \"name\": \"LGBMRegressor_1\",\n        \"model\": lightgbm.LGBMRegressor(\n            random_state=42,\n            n_estimators=100,\n            max_depth=6,\n            learning_rate=0.01,\n            num_leaves=31,\n            subsample=0.8,\n            colsample_bytree=0.8,\n            min_child_samples=20,\n            n_jobs=-1,\n            categorical_feature = lgbm_cat_features\n        ),\n        \"train\": True\n    },\n    {\n        \"name\": \"LGBMRegressor_2\",\n        \"model\": lightgbm.LGBMRegressor(\n            random_state=42,\n            n_estimators=100,\n            max_depth=8,\n            learning_rate=0.05,\n            num_leaves=40,\n            subsample=0.9,\n            colsample_bytree=0.9,\n            min_child_samples=30,\n            n_jobs=-1,\n            categorical_feature = lgbm_cat_features\n        ),\n        \"train\": True\n    },\n    {\n        \"name\": \"GradientBoostingRegressor_1\",\n        \"model\": sklearn.ensemble.GradientBoostingRegressor(\n            n_estimators=100,\n            max_depth=5,\n            learning_rate=0.05,\n            subsample=0.8,\n            min_samples_split=10,\n            min_samples_leaf=4,\n            random_state=42,\n        ),\n        \"train\": False\n    },\n    {\n        \"name\": \"GradientBoostingRegressor_2\",\n        \"model\": sklearn.ensemble.GradientBoostingRegressor(\n            n_estimators=100,\n            max_depth=6,\n            learning_rate=0.1,\n            subsample=0.9,\n            min_samples_split=20,\n            min_samples_leaf=6,\n            random_state=42\n        ),\n        \"train\": False\n    },\n    {\n        \"name\": \"CatBoostRegressor_1\",\n        \"model\": catboost.CatBoostRegressor(\n            depth = 10,\n            cat_features = cat_features,\n            objective = 'RMSE',\n            min_data_in_leaf = 18,\n            l2_leaf_reg = 9.044,\n            n_estimators=100,\n        ),\n        \"train\": True\n    },\n    {\n        \"name\": \"CatBoostRegressor_2\",\n        \"model\": catboost.CatBoostRegressor(\n            depth = 7,\n            cat_features = cat_features,\n            objective = 'RMSE',\n            min_data_in_leaf = 18,\n            l2_leaf_reg = 9.044,\n            n_estimators=100,\n        ),\n        \"train\": True\n    }\n]\n\nbest_estimators = [(model[\"name\"], model[\"model\"]) for model in models]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:44:18.898526Z","iopub.execute_input":"2024-12-29T15:44:18.898951Z","iopub.status.idle":"2024-12-29T15:44:18.913872Z","shell.execute_reply.started":"2024-12-29T15:44:18.898913Z","shell.execute_reply":"2024-12-29T15:44:18.912264Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"voting_regressor = sklearn.ensemble.StackingRegressor(\n    estimators = best_estimators, \n    final_estimator = sklearn.linear_model.Ridge(),\n    n_jobs=-1\n)\n\ny = TRAINING_DATAFRAME[\"Premium Amount\"]\nX = TRAINING_DATAFRAME.drop(\"Premium Amount\", axis=\"columns\")\nvoting_regressor.fit(X, y)\n\nout = pandas.DataFrame(voting_regressor.predict(TESTING_DATAFRAME))\nres = pandas.concat([pandas.DataFrame(testing_id), out], axis=1)\nres.columns = [\"id\", \"Premium Amount\"]\nres[[\"id\", \"Premium Amount\"]].to_csv(\"submission.csv\", index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-29T15:44:20.00373Z","iopub.execute_input":"2024-12-29T15:44:20.004103Z","execution_failed":"2024-12-29T16:02:54.846Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧑‍🏫 To be continued ","metadata":{}}]}