{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Regression with an Insurance Dataset S4E12","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport numpy as np","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:08:50.681552Z","iopub.execute_input":"2024-12-15T13:08:50.682052Z","iopub.status.idle":"2024-12-15T13:08:50.688457Z","shell.execute_reply.started":"2024-12-15T13:08:50.682012Z","shell.execute_reply":"2024-12-15T13:08:50.686985Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/playground-series-s4e12/train.csv\")\ntest_df = pd.read_csv(\"/kaggle/input/playground-series-s4e12/test.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:08:50.690682Z","iopub.execute_input":"2024-12-15T13:08:50.691078Z","iopub.status.idle":"2024-12-15T13:08:57.960491Z","shell.execute_reply.started":"2024-12-15T13:08:50.691038Z","shell.execute_reply":"2024-12-15T13:08:57.959338Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Lets begin with a basic overview of our data:","metadata":{}},{"cell_type":"code","source":"print(f\"Train dataset has {train_df.shape[0]} rows and {train_df.shape[1]} columns\")\nprint(f\"Test dataset has {test_df.shape[0]} rows and {test_df.shape[1]} columns\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:08:57.962042Z","iopub.execute_input":"2024-12-15T13:08:57.962373Z","iopub.status.idle":"2024-12-15T13:08:57.969605Z","shell.execute_reply.started":"2024-12-15T13:08:57.962341Z","shell.execute_reply":"2024-12-15T13:08:57.968037Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df.head(5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:08:57.972353Z","iopub.execute_input":"2024-12-15T13:08:57.972739Z","iopub.status.idle":"2024-12-15T13:08:58.004798Z","shell.execute_reply.started":"2024-12-15T13:08:57.972684Z","shell.execute_reply":"2024-12-15T13:08:58.003303Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:08:58.006267Z","iopub.execute_input":"2024-12-15T13:08:58.006625Z","iopub.status.idle":"2024-12-15T13:08:58.673978Z","shell.execute_reply.started":"2024-12-15T13:08:58.006568Z","shell.execute_reply":"2024-12-15T13:08:58.672519Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"It looks like we have missing data, lets take a look in more detail. Lets assume for now that the data has been processed so all missing data is set to NaN, rather than empty strings for example:","metadata":{}},{"cell_type":"code","source":"train_df[\"data\"] = \"train\"\ntest_df[\"data\"] = \"test\"\ndf = pd.concat([train_df, test_df])\n\nmissing_train = df.loc[df[\"data\"] == \"train\"].isna().sum().reset_index().rename(columns={\"index\": \"column\", 0: \"train_missing\"})\nmissing_test = df.loc[df[\"data\"] == \"test\"].isna().sum().reset_index().rename(columns={\"index\": \"column\", 0: \"test_missing\"})\n\nmissing_df = pd.merge(missing_train, missing_test, on=\"column\")\n\nmissing_df = missing_df.melt(\n    id_vars=[\"column\"], \n    value_vars=[\"train_missing\", \"test_missing\"], \n    var_name=\"Dataset\", \n    value_name=\"Missing Rows\"\n)\nmissing_df = missing_df.loc[missing_df[\"column\"] != \"data\"]\n\nmissing_df[\"Percentage\"] = missing_df.apply(\n    lambda row: (row[\"Missing Rows\"] / train_df.shape[0] * 100) if row[\"Dataset\"] == \"train_missing\" else (row[\"Missing Rows\"] / test_df.shape[0] * 100), \n    axis=1\n)\n\n\n# Create the plot\nf, ax = plt.subplots(figsize=(12, 12))\nsns.barplot(data=missing_df, y=\"column\", x=\"Missing Rows\", hue=\"Dataset\", ax=ax)\n\nfor container, dataset in zip(ax.containers, [\"train_missing\", \"test_missing\"]):\n    for bar, row in zip(container, missing_df[missing_df[\"Dataset\"] == dataset].itertuples()):\n        ax.text(\n            bar.get_width() + 0.5,  # Adjust position slightly to the right of the bar\n            bar.get_y() + bar.get_height() / 2,  # Center the text vertically\n            f\"{row._3} ({row.Percentage:.3f}%)\", \n            ha=\"left\", va=\"center\"\n        )\n\nax.set_title(\"Understanding Missing Data\")\nax.set_xlabel(\"Number of Missing Rows\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:08:58.675702Z","iopub.execute_input":"2024-12-15T13:08:58.676097Z","iopub.status.idle":"2024-12-15T13:09:01.552405Z","shell.execute_reply.started":"2024-12-15T13:08:58.676061Z","shell.execute_reply":"2024-12-15T13:09:01.551049Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Observations:**\n- There missing data between train and test is roughly the same percentage, which is a good sign.\n- There are a couple of rows where < 10 items are missing. This is worth taking a closer look at.\n- Occupation and previous claims is the most commonly missing at around 30%, which is a signifcant amount of missing data.\n\nLets take another look:","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(15,9))\nax.set_title(\"Visualising Missing Values\")\nsns.heatmap(df.isnull(), cbar=False, yticklabels=False ,ax=ax);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:09:01.554065Z","iopub.execute_input":"2024-12-15T13:09:01.554417Z","iopub.status.idle":"2024-12-15T13:10:01.529254Z","shell.execute_reply.started":"2024-12-15T13:09:01.554384Z","shell.execute_reply":"2024-12-15T13:10:01.527911Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df[\"Policy Start Date\"] = pd.to_datetime(train_df['Policy Start Date'])\ntrain_df[\"Policy Start Year\"] = train_df[\"Policy Start Date\"].dt.year\ntrain_df[\"Policy Start Month\"] = train_df[\"Policy Start Date\"].dt.month\ntrain_df[\"Policy Start Year Month\"] = train_df[\"Policy Start Year\"].astype(str) + \"-\" + train_df[\"Policy Start Month\"].astype(str)\n\ntest_df[\"Policy Start Date\"] = pd.to_datetime(test_df['Policy Start Date'])\ntest_df[\"Policy Start Year\"] = test_df[\"Policy Start Date\"].dt.year\ntest_df[\"Policy Start Month\"] = test_df[\"Policy Start Date\"].dt.month\ntest_df[\"Policy Start Year Month\"] = test_df[\"Policy Start Year\"].astype(str) + \"-\" + test_df[\"Policy Start Month\"].astype(str)\n\ndf[\"Policy Start Date\"] = pd.to_datetime(df['Policy Start Date'])\ndf[\"Policy Start Year\"] = df[\"Policy Start Date\"].dt.year\ndf[\"Policy Start Month\"] = df[\"Policy Start Date\"].dt.month\ndf[\"Policy Start Year Month\"] = df[\"Policy Start Year\"].astype(str) + \"-\" + df[\"Policy Start Month\"].astype(str)\n\n\nfloat_cols = [\"Annual Income\", \"Health Score\", \"Credit Score\", \"Premium Amount\"] \nint_cols = [\"Age\", \"Number of Dependents\", \"Previous Claims\", \"Insurance Duration\", \"Vehicle Age\", \"Policy Start Year\", \"Policy Start Month\"]\ndate_cols = [\"Policy Start Date\"]\nstr_cols = [i for i in train_df.columns if train_df[i].dtype == \"object\" and i not in date_cols]\nstr_cols = [i for i in str_cols if i not in [\"data\", \"Policy Start Year Month\"]]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:10:01.531454Z","iopub.execute_input":"2024-12-15T13:10:01.531793Z","iopub.status.idle":"2024-12-15T13:10:06.419442Z","shell.execute_reply.started":"2024-12-15T13:10:01.531752Z","shell.execute_reply":"2024-12-15T13:10:06.418338Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Lets begin by looking at our string (or categorical columns)","metadata":{}},{"cell_type":"code","source":"train_df.loc[:,str_cols].nunique()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:10:06.421398Z","iopub.execute_input":"2024-12-15T13:10:06.421761Z","iopub.status.idle":"2024-12-15T13:10:07.18659Z","shell.execute_reply.started":"2024-12-15T13:10:06.42173Z","shell.execute_reply":"2024-12-15T13:10:07.184763Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Looks like all the string columns are categories, lets have a look at them all in more detail:","metadata":{"execution":{"iopub.status.busy":"2024-12-08T18:16:28.099418Z","iopub.execute_input":"2024-12-08T18:16:28.099824Z","iopub.status.idle":"2024-12-08T18:16:28.107071Z","shell.execute_reply.started":"2024-12-08T18:16:28.099789Z","shell.execute_reply":"2024-12-08T18:16:28.105817Z"}}},{"cell_type":"code","source":"def val_count_df(df, column_name, sort_by_column_name=False):\n    value_count = df[column_name].value_counts().reset_index().rename(columns={\"index\":column_name}).set_index(column_name)\n    value_count[\"Percentage\"] = df[column_name].value_counts(normalize=True)*100\n    value_count = value_count.reset_index()\n    if sort_by_column_name:\n        value_count = value_count.sort_values(column_name)\n    return value_count","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:10:07.189925Z","iopub.execute_input":"2024-12-15T13:10:07.191303Z","iopub.status.idle":"2024-12-15T13:10:07.199134Z","shell.execute_reply.started":"2024-12-15T13:10:07.191237Z","shell.execute_reply":"2024-12-15T13:10:07.197778Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_and_display_valuecounts(df, column_name, sort_by_column_name):\n    val_count = val_count_df(df, column_name, sort_by_column_name)\n    display(val_count)\n    \n    val_count.set_index(column_name).plot.pie(y=\"Value Count\", figsize=(5,5), legend=False, ylabel=\"\");\n    \ndef plot_and_display_compare_valuecounts(df1, df2, column_name, sort_by_column_name):\n    val_count_1 = val_count_df(df1, column_name, sort_by_column_name)\n    val_count_1 = val_count_1.rename(columns={\"count\":\"train_value_count\", \"Percentage\":\"train_percentage\"})\n    val_count_2 = val_count_df(df2, column_name, sort_by_column_name)\n    val_count_2 = val_count_2.rename(columns={\"count\":\"test_value_count\", \"Percentage\":\"test_percentage\"})\n    \n    val_count = pd.merge(val_count_1, val_count_2, on=column_name, how=\"outer\")\n    val_count = val_count.fillna(0) # if the data is missing from a column, there is none so we fill with 0's\n    #display(val_count)\n    \n    val_count = val_count.drop(columns=[\"train_percentage\", \"test_percentage\"])\n    fig, axes = plt.subplots(1, 2, figsize=(12, 7))\n    for idx, (col, ax) in enumerate(zip([\"train_value_count\", \"test_value_count\"], axes)):\n        wedges, texts, autotexts = ax.pie(\n            val_count[col],\n            labels=val_count[column_name],\n            autopct=\"%1.2f%%\",\n            startangle=90,\n        )\n        for text in autotexts:\n            text.set_color(\"black\")\n            text.set_fontsize(10)\n        ax.set_title(f\"{['Train', 'Test'][idx]} Distribution of {column_name}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:10:07.20085Z","iopub.execute_input":"2024-12-15T13:10:07.201429Z","iopub.status.idle":"2024-12-15T13:10:07.2208Z","shell.execute_reply.started":"2024-12-15T13:10:07.201374Z","shell.execute_reply":"2024-12-15T13:10:07.219455Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for col in str_cols: \n    plot_and_display_compare_valuecounts(train_df, test_df, col, True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:10:07.222428Z","iopub.execute_input":"2024-12-15T13:10:07.222966Z","iopub.status.idle":"2024-12-15T13:10:12.581392Z","shell.execute_reply.started":"2024-12-15T13:10:07.222894Z","shell.execute_reply":"2024-12-15T13:10:12.578213Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Observations:\n- The same distributions are present for train and test, which could be assumed - but its good to check\n- The different categories for each variable are very balanced.","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\", message=\"When grouping with a length-1 list-like\")\nwarnings.filterwarnings(\"ignore\", message=\"use_inf_as_na option is deprecated\")\n\nf, axs = plt.subplots(len(float_cols), 2, figsize=(10,25))\nfor i, column in enumerate(float_cols):\n    sns.histplot(data=df.reset_index(drop=True), x=column, hue=\"data\", ax=axs[i,0], bins=20)\n    sns.histplot(data=df.reset_index(drop=True), x=column, hue=\"data\", ax=axs[i,1], bins=50)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:10:12.582869Z","iopub.execute_input":"2024-12-15T13:10:12.583574Z","iopub.status.idle":"2024-12-15T13:10:47.441156Z","shell.execute_reply.started":"2024-12-15T13:10:12.583524Z","shell.execute_reply":"2024-12-15T13:10:47.439731Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Observations:**\n- Train and test distributions are similar.","metadata":{}},{"cell_type":"code","source":"f, axs = plt.subplots(len(int_cols), figsize=(15,25))\nfor i, column in enumerate(int_cols):\n    temp_df = df[column].value_counts().sort_index().reset_index()\n    temp_df[column] = temp_df[column].astype(int)\n    sns.barplot(data=temp_df, x=column, y=\"count\", ax=axs[i], color=\"skyblue\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:10:47.443006Z","iopub.execute_input":"2024-12-15T13:10:47.44337Z","iopub.status.idle":"2024-12-15T13:10:49.353654Z","shell.execute_reply.started":"2024-12-15T13:10:47.443329Z","shell.execute_reply":"2024-12-15T13:10:49.352298Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Observations:\n- Most integer columns have the same counts, except previous claims.\n- 2019 and 2024 have less data (for both train and test) - presumerable as they have got data for exactly 5 years, but spread over 6 years (starting late 2019, ending late 2024)","metadata":{}},{"cell_type":"code","source":"print(f\"The Earliest Policy Start Date is {df['Policy Start Date'].min()}\")\nprint(f\"The Latest Policy Start Date is {df['Policy Start Date'].max()}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:10:49.35551Z","iopub.execute_input":"2024-12-15T13:10:49.355985Z","iopub.status.idle":"2024-12-15T13:10:49.378157Z","shell.execute_reply.started":"2024-12-15T13:10:49.355913Z","shell.execute_reply":"2024-12-15T13:10:49.376714Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Performing a correlation analysis:","metadata":{}},{"cell_type":"code","source":"plt.subplots(figsize=(10,10))\nsns.heatmap(train_df[float_cols+int_cols].corr(),annot=True, cmap=\"RdYlGn\", fmt = '0.3f', vmin=-0.5, vmax=0.5, cbar=False);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-15T13:10:49.380331Z","iopub.execute_input":"2024-12-15T13:10:49.380792Z","iopub.status.idle":"2024-12-15T13:10:50.855147Z","shell.execute_reply.started":"2024-12-15T13:10:49.38074Z","shell.execute_reply":"2024-12-15T13:10:50.853737Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Observations:**\n\n- The correlation between Year and Month can be ignored. This is from start and end dates being mid-years, for example November does not occur in 2024 data.\n- A higher annual income is correlated to a lower credit score? Perhaps higher income earners are more likely to take on more debt (mortgages, loans)?\n- The highest correlation to premium amount is with previous claims. This feature will likely be the most important for models.\n- The second highest correlation to premium amount is with credit score. With a lower credit score having a higher premium amount. This feature is also likely to be very important.","metadata":{}}]}