{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84896,"databundleVersionId":10305135,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\nGENERAL INFORMATIONS-GENEL BİLGİLER\n
","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-12-05T17:37:01.586024Z","iopub.execute_input":"2024-12-05T17:37:01.58648Z","iopub.status.idle":"2024-12-05T17:37:30.935245Z","shell.execute_reply.started":"2024-12-05T17:37:01.586443Z","shell.execute_reply":"2024-12-05T17:37:30.933956Z"}}},{"cell_type":"markdown","source":"
\r\n TÜRKÇE:\r\n \r\n\r\n ENGLISH:\r\n \r\n
\r\ndiv>\r\ndiv>\r\n>\r\n","metadata":{}},{"cell_type":"markdown","source":"
\r\n

TABLE OF CONTENTS - İÇİNDEKİLER

\r\n
    \r\n
  1. Veriyle ilgili gerekli importları yapmak - Making the necessary imports of the data
    2. \r\n
  2. Veriye Genel Bakış - Data Overview
    3. \r\n
  3. E.D.A Süreci - E.D.A Process\r\n
      \r\n
    • Veri Test-Train Olarak Ayrılır - Data is Separated as Test-Train
      • \r\n
    • Temel Göstergerelere bakılır (df.describe, df.info vb) - Basic Indicators are checked (df.describe, df.info etc.)
      • \r\n
    • Kategorik ve Sayısal Değişkenlerin Belirlenmesi, Veri Normal Dağılıyormu İncelenmesi ve Hedef Değişkenle Diğer Değişkenler Kırılıma Sokulması - Determining Categorical and Numerical Variables, Examining Whether the Data is Normally Distributed and Breaking It Down with the Target Variable and Other Variables
      • \r\n
    \r\n
    4. \r\n
  4. Veri Önişleme Süreci - Data Preprocess\r\n
      \r\n
    • Tekrarlanan verilerin silinmesi - Deleting duplicate data
      • \r\n
    • NaN Değerlerin Silinmesi - Deleting NaN Values
      • \r\n
    • Aykırı Değerlerin Bulunması ve Düzeltilmesi - Finding and Correcting Outliers
      • \r\n
    • Dominant Değerlerin Bulunup Düzeltilmesi - Finding and Correcting Dominant Values
      • \r\n
    • Korelasyon Analizi Yapılır ve Birbiriyle Yüksek Korele Olan Değişkenler Çıkartılır - Correlation Analysis is Performed and Variables That Are Highly Correlated with Each Other Are Removed
      • \r\n
    • Özellik Mühendisliği Yapılır - Feature Engineering is Done
      • \r\n
    \r\n
    5. \r\n
  5. Modelleme Aşaması - Modeling Phase
    6. \r\n
\r\n
\r\n","metadata":{}},{"cell_type":"markdown","source":"
\nVeriyle İlgili Gerekli İmportları Yapmak - Making the Necessary Imports of the Data\n
","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.impute import KNNImputer, SimpleImputer\nfrom sklearn.preprocessing import OneHotEncoder, RobustScaler\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.metrics import mean_squared_log_error\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\nimport matplotlib.pyplot as plt\nimport joblib\nimport tensorflow as tf\nfrom xgboost import XGBRegressor\nfrom scipy.stats import shapiro, kstest\nfrom scipy.stats import shapiro, kstest, zscore\n\npd.set_option('display.max_columns', None)\npd.set_option('display.max_rows', None)\npd.set_option('display.max_seq_item', None)\npd.set_option('display.float_format', '{:.2f}'.format)\n\ndf_test = pd.read_csv(\"/kaggle/input/playground-series-s4e12/test.csv\")\ndf_train = pd.read_csv(\"/kaggle/input/playground-series-s4e12/train.csv\")\nsubmission = pd.read_csv(\"/kaggle/input/playground-series-s4e12/sample_submission.csv\")\n\ndf_train.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-06T18:43:54.127965Z","iopub.execute_input":"2024-12-06T18:43:54.128427Z","iopub.status.idle":"2024-12-06T18:44:22.657391Z","shell.execute_reply.started":"2024-12-06T18:43:54.128388Z","shell.execute_reply":"2024-12-06T18:44:22.656279Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\nE.D.A Süreci - E.D.A Process\n
","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\r\n Veriyle İlgili Sorunlar-Dataset Problems
\r\n Yaş ve Policy Start Date Değişkenlerinde Veri Tipinde Değişiklik Gerekiyor-Change in Data Type is Required for Age and Policy Start Date Variables
\r\n Değişkenler NaN değerler Mevcut-Variables Have NaN \r\n
\r\n","metadata":{}},{"cell_type":"code","source":"df_train['Policy Start Date'] = pd.to_datetime(df_train['Policy Start Date'])\ndf_train.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-06T18:44:28.511362Z","iopub.execute_input":"2024-12-06T18:44:28.511974Z","iopub.status.idle":"2024-12-06T18:44:28.960594Z","shell.execute_reply.started":"2024-12-06T18:44:28.511914Z","shell.execute_reply":"2024-12-06T18:44:28.959393Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_train.describe().T","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-06T11:10:13.58096Z","iopub.execute_input":"2024-12-06T11:10:13.581485Z","iopub.status.idle":"2024-12-06T11:10:14.43853Z","shell.execute_reply.started":"2024-12-06T11:10:13.581428Z","shell.execute_reply":"2024-12-06T11:10:14.437043Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_train[\"Age\"] = df_train[\"Age\"].fillna(df_train[\"Age\"].median())\ndf_test[\"Age\"] = df_test[\"Age\"].fillna(df_test[\"Age\"].median())\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-06T18:44:32.236527Z","iopub.execute_input":"2024-12-06T18:44:32.237812Z","iopub.status.idle":"2024-12-06T18:44:32.320299Z","shell.execute_reply.started":"2024-12-06T18:44:32.237739Z","shell.execute_reply":"2024-12-06T18:44:32.31839Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_train[\"Age\"]=df_train[\"Age\"].astype(int)\ndf_test[\"Age\"]=df_test[\"Age\"].astype(int)\ndf_train.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-06T18:44:34.687182Z","iopub.execute_input":"2024-12-06T18:44:34.687581Z","iopub.status.idle":"2024-12-06T18:44:35.344506Z","shell.execute_reply.started":"2024-12-06T18:44:34.687546Z","shell.execute_reply":"2024-12-06T18:44:35.343163Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.preprocessing import OneHotEncoder, RobustScaler\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.compose import ColumnTransformer\nfrom xgboost import XGBRegressor\nfrom sklearn.metrics import mean_squared_log_error\nimport joblib\n\n# Function to calculate RMSLE\ndef calculate_rmsle(y_true, y_pred):\n # Ensure predictions and true values are non-negative\n y_true = np.maximum(y_true, 0)\n y_pred = np.maximum(y_pred, 0)\n return np.sqrt(mean_squared_log_error(y_true, y_pred))\n\ndef main_pipeline(df_train, df_test, submission):\n # Print column names to debug the issue\n print(\"Columns in df_test:\", df_test.columns)\n \n # Prepare features (X) and target (y)\n X = df_train.drop(['Premium Amount'], axis=1)\n y = df_train['Premium Amount']\n\n # Convert 'Policy Start Date' to datetime\n X['Policy Start Date'] = pd.to_datetime(X['Policy Start Date'], errors='coerce')\n df_test['Policy Start Date'] = pd.to_datetime(df_test['Policy Start Date'], errors='coerce')\n\n # Extract features from 'Policy Start Date' like year, month, and day\n X['Policy Start Year'] = X['Policy Start Date'].dt.year\n X['Policy Start Month'] = X['Policy Start Date'].dt.month\n X['Policy Start Day'] = X['Policy Start Date'].dt.day\n\n df_test['Policy Start Year'] = df_test['Policy Start Date'].dt.year\n df_test['Policy Start Month'] = df_test['Policy Start Date'].dt.month\n df_test['Policy Start Day'] = df_test['Policy Start Date'].dt.day\n\n # Drop 'Policy Start Date' after extracting features\n X = X.drop(['Policy Start Date'], axis=1)\n df_test = df_test.drop(['Policy Start Date'], axis=1)\n\n # Impute missing 'Age' values with median\n X[\"Age\"] = X[\"Age\"].fillna(X[\"Age\"].median())\n df_test[\"Age\"] = df_test[\"Age\"].fillna(df_test[\"Age\"].median())\n\n # Ensure 'Age' is integer type\n X[\"Age\"] = X[\"Age\"].astype(int)\n df_test[\"Age\"] = df_test[\"Age\"].astype(int)\n\n # Identify categorical and numerical columns\n categorical = X.select_dtypes(include='object').columns\n numerical = X.select_dtypes(exclude='object').columns\n\n # Remove 'id' from numerical columns as it shouldn't be part of the features\n numerical = numerical[numerical != 'id']\n\n print(f\"Categorical Columns: {categorical}\")\n print(f\"Numerical Columns: {numerical}\")\n\n # Plot correlation heatmap for numerical features\n correlation_matrix = X[numerical].corr()\n plt.figure(figsize=(12, 8))\n sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', fmt='.2f', linewidths=0.5)\n plt.title(\"Correlation Heatmap of Numerical Features\")\n plt.show()\n\n # Define preprocessing pipelines for categorical and numerical columns\n categorical_pipeline = Pipeline([\n ('imputer', SimpleImputer(strategy='most_frequent')), # Impute missing categorical values\n ('encoder', OneHotEncoder(handle_unknown='ignore')) # OneHotEncoder for nominal categories\n ])\n\n numerical_pipeline = Pipeline([\n ('imputer', SimpleImputer(strategy='median')), # Impute missing numerical values with the median\n ('scaler', RobustScaler()) # Scale numerical features using RobustScaler\n ])\n\n # Combine both pipelines in a ColumnTransformer\n preprocessor = ColumnTransformer([\n ('categorical', categorical_pipeline, categorical),\n ('numerical', numerical_pipeline, numerical)\n ])\n\n # Define the model (XGBoost Regressor)\n model = XGBRegressor()\n\n # Create a full pipeline with preprocessing and model\n pipeline = Pipeline([\n ('preprocessor', preprocessor),\n ('model', model)\n ])\n\n # Split data into training and validation sets\n X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.25, random_state=4)\n\n # Train the model\n pipeline.fit(X_train, y_train)\n\n # Evaluate on the validation set\n y_val_pred = pipeline.predict(X_val)\n\n # Calculate RMSLE on the validation predictions\n rmsle = calculate_rmsle(y_val, y_val_pred)\n print(f\"Validation RMSLE: {rmsle:.4f}\")\n\n # Optionally, save the model and preprocessor\n joblib.dump(pipeline, \"model_pipeline.joblib\")\n print(\"Model and preprocessor saved.\")\n\n # Drop 'id' column before preprocessing\n df_test_processed = preprocessor.transform(df_test.drop(columns=[\"id\"])) # Exclude 'id' during preprocessing\n\n # Make predictions using the model\n test_predictions = model.predict(df_test_processed)\n\n # Ensure 'id' exists before creating the submission\n if 'id' not in df_test.columns:\n print(\"The 'id' column is missing in df_test!\")\n else:\n # Create the submission file\n submission = pd.DataFrame({\n \"id\": df_test[\"id\"], # Ensure 'id' is included from df_test\n \"Premium Amount\": test_predictions\n })\n\n submission_file_path = \"submission.csv\"\n submission.to_csv(submission_file_path, index=False)\n print(f\"Submission file saved as {submission_file_path}.\")\n\n return pipeline, submission_file_path, submission\n\n# Example usage:\nmodel, submission_file_path, submission = main_pipeline(df_train, df_test, submission)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-06T19:19:36.305302Z","iopub.execute_input":"2024-12-06T19:19:36.305711Z","iopub.status.idle":"2024-12-06T19:20:00.237409Z","shell.execute_reply.started":"2024-12-06T19:19:36.305675Z","shell.execute_reply":"2024-12-06T19:20:00.236251Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submission.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-06T19:11:07.889616Z","iopub.execute_input":"2024-12-06T19:11:07.890035Z","iopub.status.idle":"2024-12-06T19:11:07.901906Z","shell.execute_reply.started":"2024-12-06T19:11:07.889996Z","shell.execute_reply":"2024-12-06T19:11:07.900201Z"}},"outputs":[],"execution_count":null}]}