{"metadata":{"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":56537,"databundleVersionId":8877088,"sourceType":"competition"},{"sourceId":8409068,"sourceType":"datasetVersion","datasetId":5004471},{"sourceId":8760924,"sourceType":"datasetVersion","datasetId":5247362},{"sourceId":8761119,"sourceType":"datasetVersion","datasetId":5263840}],"dockerImageVersionId":30699,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"},"papermill":{"default_parameters":{},"duration":351.443656,"end_time":"2024-06-20T19:03:55.489625","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-06-20T18:58:04.045969","version":"2.5.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## A copy of notebook [Keras Baseline Seq2Seq](https://www.kaggle.com/code/enzosebiane/keras-baseline-seq2seq?scriptVersionId=183823137) from the author [Enzo Sebiane](https://www.kaggle.com/enzosebiane)\n\nChanges made:\n- SEED = 42 --> 2024\n- epochs = 50 --> 100\n- Added an additional build_cnn11 layer to the model.\n- An ensemble of two best models\n- The same notebook, only the result of the ensemble was obtained on the local computer. It was interesting to find out if there is a difference in the result.","metadata":{"papermill":{"duration":0.00924,"end_time":"2024-06-20T18:58:06.966816","exception":false,"start_time":"2024-06-20T18:58:06.957576","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Model definition & Training","metadata":{"papermill":{"duration":0.009111,"end_time":"2024-06-20T19:01:20.224907","exception":false,"start_time":"2024-06-20T19:01:20.215796","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Submission","metadata":{"papermill":{"duration":0.015745,"end_time":"2024-06-20T19:01:59.284396","exception":false,"start_time":"2024-06-20T19:01:59.268651","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nimport gc\nimport numpy as np\nimport pandas as pd\nimport polars as pl\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport keras\nfrom sklearn import metrics\nfrom sklearn.preprocessing import StandardScaler\nfrom tqdm.notebook import tqdm\n\n# Set random seed for reproducibility\nSEED = 2024\nkeras.utils.set_random_seed(SEED)\ntf.random.set_seed(SEED)\ntf.config.experimental.enable_op_determinism()\n\n# Load and preprocess data\nDATA_PATH = \"/kaggle/input/leap-atmospheric-physics-ai-climsim\"\ntrain_data = pd.read_csv(os.path.join(DATA_PATH, \"train.csv\"))\ntrain_data.fillna(method='ffill', inplace=True)\n\n# Scaling data\nscaler = StandardScaler()\nscaled_features = scaler.fit_transform(train_data.drop(columns=['sample_id']))\ntrain_data_scaled = pd.DataFrame(scaled_features, columns=train_data.columns[1:])\n\n# Define the model architecture\ndef build_model(input_shape, output_shape):\n    model = keras.Sequential([\n        keras.layers.InputLayer(input_shape=input_shape),\n        keras.layers.Dense(256, activation='relu'),\n        keras.layers.BatchNormalization(),\n        keras.layers.Dropout(0.3),\n        keras.layers.Dense(128, activation='relu'),\n        keras.layers.BatchNormalization(),\n        keras.layers.Dropout(0.3),\n        keras.layers.Dense(output_shape, activation='linear')\n    ])\n    model.compile(optimizer='adam', loss='mse', metrics=['mae'])\n    return model\n\n# Hyperparameters\nBATCH_SIZE = 32\nEPOCHS = 50\n\n# Prepare datasets using tf.data API\ndef preprocess(data, targets):\n    dataset = tf.data.Dataset.from_tensor_slices((data, targets))\n    dataset = dataset.shuffle(buffer_size=1024).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n    return dataset\n\nX_train, y_train = train_data_scaled.to_numpy(), train_data['targets'].to_numpy()\ntrain_dataset = preprocess(X_train, y_train)\n\n# Build and train the model\ninput_shape = X_train.shape[1]\noutput_shape = y_train.shape[1]\n\nmodel = build_model(input_shape, output_shape)\ncallbacks = [\n    keras.callbacks.ModelCheckpoint('model.h5', save_best_only=True),\n    keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=5),\n    keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)\n]\n\nhistory = model.fit(train_dataset, epochs=EPOCHS, validation_split=0.2, callbacks=callbacks, verbose=1)\n\n# Evaluation\ny_valid = np.concatenate([yb for _, yb in ds_valid])\np_valid = model.predict(X_valid, batch_size=BATCH_SIZE)\np_valid = p_valid * stdd_y + mean_y\nscores_valid = np.array([metrics.r2_score(y_valid[:, i], p_valid[:, i]) for i in range(len(TARGETS))])\nprint(f\"Validation scores: {scores_valid}\")\n\n# Plot training history\nplt.plot(history.history['loss'], label='Training Loss')\nplt.plot(history.history['val_loss'], label='Validation Loss')\nplt.yscale('log')\nplt.legend()\nplt.show()\n","metadata":{},"execution_count":null,"outputs":[]}]}