{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nThis submission is based on the [fast.ai](https://www.fast.ai/) library.\n\nI'm finding the Fast.ai course [Practical Deep Learning](https://course.fast.ai/) and the accompanying book to be invaluable. Thanks are due to [Radek Osmulski](https://radekosmulski.com/) for pointing me to this resource. I'm also finding his book offers a number of interesting insights which have been helpful as was his notebook [EDA + training a fast.ai model + submission](https://www.kaggle.com/code/radek1/eda-training-a-fast-ai-model-submission/notebook). The image dataset I'm using is derived from the one he created [.png dataset](https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs) but uses a different directory structure. Thanks are also due to Innat for some ideas he suggested in [[Keras]: RSNA Breast Cancer Detection [Training]](https://www.kaggle.com/code/ipythonx/keras-rsna-breast-cancer-detection-training) \n\n[A Brief Intro to Mammography](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369262)<br>\n[Everything you wanted to know about mammography (part 1)](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/374288)<br>\n[Everything you wanted to know about mammography (part 2)](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/374946)<br>\n\nwere used to understand the context\n\n### Other reference material\n\n[Image Classification Tips & Tricks](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/372567)<br>\n[6 Computer Vision tricks for faster training and better models](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369155)<br>\n[Deep learning unbalanced training data? Solve it like this.](https://towardsdatascience.com/deep-learning-unbalanced-training-data-solve-it-like-this-6c528e9efea6)\n[How to Deal with Imbalanced Data using SMOTE](https://medium.com/analytics-vidhya/balance-your-data-using-smote-98e4d79fcddb)<br>\n[Practical Guide to deal with Imbalanced Classification Problems in R](https://www.analyticsvidhya.com/blog/2016/03/practical-guide-deal-imbalanced-classification-problems/)<br>\n[Imbalanced Data: How to handle Imbalanced Classification Problems](https://www.analyticsvidhya.com/blog/2017/03/imbalanced-data-classification/)<br>\n[This Machine Learning Project on Imbalanced Data Can Add Value to Your Resume](https://www.analyticsvidhya.com/blog/2016/09/this-machine-learning-project-on-imbalanced-data-can-add-value-to-your-resume/)\n\n\n[Summary of key discussions after first month](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/374248)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport cv2\n\nimport shutil\n\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nfrom pathlib import Path\nimport glob\n\nfrom fastai.data.all import *\nfrom fastai.vision.all import *","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:27:28.831915Z","iopub.execute_input":"2023-01-28T07:27:28.832386Z","iopub.status.idle":"2023-01-28T07:27:34.402393Z","shell.execute_reply.started":"2023-01-28T07:27:28.832305Z","shell.execute_reply":"2023-01-28T07:27:34.400994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Source: https://www.kaggle.com/datasets/almateya/offline-resnet50\n# Load dataset offline-resnet50\n\n!mkdir -p /root/.cache/torch/hub/checkpoints\n\nsrc = '/kaggle/input/offline-resnet50/resnet50'   #'../input/offline-resnet50/resnet50'\ndst = '/root/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth'\n\nshutil.copy(src, dst)","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:27:52.296198Z","iopub.execute_input":"2023-01-28T07:27:52.296861Z","iopub.status.idle":"2023-01-28T07:27:55.403484Z","shell.execute_reply.started":"2023-01-28T07:27:52.296823Z","shell.execute_reply":"2023-01-28T07:27:55.402395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_csv = pd.read_csv('../input/rsnabreastcancerdetection/test.csv')\ntest_csv","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:01.95917Z","iopub.execute_input":"2023-01-28T07:28:01.95982Z","iopub.status.idle":"2023-01-28T07:28:02.011219Z","shell.execute_reply.started":"2023-01-28T07:28:01.959773Z","shell.execute_reply":"2023-01-28T07:28:02.010232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### As noted in [A Brief Intro to Mammography](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369262) \n\n* For this task, we are focused on screening mammograms, which again means that only scores of BI-RADS 0, 1, or 2 are possible. One can essentially think of 0 as \"abnormal\" and 1 and 2 as \"normal.\" There can be subjectivity in assigning BI-RADS 1 or 2. For example, if there are stable findings that are almost certainly benign breast cysts, the mammogram may be assigned 1 or 2 depending on the radiologist.\n\n* Our task in the challenge is to predict cancer or no cancer, a binary value. This may be obvious, but not all BI-RADS 0 cases have cancer. The final cancer label will depend on the outcome of the diagnostic mammogram and the biopsy results, if obtained.\n\ntrain_05.csv is a modified version of the original training set which was created in Excel where:\n\n* An \"outcome\" column has been created where:<br>\n0 = A BI-RAD score of 1 or 2<br>\n1 = A BI-RAD score of 0<br>\n\n* lbl_0 is a multi-label classification showing Laterality, View, whether Cancer was diagnosed and whether the 'BIRADS' score was normal or abnormal\n* lbl_1 is similar but excludes the Cancer diagnosis\n* lbl_2 excludes Laterality\n\nSince the test set only includes MLO and CC views only rows that have these views are included\n\ntrain_06.csv only shows a 'BIRAD' score of 0","metadata":{}},{"cell_type":"code","source":"train_csv = pd.read_csv('../input/rsnabreastcancerdetection/train_06.csv')\ntrain_csv['cancer'] = train_csv['cancer'].astype(str)\n\ntrain_csv.head(5)","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:06.798614Z","iopub.execute_input":"2023-01-28T07:28:06.799059Z","iopub.status.idle":"2023-01-28T07:28:06.922655Z","shell.execute_reply.started":"2023-01-28T07:28:06.799022Z","shell.execute_reply":"2023-01-28T07:28:06.921424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**excluding rows that have no BIRADS score**","metadata":{}},{"cell_type":"code","source":"df=train_csv[train_csv['BIRADS'].notna()]","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:17.935596Z","iopub.execute_input":"2023-01-28T07:28:17.936202Z","iopub.status.idle":"2023-01-28T07:28:17.951336Z","shell.execute_reply.started":"2023-01-28T07:28:17.936156Z","shell.execute_reply":"2023-01-28T07:28:17.950109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df=df[[\"img_path\",\"cancer\"]]","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:22.228378Z","iopub.execute_input":"2023-01-28T07:28:22.229344Z","iopub.status.idle":"2023-01-28T07:28:22.24346Z","shell.execute_reply.started":"2023-01-28T07:28:22.2293Z","shell.execute_reply":"2023-01-28T07:28:22.242287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Labels","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(20,6))\nplt.subplot(1,2,1)\nax1 = sns.countplot(data=df, x='cancer')\nplt.xticks(rotation = -45)\nfor container in ax1.containers:\n    ax1.bar_label(container)\nplt.title('Distribution of Labels');","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:26.831Z","iopub.execute_input":"2023-01-28T07:28:26.83152Z","iopub.status.idle":"2023-01-28T07:28:27.154833Z","shell.execute_reply.started":"2023-01-28T07:28:26.831478Z","shell.execute_reply":"2023-01-28T07:28:27.153784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Augmentation\n\nMammograms diagnosed with cancer were augmented using [this notebook](https://www.kaggle.com/code/julianmacnamara/rsna-augmentation). Essentially each image was flipped, inverted, mirrored, autocontrasted with a cutoff of 0.5 and equalised \n\nVBA was then used to create an additional training set which was appended to train_06.csv as train_07.csv\n\nThe images were added to [rsna-images-inc-augmented](https://www.kaggle.com/datasets/julianmacnamara/rsna-images-inc-augmented)","metadata":{}},{"cell_type":"code","source":"aug_csv = pd.read_csv('../input/rsnabreastcancerdetection/train_07.csv')\naug_csv['cancer'] = aug_csv['cancer'].astype(str)\n\ndf1=aug_csv[aug_csv['BIRADS'].notna()]\n\ndf1=df1[[\"img_path\",\"cancer\"]]","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:31.904572Z","iopub.execute_input":"2023-01-28T07:28:31.905064Z","iopub.status.idle":"2023-01-28T07:28:32.029178Z","shell.execute_reply.started":"2023-01-28T07:28:31.905023Z","shell.execute_reply":"2023-01-28T07:28:32.028075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,6))\nplt.subplot(1,2,1)\nax1 = sns.countplot(data=df1, x='cancer')\nplt.xticks(rotation = -45)\nfor container in ax1.containers:\n    ax1.bar_label(container)\nplt.title('Distribution of Labels');","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:37.381251Z","iopub.execute_input":"2023-01-28T07:28:37.381858Z","iopub.status.idle":"2023-01-28T07:28:37.760707Z","shell.execute_reply.started":"2023-01-28T07:28:37.381815Z","shell.execute_reply":"2023-01-28T07:28:37.759844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df1=df1[[\"img_path\",\"cancer\"]]","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:43.637021Z","iopub.execute_input":"2023-01-28T07:28:43.637758Z","iopub.status.idle":"2023-01-28T07:28:43.64711Z","shell.execute_reply.started":"2023-01-28T07:28:43.637711Z","shell.execute_reply":"2023-01-28T07:28:43.645905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path = '../input/rsna-images-inc-augmented'","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:46.869891Z","iopub.execute_input":"2023-01-28T07:28:46.870397Z","iopub.status.idle":"2023-01-28T07:28:46.875536Z","shell.execute_reply.started":"2023-01-28T07:28:46.870355Z","shell.execute_reply":"2023-01-28T07:28:46.874444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dblock=DataBlock()","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:51.117081Z","iopub.execute_input":"2023-01-28T07:28:51.117613Z","iopub.status.idle":"2023-01-28T07:28:51.124753Z","shell.execute_reply.started":"2023-01-28T07:28:51.117571Z","shell.execute_reply":"2023-01-28T07:28:51.12348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dsets=dblock.datasets(df1)","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:54.550155Z","iopub.execute_input":"2023-01-28T07:28:54.55069Z","iopub.status.idle":"2023-01-28T07:28:54.599242Z","shell.execute_reply.started":"2023-01-28T07:28:54.550646Z","shell.execute_reply":"2023-01-28T07:28:54.598228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_x(r): return r['img_path']\ndef get_y(r): return r['cancer']\ndblock = DataBlock(get_x = get_x, get_y = get_y)\ndsets = dblock.datasets(df1)\ndsets.train[0]","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:28:58.603686Z","iopub.execute_input":"2023-01-28T07:28:58.604136Z","iopub.status.idle":"2023-01-28T07:28:58.668603Z","shell.execute_reply.started":"2023-01-28T07:28:58.604089Z","shell.execute_reply":"2023-01-28T07:28:58.667455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dblock = DataBlock(blocks=(ImageBlock, CategoryBlock),\n                   get_x = get_x,\n                   get_y = get_y,\n                   splitter=RandomSplitter (valid_pct=0.2, seed=16),\n                   item_tfms=Resize(224))\ndsets = dblock.datasets(df1)\ndsets.train[0]","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:29:24.505417Z","iopub.execute_input":"2023-01-28T07:29:24.505872Z","iopub.status.idle":"2023-01-28T07:29:27.377661Z","shell.execute_reply.started":"2023-01-28T07:29:24.505834Z","shell.execute_reply":"2023-01-28T07:29:27.3697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls = dblock.dataloaders(df1)","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:29:35.935032Z","iopub.execute_input":"2023-01-28T07:29:35.93562Z","iopub.status.idle":"2023-01-28T07:29:39.895979Z","shell.execute_reply.started":"2023-01-28T07:29:35.935575Z","shell.execute_reply":"2023-01-28T07:29:39.894351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating the learner and training","metadata":{}},{"cell_type":"code","source":"learn = vision_learner(dls, models.resnet50, metrics=accuracy, pretrained=True)","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:29:46.022927Z","iopub.execute_input":"2023-01-28T07:29:46.023706Z","iopub.status.idle":"2023-01-28T07:29:47.474938Z","shell.execute_reply.started":"2023-01-28T07:29:46.023657Z","shell.execute_reply":"2023-01-28T07:29:47.473421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# learn.lr_find()","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:29:53.068425Z","iopub.execute_input":"2023-01-28T07:29:53.068868Z","iopub.status.idle":"2023-01-28T07:31:00.367101Z","shell.execute_reply.started":"2023-01-28T07:29:53.068829Z","shell.execute_reply":"2023-01-28T07:31:00.365928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.fit_one_cycle(4, 0.00301995175)","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:31:38.945821Z","iopub.execute_input":"2023-01-28T07:31:38.946491Z","iopub.status.idle":"2023-01-28T07:37:34.700886Z","shell.execute_reply.started":"2023-01-28T07:31:38.946445Z","shell.execute_reply":"2023-01-28T07:37:34.699486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_confusion_matrix()","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:38:16.084511Z","iopub.execute_input":"2023-01-28T07:38:16.085694Z","iopub.status.idle":"2023-01-28T07:38:49.505711Z","shell.execute_reply.started":"2023-01-28T07:38:16.085642Z","shell.execute_reply":"2023-01-28T07:38:49.504057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path='/kaggle/input/rsna-test-images-as-pngs/'","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:38:54.727167Z","iopub.execute_input":"2023-01-28T07:38:54.727705Z","iopub.status.idle":"2023-01-28T07:38:54.738749Z","shell.execute_reply.started":"2023-01-28T07:38:54.727655Z","shell.execute_reply":"2023-01-28T07:38:54.737662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Source: https://www.kaggle.com/code/beezus666/dicom-to-png-to-predict\n\n# Get images\ntest_images = sorted([os.path.join(path, file) for file in os.listdir(path)])\nimages = [cv2.imread(file) for file in test_images]\n\n# pass images to fast.ai learner and get predictions\ntest_dl = learn.dls.test_dl(images)\npreds_batch, _ = learn.get_preds(dl=test_dl)\npredsdec, _, decoded = learn.get_preds(dl=test_dl, with_decoded=True)\npredsdec[:10]","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:38:59.769313Z","iopub.execute_input":"2023-01-28T07:38:59.769774Z","iopub.status.idle":"2023-01-28T07:39:00.538491Z","shell.execute_reply.started":"2023-01-28T07:38:59.769737Z","shell.execute_reply":"2023-01-28T07:39:00.53729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# extract the prediction, which is the 2nd value in the tensor above\n\narray = preds_batch.numpy()\nlist_of_lists = array.tolist()\nsecond_values = [lst[1] for lst in list_of_lists]","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:39:07.269848Z","iopub.execute_input":"2023-01-28T07:39:07.270362Z","iopub.status.idle":"2023-01-28T07:39:07.281586Z","shell.execute_reply.started":"2023-01-28T07:39:07.270312Z","shell.execute_reply":"2023-01-28T07:39:07.280256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sorted_files = sorted(os.listdir(path))\nno_extensions = [os.path.splitext(name)[0] for name in sorted_files]\n\npreds_df = pd.DataFrame(data = {'concat_id':no_extensions, 'cancer':second_values})","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:39:13.331418Z","iopub.execute_input":"2023-01-28T07:39:13.331994Z","iopub.status.idle":"2023-01-28T07:39:13.346746Z","shell.execute_reply.started":"2023-01-28T07:39:13.331948Z","shell.execute_reply":"2023-01-28T07:39:13.345482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"info_df=pd.read_csv('/kaggle/input/rsna-breast-cancer-detection/test.csv')\ninfo_df['concat_id'] = info_df['image_id'].astype(str)\ninfo_df=info_df[[\"patient_id\",\"image_id\",\"laterality\",\"concat_id\"]]","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:39:17.375038Z","iopub.execute_input":"2023-01-28T07:39:17.375503Z","iopub.status.idle":"2023-01-28T07:39:17.397Z","shell.execute_reply.started":"2023-01-28T07:39:17.375463Z","shell.execute_reply":"2023-01-28T07:39:17.395937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"merged_df = pd.merge(right=info_df, left=preds_df, on='concat_id')\nmerged_df['prediction_id'] = merged_df['patient_id'].astype(str)+'_'+merged_df['laterality'].astype(str)","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:39:23.32976Z","iopub.execute_input":"2023-01-28T07:39:23.330234Z","iopub.status.idle":"2023-01-28T07:39:23.355382Z","shell.execute_reply.started":"2023-01-28T07:39:23.33017Z","shell.execute_reply":"2023-01-28T07:39:23.354121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Select only the 'prediction_id' and 'cancer' columns\nresulting_df = merged_df[['prediction_id', 'cancer']]\n\nresulting_df = resulting_df.groupby('prediction_id', as_index=False).mean()\n\nresulting_df = resulting_df.sort_index()","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:39:41.496072Z","iopub.execute_input":"2023-01-28T07:39:41.496508Z","iopub.status.idle":"2023-01-28T07:39:41.530818Z","shell.execute_reply.started":"2023-01-28T07:39:41.496473Z","shell.execute_reply":"2023-01-28T07:39:41.529721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# shutil.rmtree(\"/kaggle/working/models\")","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:40:02.63529Z","iopub.execute_input":"2023-01-28T07:40:02.635733Z","iopub.status.idle":"2023-01-28T07:40:02.641364Z","shell.execute_reply.started":"2023-01-28T07:40:02.635695Z","shell.execute_reply":"2023-01-28T07:40:02.640266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i=resulting_df['cancer'][0]\ni","metadata":{"execution":{"iopub.status.busy":"2023-01-28T08:11:11.600119Z","iopub.execute_input":"2023-01-28T08:11:11.600613Z","iopub.status.idle":"2023-01-28T08:11:11.609078Z","shell.execute_reply.started":"2023-01-28T08:11:11.600575Z","shell.execute_reply":"2023-01-28T08:11:11.608014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"j=resulting_df['cancer'][1]\nj","metadata":{"execution":{"iopub.status.busy":"2023-01-28T08:11:47.44843Z","iopub.execute_input":"2023-01-28T08:11:47.448925Z","iopub.status.idle":"2023-01-28T08:11:47.45759Z","shell.execute_reply.started":"2023-01-28T08:11:47.448889Z","shell.execute_reply":"2023-01-28T08:11:47.456549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_df=pd.read_csv('/kaggle/input/rsna-breast-cancer-detection/sample_submission.csv')\nfinal_df.loc[0:,\"cancer\"]=i\nfinal_df.loc[1:,\"cancer\"]=j\nfinal_df","metadata":{"execution":{"iopub.status.busy":"2023-01-28T08:21:41.417641Z","iopub.execute_input":"2023-01-28T08:21:41.418093Z","iopub.status.idle":"2023-01-28T08:21:41.442692Z","shell.execute_reply.started":"2023-01-28T08:21:41.418055Z","shell.execute_reply":"2023-01-28T08:21:41.441627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-01-28T07:40:30.129655Z","iopub.execute_input":"2023-01-28T07:40:30.130128Z","iopub.status.idle":"2023-01-28T07:40:30.148106Z","shell.execute_reply.started":"2023-01-28T07:40:30.130088Z","shell.execute_reply":"2023-01-28T07:40:30.146922Z"},"trusted":true},"execution_count":null,"outputs":[]}]}