{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook contains the complete, end to end pipeline for training and successfully predicting on test!\n\nHere are the things we'll cover.\n\n1. [Training a fast.ai model](#section-one)\n2. [Using the model to predict on test](#section-two)\n3. [Making a submission](#section-three)\n\nFor training, we will use images preprocessed to PNGs that I shared here: [RSNA Mammo PNGs 256px, 384px, 512px, 768px, 1024px](https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs)\n\nLet's get started! 🚀","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"## Other resources you might find useful:\n\n* [💡 how to process DICOM images to PNGs](https://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs)\n* [📊 EDA + training a fast.ai model + submission 🚀](https://www.kaggle.com/code/radek1/eda-training-a-fast-ai-model-submission)","metadata":{}},{"cell_type":"markdown","source":"### Versions\n\n* **v10**: res18, 4 epochs trained in submission, 256px, 3 splits, `CV: 0.046/0.076 LB: 0.05`, unfortunately had a bug in best thresh calulation\n* **v11**: res18, 4 epochs trained in submission, 256px, 4 splits, ⏰ times out\n* **v15**: res18, 4 epochs trained externally, 256px, CPU (the processing of images might be the most expensive part of inference! especially with resnet18, 2 CPUs -> 4 CPUs might be the way to go)\n* **v16**: res18, 4 epochs trained externally, 256px, CPU, higher thresh (the means of predictions should be better than predictions from an individual model, a higher threshold *might* work better)\n* **v17**: res18, like `v15`, but trained with cherry picking, either performing well in general (doubt it) or overfitted to local CV\n* **v18**: tf_efficientnetv2_s, 512x512, single model\n* **v19**: tf_efficientnetv2_s, 512x512, ensemble of 4 models\n* **v20**: tf_efficientnetv2_s, like v19, but running with different pretrained weights, main difference -- this one runs on the CPU\n* **v22**: like v18, but taking max instead of mean of predictions on images, that makes more sense but who knows if it will work better in practice\n* **v23**: moving to new arch and adding VOI LUT transformation (might make the pipeline time out as it makes the processing a bit longer)\n* **v24**: like v23 but with mean of individual thresholds recorded during training\n* **v25**: like v23 but without VOI LUT processing\n* **v26**: a single model with particularly good optimized pfbeta1 of just under 0.196\n* **v27**: a single model with an optimized pfbeta1 of 0.212\n* **v28**: a single model with an optimized pfbeta1 of 0.249\n* **v30**: like v28, but with mean aggregation of predictions by `prediction_id`\n* **v31**: a single model with an optimized pfbeta1 of 0.252\n* **v32**: a single model with an optimized pfbeta1 of 0.207 trained on 1024x1024\n* **v33**: similar to v32, but trained with augmentation and results in much higher threshold (plus locally the difference between pfbeta and optimized pfbeta is much greater)\n* **v34**: like v32, but trained for one more epoch and on a different fold\n* **v35**: an ensemble of v32 and v34","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"### Updates (account of working on this -- documents the despair of an ML practitioner we so often encounter!)\n\n**12/04**: I find myself in the spot so often encountered by DL practitioners -- the model simply doesn't train! Or rather, it trains poorly to the point where one cannot be certain to what extent it is making good use of the available information.\n\nThis is an interesting case as a lot can be happening. From what I observed on the forums (special thanks go to Theo Viel and his awesome notebook: [RSNA Breast Baseline - Inference](https://www.kaggle.com/code/theoviel/rsna-breast-baseline-inference), now there is also [this notebook](https://www.kaggle.com/code/hengck23/notebooke04a738685) by hengck23), people are feeding the data to their model in a way that I wouldn't expect. Theo is only normalizing it to between 0 and 1.\n\nBut this might mean that some important information to the model is lost in the processing that I do here. Or it could be a hunderd of other small things the community has arrived at through so many RSNA competitions 🙂.\n\nThree things to do:\n\n* explore various ways of phrsing the problem within `fastai` (training with Sigmoid, Softmax, etc)\n* get the pipeline on Kaggle working with what I have\n    * to always have a working pipeline version (otherwise it is very easy to lose momentum on a project)\n    * to be verifying if local improvements generalize to LB (so far, I have a very weak handle on what is going on in this competition, the more data I collect the better)\n* after the above, see what happens if I feed data to my model either how Theo or hengck23 does it (normalized to [0,1], zero-centered and standard dev normalized)\n\n**12/05**: I stayed up for more than half the night on Saturday and Sunday to work on this as I knew I wouldn't get much time to give this over the week. There can be an all-consuming quality to DL projects if you are not careful.\n\nI reimplemented the entire pipeline in pure PyTorch... and it doesn't train in the SAME WAY as my fast.ai pipeline...\n\n```\nstep:     1    val_loss: 0.686    pf1: 0.042    auc: 0.464\nstep:   500    val_loss: 0.108    pf1: 0.023    auc: 0.479\nstep:  1000    val_loss: 0.106    pf1: 0.023    auc: 0.579\nstep:  1500    val_loss: 0.105    pf1: 0.021    auc: 0.558\nstep:  2000    val_loss: 0.107    pf1: 0.020    auc: 0.524\nstep:  2500    val_loss: 0.106    pf1: 0.024    auc: 0.599\nstep:  3000    val_loss: 0.105    pf1: 0.023    auc: 0.575\nstep:  3500    val_loss: 0.104    pf1: 0.025    auc: 0.610\nstep:  4000    val_loss: 0.103    pf1: 0.027    auc: 0.623\nstep:  4500    val_loss: 0.104    pf1: 0.027    auc: 0.612\nstep:  5000    val_loss: 0.104    pf1: 0.027    auc: 0.607\nstep:  5125    val_loss: 0.104    pf1: 0.027    auc: 0.606\nCPU times: user 12min 15s, sys: 38 s, total: 12min 53s\nWall time: 12min 56s\n```\n\nI replaced the training loop, how data is read, the data I am reading, added a metric from `torchmetrics` for another bit of sanity check and also did a couple of things that don't make sense but you essentially question everything when stuff doesn't train. The things I tried that didn't make much sense was training on larger images (why 512px should work if 256px doesn't train reasonably?) and training with sigmoid and BCE vs softmax and nll (with two classes they should be equivalent). Oh yeah, I also tried different models of course, directly from fast.ai, the one from Theo, pretrained, not pretrained, etc.\n\nAND YET THERE ARE PEOPLE ON THE LB whose models do seem to train, potentially with ease.\n\nWhat is the secret? What am I missing?\n\nThe only low hanging fruit I can think of now is addressing the class imbalance... Counting by images, there are roughly 2% of positive examples in the dataset. Maybe this can make a difference.\n\nOther than that, the only 3 components I haven't tried replacing so far are:\n* the researcher working on this (me)\n* the splitting functionality into train and val\n* my computer\n* *maybe* the dataset, as in, would my code train on cats and dogs?\n\n**12/05**: I think I found a way to train the model so that it works 😭😭😭😭😭 Don't want to jinx it though 🙂 Let me move it over to the Kaggle pipeline and let us see how it fares here 🙂 Writing [the thoughts down in this thread](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370587) was really helpful 🙏","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"<a id=\"section-one\"></a>\n# Training a fast.ai model\n","metadata":{"execution":{"iopub.status.busy":"2022-12-02T00:49:16.208242Z","iopub.execute_input":"2022-12-02T00:49:16.208915Z","iopub.status.idle":"2022-12-02T00:49:16.247884Z","shell.execute_reply.started":"2022-12-02T00:49:16.20876Z","shell.execute_reply":"2022-12-02T00:49:16.244677Z"}}},{"cell_type":"code","source":"!unzip -q ../input/timm-with-dependencies/timm_all -d timm-with-dependencies\n!pip install --no-index --find-links timm-with-dependencies timm\n!pip install /kaggle/input/dicomsdl-offline-installer/dicomsdl-0.109.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl\n\nfrom fastai.vision.learner import *\nfrom fastai.data.all import *\nfrom fastai.vision.all import *\nfrom fastai.metrics import ActivationType\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom collections import defaultdict\nimport pandas as pd\nimport numpy as np\nfrom pdb import set_trace","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:36:40.390747Z","iopub.execute_input":"2022-12-11T13:36:40.391051Z","iopub.status.idle":"2022-12-11T13:37:50.401066Z","shell.execute_reply.started":"2022-12-11T13:36:40.390984Z","shell.execute_reply":"2022-12-11T13:37:50.399966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_EPOCHS = 4\nNUM_SPLITS = 4\n\nRESIZE_TO = (1024, 1024)\n\nDATA_PATH = '/kaggle/input/rsna-breast-cancer-detection'\nTRAIN_IMAGE_DIR = '/kaggle/input/rsna-mammography-images-as-pngs/images_as_pngs_cv2_512'\nTEST_DICOM_DIR = '/kaggle/input/rsna-breast-cancer-detection/test_images'\nMODEL_PATH = '/kaggle/input/rsna-trained-model-weights/tf_effv2_s_208_402/tf_effv2_s_208_402'\n\nlabel_smoothing_weights = torch.tensor([1,10]).float()\nif torch.cuda.is_available():\n    label_smoothing_weights = label_smoothing_weights.cuda()","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:37:50.404007Z","iopub.execute_input":"2022-12-11T13:37:50.404934Z","iopub.status.idle":"2022-12-11T13:37:53.386183Z","shell.execute_reply.started":"2022-12-11T13:37:50.404894Z","shell.execute_reply":"2022-12-11T13:37:53.385213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating stratified splits for training","metadata":{}},{"cell_type":"code","source":"train_csv = pd.read_csv(f'{DATA_PATH}/train.csv')\npatient_id_any_cancer = train_csv.groupby('patient_id').cancer.max().reset_index()\nskf = StratifiedKFold(NUM_SPLITS, shuffle=True, random_state=42)\nsplits = list(skf.split(patient_id_any_cancer.patient_id, patient_id_any_cancer.cancer))","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:37:53.387844Z","iopub.execute_input":"2022-12-11T13:37:53.38824Z","iopub.status.idle":"2022-12-11T13:37:53.517987Z","shell.execute_reply.started":"2022-12-11T13:37:53.388203Z","shell.execute_reply":"2022-12-11T13:37:53.517051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Defining some helper functions","metadata":{}},{"cell_type":"markdown","source":"I am defining some functionality here to make our life easier and to get us the last 10% of the way to a really good result.\n\nGenerally, none of this code is core to training or predicting, we could skip most of it and still be able to get a well trained model.\n\n\nBut here we want to push the boundaries of performance so let's get these things in 🙂","metadata":{}},{"cell_type":"code","source":"#https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369267  \ndef pfbeta_torch(preds, labels, beta=1):\n    if preds.dim() != 2 or (preds.dim() == 2 and preds.shape[1] !=2): raise ValueError('Houston, we got a problem')\n    preds = preds[:, 1]\n    preds = preds.clip(0, 1)\n    y_true_count = labels.sum()\n    ctp = preds[labels==1].sum()\n    cfp = preds[labels==0].sum()\n    beta_squared = beta * beta\n    c_precision = ctp / (ctp + cfp)\n    c_recall = ctp / y_true_count\n    if (c_precision > 0 and c_recall > 0):\n        result = (1 + beta_squared) * (c_precision * c_recall) / (beta_squared * c_precision + c_recall)\n        return result\n    else:\n        return 0.0\n\n# https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369886    \ndef pfbeta_torch_thresh(preds, labels):\n    optimized_preds = optimize_preds(preds, labels)\n    return pfbeta_torch(optimized_preds, labels)\n\ndef optimize_preds(preds, labels=None, thresh=None, return_thresh=False, print_results=False):\n    preds = preds.clone()\n    if labels is not None: without_thresh = pfbeta_torch(preds, labels)\n    \n    if not thresh and labels is not None:\n        threshs = np.linspace(0, 1, 101)\n        f1s = [pfbeta_torch((preds > thr).float(), labels) for thr in threshs]\n        idx = np.argmax(f1s)\n        thresh, best_pfbeta = threshs[idx], f1s[idx]\n\n    preds = (preds > thresh).float()\n\n    if print_results:\n        print(f'without optimization: {without_thresh}')\n        pfbeta = pfbeta_torch(preds, labels)\n        print(f'with optimization: {pfbeta}')\n        print(f'best_thresh = {thresh}')\n    if return_thresh:\n        return thresh\n    return preds\n\nfn2label = {fn: cancer_or_not for fn, cancer_or_not in zip(train_csv['image_id'].astype('str'), train_csv['cancer'])}\n\ndef splitting_func(paths):\n    train = []\n    valid = []\n    for idx, path in enumerate(paths):\n        if int(path.parent.name) in patient_id_any_cancer.iloc[splits[SPLIT][0]].patient_id.values:\n            train.append(idx)\n        else:\n            valid.append(idx)\n    return train, valid\n\ndef label_func(path):\n    return fn2label[path.stem]\n\ndef get_items(image_dir_path):\n    items = []\n    for p in get_image_files(image_dir_path):\n        items.append(p)\n        if p.stem in fn2label and int(p.parent.name) in patient_id_any_cancer.iloc[splits[SPLIT][0]].patient_id.values:\n            if label_func(p) == 1:\n                for _ in range(5):\n                    items.append(p)\n    return items","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-11T13:37:53.521651Z","iopub.execute_input":"2022-12-11T13:37:53.52197Z","iopub.status.idle":"2022-12-11T13:37:53.585039Z","shell.execute_reply.started":"2022-12-11T13:37:53.521943Z","shell.execute_reply":"2022-12-11T13:37:53.584203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wrapping getting data and getting a model into functions -- this way our logic for training will be cleaner to read.","metadata":{}},{"cell_type":"code","source":"from timm.models.layers.adaptive_avgmax_pool import SelectAdaptivePool2d\nfrom torch.nn import Flatten\n\ndef get_dataloaders():\n    train_image_path = TRAIN_IMAGE_DIR\n\n    dblock = DataBlock(\n        blocks    = (ImageBlock, CategoryBlock),\n        get_items = get_items,\n        get_y = label_func,\n        splitter  = splitting_func,\n        batch_tfms=[Flip()],\n    )\n    dsets = dblock.datasets(train_image_path)\n    return dblock.dataloaders(train_image_path, batch_size=32)\n\ndef get_learner(arch=resnet18):\n    learner = vision_learner(\n        get_dataloaders(),\n        arch,\n        custom_head=nn.Sequential(SelectAdaptivePool2d(pool_type='avg', flatten=Flatten()), nn.Linear(1280, 2)),\n        metrics=[\n            error_rate,\n            AccumMetric(pfbeta_torch, activation=ActivationType.Softmax, flatten=False),\n            AccumMetric(pfbeta_torch_thresh, activation=ActivationType.Softmax, flatten=False)\n        ],\n        loss_func=CrossEntropyLossFlat(weight=torch.tensor([1,50]).float()),\n        pretrained=True,\n        normalize=False\n    ).to_fp16()\n    return learner","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:37:53.587124Z","iopub.execute_input":"2022-12-11T13:37:53.587799Z","iopub.status.idle":"2022-12-11T13:37:53.595877Z","shell.execute_reply.started":"2022-12-11T13:37:53.58776Z","shell.execute_reply":"2022-12-11T13:37:53.595034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating the learner and training","metadata":{}},{"cell_type":"code","source":"# This is a dependency that is needed for reading DICOM images\n\ntry:\n    import pylibjpeg\nexcept:\n    !rm -rf /root/.cache/torch/hub/checkpoints/\n    !mkdir -p /root/.cache/torch/hub/checkpoints/\n    !pip install /kaggle/input/rsna-2022-whl/{pydicom-2.3.0-py3-none-any.whl,pylibjpeg-1.4.0-py3-none-any.whl,python_gdcm-3.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl}\n    !pip install /kaggle/input/rsna-2022-whl/{torch-1.12.1-cp37-cp37m-manylinux1_x86_64.whl,torchvision-0.13.1-cp37-cp37m-manylinux1_x86_64.whl}\n\n# copying the pretrained weights\n\nif not os.path.exists('/root/.cache/torch/hub/checkpoints/'):\n        os.makedirs('/root/.cache/torch/hub/checkpoints/')\n!cp '/kaggle/input/pretrained-model-weights-for-fastai/resnet18-f37072fd.pth' '/root/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth'\n!cp '/kaggle/input/pretrained-model-weights-for-fastai/tf_efficientnetv2_s-eb54923e.pth' '/root/.cache/torch/hub/checkpoints/tf_efficientnetv2_s-eb54923e.pth'","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:37:53.597275Z","iopub.execute_input":"2022-12-11T13:37:53.597892Z","iopub.status.idle":"2022-12-11T13:39:40.524954Z","shell.execute_reply.started":"2022-12-11T13:37:53.597856Z","shell.execute_reply":"2022-12-11T13:39:40.523668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\npreds, labels = [], []\n\nSPLIT = 0 # our learner needs this to construct its dataloaders...\nlearn = get_learner('tf_efficientnetv2_s')\n\n# instead of training, to conserve pipeline time, I am uploading models trained locally\n# uncomment the lines below for training\n  \n# for SPLIT in range(NUM_SPLITS):\n#     learn = get_learner()\n#     learn.unfreeze()\n#     learn.fit_one_cycle(NUM_EPOCHS, 1e-4, pct_start=0.1)\n#     learn.save(f'{MODEL_PATH}/{SPLIT}')\n        \n#     output = learn.get_preds()\n#     preds.append(output[0])\n#     labels.append(output[1])","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:39:40.526957Z","iopub.execute_input":"2022-12-11T13:39:40.527385Z","iopub.status.idle":"2022-12-11T13:41:51.947327Z","shell.execute_reply.started":"2022-12-11T13:39:40.527351Z","shell.execute_reply":"2022-12-11T13:41:51.945934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# threshold = optimize_preds(torch.cat(preds), torch.cat(labels), return_thresh=True, print_results=True)\nthreshold = 0.402","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:41:51.948973Z","iopub.execute_input":"2022-12-11T13:41:51.951477Z","iopub.status.idle":"2022-12-11T13:41:51.956585Z","shell.execute_reply.started":"2022-12-11T13:41:51.951433Z","shell.execute_reply":"2022-12-11T13:41:51.955205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predicting on test<a id=\"section-two\">","metadata":{}},{"cell_type":"code","source":"# import pydicom\n# from pydicom.pixel_data_handlers.util import apply_voi_lut\nimport dicomsdl\n    \nfrom pathlib import Path\nimport multiprocessing as mp\nimport cv2\n\n!rm -rf test_resized_{RESIZE_TO[0]}\n\ndef dicom_file_to_ary(path):\n    dcm_file = dicomsdl.open(str(path))\n    data = dcm_file.pixelData()\n\n    data = (data - data.min()) / (data.max() - data.min())\n\n    if dcm_file.getPixelDataInfo()['PhotometricInterpretation'] == \"MONOCHROME1\":\n        data = 1 - data\n\n    data = cv2.resize(data, RESIZE_TO)\n    data = (data * 255).astype(np.uint8)\n    return data\n\ndirectories = list(Path(TEST_DICOM_DIR).iterdir())\n\ndef process_directory(directory_path):\n    parent_directory = str(directory_path).split('/')[-1]\n    !mkdir -p test_resized_{RESIZE_TO[0]}/{parent_directory}\n    for image_path in directory_path.iterdir():\n        processed_ary = dicom_file_to_ary(image_path)\n        cv2.imwrite(\n            f'test_resized_{RESIZE_TO[0]}/{parent_directory}/{image_path.stem}.png',\n            processed_ary\n        )\n\nwith mp.Pool(mp.cpu_count()) as p:\n    p.map(process_directory, directories)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-11T13:41:51.95815Z","iopub.execute_input":"2022-12-11T13:41:51.958542Z","iopub.status.idle":"2022-12-11T13:41:56.990316Z","shell.execute_reply.started":"2022-12-11T13:41:51.958508Z","shell.execute_reply":"2022-12-11T13:41:56.988893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\npreds_all = []\n\ntest_dl = learn.dls.test_dl(get_image_files(f'test_resized_{RESIZE_TO[0]}'))\nfor SPLIT in range(NUM_SPLITS):\n    learn.load(f'{MODEL_PATH}/{SPLIT}')\n    preds, _ = learn.get_preds(dl=test_dl)\n    preds_all.append(preds)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:41:56.994585Z","iopub.execute_input":"2022-12-11T13:41:56.994937Z","iopub.status.idle":"2022-12-11T13:42:09.296422Z","shell.execute_reply.started":"2022-12-11T13:41:56.994904Z","shell.execute_reply":"2022-12-11T13:42:09.295252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = torch.zeros_like(preds_all[0])\nfor pred in preds_all:\n    preds += pred\n\npreds /= NUM_SPLITS\n\n\npreds = optimize_preds(preds, thresh=threshold)\nimage_ids = [path.stem for path in test_dl.items]\n\nimage_id2pred = defaultdict(lambda: 0)\nfor image_id, pred in zip(image_ids, preds[:, 1]):\n    image_id2pred[int(image_id)] = pred.item()","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:42:09.300997Z","iopub.execute_input":"2022-12-11T13:42:09.302779Z","iopub.status.idle":"2022-12-11T13:42:09.314005Z","shell.execute_reply.started":"2022-12-11T13:42:09.30274Z","shell.execute_reply":"2022-12-11T13:42:09.312887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-three\"></a>\n# Making a submission","metadata":{}},{"cell_type":"code","source":"test_csv = pd.read_csv('/kaggle/input/rsna-breast-cancer-detection/test.csv')\n\nprediction_ids = []\npreds = []\n\nfor _, row in test_csv.iterrows():\n    prediction_ids.append(row.prediction_id)\n    preds.append(image_id2pred[row.image_id])\n\nsubmission = pd.DataFrame(data={'prediction_id': prediction_ids, 'cancer': preds}).groupby('prediction_id').max().reset_index()\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:42:09.315357Z","iopub.execute_input":"2022-12-11T13:42:09.315951Z","iopub.status.idle":"2022-12-11T13:42:09.369409Z","shell.execute_reply.started":"2022-12-11T13:42:09.315808Z","shell.execute_reply":"2022-12-11T13:42:09.368338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T13:42:09.373723Z","iopub.execute_input":"2022-12-11T13:42:09.374142Z","iopub.status.idle":"2022-12-11T13:42:09.386693Z","shell.execute_reply.started":"2022-12-11T13:42:09.374108Z","shell.execute_reply":"2022-12-11T13:42:09.385713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And that's it! Thank you very much for reading! 🙂\n\n**If you enjoyed the notebook, please upvote! 🙏 Thank you, appreciate your support!**\n\nHappy Kaggling 🥳\n","metadata":{}}]}