{"cells":[{"metadata":{},"cell_type":"markdown","source":"In the notebook \"[Cleaning the data for rapid prototyping](https://www.kaggle.com/jhoward/cleaning-the-data-for-rapid-prototyping-fastai)\" I showed how to create a small, fast, ready-to-use dataset for prototyping our models. The dataset created in that notebook, along with the metadata files it uses, are now [available here](https://www.kaggle.com/jhoward/rsna-hemorrhage-jpg).\n\nSo let's use them to create a model! In this notebook we'll see the whole journey from pre-training using progressive resizing on our prototyping sample, through to fine-tuning on the full dataset, and then submitting to the competition.\n\nI'm intentionally not doing any tricky modeling in this notebook, because I want to show the power of simple techniques and simples architectures. You should take this as a starting point and experiment! e.g. try data augmentation methods, architectures, preprocessing approaches, using the DICOM metadata, and so forth...\n\nWe'll be using the fastai.medical.imaging library here - for more information about this see the notebook [Some DICOM gotchas to be aware of](https://www.kaggle.com/jhoward/some-dicom-gotchas-to-be-aware-of-fastai). We'll also use the same basic setup that's in the notebook.\n\n*Update: I'm out of GPU hours and Kaggle isn't freezing when running the current version of the notebook. To see a complete run, see [this version](https://www.kaggle.com/jhoward/from-prototyping-to-submission-fastai?scriptVersionId=22577538). I've commented out the GPU calls in this run so I can run it end to end.*"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install torch torchvision feather-format kornia pyarrow --upgrade   > /dev/null\n!pip install git+https://github.com/fastai/fastai2 > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastai2.basics           import *\nfrom fastai2.vision.all       import *\nfrom fastai2.medical.imaging  import *\nfrom fastai2.callback.tracker import *\nfrom fastai2.callback.all     import *\n\nnp.set_printoptions(linewidth=120)\nmatplotlib.rcParams['image.cmap'] = 'bone'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First we read in the metadata files (linked in the introduction)."},{"metadata":{"trusted":true},"cell_type":"code","source":"path = Path('../input/rsna-intracranial-hemorrhage-detection/rsna-intracranial-hemorrhage-detection/')\npath_trn = path/'stage_2_train'\npath_tst = path/'stage_2_test'\n\npath_inp = Path('../input')\npath_xtra = path_inp/'rsna-hemorrhage-jpg'\npath_meta = path_xtra/'meta'/'meta'\npath_jpg = path_xtra/'train_jpg'/'train_jpg'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_comb = pd.read_feather(path_meta/'comb.fth').set_index('SOPInstanceUID')\ndf_tst  = pd.read_feather(path_meta/'df_tst.fth').set_index('SOPInstanceUID')\ndf_samp = pd.read_feather(path_meta/'wgt_sample.fth').set_index('SOPInstanceUID')\nbins = (path_meta/'bins.pkl').load()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train vs valid"},{"metadata":{},"cell_type":"markdown","source":"To get better validation measures, we should split on patients, not just on studies, since that's how the test set is created.\n\nHere's a list of random patients:"},{"metadata":{"trusted":true},"cell_type":"code","source":"set_seed(42)\npatients = df_comb.PatientID.unique()\npat_mask = np.random.random(len(patients))<0.8\npat_trn = patients[pat_mask]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can use that to take just the patients in a dataframe that match that mask:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def split_data(df):\n    idx = L.range(df)\n    mask = df.PatientID.isin(pat_trn)\n    return idx[mask],idx[~mask]\n\nsplits = split_data(df_samp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's double-check that for a patient in the training set that their images are all in the first split."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_trn = df_samp.iloc[splits[0]]\np1 = L.range(df_samp)[df_samp.PatientID==df_trn.PatientID[0]]\nassert len(p1) == len(set(p1) & set(splits[0]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare sample DataBunch"},{"metadata":{},"cell_type":"markdown","source":"We will grab our sample filenames for the initial pretraining."},{"metadata":{"trusted":true},"cell_type":"code","source":"def filename(o): return os.path.splitext(os.path.basename(o))[0]\n\nfns = L(list(df_samp.fname)).map(filename)\nfn = fns[0]\nfn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We need to create a `DataBunch` that contains our sample data, so we need a function to convert a filename (pointing at a DICOM file) into a path to our sample JPEG files:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def fn2image(fn): return PILCTScan.create((path_jpg/fn).with_suffix('.jpg'))\nfn2image(fn).show();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We also need to be able to grab the labels from this, which we can do by simply indexing into our sample `DataFrame`."},{"metadata":{"trusted":true},"cell_type":"code","source":"htypes = ['any','epidural','intraparenchymal','intraventricular','subarachnoid','subdural']\ndef fn2label(fn): return df_comb.loc[fn][htypes].values.astype(np.float32)\nfn2label(fn)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you have a larger GPU or more workers, change batchsize and number-of-workers here:"},{"metadata":{"trusted":true},"cell_type":"code","source":"bs,nw = 128,4","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We're going to use fastai's new [Transform Pipeline API](http://dev.fast.ai/pets.tutorial.html) to create the DataBunch, since this is extremely flexible, which is great for intermediate and advanced Kagglers. (Beginners will probably want to stick with the Data Blocks API). We create two transform pipelines, one to open the image file, and one to look up the label and create a tensor of categories."},{"metadata":{"trusted":true},"cell_type":"code","source":"tfms = [[fn2image], [fn2label,EncodedMultiCategorize(htypes)]]\ndsrc = Datasets(fns, tfms, splits=splits)\nnrm = Normalize(tensor([0.6]).cuda(),tensor([0.25]).cuda())\naug = aug_transforms(p_lighting=0.)\nbatch_tfms = [IntToFloatTensor(), nrm, *aug]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To support progressive resizing (one of the most useful tricks in the deep learning practitioner's toolbox!) we create a function that returns a dataset resized to a requested size:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_data(bs, sz):\n    return dsrc.dataloaders(bs=bs, num_workers=nw,after_item=[ToTensor],after_batch=batch_tfms+[AffineCoordTfm(size=sz)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's try it out!"},{"metadata":{"trusted":true},"cell_type":"code","source":"dbch = get_data(128, 96)\nxb,yb = to_cpu(dbch.one_batch())\ndbch.show_batch(max_n=4, figsize=(9,6))\nxb.mean(),xb.std(),xb.shape,len(dbch.train_ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's track the accuracy of the *any* label as our main metric, since it's easy to interpret."},{"metadata":{"trusted":true},"cell_type":"code","source":"def accuracy_any(inp, targ, thresh=0.5, sigmoid=True):\n    inp,targ = flatten_check(inp[:,0],targ[:,0])\n    if sigmoid: inp = inp.sigmoid()\n    return ((inp>thresh)==targ.bool()).float().mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The loss function in this competition is weighted, so let's train using that loss function too."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_loss(scale=1.0):\n    loss_weights = tensor(2.0, 1, 1, 1, 1, 1).cuda()*scale\n    return BaseLoss(nn.BCEWithLogitsLoss, pos_weight=loss_weights, floatify=True, flatten=False, \n        is_2d=False, activation=torch.sigmoid)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll scale the loss initially to account for our sampling (since the original data had 14% rows with a positive label, and we resampled it to 50/50)."},{"metadata":{"trusted":true},"cell_type":"code","source":"loss_func = get_loss(0.14*2)\nopt_func = partial(Adam, wd=0.01, eps=1e-3)\nmetrics=[accuracy_multi,accuracy_any]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we're ready to create our learner. We can use mixed precision (fp16) by simply adding a call to `to_fp16()`!"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_learner():\n    dbch = get_data(128,128)\n    learn = cnn_learner(dbch, xresnet50, loss_func=loss_func, opt_func=opt_func, metrics=metrics)\n    return learn.to_fp16()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = get_learner()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Leslie Smith's famous LR finder will give us a reasonable learning rate suggestion."},{"metadata":{"trusted":true},"cell_type":"code","source":" lrf = learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pretrain on sample"},{"metadata":{},"cell_type":"markdown","source":"Here's our main routine for changing the size of the images in our DataBunch, doing one fine-tuning of the final layers, and then training the whole model for a few epochs."},{"metadata":{"trusted":true},"cell_type":"code","source":"def do_fit(bs,sz,epochs,lr, freeze=True):\n    learn.dbunch = get_data(bs, sz)\n    if freeze:\n        if learn.opt is not None: learn.opt.clear_state()\n        learn.freeze()\n        learn.fit_one_cycle(1, slice(lr))\n    learn.unfreeze()\n    learn.fit_one_cycle(epochs, slice(lr))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can pre-train at different sizes."},{"metadata":{"trusted":true},"cell_type":"code","source":"do_fit(128, 96, 4, 1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"do_fit(128, 160, 3, 1e-3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Scale up to full dataset"},{"metadata":{},"cell_type":"markdown","source":"Now let's fine tune this model on the full dataset. We'll need all the filenames now, not just the sample."},{"metadata":{"trusted":true},"cell_type":"code","source":"fns = L(list(df_comb.fname)).map(filename)\nsplits = split_data(df_comb)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These functions are copied nearly verbatim from our [earlier cleanup notebook](https://www.kaggle.com/jhoward/cleaning-the-data-for-rapid-prototyping-fastai), so have a look there for details."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fix_pxrepr(dcm):\n    if dcm.PixelRepresentation != 0 or dcm.RescaleIntercept<-100: return\n    x = dcm.pixel_array + 1000\n    px_mode = 4096\n    x[x>=px_mode] = x[x>=px_mode] - px_mode\n    dcm.PixelData = x.tobytes()\n    dcm.RescaleIntercept = -1000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def dcm_tfm(fn): \n    fn = (path_trn/fn).with_suffix('.dcm')\n    try:\n        x = fn.dcmread()\n        fix_pxrepr(x)\n    except Exception as e:\n        print(fn,e)\n        raise SkipItemException\n    if x.Rows != 512 or x.Columns != 512: x.zoom_to((512,512))\n    px = x.scaled_px\n    return TensorImage(px.to_3chan(dicom_windows.brain,dicom_windows.subdural, bins=bins))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dcm = dcm_tfm(fns[0])\nshow_images(dcm)\ndcm.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have some slight changes to our data source"},{"metadata":{"trusted":true},"cell_type":"code","source":"tfms = [[dcm_tfm], [fn2label,EncodedMultiCategorize(htypes)]]\ndsrc = Datasets(fns, tfms, splits=splits)\nbatch_tfms = [nrm, *aug]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_data(bs, sz):\n    return dsrc.dataloaders(bs=bs, num_workers=nw, after_batch=batch_tfms+[AffineCoordTfm(size=sz)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can test it out:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dbch = get_data(64,256)\nx,y = to_cpu(dbch.one_batch())\ndbch.show_batch(max_n=4)\nx.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We need to remove the sample scaling from our loss function, since we're using the full dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.loss_func = get_loss(1.0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can now fine-tune the final layers."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fit_tune(bs, sz, epochs, lr):\n    dbch = get_data(bs, sz)\n    learn.dbunch = dbch\n    learn.opt.clear_state()\n    learn.unfreeze()\n    learn.fit_one_cycle(epochs, slice(lr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fit_tune(64, 256, 2, 1e-3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare for submission"},{"metadata":{},"cell_type":"markdown","source":"Now we're ready to submit. We can use the handy `test_dl` function to get an inference `DataLoader` ready, then we can check it looks OK."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_fns = [(path_tst/f'{filename(o)}.dcm').absolute() for o in df_tst.fname.values]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"testFns = L(list(test_fns)).map(filename)\ntst = test_set(dsrc, testFns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We pass that to `get_preds` to get our predictions, and then clamp them just in case we have some extreme values."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.dls.device","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"next(learn.model.parameters()).is_cuda","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#preds,targs = learn.get_preds(dl=tst.tls)\npreds_clipped = preds.clamp(.0001, .999)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'm too lazy to write a function that creates a submission file, so this code is stolen from Radek, with minor changes."},{"metadata":{"trusted":true},"cell_type":"code","source":"ids = []\nlabels = []\n\nfor idx,pred in zip(df_tst.index, preds_clipped):\n    for i,label in enumerate(htypes):\n        ids.append(f\"{idx}_{label}\")\n        predicted_probability = '{0:1.10f}'.format(pred[i].item())\n        labels.append(predicted_probability)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_csv = pd.DataFrame({'ID': ids, 'Label': labels})\ndf_csv.to_csv(f'submission.csv', index=False)\ndf_csv.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run the code below if you want a link to download the submission file."},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import FileLink, FileLinks\nFileLink('submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"}},"nbformat":4,"nbformat_minor":4}