![\"https://i.ibb.co/k35LJVp/Selection-517.png\"](\"https://i.ibb.co/k35LJVp/Selection-517.png\")
many kagglers asked about my hardware to train large models. Here is it:
\n\"All experiment results are only made possible by the Z8 by HP workstation with 2x A6000 48GB Nvidia GPU cards. With fast and large GPU cards, you can enjoy designing models and sleep early without worrying about out-of-memory errors :) \"
\n
sample code:\nhttps://www.kaggle.com/code/hengck23/notebooke04a738685 \nhttps://www.kaggle.com/datasets/hengck23/for-tpu-efficientb4-debug \nmany kagglers asked about my hardware to train large models. Here is it:
\n\"All experiment results are only made possible by the Z8 by HP workstation with 2x A6000 48GB Nvidia GPU cards. With fast and large GPU cards, you can enjoy designing models and sleep early without worrying about out-of-memory errors :) \"
\n
sample code:\nhttps://www.kaggle.com/code/hengck23/notebooke04a738685 \nhttps://www.kaggle.com/datasets/hengck23/for-tpu-efficientb4-debug \nexample of using rejection-based approach.
\nyou can improve accuracy with less computation.
LB : 0.58\n test_df0 = test_df.copy()\n probability0 = do_predict(net0, test_df0)\n test_df0.loc[:,'cancer_p0']=probability0\n\n\n #---\n t = np.percentile(probability0,65) #reject 65%\n test_df1 = test_df0[test_df0.cancer_p0>t].reset_index(drop=True)\n probability1 = do_predict(net1, test_df1)\n test_df1.loc[:,'cancer_p1']=probability1\n\n # ---\n\n\nLB : 0.59 (cv increase +0.02)\n df = test_df0.merge(test_df1[['image_id','cancer_p1']],on='image_id',how='left')\n df.loc[df.cancer_p1.isna(),'cancer_p1'] = df.cancer_p0\n probability = (df.cancer_p0.values + df.cancer_p1.values)/2\n\nnet0 and net1 are trained using same fold but different seed\n![\"https://i.ibb.co/6szKVLc/Selection-613.png\"](\"https://i.ibb.co/6szKVLc/Selection-613.png\")
tip:
\nnet1 can also be a network that uses higher resolution, etc
\n[1]Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
\nuse larger learning rate to learn a coarse model.
\nthen finetune using different seed (+ different hyperparameters, rate, weighing, etc)
ensemble all using the method in [1], i.e. just average the weights of the best k-models in greedy way.
", "rawMarkdown": "example of using rejection-based approach.\nyou can improve accuracy with less computation.\n\n```\nLB : 0.58\n test_df0 = test_df.copy()\n probability0 = do_predict(net0, test_df0)\n test_df0.loc[:,'cancer_p0']=probability0\n\n\n #---\n t = np.percentile(probability0,65) #reject 65%\n test_df1 = test_df0[test_df0.cancer_p0>t].reset_index(drop=True)\n probability1 = do_predict(net1, test_df1)\n test_df1.loc[:,'cancer_p1']=probability1\n\n # ---\n\n\nLB : 0.59 (cv increase +0.02)\n df = test_df0.merge(test_df1[['image_id','cancer_p1']],on='image_id',how='left')\n df.loc[df.cancer_p1.isna(),'cancer_p1'] = df.cancer_p0\n probability = (df.cancer_p0.values + df.cancer_p1.values)/2\n\nnet0 and net1 are trained using same fold but different seed\n\n```\n\n\n\n---\n\ntip:\n\nnet1 can also be a network that uses higher resolution, etc \n\n\n---\n[1]Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time\n\nuse larger learning rate to learn a coarse model.\nthen finetune using different seed (+ different hyperparameters, rate, weighing, etc)\n\nensemble all using the method in [1], i.e. just average the weights of the best k-models in greedy way.\n", "votes": 13 }, { "id": 2096760, "postDate": "2023-01-12T08:43:06.620Z", "content": "transformer is the king!
\n![\"Selection-516\"](\"https://i.ibb.co/sHqDrLs/Selection-516.png\")
\n![\"Selection-515\"](\"https://i.ibb.co/McW8VYg/Selection-515.png\")
\n\n",
"votes": 13,
"replies": [
{
"id": 2096777,
"postDate": "2023-01-12T08:56:26.153Z",
"content": "Havent heard about NextVIT thanks for sharing, I do not even find it in timm :)
\nBTW I am surprised of your f1 score of 0.5 given that AUC is 0.9.
", "rawMarkdown": "Havent heard about NextVIT thanks for sharing, I do not even find it in timm :)\n\nBTW I am surprised of your f1 score of 0.5 given that AUC is 0.9. ", "replies": [ { "id": 2096816, "postDate": "2023-01-12T09:20:57.397Z", "content": "ROC-AUC is actually also not stable. you can get good F1 and AUC at CV by over sampling positive class and very strong model (e.g. imagenet top-1 0.86). But then i find that it is actually overfitting.
\nfor AUC of 0.90, F1 can varies a lot
", "rawMarkdown": "ROC-AUC is actually also not stable. you can get good F1 and AUC at CV by over sampling positive class and very strong model (e.g. imagenet top-1 0.86). But then i find that it is actually overfitting. \n\nfor AUC of 0.90, F1 can varies a lot", "votes": 1 }, { "id": 2096832, "postDate": "2023-01-12T09:31:40.157Z", "content": "I see slight variation but not that big.
\nWhat is ve class?
", "rawMarkdown": "I see slight variation but not that big.\n\nWhat is ve class?" }, { "id": 2096835, "postDate": "2023-01-12T09:32:32.033Z", "content": "it is positive class
", "rawMarkdown": "it is positive class" }, { "id": 2097902, "postDate": "2023-01-13T03:44:37.150Z", "content": "\n\n\nmanged to get tensorRT working!
\nperformance on local PC:
\nnum of images process 10935\n\npytorch fp16\n-----\ntime = 20 min 26 sec\nauc = 0.8925856621368753\nf1score.max() = 0.4968558969524153\n@threshold 0.3061224489795918\n\ntensorRT fp16\n-----\ntime = 9 min 05 sec\nauc = 0.8926157748322697\nf1score.max() = 0.49423766814841186\n@threshold 0.3061224489795918\nperformance on kaggle notebook to come later …
", "rawMarkdown": "~~tensorRT is gpu specific~~\n\n~~unfortunately, if i use pytorch-tensorRT on kaggle notebook, it runs out of memory for my large input size when compile nn.Module to trt engine ....~~\n\n~~need to think of something else~~\n\nmanged to get tensorRT working!\n\nperformance on local PC:\n\n```\nnum of images process 10935\n\npytorch fp16\n-----\ntime = 20 min 26 sec\nauc = 0.8925856621368753\nf1score.max() = 0.4968558969524153\n@threshold 0.3061224489795918\n \ntensorRT fp16\n-----\ntime = 9 min 05 sec\nauc = 0.8926157748322697\nf1score.max() = 0.49423766814841186\n@threshold 0.3061224489795918\n```\n\nperformance on kaggle notebook to come later ...\n\n", "votes": 1 }, { "id": 2101298, "postDate": "2023-01-15T19:43:05.330Z", "content": "If oversampling positive class will cause over fitting, may I ask how did you deal with the unbalance of neg/pos? BCE loss with weight? I read your discussion about multi-view. Your neg/pos is about 7:1. Thanks!
", "rawMarkdown": "If oversampling positive class will cause over fitting, may I ask how did you deal with the unbalance of neg/pos? BCE loss with weight? I read your discussion about multi-view. Your neg/pos is about 7:1. Thanks!" }, { "id": 2108317, "postDate": "2023-01-20T12:28:38.563Z", "content": "\n\nBut then i find that it is actually overfitting
\n
How did you find this out?
", "rawMarkdown": "> But then i find that it is actually overfitting\n\nHow did you find this out?" }, { "id": 2108339, "postDate": "2023-01-20T12:57:08.803Z", "content": "\"But then i find that it is actually overfitting\"
\nHow did you find this out?
The curve below and the LB results
\nhttps://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/378521
let's do a thought experiment
\nyou should at least note the following:
\nOh, I see, I get it, thanks!
", "rawMarkdown": "Oh, I see, I get it, thanks!" } ] }, { "id": 2096785, "postDate": "2023-01-12T09:05:01.823Z", "content": "CV of .497 👌! Is that cross-validation over how many folds? Thanks for the NextVIT architecture advice 👍
", "rawMarkdown": "CV of .497 👌! Is that cross-validation over how many folds? Thanks for the NextVIT architecture advice 👍", "replies": [ { "id": 2096815, "postDate": "2023-01-12T09:17:43.723Z", "content": "only one fold for CV and LB
", "rawMarkdown": "only one fold for CV and LB" }, { "id": 2096837, "postDate": "2023-01-12T09:34:03.063Z", "content": "okay that makes more sense then
", "rawMarkdown": "okay that makes more sense then" }, { "id": 2096866, "postDate": "2023-01-12T09:56:08.937Z", "content": "Alright thanks for info.
\nFor my training and testing the diff. in scores between folds and the sensitive threshold optimizing in numbers and math, give fear for a large shakeup.
\nHope you others have a more stable solutions and scores between the folds than I have ;)
one fold nextvit-b at 1536x960 already takes 9hr to run for the submission.
\nhence i am unable to run for an ensemble of multiple folds.
now i am solving this problem. (e.g. an early rejector, faster voi_lut_apply(), tensorRT or change a faster vision transformer).
\nI also try other transformer like CoAT, PVTv2 and the results are generally good in CV (i haven't make submission for these yet)
", "rawMarkdown": "one fold nextvit-b at 1536x960 already takes 9hr to run for the submission. \nhence i am unable to run for an ensemble of multiple folds.\n\nnow i am solving this problem. (e.g. an early rejector, faster voi\\_lut\\_apply(), tensorRT or change a faster vision transformer).\n\nI also try other transformer like CoAT, PVTv2 and the results are generally good in CV (i haven't make submission for these yet)" }, { "id": 2098405, "postDate": "2023-01-13T14:18:21.227Z", "content": "thanks for the info, how does base compare to the small nextvit?
", "rawMarkdown": "thanks for the info, how does base compare to the small nextvit? " } ] }, { "id": 2097098, "postDate": "2023-01-12T13:20:04.460Z", "content": "Thanks again for sharing!
\nI am immediately trying out nextvit and I found that it is very difficult to train nextvit.
\nWith the same parameters as before, I had early gradient explosions, and when I reduced the learning rate, it converged but performance dropped.
Do you have any good tips for training?
", "rawMarkdown": "Thanks again for sharing!\n\nI am immediately trying out nextvit and I found that it is very difficult to train nextvit.\nWith the same parameters as before, I had early gradient explosions, and when I reduced the learning rate, it converged but performance dropped.\n\nDo you have any good tips for training?", "replies": [ { "id": 2097248, "postDate": "2023-01-12T14:37:58.990Z", "content": "the parameters for training transformer (nextvit and others) are different from cnn
\nin general, learning rate and loss (oversampling or margin or +ve weighing) determines results
i use the following:
\nimage_height = 1536\nimage_width = 960\n\npretain = '/home/titanx/hengck/share1/data/pretrain_model/nextvit_base_in1k_384.pth'\nnextvit_base()\nF.binary_cross_entropy_with_logits(cancer,batch['cancer'])\n\n\n batch_size = 8 \n ratio = 8 (1 pos sample in every 8 train sample)\n\n\n def scheduler(epoch): \n\n num_epoch = 6\n start_lr = 5e-5\n min_lr = 1e-5\n lr = (num_epoch-epoch)/num_epoch * (start_lr-min_lr) + min_lr\n lr = max(min_lr,lr)\n return lr\n\n\n optimizer = Lookahead(RAdam(filter(lambda p: p.requires_grad, net.parameters()),lr=-1), alpha=0.5, k=5)\ntrain log (validation is without TTA)
\n** start training here! **\n batch_size = 8, ratio = 8\n experiment = ['nextvit-b-1536', 'run_train_fold0.py']\n |-------------------------- VALID-------------------|---------------- TRAIN/BATCH --------\nrate iter epoch | loss auc, f1 th sen spec f1 mean | loss | time \n---------------------------------------------------------------------------------------------------------------------\n0.00e+0 00000000* 0.00 | 0.674 0.491 0.0437 0.490 0.539 0.481 0.043 | 0.000 0.000 0.000 | 0 hr 05 min\n4.33e-5 00006122* 1.00 | 0.194 0.759 0.2096 0.735 0.199 0.984 0.240 | 0.342 0.000 0.000 | 1 hr 27 min\n3.67e-5 00012244* 2.00 | 0.118 0.798 0.2516 0.531 0.249 0.984 0.349 | 0.271 0.000 0.000 | 2 hr 50 min\n3.00e-5 00018366* 3.00 | 0.091 0.815 0.3387 0.592 0.245 0.996 0.433 | 0.230 0.000 0.000 | 4 hr 13 min\n2.33e-5 00024488* 4.00 | 0.099 0.837 0.3718 0.796 0.278 0.995 0.465 | 0.179 0.000 0.000 | 5 hr 35 min\n1.67e-5 00030610* 5.00 | 0.102 0.834 0.3665 0.449 0.369 0.986 0.460 | 0.185 0.000 0.000 | 6 hr 58 min\n1.00e-5 00036732* 6.00 | 0.089 0.817 0.3350 0.122 0.320 0.987 0.452 | 0.166 0.000 0.000 | 8 hr 21 min\n1.00e-5 00042854* 7.00 | 0.094 0.819 0.3711 0.653 0.282 0.995 0.508 | 0.153 0.000 0.000 | 9 hr 44 min\n1.00e-5 00048976* 8.00 | 0.091 0.802 0.3829 0.163 0.307 0.993 0.473 | 0.154 0.000 0.000 | 11 hr 07 min\nfinally apply TTA and swa for
\ndef make_swa():\n out_file = fold_dir + f'/checkpoint/swa.model.pth' \n iteration = [\n '00048976',\n '00042854',\n '00036732',\n '00036732',\n '00030610',\n '00024488',\n ]\n state_dict = None\n for i in iteration:\n f = fold_dir + f'/checkpoint/{i}.model.pth'\n print(f)\n f = torch.load(f, map_location=lambda storage, loc: storage)\n if state_dict is None:\n state_dict = f['state_dict']\n else:\n key = list(f['state_dict'].keys())\n for k in key:\n state_dict[k] = state_dict[k] + f['state_dict'][k]\n\n for k in key:\n state_dict[k] = state_dict[k] / len(iteration)\n print('')\n\n print(out_file)\n torch.save({'state_dict': state_dict}, out_file)\ncpu augmentation is too slow. so i use kornia gpu augmentation
\nclass DataAugmentation1(nn.Module):\n def __init__(self,):\n super().__init__()\n self.flip = nn.Sequential(\n RandomHorizontalFlip(p=0.5),\n RandomVerticalFlip(p=0.5),\n )\n\n p=0.8\n self.transform_geometry = ImageSequential(\n RandomAffine(degrees=20, translate=0.1, scale=[0.8,1.2], shear=20, p=p),\n RandomThinPlateSpline(scale=0.25, p=p),\n random_apply=1, #choose 1\n )\n\n p=0.5\n self.transform_intensity = ImageSequential(\n RandomGamma(gamma=(0.5, 1.5), gain=(0.5, 1.2), p=p),\n RandomContrast(contrast=(0.8,1.2), p=p),\n RandomBrightness(brightness=(0.8,1.2), p=p),\n random_apply=1, #choose 1\n )\n\n p=0.5\n self.transform_other = ImageSequential(\n MyRoll(p=0.1), #Mosaic Augmentation using only one image, implemented by using pytorch roll , i.e. cyclic shift\n MyCutOut(num_block=5, block_size=[0.1, 0.2], fill='constant', p=0.1),\n random_apply=1, #choose 1\n )\n\n\n @torch.no_grad() # disable gradients for effiency\n def forward(self, x):\n x = self.flip(x) # BxCxHxW\n x = self.transform_geometry(x)\n x = self.transform_intensity(x)\n x = self.transform_other(x)\n return x\nThank you for being quite detailed!
\nIt's quite different from my setting, so I'm learning a lot.
can you tell what is f1 in the log above? is it non-thresholded pF1?
", "rawMarkdown": "can you tell what is f1 in the log above? is it non-thresholded pF1?" }, { "id": 2098098, "postDate": "2023-01-13T08:16:50.987Z", "content": "max threshold f1
", "rawMarkdown": "max threshold f1" }, { "id": 2144080, "postDate": "2023-02-14T18:11:39.030Z", "content": "Can you please explain why lr = -1 in optimizer?
", "rawMarkdown": "Can you please explain why lr = -1 in optimizer?", "votes": 1 } ] }, { "id": 2097365, "postDate": "2023-01-12T16:06:36.973Z", "content": "Does SWA stands for Stochastic Weight Averaging?
\nNice score!
OK. I see …. :)
", "rawMarkdown": "OK. I see .... :) " } ] }, { "id": 2101305, "postDate": "2023-01-15T19:53:03.410Z", "content": "Have you encountered any problem during training - loss NaN (I use oryginal naxtViT repo)?
", "rawMarkdown": "Have you encountered any problem during training - loss NaN (I use oryginal naxtViT repo)? ", "replies": [ { "id": 2101474, "postDate": "2023-01-16T00:10:33.573Z", "content": "maybe try use smaller learning rate. 5e-5 to 1e-5.
", "rawMarkdown": "maybe try use smaller learning rate. 5e-5 to 1e-5." }, { "id": 2101811, "postDate": "2023-01-16T07:09:42.660Z", "content": "yes, I started from 5e-5 but unfortunately exploded during second epoch. I will try today to do more experiments.
", "rawMarkdown": "yes, I started from 5e-5 but unfortunately exploded during second epoch. I will try today to do more experiments." }, { "id": 2101847, "postDate": "2023-01-16T08:01:41.480Z", "content": "increased pos sampling e.g. from 1 pos in batch=8 to 1 in 4.
\nuse 3e-5
I followed your suggestion (starting lr 3e-5) and no NaN appeared. Thank you so much.
\nOne more question - do you balance loss function using weight? As I can see my model is good after 1-2 epochs and then started to overfit for \"no cancer\".
Hi! I have the same question about the loss with balanced weight… Also, what's the good metric to decide a good model? AUC or f1? because in my eval, the highest AUC is around 0.975, which is the same proportion of neg/pos…
", "rawMarkdown": "Hi! I have the same question about the loss with balanced weight... Also, what's the good metric to decide a good model? AUC or f1? because in my eval, the highest AUC is around 0.975, which is the same proportion of neg/pos... " }, { "id": 2102225, "postDate": "2023-01-16T13:31:04.297Z", "content": "e.g.
\nset rate=5e-5:
\ncannot learn validation f1 is 0.05
set rate=3e-5:
\nvalidation f1 is 0.10 for some first iterations, then fall back to previous case of 0.05 in later iterations
set rate=1e-5:
\nsuccess! validation f1 improves as iterations proceeds
if it doesn't work even if you set rate=1e-6, then the model cannot be train with you current setup.
\nyou have to change oversampling, loss (intermediate aux loss, or other loss than BCE) or use a less complex version (e.g. small, tiny version, etc)
My nextvit (small) is fighting …. 😂 AUC_ROC -> 0.86 / probf1 ~0.36 (val)
\n![\"\"](\"https://i.ibb.co/QYfPK1c/W-B-Chart-17-01-2023-14-56-18.png\")
![\"\"](\"https://i.ibb.co/4d39pWS/prec.png\")
\n![\"\"](\"https://i.ibb.co/wMgP9NZ/sep.png\")
any submission results?
\nthere could be overfitting
\nbut maybe LB results cab be good it is hard to judge.
\nyou can select a few intermediate model and submit too
may I ask what is the metric to choose best model in this competition? AUC? pf1? f1?
\nBecause sometimes I found AUC is decreasing but the pf1 is increasing.
\nThanks for your answer!
Let me do some more experiment and will submit. I will let you know.
", "rawMarkdown": "Let me do some more experiment and will submit. I will let you know." } ] } ] }, { "id": 2066638, "postDate": "2022-12-15T22:58:11.623Z", "content": "it turns that i have lower LB than other kagglers using efficientnet.
\ni have forgotten to set the drop path rate. Here is the fixed:
efficientnet_b2(pretrained=True, drop_rate = 0.3, drop_path_rate = 0.2)\n\nrefer to timm efficientnet source code for settings of drop_rate, drop_path_rate\nyou can use single-fold model (i.e. just one checkpoint file) to get LB >0.51. more on that later,
", "rawMarkdown": "it turns that i have lower LB than other kagglers using efficientnet.\ni have forgotten to set the drop path rate. Here is the fixed:\n\n```\n\nefficientnet_b2(pretrained=True, drop_rate = 0.3, drop_path_rate = 0.2)\n\nrefer to timm efficientnet source code for settings of drop_rate, drop_path_rate\n```\n\nyou can use single-fold model (i.e. just one checkpoint file) to get LB >0.51. more on that later,", "votes": 14 }, { "id": 2098821, "postDate": "2023-01-13T21:42:00.453Z", "content": "tensorRT timming is out !!!
\nas claimed in the paper, nextVIT is the fastest tensorRT vision transformer (same speed as efficientnet)
\nkaggle p100 notebook timing:\n\n--------------------------------------------------\npublic LB submission tensorRT 4 hr : LB 0.56\n\n--------------------------------------------------\nlocal cv 10939 images\n\nauc 0.8936061433377303\nf1score 0.49148140396595846\n@threshold 0.30612\n\n\ntotal (end-to-end from dashboard)\nDisk 16.4/73.1 GB\n1 hr 25 min\n\nbreakdown:\n\n1. install tensorRT, etc 5 min\n\n2. decode 10939 dicom images (keep aspect to 1539, use voi_lut-apply_32fp):\nnvjpeg2k (j2k, 5118 images) 28 min\ndicomsdl (non-j2k, 5917 images, 2 thread) 24 min\n\n3. detect breast box (resnet34 segentation) 2 min\n\n4. tensorRT fp16 nextVIT-B (1539x960)\none fold, original + hflip_TTA\n\nCPU utilisation 120%, 13/13 GB\nGPU utilisation 99% , 4.5/15 GB\n23 min 21 sec (7.80799 images per sec)\n\n\n=============================================\nreference (without tensorRT):\n\nsubmission 9hr : LB 0.56\n\n4. merged_bn fp16 (1539x960)\nCPU utilisation 108%, 13/13 GB\nGPU utilisation 100% , 8.5/15 GB\n95 min 32 sec (1.90819 images per sec)\nHow are you loading all images in 4 hours lol :)
\nThanks for sharing - transformer based architectures usually benefit the most from such compiles.
\nI am curious if anyone got Pytorch 2.0 running in kaggle kernels and have checked how close it comes to tensorrt.
", "rawMarkdown": "How are you loading all images in 4 hours lol :)\n\nThanks for sharing - transformer based architectures usually benefit the most from such compiles.\n\nI am curious if anyone got Pytorch 2.0 running in kaggle kernels and have checked how close it comes to tensorrt.", "votes": 2, "replies": [ { "id": 2098829, "postDate": "2023-01-13T21:57:31.520Z", "content": "voi_lut_apply_32fp() eats my time.
\nwithout it loading images is 3hr.
maybe i can improve a little by apply resize first and then the voi lut porcessing on smaller resized image.
", "rawMarkdown": "voi\\_lut\\_apply\\_32fp() eats my time. \nwithout it loading images is 3hr.\n\nmaybe i can improve a little by apply resize first and then the voi lut porcessing on smaller resized image." }, { "id": 2098965, "postDate": "2023-01-14T01:42:01.293Z", "content": "\"I am curious if anyone got Pytorch 2.0 running in kaggle kernels and have checked how close it comes to tensorrt.\"
\ni would think tensorRT is better
Yes I also saw this, but reality and graphs are sometimes different. SO Im just curious if someone tried.
", "rawMarkdown": "Yes I also saw this, but reality and graphs are sometimes different. SO Im just curious if someone tried.", "votes": 1 } ] }, { "id": 2098955, "postDate": "2023-01-14T01:25:21.517Z", "content": "tensorRT engine trt file generation code is here:
\nhttps://www.kaggle.com/code/hengck23/4hr-tensorrt-nextvit-example
i think if you edit the code for 2x T4 GPU, you will get the same LB results in 1.5hr
\ntime to try a bigger 2048 image on transformer …
2048x1280 for tensorrt NextVIT-S (smaller variant, imagenet 83.6) takes 5h.
\nhowever, results were as good?
\nlocalCV 0.4856957958 @th=0.387755102
\nLB 0.46
set dropout to 0. instead of 0
\none is float, the other is int
Many thanks! I'll try it.
", "rawMarkdown": "Many thanks! I'll try it." }, { "id": 2100673, "postDate": "2023-01-15T10:52:17.483Z", "content": "the error still exist after setting dropout to 0. instead of 0
", "rawMarkdown": "the error still exist after setting dropout to 0. instead of 0" }, { "id": 2100676, "postDate": "2023-01-15T10:54:21.403Z", "content": "can you post the full error message
", "rawMarkdown": "can you post the full error message" }, { "id": 2100684, "postDate": "2023-01-15T11:00:51.820Z", "content": "RuntimeError Traceback (most recent call last)
\n/tmp/ipykernel_23/1028975912.py in
\n 25 enabled_precisions={torch.half}, # Run with FP16
\n 26 workspace_size=1 << 32,\n---> 27 require_full_compilation=True,
\n 28 )
\n 29 torch.jit.save(trt_model_fp16, 'kaggle-nextvit-b-1536-gpu-aug0-01-swa.trt_fp16.ts')
/opt/conda/lib/python3.7/site-packages/torch_tensorrt/_compile.py in compile(module, ir, inputs, enabled_precisions, **kwargs)
\n 122 \"Module was provided as a torch.nn.Module, trying to script the module with torch.jit.script. In the event of a failure please preconvert your module to TorchScript\",
\n 123 )
\n--> 124 ts_mod = torch.jit.script(module)
\n 125 return torch_tensorrt.ts.compile(
\n 126 ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in script(obj, optimize, _frames_up, _rcb, example_inputs)
\n 1285 obj = call_prepare_scriptable_func(obj)
\n 1286 return torch.jit._recursive.create_script_module(
\n-> 1287 obj, torch.jit._recursive.infer_methods_to_compile
\n 1288 )
\n 1289
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module(nn_module, stubs_fn, share_types, is_tracing)
\n 456 if not is_tracing:
\n 457 AttributeTypeIsSupportedChecker().check(nn_module)
\n--> 458 return create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 459
\n 460 def create_script_module_impl(nn_module, concrete_type, stubs_fn):
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 518
\n 519 # Actually create the ScriptModule, initializing it with the function we just defined
\n--> 520 script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
\n 521
\n 522 # Compile methods if necessary
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in _construct(cpp_module, init_fn)
\n 613 \"\"\"
\n 614 script_module = RecursiveScriptModule(cpp_module)
\n--> 615 init_fn(script_module)
\n 616
\n 617 # Finalize the ScriptModule: replace the nn.Module state with our
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in init_fn(script_module)
\n 496 else:
\n 497 # always reuse the provided stubs_fn to infer the methods to compile
\n--> 498 scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
\n 499
\n 500 cpp_module.setattr(name, scripted)
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 518
\n 519 # Actually create the ScriptModule, initializing it with the function we just defined
\n--> 520 script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
\n 521
\n 522 # Compile methods if necessary
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in _construct(cpp_module, init_fn)
\n 613 \"\"\"
\n 614 script_module = RecursiveScriptModule(cpp_module)
\n--> 615 init_fn(script_module)
\n 616
\n 617 # Finalize the ScriptModule: replace the nn.Module state with our
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in init_fn(script_module)
\n 496 else:
\n 497 # always reuse the provided stubs_fn to infer the methods to compile
\n--> 498 scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
\n 499
\n 500 cpp_module.setattr(name, scripted)
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 518
\n 519 # Actually create the ScriptModule, initializing it with the function we just defined
\n--> 520 script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
\n 521
\n 522 # Compile methods if necessary
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in _construct(cpp_module, init_fn)
\n 613 \"\"\"
\n 614 script_module = RecursiveScriptModule(cpp_module)
\n--> 615 init_fn(script_module)
\n 616
\n 617 # Finalize the ScriptModule: replace the nn.Module state with our
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in init_fn(script_module)
\n 496 else:
\n 497 # always reuse the provided stubs_fn to infer the methods to compile
\n--> 498 scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
\n 499
\n 500 cpp_module.setattr(name, scripted)
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 518
\n 519 # Actually create the ScriptModule, initializing it with the function we just defined
\n--> 520 script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
\n 521
\n 522 # Compile methods if necessary
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in _construct(cpp_module, init_fn)
\n 613 \"\"\"
\n 614 script_module = RecursiveScriptModule(cpp_module)
\n--> 615 init_fn(script_module)
\n 616
\n 617 # Finalize the ScriptModule: replace the nn.Module state with our
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in init_fn(script_module)
\n 496 else:
\n 497 # always reuse the provided stubs_fn to infer the methods to compile
\n--> 498 scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
\n 499
\n 500 cpp_module.setattr(name, scripted)
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 518
\n 519 # Actually create the ScriptModule, initializing it with the function we just defined
\n--> 520 script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
\n 521
\n 522 # Compile methods if necessary
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in _construct(cpp_module, init_fn)
\n 613 \"\"\"
\n 614 script_module = RecursiveScriptModule(cpp_module)
\n--> 615 init_fn(script_module)
\n 616
\n 617 # Finalize the ScriptModule: replace the nn.Module state with our
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in init_fn(script_module)
\n 496 else:
\n 497 # always reuse the provided stubs_fn to infer the methods to compile
\n--> 498 scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
\n 499
\n 500 cpp_module.setattr(name, scripted)
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 522 # Compile methods if necessary
\n 523 if concrete_type not in concrete_type_store.methods_compiled:
\n--> 524 create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
\n 525 # Create hooks after methods to ensure no name collisions between hooks and methods.
\n 526 # If done before, hooks can overshadow methods that aren't exported.
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
\n 373 property_rcbs = [p.resolution_callback for p in property_stubs]
\n 374
\n--> 375 concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
\n 376
\n 377 def create_hooks_from_stubs(concrete_type, hook_stubs, pre_hook_stubs):
RuntimeError: Can't redefine method: forward on class: torch.torch.nn.modules.dropout.Dropout (of Python compilation unit at: 0x5643bd433f20)
", "rawMarkdown": "this is full error message.\n---------------------------------------------------------------------------\nRuntimeError Traceback (most recent call last)\n/tmp/ipykernel_23/1028975912.py inthis error don't occurs at my side. i also try torch.jit.trace(model, x) which return successful results.
\ncheck that you are using eval()
\ni suggest the either of the followings:
there are several versions of nextvit.py file from the github. i am using the image classification one.
follow the instruction from the github (export_tensorrt_engine.py), try to create onnx file. This checks your system(version, etc), torch.onnx also calls torch jit script.
simply edit your code. replace nn.Dropout with nn.Identity. If there is problem with dropPath create afunction/class that just let input pass through. (dropout and droppath are not used in eval() mode)
Thanks for your oppions.
", "rawMarkdown": "Thanks for your oppions." }, { "id": 2100931, "postDate": "2023-01-15T13:56:41.310Z", "content": "could you share a notebook for generating trt engine file of NextVitNet?
", "rawMarkdown": "could you share a notebook for generating trt engine file of NextVitNet?" } ] }, { "id": 2100622, "postDate": "2023-01-15T10:01:06.630Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8719748%2F3ceef46ac120c13cfdc04a3da3d8d84c%2F111.jpg?generation=1673776841164994&alt=media\")
\nhi, I got above error when generating trt engine file. Could you tell me how to solve it?
T4x2 is King, I just switched my pipeline to it, and preprocessing is x1.24 faster. Preprocessing the training set I improve from 3h 26m to 2h 46m !!
\nprocessing the first 5118 j2k and 5917 non-j2k (to compare to your numbers), the pipeline does 9.2 minutes, and 20 minutes respectively! (time also includes yolov5 inference on 640px imgs)
\nI get this small speedup by doing all the resizing and windowing operations in different threads on EACH GPU!
\njust a note: the apply_voi_lut == apply_windowing since I believe that none of the images in the training set have a VOI lookup table.
", "rawMarkdown": "T4x2 is King, I just switched my pipeline to it, and preprocessing is x1.24 faster. Preprocessing the training set I improve from 3h 26m to 2h 46m !! \n\nprocessing the first 5118 j2k and 5917 non-j2k (to compare to your numbers), the pipeline does 9.2 minutes, and 20 minutes respectively! (time also includes yolov5 inference on 640px imgs)\n\nI get this small speedup by doing all the resizing and windowing operations in different threads on EACH GPU!\n\njust a note: the apply_voi_lut == apply_windowing since I believe that none of the images in the training set have a VOI lookup table. " } ] }, { "id": 2107692, "postDate": "2023-01-20T01:01:26.237Z", "content": "there is a novel method to fight rare (imbalance) class
\n[1] Background Splitting: Finding Rare Classes in a Sea of Background
\nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Mullapudi_Background_Splitting_Finding_Rare_Classes_in_a_Sea_of_Background_CVPR_2021_paper.pdf
the idea i simple randomly assign label to your background images. hence even if you sample a batch of all negative images, you will not learn to predict same class
\nhttps://www.youtube.com/watch?v=I6-8mrp99sI
![\"https://i.ibb.co/PQST7mG/Selection-594.png\"](\"https://i.ibb.co/PQST7mG/Selection-594.png\")
That's a cool idea. I'll be very curious to hear how well it works.
", "rawMarkdown": "That's a cool idea. I'll be very curious to hear how well it works.", "votes": 4 } ] }, { "id": 2076917, "postDate": "2022-12-27T02:37:49.950Z", "content": "i find the treasure !!!!
\n![\"https://i.ibb.co/6tGVML0/Selection-315.png\"](\"https://i.ibb.co/6tGVML0/Selection-315.png\")
A amazing AUC.
", "rawMarkdown": "A amazing AUC.", "votes": 1, "replies": [ { "id": 2076990, "postDate": "2022-12-27T05:26:18.747Z", "content": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8456839/
", "rawMarkdown": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8456839/", "votes": 5 }, { "id": 2076997, "postDate": "2022-12-27T05:39:50.727Z", "content": "Thanks you.
", "rawMarkdown": "Thanks you. " }, { "id": 2076999, "postDate": "2022-12-27T05:45:35.963Z", "content": "more baseline models and results later …
\nMVCCL model for ADMANI dataset
\nhttps://www.kaggle.com/code/hengck23/mvccl-model-for-admani-dataset
\"Our experiments were performed on Swinburne supercomputer OzSTAR* with a cluster of NVIDIA Tesla P100 GPUs\" .. OzSTAR comprises over 5,000 processing cores, 230 GPUs, a collective 25 Terabytes of system memory and access to over 6 Petabytes of storage.
", "rawMarkdown": "\"Our experiments were performed on Swinburne supercomputer OzSTAR* with a cluster of NVIDIA Tesla P100 GPUs\" .. OzSTAR comprises over 5,000 processing cores, 230 GPUs, a collective 25 Terabytes of system memory and access to over 6 Petabytes of storage." }, { "id": 2077034, "postDate": "2022-12-27T06:46:26.990Z", "content": "this is because they have 3 million images
\nthey resolution is about 2600
\n\nthis is because they have 3 million images
\n
\n they resolution is about 2600
I am referring to the link https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8456839/
\nThis is the paper that the above screen cap came from. Thanks Remek for the cite.
\nI don't see any reference to 'millions' of images in the study. Perhaps I missed it though.
", "rawMarkdown": "> this is because they have 3 million images\n> they resolution is about 2600\n\nI am referring to the link https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8456839/ \n\nThis is the paper that the above screen cap came from. Thanks Remek for the cite.\n\nI don't see any reference to 'millions' of images in the study. Perhaps I missed it though." }, { "id": 2077041, "postDate": "2022-12-27T06:57:08.987Z", "content": "@ynhuhu Another thing to consider is that the AUC you see above is on the patches. Here are the top line results from the paper:
\n\n", "rawMarkdown": "@ynhuhu Another thing to consider is that the AUC you see above is on the patches. Here are the top line results from the paper:\n\n\n>Results\nOur evaluation uses the area under curve (AUC) and accuracy (ACC) for performance measurement. The best evaluation result, based on 349 test cases (930 test images), was an AUC of 0.8979 [95% confidence interval (CI) 0.873, 0.923] and ACC of 0.8178 [95% CI 0.785, 0.850]. \n", "votes": 1 }, { "id": 2077053, "postDate": "2022-12-27T07:09:24.097Z", "content": "Results
\n
\n Our evaluation uses the area under curve (AUC) and accuracy (ACC) for performance measurement. The best evaluation result, based on 349 test cases (930 test images), was an AUC of 0.8979 [95% confidence interval (CI) 0.873, 0.923] and ACC of 0.8178 [95% CI 0.785, 0.850].
Ye, you are right.
", "rawMarkdown": "Ye, you are right. " } ] } ] }, { "id": 2069441, "postDate": "2022-12-19T01:53:45.930Z", "content": "![\"https://i.ibb.co/jGmX2RS/Selection-228.png\"](\"https://i.ibb.co/jGmX2RS/Selection-228.png\")
in some of the video and websites i have read, we screen mammography images by comparing left and right images side-by-side (see image above). Instead of predicting based on single image, we use stitch of of R-L image as single input. Alternatively, we can have 2 view (2x single input) and fused them later.
\nThe advantage is that we both have same breast density, so abnormality can stand out better
", "rawMarkdown": "\n\nin some of the video and websites i have read, we screen mammography images by comparing left and right images side-by-side (see image above). Instead of predicting based on single image, we use stitch of of R-L image as single input. Alternatively, we can have 2 view (2x single input) and fused them later.\n\nThe advantage is that we both have same breast density, so abnormality can stand out better", "votes": 9, "replies": [ { "id": 2069467, "postDate": "2022-12-19T02:32:32.970Z", "rawMarkdown": "", "votes": -10, "isDeleted": true } ] }, { "id": 2061348, "postDate": "2022-12-11T03:12:28.180Z", "content": "i see some (very) good improvement in using 16-bit png (instead of 8bit).
\nThis is because i am not applying windowing or VOI LUT in the dim data, which itself is tricky.
It is difficult to large scale experiment (especially for kaggle submission)
\nOther kagglers may want to verify this.
![\"https://i.ibb.co/HhXp1j3/Selection-519.png\"](\"https://i.ibb.co/HhXp1j3/Selection-519.png\")
def read_dicom_as_image(dcm_file):\n dicom = pydicom.dcmread(dcm_file)\n image = dicom.pixel_array \n image = (image - image.min()) / (image.max() - image.min()+1e-6) #this cast to float32\n if dicom.PhotometricInterpretation == 'MONOCHROME1':\n image = 1 - image\n\n return image\n\ndef parallel_process(dcm_file):\n patient_id = dcm_file.split('/')[-2]\n image_id = dcm_file.split('/')[-1][:-4]\n image = read_dicom_as_image(dcm_file)\n image = cv2.resize(image, (image_size, image_size), interpolation=cv2.INTER_LINEAR)\n image = (image * 65535).astype(np.uint16)\n\n os.makedirs(f'{png_dir}/{patient_id}', exist_ok=True)\n cv2.imwrite(f'{png_dir}/{patient_id}/{image_id}.png',image)\n\n\nif 1:\n Parallel(n_jobs=10)(\n delayed(parallel_process)(f)\n for f in tqdm(dcm_file)\n )\ni suspect my previous improvement of 2048 is actually from the intensity improvement
\nlearnable windowing
\nPractical Window Setting Optimization for Medical Image Deep Learning
\nhttps://github.com/MGH-LMIC/windows_optimization
CT Window Trainable Neural Network for Improving Intracranial Hemorrhage Detection
\nhttps://ars.els-cdn.com/content/image/1-s2.0-S093336571930939X-gr2.jpg
I'm not sure how you saved 2048 x 2048 uint16 images. Cuz, only 13k images take >20GB space, so for total data, it would be nearly >100GB space. In that case, kaggle should throw error. Could you please share how you saved uint16 images?
You could do it off kaggle, right. Also, batching on kaggle for inference.
", "rawMarkdown": "You could do it off kaggle, right. Also, batching on kaggle for inference." }, { "id": 2062268, "postDate": "2022-12-11T23:11:24.313Z", "content": "i do not save at kaggle inference.
\ni haven't tried uint16 for 2048 yet. i am running experiments for 1024 for now
@hengck23 Why it's for?
\nimage = (image * 65535).astype(np.uint16)\nSeems like image is normalized normalized beforehand (usually between 0-1), to cast uint16 you better to multiply these values by 65535 (which is the max value of uint16) before mapping them to rounded uint16 values.
", "rawMarkdown": "Seems like image is normalized normalized beforehand (usually between 0-1), to cast uint16 you better to multiply these values by 65535 (which is the max value of uint16) before mapping them to rounded uint16 values." }, { "id": 2064198, "postDate": "2022-12-13T15:15:34.003Z", "content": "this is to save as 16-bit png.
\nif you don't save image for inference, the casting is not required
![\"https://i.ibb.co/JdV5v8Y/Selection-174.png\"](\"https://i.ibb.co/JdV5v8Y/Selection-174.png\")
What do you think about normalizing with 16-bit dataset mean/std? @hengck23
", "rawMarkdown": "What do you think about normalizing with 16-bit dataset mean/std? @hengck23 " }, { "id": 2094504, "postDate": "2023-01-10T19:37:10.733Z", "content": "only experiment will confirms results.
\nand you don't have to stick to one processing. you can use different processing in ensemble
This is interesting. If you don't convert to 8bit, you have more information available. But do you still start from the pretrained models on 8 bit images?
", "rawMarkdown": "This is interesting. If you don't convert to 8bit, you have more information available. But do you still start from the pretrained models on 8 bit images?", "replies": [ { "id": 2094506, "postDate": "2023-01-10T19:38:04.793Z", "content": "no.
\nit is like if pretrain model is train on 224x224 image size, you can finetune it for larger and smaller size
", "rawMarkdown": "no.\n\nit is like if pretrain model is train on 224x224 image size, you can finetune it for larger and smaller size", "votes": 1 } ] } ] }, { "id": 2067573, "postDate": "2022-12-16T20:53:11.017Z", "content": "the fastest you can go is slightly less than 3hr
\ninput1024, one model single-fold efficientnetb4
![\"https://i.ibb.co/x5DN900/Selection-206.png\"](\"https://i.ibb.co/x5DN900/Selection-206.png\")
#share ----\ndef normalised_to_8bit(image, photometric_interpretation):\n xmin = image.min()\n xmax = image.max() \n norm = np.empty_like(image, dtype=np.uint8)\n dicomsdl.util.convert_to_uint8(image, norm, xmin, xmax)\n if photometric_interpretation == 'MONOCHROME1':\n norm = 255 - norm\n return norm\n\n\n# j2k ----\nj2k_decoder = nvjpeg2k.Decoder()\ndef process_j2k(df, dcm_dir, image_dir, image_size):\n for t, d in tqdm(df.iterrows()):\n dcm_file = f'{dcm_dir}/{d.patient_id}/{d.image_id}.dcm'\n ds = pydicom.dcmread(dcm_file)\n offset = ds.PixelData.find(b'\\x00\\x00\\x00\\x0C')\n jpeg_stream = bytearray(ds.PixelData[offset:]) \n m = j2k_decoder.decode(jpeg_stream) \n\n\n # resize and save as png\n m = normalised_to_8bit(m, ds.PhotometricInterpretation)\n m = cv2.resize(m, dsize=(image_size, image_size), interpolation=cv2.INTER_LINEAR)\n cv2.imwrite(f'{image_dir}/{d.patient_id}/{d.image_id}.png', m)\n\n...\n\n#non j2k ----\n\ndef dicomsdl_parallel_process_fn(d, dcm_dir, image_dir, image_size):\n dcm_file = f'{dcm_dir}/{d.patient_id}/{d.image_id}.dcm'\n ds = dicomsdl.open(dcm_file)\n info = ds.getPixelDataInfo() \n m = np.empty(shape=[info['Rows'], info['Cols']], dtype=info['dtype'])\n ds.copyFrameData(0, m) \n\n # resize and save as png\n m = normalised_to_8bit(m, ds.PhotometricInterpretation)\n m = cv2.resize(m, dsize=(image_size, image_size), interpolation=cv2.INTER_LINEAR)\n cv2.imwrite(f'{image_dir}/{d.patient_id}/{d.image_id}.png', m)\n\ndef process_non_j2k(df, dcm_dir, image_dir, image_size, n_jobs): \n Parallel(n_jobs=n_jobs)(\n delayed(dicomsdl_parallel_process_fn)(d, dcm_dir, image_dir, image_size)\n for t,d in tqdm(df.iterrows())\n )\nyou probably need to retrain with images generated by dicomsdl.util.convert_to_uint8()
", "rawMarkdown": "the fastest you can go is slightly less than 3hr\ninput1024, one model single-fold efficientnetb4\n\n\n \n```\n#share ----\ndef normalised_to_8bit(image, photometric_interpretation):\n xmin = image.min()\n xmax = image.max() \n norm = np.empty_like(image, dtype=np.uint8)\n dicomsdl.util.convert_to_uint8(image, norm, xmin, xmax)\n if photometric_interpretation == 'MONOCHROME1':\n norm = 255 - norm\n return norm\n\n\n# j2k ----\nj2k_decoder = nvjpeg2k.Decoder()\ndef process_j2k(df, dcm_dir, image_dir, image_size):\n for t, d in tqdm(df.iterrows()):\n dcm_file = f'{dcm_dir}/{d.patient_id}/{d.image_id}.dcm'\n ds = pydicom.dcmread(dcm_file)\n offset = ds.PixelData.find(b'\\x00\\x00\\x00\\x0C')\n jpeg_stream = bytearray(ds.PixelData[offset:]) \n m = j2k_decoder.decode(jpeg_stream) \n\n\n # resize and save as png\n m = normalised_to_8bit(m, ds.PhotometricInterpretation)\n m = cv2.resize(m, dsize=(image_size, image_size), interpolation=cv2.INTER_LINEAR)\n cv2.imwrite(f'{image_dir}/{d.patient_id}/{d.image_id}.png', m)\n\n...\n\n#non j2k ----\n \ndef dicomsdl_parallel_process_fn(d, dcm_dir, image_dir, image_size):\n dcm_file = f'{dcm_dir}/{d.patient_id}/{d.image_id}.dcm'\n ds = dicomsdl.open(dcm_file)\n info = ds.getPixelDataInfo() \n m = np.empty(shape=[info['Rows'], info['Cols']], dtype=info['dtype'])\n ds.copyFrameData(0, m) \n\n # resize and save as png\n m = normalised_to_8bit(m, ds.PhotometricInterpretation)\n m = cv2.resize(m, dsize=(image_size, image_size), interpolation=cv2.INTER_LINEAR)\n cv2.imwrite(f'{image_dir}/{d.patient_id}/{d.image_id}.png', m)\n\ndef process_non_j2k(df, dcm_dir, image_dir, image_size, n_jobs): \n Parallel(n_jobs=n_jobs)(\n delayed(dicomsdl_parallel_process_fn)(d, dcm_dir, image_dir, image_size)\n for t,d in tqdm(df.iterrows())\n )\n\n```\n\nyou probably need to retrain with images generated by dicomsdl.util.convert_to_uint8()\n\n \n\n\n", "votes": 8, "replies": [ { "id": 2082166, "postDate": "2023-01-01T07:37:53.707Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2067719, "postDate": "2022-12-17T04:28:46.743Z", "content": "in theory, you can stitch four 1024 images (LCC,RCC,LMLO,RMLO) into a single 2048 image. Then input this into a single network and make 2 predictions for L,R
", "rawMarkdown": "in theory, you can stitch four 1024 images (LCC,RCC,LMLO,RMLO) into a single 2048 image. Then input this into a single network and make 2 predictions for L,R", "votes": 7, "replies": [ { "id": 2079570, "postDate": "2022-12-29T12:34:48.603Z", "content": "Multiple-instance learning could be useful.
", "rawMarkdown": "Multiple-instance learning could be useful." } ] }, { "id": 2059408, "postDate": "2022-12-08T20:15:54.477Z", "content": "[paper] Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening- Nan Wu
\nhttps://github.com/nyukat/breast_cancer_classifier
extensive experiments for multi-view prediction (over 1 million images)
\n![\"Selection-142\"](\"https://i.ibb.co/sRbJLjK/Selection-142.png\")
\n![\"Selection-141\"](\"https://i.ibb.co/M6PPgKr/Selection-141.png\")
\n",
"votes": 7,
"replies": [
{
"id": 2059420,
"postDate": "2022-12-08T20:27:19.103Z",
"content": "Very nice concept. Good inspiration! 👍
", "rawMarkdown": "Very nice concept. Good inspiration! 👍" } ] }, { "id": 2120459, "postDate": "2023-01-29T16:00:55.473Z", "content": "i tried many multiple images prediction method and below is the only one that works.
\nFor nextvit-B local CV improves from 0.49 (single-image predict + mean) to 0.511 (multi-image predict) for the my first experiments:
I got improvement at the first run without adjusting of hyper parameters and pipeline!
\nhere are more information:
\nPART ONE: paper review and description of method
![\"https://i.ibb.co/yy8bBGd/Selection-724.png\"](\"https://i.ibb.co/yy8bBGd/Selection-724.png\")
[1] COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Prediction - A. Sriram (facebook AI), arXiv 2020
\nhttps://github.com/facebookresearch/CovidPrognosis
PART TWO: modification for kaggle breast mammography and example notebook
\ndummy code is up: https://www.kaggle.com/code/hengck23/example-of-multi-image-prediction
PART THREE: results and analysis
\nto be updated
first experiment results
\nMIP prediction\n\nget_f1score(cancer_p[site_id==1], cancer_t[site_id==1], mode='max')\nOut[10]: (0.4852269914108315, 0.4081632653061224) # (f1score, threshold)\n\nget_f1score(cancer_p[site_id==2], cancer_t[site_id==2], mode='max')\nOut[11]: (0.5681399603886713, 0.24489795918367346)\n\nget_f1score(cancer_p, cancer_t, mode='max')\nOut[12]: (0.5106197826424658, 0.2857142857142857)\nauc = 0.8344857204208718\nbce loss = 0.081253260\n![\"https://i.ibb.co/yQ9KQdz/Selection-728.png\"](\"https://i.ibb.co/yQ9KQdz/Selection-728.png\")
MIP trained with frozen image encoder + augmentation
\nuse age, site id, view
\n![\"https://i.ibb.co/wWtMJsQ/Selection-739.png\"](\"https://i.ibb.co/wWtMJsQ/Selection-739.png\")
\nunfrozen image encoder (last layer only) + augmentation : worse (overfitting observed)
unfrozen image encoder (last layer only) + no augmentation : not that bad??? … to repeat experiment again
", "rawMarkdown": "MIP trained with frozen image encoder + augmentation\nuse age, site id, view\n\n\n~~next update is to trained with unfrozen image encoder~~\nunfrozen image encoder (last layer only) + augmentation : worse (overfitting observed)\n\n\nunfrozen image encoder (last layer only) + no augmentation : not that bad??? ... to repeat experiment again", "votes": 2 }, { "id": 2123664, "postDate": "2023-01-31T16:19:21.270Z", "content": "Looks really good! Any sub to LB?
\nIt appeared that my bug in code had no influence on score. Still one model only 0.57. Looking for new ways to cross over 0.6 😂
Hello @remekkinas Could you tell me which image size did you use to get 0.57 LB? Thanks
", "rawMarkdown": "Hello @remekkinas Could you tell me which image size did you use to get 0.57 LB? Thanks" }, { "id": 2123989, "postDate": "2023-01-31T19:24:57.640Z", "content": "From beggining of the competiton I use rule (SIZE, SIZE // 2). Size depends on network architecture I use. Now is (4 * 384, 2 * 384).
", "rawMarkdown": "From beggining of the competiton I use rule (SIZE, SIZE // 2). Size depends on network architecture I use. Now is (4 * 384, 2 * 384). " }, { "id": 2138174, "postDate": "2023-02-10T16:06:57.800Z", "content": "@hengck23 did you ever submit this model to LB?
", "rawMarkdown": "@hengck23 did you ever submit this model to LB?" }, { "id": 2145791, "postDate": "2023-02-15T11:12:01.113Z", "content": "yes i did. surprsingly …. +0.03 in CV, -0.01 in LB
", "rawMarkdown": "yes i did. surprsingly .... +0.03 in CV, -0.01 in LB ", "votes": 1 }, { "id": 2146126, "postDate": "2023-02-15T16:16:50.693Z", "content": "I had LB … CV correlation. The worse score in CV the best on LB. This is certainly joke but I am really afraid how to chose final submission. I know that my submissions are shaky… and do not count on anything.
", "rawMarkdown": "I had LB … CV correlation. The worse score in CV the best on LB. This is certainly joke but I am really afraid how to chose final submission. I know that my submissions are shaky… and do not count on anything. ", "votes": 4 }, { "id": 2146231, "postDate": "2023-02-15T17:55:37.040Z", "content": "Our first 2 folds tracked public LB very closely so occassionally I would submit 2 folds' predictions to pump our public position 😂. Also, multi-image models boosted local validation pF1 at the cost of much higher optimal thresholds, which ended up in worse LB scores
", "rawMarkdown": "Our first 2 folds tracked public LB very closely so occassionally I would submit 2 folds' predictions to pump our public position 😂. Also, multi-image models boosted local validation pF1 at the cost of much higher optimal thresholds, which ended up in worse LB scores", "votes": 5 }, { "id": 2146426, "postDate": "2023-02-15T21:21:39.920Z", "content": "\"multi-image models boosted local validation pF1 at the cost of much higher optimal thresholds\"
\nhigher threshold is over fitting i i i think
\nit seems that there is not enough data to do multiview prediction, it is quite a pity.
i have to remove transformer and adopted simpler fusion like mean,max, attention pool, etc …
\nalso i do not have proper augmentation for multi-view (the views are related, so are the augmentation)
\non a site note, if i perform multi-view learning on external vindr-mammo dataset with patch prediction (i.e. convert segmentation pixel label to patch label), the class activation maps do show better results on validation set.
", "rawMarkdown": "\"multi-image models boosted local validation pF1 at the cost of much higher optimal thresholds\"\n\nhigher threshold is over fitting i i i think \nit seems that there is not enough data to do multiview prediction, it is quite a pity.\n\ni have to remove transformer and adopted simpler fusion like mean,max, attention pool, etc ...\n\nalso i do not have proper augmentation for multi-view (the views are related, so are the augmentation)\n\n---\n\non a site note, if i perform multi-view learning on external vindr-mammo dataset with patch prediction (i.e. convert segmentation pixel label to patch label), the class activation maps do show better results on validation set.", "votes": 1 }, { "id": 2146870, "postDate": "2023-02-16T08:12:27.790Z", "content": "When I tried your multi-image transformer, I had to freeze 70-80% of backbone blocks and used a very small lr to make it trainable. I also applied the same augmentation to all images per side, like this (albumentations is quite convenient)
\ntransformed = augment(images=image0, images1=image1 ...)\nI agree that we simply don't have enough data for multi-view training 😅. Simpler fusion models like boosted trees, mean, max might work better.
", "rawMarkdown": "When I tried your multi-image transformer, I had to freeze 70-80% of backbone blocks and used a very small lr to make it trainable. I also applied the same augmentation to all images per side, like this (albumentations is quite convenient)\n```\ntransformed = augment(images=image0, images1=image1 ...)\n```\nI agree that we simply don't have enough data for multi-view training 😅. Simpler fusion models like boosted trees, mean, max might work better.", "votes": 1 } ] } ] }, { "id": 2108173, "postDate": "2023-01-20T10:10:07.493Z", "content": "this is a cheap way to improve resolution and works for me at local CV
\n![\"https://i.ibb.co/nsPnr58/Selection-602.png\"](\"https://i.ibb.co/nsPnr58/Selection-602.png\")
Nice. As far as I understand gradient is updated from main loss?
", "rawMarkdown": "Nice. As far as I understand gradient is updated from main loss?", "replies": [ { "id": 2108234, "postDate": "2023-01-20T11:09:38.887Z", "content": "gradient is updated from main loss?
\nno. all loss
", "rawMarkdown": "gradient is updated from main loss?\n\nno. all loss", "votes": 3 }, { "id": 2108239, "postDate": "2023-01-20T11:13:30.470Z", "content": "Thank you for explanation.
", "rawMarkdown": "Thank you for explanation. " }, { "id": 2108280, "postDate": "2023-01-20T11:52:28.547Z", "content": "did you try different weighting for these losses ?
", "rawMarkdown": "did you try different weighting for these losses ?" }, { "id": 2108977, "postDate": "2023-01-21T02:16:06.237Z", "content": "I think this is a mix of two good ideas for aux loss + keeping hi-rez features. I tried something similar using resnet but it didn't improve my local CV.
\n@remekkinas this reminds me of the paper (Going deeper with convolutions) from a while back, I recommend checking it out! It is also the first paper that I saw that references a meme 😄😄
\n@left13 aux loss weighting might be interesting to try out, they also do it in the paper I linked above
\n\"During training, their loss gets added to the total loss of the network with a discount weight (the losses of the auxiliary classifiers were weighted by 0.3). At inference time, these auxiliary networks are discarded.\" [Going deeper with convolutions, 6]
I think if we were training from scratch it'll be a small boost to training time but I am interested in how each part affects CV for this competition since we are only fine-tuning the model
", "rawMarkdown": "I think this is a mix of two good ideas for aux loss + keeping hi-rez features. I tried something similar using resnet but it didn't improve my local CV.\n\n@remekkinas this reminds me of the paper ([Going deeper with convolutions](https://arxiv.org/pdf/1409.4842.pdf)) from a while back, I recommend checking it out! It is also the first paper that I saw that references a meme 😄😄\n\n@left13 aux loss weighting might be interesting to try out, they also do it in the paper I linked above\n\"During training, their loss gets added to the total loss of the network with a discount weight (the losses of the auxiliary classifiers were weighted by 0.3). At inference time, these auxiliary networks are discarded.\" [Going deeper with convolutions, 6]\n\nI think if we were training from scratch it'll be a small boost to training time but I am interested in how each part affects CV for this competition since we are only fine-tuning the model", "votes": 2 } ] }, { "id": 2110137, "postDate": "2023-01-22T02:08:28.810Z", "content": "if you want to use tensorrt for this model, please note the pytorch resize bug
\nhttps://github.com/pytorch/TensorRT/pull/1561
\nyou will have to modify the tensorrt py file
alternatively, just convert the encoder to trt engine and use pytorch nn module for the rest
", "rawMarkdown": "if you want to use tensorrt for this model, please note the pytorch resize bug\nhttps://github.com/pytorch/TensorRT/pull/1561\nyou will have to modify the tensorrt py file\n\nalternatively, just convert the encoder to trt engine and use pytorch nn module for the rest", "votes": 1 } ] }, { "id": 2077872, "postDate": "2022-12-27T23:34:30.137Z", "content": "finally, one paper that compares oversampling, undersampling and weighted class
\n![\"Selection-322\"](\"https://i.ibb.co/8NLc14R/Selection-322.png\")
\n![\"Selection-321\"](\"https://i.ibb.co/g3bzPHg/Selection-321.png\")
[1] Comparing Techniques for Class Imbalance in Deep LearningComparing Techniques for Class Imbalance in Deep Learning
\nClassification of Breast CancerClassification of Breast Cancer
the only conclusion is that no conclusion can be made
", "rawMarkdown": "finally, one paper that compares oversampling, undersampling and weighted class\n\n\n\n[1] Comparing Techniques for Class Imbalance in Deep LearningComparing Techniques for Class Imbalance in Deep Learning\nClassification of Breast CancerClassification of Breast Cancer\n\n\nthe only conclusion is that no conclusion can be made",
"votes": 5,
"replies": [
{
"id": 2078070,
"postDate": "2022-12-28T04:27:24.450Z",
"content": "\nI think the conclusion was generally that augmentation / pseudo labelling, especially domain specific augmentation (see the paper regarding how it does artifacting, makes a lot of sense) is effective when dealing with class imbalance. Imho, this is smart oversampling.
", "rawMarkdown": "https://www.techrxiv.org/articles/preprint/Comparing_Techniques_for_Class_Imbalance_in_Deep_Learning_Classification_of_Breast_Cancer/21400632\n\nI think the conclusion was generally that augmentation / pseudo labelling, especially domain specific augmentation (see the paper regarding how it does artifacting, makes a lot of sense) is effective when dealing with class imbalance. Imho, this is smart oversampling.", "votes": 2 }, { "id": 2087063, "postDate": "2023-01-05T09:38:21.357Z", "content": "Hi,
\nIt seems that, choosing sampling strategies should be depends on the dataset/scenarios, and should be hyperparameters.
\nBut there is one important thing the paper not answer, the sampling rate.
\nE.g. in over-sampling(ROS), higher pos. to neg. ratio would make the model overfit to pos.
\nThere is one recent paper[1], demonstrating this phenomena.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11665724%2F23f7d85d5c2233fe1dbe4cad50a4e72e%2F2023-01-05%205.31.30.png?generation=1672911121062365&alt=media\")
\nNote that, for the 4-layer case, pos:neg = 20:80(ROS-4) is the best choice.
[1] The Effects of Data Sampling with Deep Learning and Highly Imbalanced Big Data, Information System Frontiers, Springer, 2020
", "rawMarkdown": "Hi, \nIt seems that, choosing sampling strategies should be depends on the dataset/scenarios, and should be hyperparameters.\nBut there is one important thing the paper not answer, the sampling rate.\nE.g. in over-sampling(ROS), higher pos. to neg. ratio would make the model overfit to pos.\nThere is one recent paper[1], demonstrating this phenomena.\n\nNote that, for the 4-layer case, pos:neg = 20:80(ROS-4) is the best choice.\n\n[1] The Effects of Data Sampling with Deep Learning and Highly Imbalanced Big Data, Information System Frontiers, Springer, 2020", "replies": [ { "id": 2087167, "postDate": "2023-01-05T11:38:44.870Z", "content": "![\"https://i.ibb.co/nM5HBTx/Selection-476.png\"](\"https://i.ibb.co/nM5HBTx/Selection-476.png\")
most of the paper may not be useful because results of imbalance data is very dependent on the data itself.
\nhere is how you should analyze. see the results of the validation above, the question you should ask is that \"can you draw the distribution of an unknown test LB data?
\n[1] the distribution of the neg validation is smooth and predictable. How i think the test LB neg is close to the black dotted line.
\n[2] for the pos validation, it is multi modal, unpredictable , it even have empty bins. That is why results is unpredictable.
\non a side note:
\nif you are trying to use kaggle data to create a stable CV-LB, you should ask:
\n\"choosing sampling strategies should be depends on the dataset/scenarios, and should be hyperparameters.\"
\nthis is correct. it also depends on model.
", "rawMarkdown": "\"choosing sampling strategies should be depends on the dataset/scenarios, and should be hyperparameters.\"\n\nthis is correct. it also depends on model." }, { "id": 2087382, "postDate": "2023-01-05T15:03:33.950Z", "content": "Thanks, your comments are very inspiring
", "rawMarkdown": "Thanks, your comments are very inspiring" } ] } ] }, { "id": 2076911, "postDate": "2022-12-27T02:03:43.957Z", "content": "is the cat out of the bag …. anyone caught the cat ???
\nhttps://pubs.rsna.org/doi/pdf/10.1148/ryai.220072
\nADMANI: Annotated Digital Mammograms and Associated Non-Image Datasets
\nPublished Online:Dec 21 2022
![\"https://i.ibb.co/6YSxq3h/Selection-312.png\"](\"https://i.ibb.co/6YSxq3h/Selection-312.png\")
\" A subset of 40,000 images from 10,000 episodes will be provided for the
\nRadiological Society of North America Mammography Breast Cancer Detection AI Challenge,
\nlaunching on November 28th. The challenge training dataset will be made public when the
\nchallenge is launched and will remain available to researchers when the challenge concludes.
\nThe 10,000 episodes will be randomly selected from the dataset from a three-year period.\"
https://www.rsna.org/education/ai-resources-and-training/ai-image-challenge
\n\"The dataset was contributed by mammography screening programs in Australia and the U.S. It includes detailed labels, with radiologists’ evaluations and follow-up pathology results for suspected malignancies.\"
aka site1 and site2
\nmaybe NYU + ADMANI???
\nbaseline results:
\nhere is a paper that compares results with and without NYU pretrain model:
[1] Evaluation of deep learning-based artificial intelligence techniques for breast cancer detection on mammograms: Results from a retrospective study using a BreastScreen Victoria dataset (part of ADMANI)
\n[2] Multi-view Local Co-occurrence and Global Consistency Learning Improve Mammogram Classification Generalisation
", "rawMarkdown": "is the cat out of the bag .... anyone caught the cat ???\n\nhttps://pubs.rsna.org/doi/pdf/10.1148/ryai.220072\nADMANI: Annotated Digital Mammograms and Associated Non-Image Datasets\nPublished Online:Dec 21 2022\n\n\n\n\" A subset of 40,000 images from 10,000 episodes will be provided for the\nRadiological Society of North America Mammography Breast Cancer Detection AI Challenge,\nlaunching on November 28th. The challenge training dataset will be made public when the\nchallenge is launched and will remain available to researchers when the challenge concludes.\nThe 10,000 episodes will be randomly selected from the dataset from a three-year period.\"\n\nhttps://www.rsna.org/education/ai-resources-and-training/ai-image-challenge\n\"The dataset was contributed by mammography screening programs in Australia and the U.S. It includes detailed labels, with radiologists’ evaluations and follow-up pathology results for suspected malignancies.\"\n\naka site1 and site2\n\nmaybe NYU + ADMANI???\n\nbaseline results:\nhere is a paper that compares results with and without NYU pretrain model:\n\n[1] Evaluation of deep learning-based artificial intelligence techniques for breast cancer detection on mammograms: Results from a retrospective study using a BreastScreen Victoria dataset (part of ADMANI)\n\n[2] Multi-view Local Co-occurrence and Global Consistency Learning Improve Mammogram Classification Generalisation\n", "votes": 5 }, { "id": 2084837, "postDate": "2023-01-03T19:36:40.053Z", "content": "high quality \"Breast Micro-Calcifications Dataset with Precisely Annotated Sequential Mammograms\"
\ndataset:
\nhttps://zenodo.org/record/5036062
paper:
\n[1] Loizidou, K., Skouroumouni, G., Pitris, C. et al. Digital subtraction of temporally sequential mammograms for improved detection and classification of microcalcifications. Eur Radiol Exp 5, 40 (2021). https://doi.org/10.1186/s41747-021-00238-w
papers that uses this dataset:
\nhttps://scholar.google.com/citations?user=qadXBKAAAAAJ&hl=en
some analysis
\n![\"https://i.ibb.co/F8vKhbt/Selection-106.png\"](\"https://i.ibb.co/F8vKhbt/Selection-106.png\")
the sorted probability (red-black) graph gives you an idea how the sample prediction values fluctuate with different models.
\nthis is important when you are making ensemble and choosing threshold value in pfbeta binarization.
\nthe threshold must be stable over the fluctuation.
Thanks! I see that you are using balance sampler, would be interesting compare with bce with loss weight with the same weights.
", "rawMarkdown": "Thanks! I see that you are using balance sampler, would be interesting compare with bce with loss weight with the same weights." }, { "id": 2054865, "postDate": "2022-12-04T13:59:15.230Z", "content": "![\"https://i.ibb.co/F0XkGHD/Selection-121.png\"](\"https://i.ibb.co/F0XkGHD/Selection-121.png\")
i try to understand why resolution 1024 is better than 512. I show some CAM activation results for 1024 here (but i am not sure if the model is correct)
\n![\"Selection-117\"](\"https://i.ibb.co/0Bt7mG7/Selection-117.png\")
\n![\"Selection-116\"](\"https://i.ibb.co/4tMnr9d/Selection-116.png\")
\n",
"votes": 8
},
{
"id": 2055654,
"postDate": "2022-12-05T09:31:50.267Z",
"content": "I am not familiar with mammography findings, but perhaps the visibility of calcification, which is typical of malignant findings, is very different between 512 and 1024?
\nBoth of the images you posted seem to focus on the fine calcifications, which are both parapetitive.
EDIT: I don't think calcification is completely specific, as it can be seen not only in breast cancer but also in benign diseases.
", "rawMarkdown": "I am not familiar with mammography findings, but perhaps the visibility of calcification, which is typical of malignant findings, is very different between 512 and 1024?\nBoth of the images you posted seem to focus on the fine calcifications, which are both parapetitive.\n\nEDIT: I don't think calcification is completely specific, as it can be seen not only in breast cancer but also in benign diseases.", "votes": 5 }, { "id": 2056120, "postDate": "2022-12-05T18:40:11.017Z", "content": "I can't speak to this specific image, but to the point @yosukeyama made it's expected that the presence of some very small features (the calcifications) might favor models that use larger images. This article might provide useful context: https://radiopaedia.org/articles/breast-imaging-reporting-and-data-system-bi-rads
", "rawMarkdown": "I can't speak to this specific image, but to the point @yosukeyama made it's expected that the presence of some very small features (the calcifications) might favor models that use larger images. This article might provide useful context: https://radiopaedia.org/articles/breast-imaging-reporting-and-data-system-bi-rads", "votes": 4 }, { "id": 2056132, "postDate": "2022-12-05T19:05:44.347Z", "content": "Addition to the dim. questions, can we lose some important grey level information in the png conversion from dicom and maybe it would be better using e.g. NIfTI-format, or do we have all information needed by using windowing features?
", "rawMarkdown": "Addition to the dim. questions, can we lose some important grey level information in the png conversion from dicom and maybe it would be better using e.g. NIfTI-format, or do we have all information needed by using windowing features?", "votes": 2 }, { "id": 2057734, "postDate": "2022-12-07T10:11:06.943Z", "content": "i show CAM heatmap for an external dataset with lesion annotation
\nhttps://vindr.ai/datasets/mammo
![\"0-991211-9b28bca8f8312283e4fd9f093646b04b\"](\"https://i.ibb.co/p1m1dQj/0-991211-9b28bca8f8312283e4fd9f093646b04b.png\")
\n![\"0-958984-dec4540f406b4e556983b4f76504ee30\"](\"https://i.ibb.co/RQqbFH5/0-958984-dec4540f406b4e556983b4f76504ee30.png\")
\n![\"0-854492-19acc4b912b5637af651392bc1fe6b6e\"](\"https://i.ibb.co/SxCbhrg/0-854492-19acc4b912b5637af651392bc1fe6b6e.png\")
\n![\"0-735352-f54b07517cb46e8a59c1c10748bdb5ed\"](\"https://i.ibb.co/xXWhwR9/0-735352-f54b07517cb46e8a59c1c10748bdb5ed.png\")
\n![\"0-682617-735851f234a657318773c4cbbe4969cf\"](\"https://i.ibb.co/n1P216J/0-682617-735851f234a657318773c4cbbe4969cf.png\")
\n![\"0-587891-d83b16559c3ad828bd86db23e4f11243\"](\"https://i.ibb.co/yYTWPm2/0-587891-d83b16559c3ad828bd86db23e4f11243.png\")
\n![\"0-425781-6057b80f4f3e7d18f2bac341f7a64e07\"](\"https://i.ibb.co/8gnXq6y/0-425781-6057b80f4f3e7d18f2bac341f7a64e07.png\")
\n![\"0-371338-83be060130997ca7b67b3979978a5d29\"](\"https://i.ibb.co/Mp5xHsT/0-371338-83be060130997ca7b67b3979978a5d29.png\")
\n![\"0-347656-31fcc94f3079f2b234c6e4304ab540e3\"](\"https://i.ibb.co/Yks2WMy/0-347656-31fcc94f3079f2b234c6e4304ab540e3.png\")
\n![\"0-289062-e45c5993ab3a5c28b0f5ed32a0c204b9\"](\"https://i.ibb.co/JRhKCvH/0-289062-e45c5993ab3a5c28b0f5ed32a0c204b9.png\")
\n![\"0-262451-07e191bc54c3378f9fdf23ddecc47420\"](\"https://i.ibb.co/YX5vDQL/0-262451-07e191bc54c3378f9fdf23ddecc47420.png\")
i am glad that for high predicted probability, the model (single fold 1024) performs quite well.
\nhowever t there are many misses. I think this is due to lack of kaggle train data.
use of external will play an important part in this competition
", "rawMarkdown": "i show CAM heatmap for an external dataset with lesion annotation\nhttps://vindr.ai/datasets/mammo\n\n\n\n\n\n\n\n\n\n\n\n\n\n\ni am glad that for high predicted probability, the model (single fold 1024) performs quite well.\nhowever t there are many misses. I think this is due to lack of kaggle train data.\n\nuse of external will play an important part in this competition\n",
"votes": 7
},
{
"id": 2058004,
"postDate": "2022-12-07T14:37:27.680Z",
"content": "2048 is the king?
\n![\"https://i.ibb.co/8rk2PyZ/Selection-134.png\"](\"https://i.ibb.co/8rk2PyZ/Selection-134.png\")
anyone has good suggestion for a low-memory, low flop model that might work well for 2048?
", "rawMarkdown": "2048 is the king?\n\n\n\nanyone has good suggestion for a low-memory, low flop model that might work well for 2048?", "votes": 3 }, { "id": 2058540, "postDate": "2022-12-08T02:27:18.853Z", "content": "check the distribution diagram
\nhttps://miro.medium.com/max/720/1*yF319EgJVzag9pd2ZL4D4Q.webp
it compare bce loss and soft-f1 loss
\n", "rawMarkdown": "check the distribution diagram\nhttps://miro.medium.com/max/720/1*yF319EgJVzag9pd2ZL4D4Q.webp\n\nit compare bce loss and soft-f1 loss\n\nhttps://towardsdatascience.com/the-unknown-benefits-of-using-a-soft-f1-loss-in-classification-systems-753902c0105d\n\n" }, { "id": 2058880, "postDate": "2022-12-08T09:28:06.033Z", "content": "\n\nanyone has good suggestion for a low-memory, low flop model that might work well for 2048?
\n
Teacher-student architectur and knowledge distillation?
", "rawMarkdown": "> anyone has good suggestion for a low-memory, low flop model that might work well for 2048?\n\nTeacher-student architectur and knowledge distillation?" }, { "id": 2058958, "postDate": "2022-12-08T11:00:11.143Z", "content": "![\"https://i.ibb.co/rwkSLfG/Selection-138.png\"](\"https://i.ibb.co/rwkSLfG/Selection-138.png\")
i added a head to learn the best probability calibration for maximizing kaggle metric F1.
\nIn the end, it is the same as hard binary thresholding
Good explanation, well🙌
", "rawMarkdown": "Good explanation, well🙌" } ] }, { "id": 2162027, "postDate": "2023-02-28T00:07:40.403Z", "content": "results are out!
\n3 fold nextvit-b has private lb 0.48.
\n1 fold nextvit-b has private lb 0.45.
but unfornately, i didn't select that
\n![\"https://i.ibb.co/bWXKr73/Selection-999-1157.png\"](\"https://i.ibb.co/bWXKr73/Selection-999-1157.png\")
May I ask why you chose 1-fold over 3-fold solution?
", "rawMarkdown": "May I ask why you chose 1-fold over 3-fold solution?" } ] }, { "id": 2113008, "postDate": "2023-01-24T02:53:04.297Z", "content": "sometimes you want to modify timms model without messing of the original code. Here is a way to do it:
\nfrom timm.models.convnext import _create_convnext\n\ndef convnext_tiny(pretrained=False, **kwargs):\n model_args = dict(depths=(3, 3, 9, 3), dims=(96, 192, 384, 768), **kwargs)\n model = _create_convnext('convnext_tiny.in12k_ft_in1k_384', pretrained=pretrained, **model_args)\n setattr(model, 'depths', [3, 3, 9, 3])\n return model\n\n\n#modify to output all layers\nclass Encoder(nn.Module):\n def __init__(self, ):\n super(Encoder, self).__init__()\n e = convnext_tiny(pretrained=True)\n self.stem = e.stem\n self.stage1 = e.stages[ 0 : e.depths[0]]\n self.stage2 = e.stages[e.depths[0] : e.depths[1]]\n self.stage3 = e.stages[e.depths[1] : e.depths[2]]\n self.stage4 = e.stages[e.depths[2] : e.depths[3]]\n self.norm_pre = e.norm_pre\n del e\n\n def forward(self, x):\n x0 = self.stem(x)\n x1 = self.stage1(x0)\n x2 = self.stage2(x1)\n x3 = self.stage3(x2)\n x4 = self.stage4(x3)\n return [x1,x2,x3,x4]\nin order to improve the detection rate, one may want to flip some labels:
\ne.g.
\ndifficult_negative_case =1, biopsy=1 --> cancer =1
\n(since it is sent for biopsy, it should be visually close to malignant )
tried that but no improvement. might need to add density type as well
", "rawMarkdown": "tried that but no improvement. might need to add density type as well", "votes": 1 } ] }, { "id": 2084695, "postDate": "2023-01-03T17:18:02.053Z", "content": "https://github.com/nyukat/BIRADS_classifier
\nhttps://cs.nyu.edu/~kgeras/reports/datav1.0.pdf
How to map external data to kaggle label
\n\"As BI-RADS 0 and BI-RADS 1 and BI-RADS 2 should be the only BI-RADS categories used in screening mammography,
\nwe condensed all BI-RADS categories into three classes for the purposes of training our model.\"
BI-RADS 0, 4a/b/c and 5 were mapped to a new ‘BI-RADS 0’ as each indicates a possibility of malignancy.
\nBI-RADS 1 is retained at ‘BI-RADS1’.
\nBI-RADS 2 and 3 are mapped to a new ‘BI-RADS 2’, as they both indicate benign findings.
This procedure resulted in a dataset consisting of a single BI-RADS label over three classes for each of our valid screening mammography exams.\"
\nBI-RADS categories:
\n0 (‘incomplete’),
\n1 (‘negative’),
\n2 (‘benign’),
\n3 (‘probably benign’),
\n4a (‘low suspicious’),
\n4b (‘moderate sus-picious’),
\n4c (‘high suspicious’)
\n5 (‘highly suggestive of malignancy’)
\n6 (‘known biopsy with proven malignancy’)
The paper you link to is from 2019, correct? That's 3 years ago. I wonder if they've enhanced the labelling given what's occurred since then.
", "rawMarkdown": "The paper you link to is from 2019, correct? That's 3 years ago. I wonder if they've enhanced the labelling given what's occurred since then." } ] }, { "id": 2082873, "postDate": "2023-01-02T03:44:08.533Z", "content": "results on large scale external data (vindr) is pretty much the same as kaggle data:
\n![\"Picture1\"](\"https://i.ibb.co/3CBcs2N/Picture1.png\")
",
"votes": 3,
"replies": [
{
"id": 2083835,
"postDate": "2023-01-02T22:38:43.980Z",
"content": "Hey, I am a novice and I have a question, did you train on Kaggle and predict on Vindr? I am trying to figure out a good CV step because I have doubts about mine. Since we are thresholding, would holdout and N-fold CV be better to find a threshold for the competition dataset? And can we use Vindr as the holdout if it is similar as you mentioned?
\nIf the Vindr dataset is very similar to Kaggle's I am sure we can use it to validate models (I think this won't go against their license since we are not training on it).
", "rawMarkdown": "Hey, I am a novice and I have a question, did you train on Kaggle and predict on Vindr? I am trying to figure out a good CV step because I have doubts about mine. Since we are thresholding, would holdout and N-fold CV be better to find a threshold for the competition dataset? And can we use Vindr as the holdout if it is similar as you mentioned?\n\nIf the Vindr dataset is very similar to Kaggle's I am sure we can use it to validate models (I think this won't go against their license since we are not training on it).", "votes": 2, "replies": [ { "id": 2083921, "postDate": "2023-01-03T01:08:00.367Z", "content": "[1] Can Vindr be used in the solution at all?
\ni haven't considered this yet. my current objective is to get a good way to understand the problem, training and metrics first. maybe it is possible that Vindr is not allowed and i will deal with that later. Vindr is currently investigated, because i can get benchmark results form paper and there are more annotation like bounding box, etc for experiments.
\n[2] did you train on Kaggle and predict on Vindr?
\nresults are based on training = Vindr, testing = Vindr. I am just repeating the paper results to confirm my pipeline is correct.
\nEven with Vindr, the metrics are not stable (both AUC and F1 score). Vindr is still not large enough. I think the reason is as follows:
\npure visual diagnostics from image mammography has its limitation. Positive predictive value (PPV), aka. precision, of radiologist in screening is not high. that is why biopsy is needed to confirm cancer cases.
that is why just only based on visual evidence, given a choice to improve either sensitivity (tp rate) or specificity (tn rate), it is \"easier\" for the model to choose specificity. You can better metrics because there are much more neg samples (imbalance) and more likely to be correct (given same detectable visual abnormality, it is more likely to be benign)
you can google for PPV for abnormality in mammography screening for more information
\nHence there is probably no good metric. and you are not likely to see good CV-LB as you would usually would in previous problems with better data.
\nYou will also see that for papers that conduct experiments on various datasets (instead of one), metrics on various datasets various. Results are more stable for those that uses millions of training images (from 1 million to 4 million).
\nsince you are a student, i strong encourage you to apply for those million image dataset like OPTIMAM, CSAW, AMAINDA and conduct experiments on those (but note that they cannot be used for kaggle solutions if you are aiming for prize). These are open to public but by need email request. You will get a bigger picture on stable CV.
\nThe next best thing is to read papers on those big dataset.
\nThen how to approach this competition?
\ndiversity:
\n\"if diverse solution agrees on diverse datasets with diverse metrics, they are probably more correct\".
", "rawMarkdown": "[1] Can Vindr be used in the solution at all?\n\ni haven't considered this yet. my current objective is to get a good way to understand the problem, training and metrics first. maybe it is possible that Vindr is not allowed and i will deal with that later. Vindr is currently investigated, because i can get benchmark results form paper and there are more annotation like bounding box, etc for experiments.\n\n[2] did you train on Kaggle and predict on Vindr?\n\nresults are based on training = Vindr, testing = Vindr. I am just repeating the paper results to confirm my pipeline is correct.\n\n---\n\nEven with Vindr, the metrics are not stable (both AUC and F1 score). Vindr is still not large enough. I think the reason is as follows:\n\n- pure visual diagnostics from image mammography has its limitation. Positive predictive value (PPV), aka. precision, of radiologist in screening is not high. that is why biopsy is needed to confirm cancer cases.\n\n- that is why just only based on visual evidence, given a choice to improve either sensitivity (tp rate) or specificity (tn rate), it is \"easier\" for the model to choose specificity. You can better metrics because there are much more neg samples (imbalance) and more likely to be correct (given same detectable visual abnormality, it is more likely to be benign)\n\nyou can google for PPV for abnormality in mammography screening for more information \n\nHence there is probably no good metric. and you are not likely to see good CV-LB as you would usually would in previous problems with better data. \n\n---\n\nYou will also see that for papers that conduct experiments on various datasets (instead of one), metrics on various datasets various. Results are more stable for those that uses millions of training images (from 1 million to 4 million).\n\nsince you are a student, i strong encourage you to apply for those million image dataset like OPTIMAM, CSAW, AMAINDA and conduct experiments on those (but note that they cannot be used for kaggle solutions if you are aiming for prize). These are open to public but by need email request. You will get a bigger picture on stable CV. \n\nThe next best thing is to read papers on those big dataset.\n\n---\n\nThen how to approach this competition?\n\ndiversity:\n- different models and different solutions (not just change the backbone)\n- more different datasets to train or validate (one or two or even three are not enough)\n\n\"if diverse solution agrees on diverse datasets with diverse metrics, they are probably more correct\".\n ", "votes": 4 }, { "id": 2083930, "postDate": "2023-01-03T01:18:52.607Z", "content": "Thanks for your insights, I will continue learning!
", "rawMarkdown": "Thanks for your insights, I will continue learning!" }, { "id": 2084004, "postDate": "2023-01-03T04:07:33.927Z", "content": "![\"https://i.imgur.com/poP7qN6.png\"](\"https://i.imgur.com/poP7qN6.png\")
an example of repeating results. you can see how the results varies
", "rawMarkdown": "\n\nan example of repeating results. you can see how the results varies", "votes": 1 } ] } ] }, { "id": 2057901, "postDate": "2022-12-07T12:42:41.050Z", "content": "mixed results of dicom intensity windowing
\n![\"https://i.ibb.co/t4TFG9z/Selection-127.png\"](\"https://i.ibb.co/t4TFG9z/Selection-127.png\")
Applying window operation on the fly is too expensive though. Maybe you can try stacking 2 different windowed output on channel dimension.
", "rawMarkdown": "Applying window operation on the fly is too expensive though. Maybe you can try stacking 2 different windowed output on channel dimension." } ] }, { "id": 2065229, "postDate": "2022-12-14T12:25:30.160Z", "content": "how to set pos weight in loss to maximize f1 score for imbalanced class
\nhttp://ethen8181.github.io/machine-learning/model_selection/imbalanced/imbalanced_metrics.html
Balance using Sampler or loss weights … or both?
", "rawMarkdown": "Balance using Sampler or loss weights ... or both?" }, { "id": 2065446, "postDate": "2022-12-14T17:02:07.163Z", "content": "still experimenting. i not sure if there will be a shakeup because for the same single one-fold model + the same data that is trained differently, i can get very different lb scores of 0.39 to 0.51.
", "rawMarkdown": "still experimenting. i not sure if there will be a shakeup because for the same single one-fold model + the same data that is trained differently, i can get very different lb scores of 0.39 to 0.51." }, { "id": 2065958, "postDate": "2022-12-15T08:34:21.677Z", "content": "Do you use BalancedSampler presented in your code? If yes I think that a bit of instability in score could cause this line:
\n pos_index = np.random.choice(pos_index, self.length//self.r).reshape(-1,1)\nwhat do you think?
", "rawMarkdown": "Do you use BalancedSampler presented in your code? If yes I think that a bit of instability in score could cause this line:\n\n```\n pos_index = np.random.choice(pos_index, self.length//self.r).reshape(-1,1)\n```\n\nwhat do you think?", "votes": 1 } ] }, { "id": 2059570, "postDate": "2022-12-09T03:16:02.913Z", "content": "validation results are complementary. This means that 2048 model is improving different samples from 1024.
\nin particular, AUC of large image is much lower (probably improving the previous low p=0 region)
\npfbeta of large model is better (probably improving the previous mid p=0.5 region)
image-wise validation results (fold-0)\n\n\neffb2-2048\nbce_loss AUC pfbeta\n0.09397 0.82092 0.19947 \n\neffb6-1024\nbce_loss AUC pfbeta\n0.106 0.7949 0.263 \nLB results of 2048
\n![\"https://i.ibb.co/qpqHVhT/Selection-146.png\"](\"https://i.ibb.co/qpqHVhT/Selection-146.png\")
when ensemble of multiple input size are used, i first convert dicom to the largest size png with cv2.resize().
\nthen, pytorch F.interpolate() function is used to create the smaller size image in net forward().
\nthis is different from training and maybe the cause of poor performance?
interesting paper
\ncvpr2022:
\nEfficient Classification of Very Large Images with Tiny Objects
\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Kong_Efficient_Classification_of_Very_Large_Images_With_Tiny_Objects_CVPR_2022_paper.pdf
Thanks for sharing @hengck23 great analysis!
", "rawMarkdown": "Thanks for sharing @hengck23 great analysis!", "votes": 4 }, { "id": 2054612, "postDate": "2022-12-04T09:29:51.753Z", "content": "@hengck23 I am learning a lot from these. Much appreciated.
", "rawMarkdown": "@hengck23 I am learning a lot from these. Much appreciated.", "votes": 4 }, { "id": 2149480, "postDate": "2023-02-18T10:58:31.413Z", "content": "some cam map thoughts:
\n![\"Selection-999-757\"](\"https://i.ibb.co/HnVCCFK/Selection-999-757.png\")
\n![\"Selection-999-758\"](\"https://i.ibb.co/zSyTmmY/Selection-999-758.png\")
![\"Selection-999-759\"](\"https://i.ibb.co/kxQhm8w/Selection-999-759.png\")
\n![\"Selection-999-760\"](\"https://i.ibb.co/F6kzp8S/Selection-999-760.png\")
\n\n\n\n\n\n",
"votes": 1,
"replies": [
{
"id": 2149544,
"postDate": "2023-02-18T12:23:21.710Z",
"content": "i suddenly have interesting question. maybe someone can take this as future research.
\nif i purposely mislabelled a negative image as positive for training, what would the CAM heatmap show? waht is the implication?
how about the reverse? mislabelled pos as neg?
", "rawMarkdown": "i suddenly have interesting question. maybe someone can take this as future research.\nif i purposely mislabelled a negative image as positive for training, what would the CAM heatmap show? waht is the implication?\n\nhow about the reverse? mislabelled pos as neg?", "replies": [ { "id": 2152260, "postDate": "2023-02-20T17:17:00.357Z", "content": "if may be difficult to apply multi-view learning, but you can transfer/distill knowledge via manual annotation !!!!
\n![\"https://i.ibb.co/56jy09J/Selection-999-843.png\"](\"https://i.ibb.co/56jy09J/Selection-999-843.png\")
i started some experiments on multi-view. this is very tricky as you are basically reducing your train and validation data since it now become
\nnum of train data = num of original train data / num of view\n@hengck23
\nThanks for this really useful and interesting discussion. By the way, have models on multi-view worked well?
i haven try this but 2 ideas work for finetunning NLP transformers may work here
\nboth method try to flatten the valley of the resulting loss landscape to imporve robustenss and fight overfitting.
\n(or in the input space sense, try to create virtual neighbours for each train samples)
reference
\nhttps://github.com/juntang-zhuang/GSAM
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
\nhttps://arxiv.org/abs/2006.03654
\n\"In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understanding\"
\nhttps://github.com/microsoft/DeBERTa/blob/master/DeBERTa/sift/sift.py
Have you read \"Deep is a Luxury We Don't Have\"? They improve on the GMIC arch and show that ASAM improves generalization. Results look good and code is released but its under a company so I doubt weights would be released.
", "rawMarkdown": "Have you read \"[Deep is a Luxury We Don't Have](https://arxiv.org/abs/2208.06066)\"? They improve on the GMIC arch and show that ASAM improves generalization. Results look good and [code is released](https://github.com/whiterabbit-ai/hct) but its under a company so I doubt weights would be released.", "votes": 1, "replies": [ { "id": 2105192, "postDate": "2023-01-18T10:47:57.453Z", "content": "I have tried it and failed to achieve AUC > 0.75 on kaggle dataset
\nAlso it takes a long time to load images in 3k resolution,
\nso you will have to either wait couple of days or implement DALI loading pipeline
The reason I think is that there is no pretrained weights and kaggle dataset is rather small to use networks without pretraining, but if I remember correctly their neural net on 3k resolution & batch 16 takes only 10Gb of GPU VRAM with amp, so you can give it a try
", "rawMarkdown": "I have tried it and failed to achieve AUC > 0.75 on kaggle dataset\n\nAlso it takes a long time to load images in 3k resolution, \nso you will have to either wait couple of days or implement DALI loading pipeline\n\nThe reason I think is that there is no pretrained weights and kaggle dataset is rather small to use networks without pretraining, but if I remember correctly their neural net on 3k resolution & batch 16 takes only 10Gb of GPU VRAM with amp, so you can give it a try" }, { "id": 2105370, "postDate": "2023-01-18T13:04:50.827Z", "content": "Yeah without weights I don't expect it to produce good results for our uses but the paper is a good read and I learned a lot of new things. My experience with GMIC isn't well either, I think with the same resolution I only got to AUC ~ .85 single image but the recall wasn't great so my F1 score was about .3~4.
", "rawMarkdown": "Yeah without weights I don't expect it to produce good results for our uses but the paper is a good read and I learned a lot of new things. My experience with GMIC isn't well either, I think with the same resolution I only got to AUC ~ .85 single image but the recall wasn't great so my F1 score was about .3~4. " } ] } ] }, { "id": 2079089, "postDate": "2022-12-29T01:20:09.167Z", "content": "to crop or not to crop
\n![\"Selection-334\"](\"https://i.ibb.co/pZLRcm3/Selection-334.png\")
",
"votes": 1
},
{
"id": 2074114,
"postDate": "2022-12-23T18:07:27.113Z",
"content": "![\"https://i.ibb.co/Z8kbcqx/Selection-273.png\"](\"https://i.ibb.co/Z8kbcqx/Selection-273.png\")
\ni wonder if we have enough data for transformer solution
modified from gigapixel transformer paper
\nhttps://paperswithcode.com/paper/scaling-vision-transformers-to-gigapixel-1
![\"https://i.ibb.co/tZ680Lr/Selection-275.png\"](\"https://i.ibb.co/tZ680Lr/Selection-275.png\")
![\"https://i.ibb.co/3NQ7L1Y/Selection-278.png\"](\"https://i.ibb.co/3NQ7L1Y/Selection-278.png\")
probably useful to detect asymmetry abnormality between left and right
", "rawMarkdown": "\n\nprobably useful to detect asymmetry abnormality between left and right\n" }, { "id": 2074142, "postDate": "2022-12-23T19:03:14.740Z", "content": "What do you think about AE (trained only on background) and then using it for anomaly detection? I think that for each cancer photo loss should be higher then for background.
", "rawMarkdown": "What do you think about AE (trained only on background) and then using it for anomaly detection? I think that for each cancer photo loss should be higher then for background. " }, { "id": 2074275, "postDate": "2022-12-23T23:58:37.477Z", "content": "this are some smiliar work on this:
\nhttps://www.nature.com/articles/s41598-021-89626-1
\n![\"https://i.ibb.co/kKjCsS4/Selection-279.png\"](\"https://i.ibb.co/kKjCsS4/Selection-279.png\")
masked image modeling (aka, masked token pretraining for transformer e.g. https://github.com/facebookresearch/mae) are essentially auto encoders. you can try that
\nUnsupervised Anomaly Detection in Medical Images with a Memory-augmented Multi-level Cross-attentional Masked Autoencoder
\nhttps://yutianyt.com/publications/
another great work
\nMulti-view Local Co-occurrence and Global Consistency Learning Improve Mammogram Classification Generalisation
\nhttps://yutianyt.com/publications/
![\"https://i.ibb.co/sjtJqnL/iRPE.png\"](\"https://i.ibb.co/sjtJqnL/iRPE.png\")
\nhttps://houwenpeng.com/publication.html
Rethinking and Improving Relative Position Encoding for Vision Transformer
", "rawMarkdown": "\nhttps://houwenpeng.com/publication.html\n\nRethinking and Improving Relative Position Encoding for Vision Transformer" }, { "id": 2075578, "postDate": "2022-12-25T15:50:59.347Z", "content": "example code and implementation
\n![\"Selection-293\"](\"https://i.ibb.co/LktcRj5/Selection-293.png\")
\n![\"Selection-291\"](\"https://i.ibb.co/h8kNMXy/Selection-291.png\")
\n![\"Selection-292\"](\"https://i.ibb.co/D16fQjR/Selection-292.png\")
![\"Selection-294\"](\"https://i.ibb.co/7JcK0DK/Selection-294.png\")
\n\n\n\n---\n",
"votes": 2
},
{
"id": 2075587,
"postDate": "2022-12-25T16:09:41.920Z",
"content": "Looks great! I am working now on AE using ViT (as a decoder). Hope next week will be ready for tests.
", "rawMarkdown": "Looks great! I am working now on AE using ViT (as a decoder). Hope next week will be ready for tests.", "votes": 1 }, { "id": 2079075, "postDate": "2022-12-29T00:30:06Z", "content": "\ni find a paper that does auto encoder (reconstruction loss)![\"https://i.ibb.co/Mk3T27y/Selection-326.png\"](\"https://i.ibb.co/Mk3T27y/Selection-326.png\")
https://www.frontiersin.org/articles/10.3389/fradi.2021.796078/full
\nThank you very much for providing this publication. I will read it.
\nNow I am developing AE. The main idea is:
Still work on dataset as well.
", "rawMarkdown": "Thank you very much for providing this publication. I will read it. \nNow I am developing AE. The main idea is:\n- split image into patches and augment them with positional embedding\n- output sequence is fed to the transformer encoder\n- encoded features are summed into a reconstruction vector \n- reconstruction vector is fed to decoder\n- encoded features are fed into an Gaussian approximation network\n\nStill work on dataset as well. " }, { "id": 2079629, "postDate": "2022-12-29T14:03:14.140Z", "content": "Are you doing multi-task learning or just plugging in pretrained heads to your model? @hengck23
", "rawMarkdown": "Are you doing multi-task learning or just plugging in pretrained heads to your model? @hengck23 " }, { "id": 2082758, "postDate": "2023-01-01T22:28:38.853Z", "content": "\"Specifically, a well-trained Auto-Encoder (AE) is employed to explore common anatomical structures from large sets of mammographic images with weak annotations. Moreover, to push forward segmentation performance, KD is introduced to transfer anatomy features representation ability from the AE (Teacher) to the segmentation network, i.e., U-Net (Student) so that the segmentation network becomes more sensitive to other anatomy structures when the available data is limited.\"
\nhttps://journals.plos.org/plosone/article?id=10.1371/journal.pone.0256830
\nAAWS-Net: Anatomy-aware weakly-supervised learning network for breast mass segmentation
![\"https://i.ibb.co/qBBhsty/journal-pone-0256830-g001.png\"](\"https://i.ibb.co/qBBhsty/journal-pone-0256830-g001.png\")
Hi, could u please tell me which paper is this image from? thx:)
", "rawMarkdown": "Hi, could u please tell me which paper is this image from? thx:)" } ] }, { "id": 2072979, "postDate": "2022-12-22T15:00:21.430Z", "content": "should the threshold for f1score binarisation varies with breast density (and/or BI-RADS)?
", "rawMarkdown": "should the threshold for f1score binarisation varies with breast density (and/or BI-RADS)?", "votes": 1, "replies": [ { "id": 2073046, "postDate": "2022-12-22T16:08:41.213Z", "content": "It should, the threshold will change with models predictions, and since predictions themselves can be in different ranges, ranges which could vary with change in breast density and BI-RADS, I think variation of BI-RADS will have more effect than breast density, although needs further testing
", "rawMarkdown": "It should, the threshold will change with models predictions, and since predictions themselves can be in different ranges, ranges which could vary with change in breast density and BI-RADS, I think variation of BI-RADS will have more effect than breast density, although needs further testing", "votes": 1 } ] }, { "id": 2146431, "postDate": "2023-02-15T21:31:24.823Z", "content": "i find some direct differentiable loss for AUC-PR and Fbeta score.
\n(in pytorch and tensorflow)
the idea is simple.
\nyou find surrogate function for FPR and TPR, which have to learn the a \"threshold parameter\"
\nFPR, TPR themselves are function of a deep net, which have \"net parameter\"
so the optimization become a min, max problem for \"threshold parameter\" and \"net parameter\"
\nyou can solved it via sdg with weighted classification loss
\n(or you can solve in globally using linear programming)
paper:
\nhttps://github.com/Shlomix/global_objectives_pytorch/
\nhttps://github.com/facebookresearch/pytext/blob/main/pytext/loss/loss.py
paper:
\n[1] Scalable Learning of Non-Decomposable Objectives
\nhttps://arxiv.org/abs/1608.04802
but we have an issue of imbalance. so i not sure if the method would work as well.
", "rawMarkdown": "i find some direct differentiable loss for AUC-PR and Fbeta score.\n(in pytorch and tensorflow)\n\nthe idea is simple.\nyou find surrogate function for FPR and TPR, which have to learn the a \"threshold parameter\" \nFPR, TPR themselves are function of a deep net, which have \"net parameter\" \n\nso the optimization become a min, max problem for \"threshold parameter\" and \"net parameter\" \nyou can solved it via sdg with weighted classification loss\n(or you can solve in globally using linear programming)\n\npaper:\nhttps://github.com/Shlomix/global_objectives_pytorch/\nhttps://github.com/facebookresearch/pytext/blob/main/pytext/loss/loss.py\n\n\n\npaper:\n[1] Scalable Learning of Non-Decomposable Objectives\nhttps://arxiv.org/abs/1608.04802\n\n\n---\n\nbut we have an issue of imbalance. so i not sure if the method would work as well.", "votes": 2 }, { "id": 2058060, "postDate": "2022-12-07T15:38:48.123Z", "content": "@hengck23 Your CV/LB gap seems huge. I'm having ~0.01 difference.
\nAny idea why ?
EDIT : Fixed a bug in my inference pipeline, I have a huge CV/LB gap as well now (0.04)
", "rawMarkdown": "@hengck23 Your CV/LB gap seems huge. I'm having ~0.01 difference.\nAny idea why ?\n\nEDIT : Fixed a bug in my inference pipeline, I have a huge CV/LB gap as well now (0.04)", "votes": 1, "replies": [ { "id": 2058101, "postDate": "2022-12-07T16:04:02.807Z", "content": "Is he showing it? I see LB scores for the ensemble but no CV. He's updating the image quite a bit though, so maybe we're looking at something different.
", "rawMarkdown": "Is he showing it? I see LB scores for the ensemble but no CV. He's updating the image quite a bit though, so maybe we're looking at something different." }, { "id": 2058123, "postDate": "2022-12-07T16:20:58.453Z", "content": "I think the binarised - max pfbeta column is his local validation score
Theo, I think @threshold might be the cv? If you look at some of the other images, he has some largish numbers reported for @threshold (eg, .632)
", "rawMarkdown": "Theo, I think @threshold might be the cv? If you look at some of the other images, he has some largish numbers reported for @threshold (eg, .632)" }, { "id": 2058319, "postDate": "2022-12-07T19:48:44.137Z", "content": "\"binarised - max pfbeta\" is the local CV.
\n\"@threshold \" is the threshold value where max occur.
there are different ways to aggregate and threshold:
\ne.g. threshold then aggregate, or aggregate then threshold
these methods are not optimal. the best method is to learn to predict from multiple views (e.g. cross-view attention, lstm, etc)
", "rawMarkdown": "\"binarised - max pfbeta\" is the local CV.\n\"@threshold \" is the threshold value where max occur.\n\nthere are different ways to aggregate and threshold:\ne.g. threshold then aggregate, or aggregate then threshold\n\nthese methods are not optimal. the best method is to learn to predict from multiple views (e.g. cross-view attention, lstm, etc)", "votes": 1 }, { "id": 2059050, "postDate": "2022-12-08T12:34:10.270Z", "content": "Thank you for your analysis and updates. Could you please clarify what you mean about the order of aggregate and threshold? If that's a max followed by thresh how come is different the other way around? Maybe I'm mistaken.
", "rawMarkdown": "Thank you for your analysis and updates. Could you please clarify what you mean about the order of aggregate and threshold? If that's a max followed by thresh how come is different the other way around? Maybe I'm mistaken." }, { "id": 2065448, "postDate": "2022-12-14T17:04:39.113Z", "content": "\n\"@hengck23 Your CV/LB gap seems huge. I'm having ~0.01 difference.
\nAny idea why ?\"
i confirm this is due to different sampling, loss, post-processing (recalibration) strategy
", "rawMarkdown": "@theoviel \n\n\"@hengck23 Your CV/LB gap seems huge. I'm having ~0.01 difference.\nAny idea why ?\"\n\ni confirm this is due to different sampling, loss, post-processing (recalibration) strategy", "votes": 1 } ] }, { "id": 2056581, "postDate": "2022-12-06T09:10:50.410Z", "content": "Thank You for sharing @hengck23 Always Appreciated!
", "rawMarkdown": "Thank You for sharing @hengck23 Always Appreciated!", "votes": 1 }, { "id": 2100897, "postDate": "2023-01-15T13:35:39.203Z", "content": "i made a bug and notebbok gets into infinite loop at submission.
\nthis eats up my GPU hours at the beginning of the week and was unable to verify the trick below.
the reshaping of power 0.5 trick consistently improves my CV by 0.01
\nKagglers may want to verify this for LB.
before\n valid_df.loc[:, 'cancer_p'] = cancer_p\n valid_df.loc[:, 'cancer_t'] = cancer_t\n gb = valid_df[['site_id', 'patient_id','laterality', 'cancer_t', 'cancer_p']].groupby(['patient_id', 'laterality']).mean()\n f1score, precision, recall, threshold = get_f1score(gb.cancer_p, gb.cancer_t)\n\n\nafter\n valid_df.loc[:, 'cancer_p'] = cancer_p**0.5 \n valid_df.loc[:, 'cancer_t'] = cancer_t\n gb = valid_df[['site_id', 'patient_id','laterality', 'cancer_t', 'cancer_p']].groupby(['patient_id', 'laterality']).mean()\n f1score, precision, recall, threshold = get_f1score(gb.cancer_p, gb.cancer_t)\nkeep only the most important patch!
\nWACV2023 paper
\nhttps://github.com/yueliukth/PatchDropout
![\"https://i.ibb.co/8YpxJGS/Selection-518.png\"](\"https://i.ibb.co/8YpxJGS/Selection-518.png\")
[1] PatchDropout: Economizing Vision Transformers Using Patch Dropout
\nhttps://openaccess.thecvf.com/content/WACV2023/papers/Liu_PatchDropout_Economizing_Vision_Transformers_Using_Patch_Dropout_WACV_2023_paper.pdf
seems that site1 and site 2 are biased. do we need to separate them?
\nsee bottom of notebook
\nhttps://www.kaggle.com/code/hengck23/mvccl-model-for-admani-dataset/notebook
\"there is starter code of baseline pure efficient-b0 single view here: https://www.kaggle.com/datasets/hengck23/pure-effb0-single-view-starter-for-mvccl-admani\"
", "rawMarkdown": "seems that site1 and site 2 are biased. do we need to separate them?\n\nsee bottom of notebook\nhttps://www.kaggle.com/code/hengck23/mvccl-model-for-admani-dataset/notebook\n\n\"there is starter code of baseline pure efficient-b0 single view here: https://www.kaggle.com/datasets/hengck23/pure-effb0-single-view-starter-for-mvccl-admani\"", "votes": 2, "replies": [ { "id": 2086357, "postDate": "2023-01-04T18:12:36.467Z", "content": "Hey, I noticed in this notebook the GCM has only 1 mapping function when the paper describes two for each view. I think I am missing something but shouldn't these be two different functions rather than one projection function? It wouldn't make sense to me that g_m can be mapped back to itself with the same function
\nEDIT: I know the paper is vague about it but I'll experiment with both and see.
", "rawMarkdown": "Hey, I noticed in this notebook the GCM has only 1 mapping function when the paper describes two for each view. I think I am missing something but shouldn't these be two different functions rather than one projection function? It wouldn't make sense to me that g_m can be mapped back to itself with the same function\n\nEDIT: I know the paper is vague about it but I'll experiment with both and see.", "votes": 1, "replies": [ { "id": 2086521, "postDate": "2023-01-04T20:17:42.310Z", "content": "you can also try
\na = W*b\nb =inv(W)*a\ncross-view attention and transformer
\n\n[1] https://conferences.miccai.org/2022/papers/523-Paper1238.html
\n[2] https://arxiv.org/pdf/2103.11390.pdf
![\"Selection-383\"](\"https://i.ibb.co/Rb7WdK6/Selection-383.png\")
\n![\"Selection-382\"](\"https://i.ibb.co/M5Zt6M7/Selection-382.png\")
\n![\"Selection-381\"](\"https://i.ibb.co/wsHYM2q/Selection-381.png\")
it seems that you can simply flip (standardize to L) and stack the images for alignment
\nall images are aligned to the common detected nipple point
![\"https://i.ibb.co/Q7BhwpN/Selection-300.png\"](\"https://i.ibb.co/Q7BhwpN/Selection-300.png\")
related: [1] MommiNet-v2: Mammographic multi-view mass identification networks
", "rawMarkdown": "it seems that you can simply flip (standardize to L) and stack the images for alignment\nall images are aligned to the common detected nipple point\n\n\n\nrelated: [1] MommiNet-v2: Mammographic multi-view mass identification networks\n", "votes": 2, "replies": [ { "id": 2075923, "postDate": "2022-12-26T02:19:42.067Z", "content": "In some of the photos, I noticed that the \"nipple\" point is actually also marked on the image with a circle. I am guessing that is used for the alignment.
", "rawMarkdown": "In some of the photos, I noticed that the \"nipple\" point is actually also marked on the image with a circle. I am guessing that is used for the alignment. " }, { "id": 2076621, "postDate": "2022-12-26T16:31:07.023Z", "content": "check this work as well for using stacking as alignment
\nhttps://arxiv.org/pdf/1907.13057.pdf
", "rawMarkdown": "check this work as well for using stacking as alignment\n\nhttps://arxiv.org/pdf/1907.13057.pdf", "votes": 1 } ] }, { "id": 2073809, "postDate": "2022-12-23T12:55:20.397Z", "content": "![\"https://i.ibb.co/PZffjtW/Selection-270.png\"](\"https://i.ibb.co/PZffjtW/Selection-270.png\")
\ninteresting results:
\nprediction on kaggle dataset using models trained on different external data
Really great topic you created! 👍
\nI have learned from this topic a lot.
Is it Grad-CAM or Activation map?
", "rawMarkdown": "Really great topic you created! 👍\nI have learned from this topic a lot. \n\nIs it Grad-CAM or Activation map?", "votes": 2, "replies": [ { "id": 2073822, "postDate": "2022-12-23T12:59:17.480Z", "content": "segmentation (1/32 scaled heatmap)results
", "rawMarkdown": "segmentation (1/32 scaled heatmap)results", "votes": 1 } ] }, { "id": 2082383, "postDate": "2023-01-01T14:12:24.840Z", "content": "Did you train on external data on your own or use pretrained models?
", "rawMarkdown": "Did you train on external data on your own or use pretrained models?" } ] }, { "id": 2072341, "postDate": "2022-12-22T01:35:13.687Z", "content": "some external data results
\n![\"Selection-254\"](\"https://i.ibb.co/yP2K61c/Selection-254.png\")
\n![\"Selection-253\"](\"https://i.ibb.co/pzbv5yX/Selection-253.png\")
\n![\"Selection-252\"](\"https://i.ibb.co/jw7SJn6/Selection-252.png\")
\n\n",
"votes": 2,
"replies": [
{
"id": 2072393,
"postDate": "2022-12-22T04:35:48.613Z",
"content": "here is way external data could be the key:
\n![\"Selection-256\"](\"https://i.ibb.co/d4fgtnp/Selection-256.png\")
another observation:
\nwhile it is difficult to learn some models (e.g. vision transformer, resnet (without se)) on image label on kaggle dataset, it is easy to use the same model to learn heatmap prediction with heatmap ground truth generated by hand or external model
\n\nanother observation:\nwhile it is difficult to learn some models (e.g. vision transformer, resnet (without se)) on image label on kaggle dataset, it is easy to use the same model to learn heatmap prediction with heatmap ground truth generated by hand or external model",
"replies": [
{
"id": 2072447,
"postDate": "2022-12-22T05:41:03.653Z",
"content": "I am a bit new to DS, by heatmap do you mean like a segmentation model's raw output (not binarized)?
", "rawMarkdown": "I am a bit new to DS, by heatmap do you mean like a segmentation model's raw output (not binarized)?" }, { "id": 2072462, "postDate": "2022-12-22T05:58:41.170Z", "content": "https://www.lri.fr/~gcharpia/deeppractice/2020/tpx_for_mva_dl_course_2020.pdf
\n![\"https://i.ibb.co/bvvzbHg/Selection-221.png\"](\"https://i.ibb.co/bvvzbHg/Selection-221.png\")
the 11x16 output in the figure is the heatmap.
\non a side note:
\nHeatmap predictor can be trained using image crop (you need not use whole image).
\nHeatmap predictor can train on very high resolution (e.g. crops of 2048x2048).
But if you enough memory, you can use whole image.
\nDifference between segmentation and heatmap predictor is the output size. since we only need patch predictor and not pixel predictor, there is no decoder.
\nbut you can use a segmentation instead of heat predictor if you want
", "rawMarkdown": "https://www.lri.fr/~gcharpia/deeppractice/2020/tpx_for_mva_dl_course_2020.pdf\n\n\nthe 11x16 output in the figure is the heatmap.\n\non a side note:\nHeatmap predictor can be trained using image crop (you need not use whole image). \nHeatmap predictor can train on very high resolution (e.g. crops of 2048x2048).\n\nBut if you enough memory, you can use whole image.\n\nDifference between segmentation and heatmap predictor is the output size. since we only need patch predictor and not pixel predictor, there is no decoder.\n\nbut you can use a segmentation instead of heat predictor if you want", "votes": 3 }, { "id": 2072506, "postDate": "2022-12-22T06:52:30.677Z", "content": "One thing you need to keep an eye on if you're trying to win the comp is processing time. It's a massive impediment in this comp due to the large # of images you have to infer on and the limited time you have.
\nit's one of the reasons I'm looking at yolo so much. It also has some really great stuff for training like automatic augmentation. the wandb integration is pretty sweet as well.
\nIt's also a good reason to start looking into things like cython .. I think the ability to do very optimal perf wise processing could be helpful here.
\nI'd love to know if the time/perf constraint is a feature and not a bug of this comp. And if it is a feature, what's the full reasoning behind it.
", "rawMarkdown": "One thing you need to keep an eye on if you're trying to win the comp is processing time. It's a massive impediment in this comp due to the large # of images you have to infer on and the limited time you have.\n\nit's one of the reasons I'm looking at yolo so much. It also has some really great stuff for training like automatic augmentation. the wandb integration is pretty sweet as well.\n\nIt's also a good reason to start looking into things like cython .. I think the ability to do very optimal perf wise processing could be helpful here.\n\nI'd love to know if the time/perf constraint is a feature and not a bug of this comp. And if it is a feature, what's the full reasoning behind it." } ] } ] }, { "id": 2150972, "postDate": "2023-02-19T18:00:04.150Z", "content": "For me on CV 0.32 single fold with optimal th@0.5. I was expecting a much better result on LB as I've seen alot of kagglers wrote, but got only 0.34. Anyone has a clue of what is happening ?
", "rawMarkdown": "For me on CV 0.32 single fold with optimal th@0.5. I was expecting a much better result on LB as I've seen alot of kagglers wrote, but got only 0.34. Anyone has a clue of what is happening ?" }, { "id": 2146941, "postDate": "2023-02-16T09:00:24.653Z", "content": "there is something very strange about site1 and site2 data
\ni wonder did anyone try to train a classifier to discrminate site1 and site2. if the classiifer can differentiate them, then they are different.
\nnote that site2 has slightly more images. So it is possible that the model will take care of the majority site and sacrify the smaller one
", "rawMarkdown": "there is something very strange about site1 and site2 data\n- strong model seems to improve site2 (e.g. over 0.62) at the expense of dropping site1 (e.g. just over 0.30)\n- mid strong model has bout 0.58 for site2 and 0.46 for site1\n\ni wonder did anyone try to train a classifier to discrminate site1 and site2. if the classiifer can differentiate them, then they are different.\n\nnote that site2 has slightly more images. So it is possible that the model will take care of the majority site and sacrify the smaller one " }, { "id": 2129867, "postDate": "2023-02-05T00:24:23.577Z", "content": "Thank you for your contribution.
\nNoob question: (EfficientNetB2 and EfficientNetB4) are EfficientNet V1 or EfficientNetV2 ?
v1 👍
\n(to little characters to post)
", "rawMarkdown": "v1 👍\n\n(to little characters to post)", "votes": 1 } ] }, { "id": 2129810, "postDate": "2023-02-04T22:17:54.337Z", "content": "Thank you for your contribution 🙌
\n1, Could you explain the different between 2 metrics: \"pfbeta\" and \"max pfbeta\"?
\n2, In the \"aggregate by max() per patient-laterality\" part:
\nLet's take an example: In my valid set, I have 1000 patients, each patients has 4 images, 2 images for each \"patient-laterality\".
\n****So how pfbeta (logged to your table result) has calculated? **
\n**I guess: 1000 patients * 4 = 4000 images => 4000 proba predicted => max() for each patient-laterality -> 2000 proba for 2000 (patient-laterality)s => 2000 pfbeta s for each (patient-laterality) -> then take mean of them
Thanks in advance
", "rawMarkdown": "Thank you for your contribution 🙌\n1, Could you explain the different between 2 metrics: \"pfbeta\" and \"max pfbeta\"? \n2, In the \"aggregate by max() per patient-laterality\" part:\nLet's take an example: In my valid set, I have 1000 patients, each patients has 4 images, 2 images for each \"patient-laterality\". \n****So how pfbeta (logged to your table result) has calculated? **\n**I guess: 1000 patients * 4 = 4000 images => 4000 proba predicted => max() for each patient-laterality -> 2000 proba for 2000 (patient-laterality)s => 2000 pfbeta s for each (patient-laterality) -> then take mean of them\n\nThanks in advance ", "replies": [ { "id": 2130121, "postDate": "2023-02-05T06:46:24.800Z", "content": "gb = site_df[['patient_id', 'laterality', 'cancer_t', 'cancer_p']].groupby(['patient_id', 'laterality']).mean()\ngb = site_df[['patient_id', 'laterality', 'cancer_t', 'cancer_p']].groupby(['patient_id', 'laterality']).max()\nwhere cancer_t is ground true, cancer_p is probalility of cancer. I am refering to script provided by @hengck23
\nthen metric (probf1score) is computed on gb dataframe
", "rawMarkdown": "1. max, avg aggregation type\n\n```\ngb = site_df[['patient_id', 'laterality', 'cancer_t', 'cancer_p']].groupby(['patient_id', 'laterality']).mean()\n```\n\n```\ngb = site_df[['patient_id', 'laterality', 'cancer_t', 'cancer_p']].groupby(['patient_id', 'laterality']).max()\n```\n\nwhere cancer_t is ground true, cancer_p is probalility of cancer. I am refering to script provided by @hengck23 \n\nthen metric (probf1score) is computed on gb dataframe" } ] }, { "id": 2123551, "postDate": "2023-01-31T15:21:07.247Z", "content": "came across a question relevant to this kaggle competition
\n", "rawMarkdown": "came across a question relevant to this kaggle competition\n\n\nhttps://d1.awsstatic.com/training-and-certification/docs-ml/AWS-Certified-Machine-Learning-Specialty_Sample-Questions.pdf", "replies": [ { "id": 2123557, "postDate": "2023-01-31T15:25:05.310Z", "content": "![\"Selection-817\"](\"https://i.ibb.co/C2r0Gdf/Selection-817.png\")
![\"Selection-816\"](\"https://i.ibb.co/2tZh8vJ/Selection-816.png\")
for me it is different parameters converge to about same stable loss values for for different validation data in this kaggle competition …
", "rawMarkdown": "\n\n\n\n\nfor me it is different parameters converge to about same stable loss values for for different validation data in this kaggle competition ..."
}
]
},
{
"id": 2113225,
"postDate": "2023-01-24T06:44:13.953Z",
"content": "it is noted that mean of prediction per breast is better than individual image.
\nthis means that mean prediction is a relieable self supervision signal for self supevised image learning.
in thoery if we apply this to online learning on hidden test data, maybe it can lead to better results?
\nof course, there will resources problem
i am wondering is there any CNN single fold or (single model) that can hit LB 0.58~0.60?
\nI am thinking of discarding CNN model and just use vision transformer in my solution.
CNN Single fold gives me 0.58 in LB
", "rawMarkdown": "CNN Single fold gives me 0.58 in LB", "votes": 7, "replies": [ { "id": 2112315, "postDate": "2023-01-23T14:42:36.463Z", "content": "@arunodhayan amazing! What is your CNN model?
", "rawMarkdown": "@arunodhayan amazing! What is your CNN model?", "votes": 1 } ] }, { "id": 2110584, "postDate": "2023-01-22T09:32:20.913Z", "content": "Single CNN model, 4 folds essemble, LB 0.63
", "rawMarkdown": "Single CNN model, 4 folds essemble, LB 0.63", "votes": 15, "replies": [ { "id": 2110595, "postDate": "2023-01-22T09:48:38.780Z", "content": "@kevin1742064161 @arunodhayan
\nThanks
All are good masterpieces!
", "rawMarkdown": "@kevin1742064161 @arunodhayan \nThanks\n\nAll are good masterpieces!", "votes": 1 } ] } ] }, { "id": 2107235, "postDate": "2023-01-19T16:49:32.827Z", "content": "i made a bug and fold that for one fold you can re-run with different seed and ensemble for better (and more stable?) results
", "rawMarkdown": "i made a bug and fold that for one fold you can re-run with different seed and ensemble for better (and more stable?) results" }, { "id": 2099444, "postDate": "2023-01-14T12:54:09.803Z", "content": "Great work, that's fast! What's the local score for LB 0.58, still one fold? With that speed more fold might be tested :)
", "rawMarkdown": "Great work, that's fast! What's the local score for LB 0.58, still one fold? With that speed more fold might be tested :)", "replies": [ { "id": 2099517, "postDate": "2023-01-14T13:33:21.603Z", "content": "see the comments
\n![\"https://i.ibb.co/2yvwbPL/Selection-815.png\"](\"https://i.ibb.co/2yvwbPL/Selection-815.png\")
https :// www. kaggle .com / code / hengck23/3hr-tensorrt-nextvit-example/comments\n0.58 should be round up of 0.57
", "rawMarkdown": "see the comments \n```\nhttps :// www. kaggle .com / code / hengck23/3hr-tensorrt-nextvit-example/comments\n```\n\n0.58 should be round up of 0.57", "replies": [ { "id": 2099550, "postDate": "2023-01-14T13:46:42.263Z", "content": "
\nAlright the score in the notebook and the other a comment for the LB score 👍
Hi, could you explain to me the binarised concept please?.
\nThank you in advance
", "rawMarkdown": "Hi, could you explain to me the binarised concept please?.\n\nThank you in advance" }, { "id": 2082104, "postDate": "2023-01-01T05:33:56.760Z", "content": "how to use age information?
\ninstead of consider absolute value, consider something like >30 years and <30 years, etc.
\nrefer to radioloogist workflow from the web
![\"https://i.ibb.co/6FgwVM9/Selection-465.png\"](\"https://i.ibb.co/6FgwVM9/Selection-465.png\")
cancer distribution by age and site.
\ni wonder what causes the peak. I think it is probably the screening programs (e.g. people of certain age are encouraged for screening by the government ) or the workflow (e.g. i always see \"if patient is above age xx, then do this, else do that\")
", "rawMarkdown": "\n\ncancer distribution by age and site.\n\ni wonder what causes the peak. I think it is probably the screening programs (e.g. people of certain age are encouraged for screening by the government ) or the workflow (e.g. i always see \"if patient is above age xx, then do this, else do that\")", "votes": 2 } ] }, { "id": 2082096, "postDate": "2023-01-01T05:21:42.590Z", "content": "where are the cancer:
\nhttps://www.youtube.com/watch?v=bH11DhzJRmA&t=2773s
\nBreast Micro-calcifications : All You Need to Know | Mammography | Dr. Terry Minuk
Breast Imaging: Calcifications [Basic Radiology]
\nhttps://www.youtube.com/watch?v=7d3nY1ZMr9Q
you basically can learn the relative position encoding for cross attention
\n![\"https://i.ibb.co/tsQXDCH/Selection-408.png\"](\"https://i.ibb.co/tsQXDCH/Selection-408.png\")
x1 and x2 = L and R or MLO and CC?
", "rawMarkdown": "x1 and x2 = L and R or MLO and CC?", "replies": [ { "id": 2081690, "postDate": "2022-12-31T14:11:20.613Z", "content": "L and R of same view
", "rawMarkdown": "L and R of same view" } ] } ] }, { "id": 2081227, "postDate": "2022-12-31T00:59:01.193Z", "content": "\"The difficulty of a problem can be indicated by how quickly a human can complete the task.\"
\nonly 0.5 sec for screening mammogram !!!
\n[1] Radiologists can detect the ‘gist’ of breast cancer before any overt signs of cancer appear
\nhttps://www.nature.com/articles/s41598-018-26100-5
\". Our findings suggest that readers can distinguish patients who were diagnosed with cancer, from individuals without breast cancer (normal category), at above-chance levels based on a half-second glimpse of the mammogram even before any lesion becomes visible on the mammogram\"
\nhuman expert here can be interpreted as having experience (i.e. see many data before) or having natural ability to spot some image characteristics (i.e. some good network architecture that model data prior distribution well)
", "rawMarkdown": "\"The difficulty of a problem can be indicated by how quickly a human can complete the task.\"\n\nonly 0.5 sec for screening mammogram !!!\n\n[1] Radiologists can detect the ‘gist’ of breast cancer before any overt signs of cancer appear\nhttps://www.nature.com/articles/s41598-018-26100-5\n\n\". Our findings suggest that readers can distinguish patients who were diagnosed with cancer, from individuals without breast cancer (normal category), at above-chance levels based on a half-second glimpse of the mammogram even before any lesion becomes visible on the mammogram\"\n\nhuman expert here can be interpreted as having experience (i.e. see many data before) or having natural ability to spot some image characteristics (i.e. some good network architecture that model data prior distribution well) ", "replies": [ { "id": 2081229, "postDate": "2022-12-31T01:00:53.277Z", "content": "on a very far side note, how i use gpt-chat:
\n![\"https://i.ibb.co/z7ySzws/Selection-388.png\"](\"https://i.ibb.co/z7ySzws/Selection-388.png\")
Interesting metrics - https://pubmed.ncbi.nlm.nih.gov/35434976/
\n\n\nResults: The AUROC of the AI alone, BSR (average across five readers), and GR (average across five readers) groups was 0.915 (95% confidence interval, 0.876-0.954), 0.813 (0.756-0.870), and 0.684 (0.616-0.752), respectively. With AI assistance, the AUROC significantly increased to 0.884 (0.840-0.928) and 0.833 (0.779-0.887) in the BSR and GR groups, respectively (p = 0.007 and p < 0.001, respectively). Sensitivity was improved by AI assistance in both groups (74.6% vs. 88.6% in BSR, p < 0.001; 52.1% vs. 79.4% in GR, p < 0.001), but the specificity did not differ significantly (66.6% vs. 66.4% in BSR, p = 0.238; 70.8% vs. 70.0% in GR, p = 0.689). The average reading time pooled across readers was significantly decreased by AI assistance for BSRs (82.73 vs. 73.04 seconds, p < 0.001) but increased in GRs (35.44 vs. 42.52 seconds, p < 0.001).
\n
ie:
\nAI alone - 0.915 AUROC
\nAI + Breast Specialist Radiologist (BSR) - 0.884 AUROC
\nAI + General Radiologist (GR) - 0.833 AUROC
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2F13608b3ddd45bde3f37d7eb3ef3a0533%2Fbsrmetrics.png?generation=1672469521043553&alt=media\")
there is something interesting about the dicom tag …
\n![\"https://i.ibb.co/ZLQj0Ck/Selection-231.png\"](\"https://i.ibb.co/ZLQj0Ck/Selection-231.png\")
\n![\"https://i.ibb.co/ZdDWmnQ/Selection-232.png\"](\"https://i.ibb.co/ZdDWmnQ/Selection-232.png\")
plt.plot(dicom_tag_df.ContentTime.index[dicom_tag_df.cancer==1], dicom_tag_df.ContentTime[dicom_tag_df.cancer==1],'o' )\nplt.plot(dicom_tag_df.ContentTime.index[dicom_tag_df.cancer==0], dicom_tag_df.ContentTime[dicom_tag_df.cancer==0],'.' )\nI am able to recreate the first plot, what does the second plot represent (the one with only 225 values on x axis)?
", "rawMarkdown": "I am able to recreate the first plot, what does the second plot represent (the one with only 225 values on x axis)?", "replies": [ { "id": 2070401, "postDate": "2022-12-20T01:06:22.437Z", "content": "machine id
", "rawMarkdown": "machine id", "votes": 1 } ] }, { "id": 2081191, "postDate": "2022-12-30T22:58:02.503Z", "content": "What is the meaning of the x-axis in these plots? How did you assign the index in your dataframe?
\nThe DICOM docs indicate that ContentTime is actually a string in the format HHMMSS, which may explain the pattern in the first plot.
", "rawMarkdown": "What is the meaning of the x-axis in these plots? How did you assign the index in your dataframe?\n\nThe [DICOM docs](https://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_6.2.html#:~:text=8%20bytes%20fixed-,TM,-Time) indicate that ContentTime is actually a string in the format HHMMSS, which may explain the pattern in the first plot." } ] }, { "id": 2069578, "postDate": "2022-12-19T06:20:06.457Z", "content": "Hi @hengck23 What is different of per image with per patient-laterality?
", "rawMarkdown": "Hi @hengck23 What is different of per image with per patient-laterality?" }, { "id": 2063377, "postDate": "2022-12-12T20:26:56.597Z", "content": "@hengck23 I looked into your code - could you explain line
\ncancer = torch.nan_to_num(cancer)
in inference prart? Any issues with nan's and efficientnet?
", "rawMarkdown": "@hengck23 I looked into your code - could you explain line \n\n```cancer = torch.nan_to_num(cancer)```\n\nin inference prart? Any issues with nan's and efficientnet?", "replies": [ { "id": 2063386, "postDate": "2022-12-12T20:56:33.007Z", "content": "i haven't checked. but i suspect sigmoid in FP16 (turn last logit to probability).
\na better solution is
cancer = torch.sigmoid(cancer.float())\nI noticed that if you use amp autocast in inference prart Efficientnet is able to generate NaN in logits. Just leaving only autocast in training loop solve this issue.
", "rawMarkdown": "I noticed that if you use amp autocast in inference prart Efficientnet is able to generate NaN in logits. Just leaving only autocast in training loop solve this issue.", "replies": [ { "id": 2066692, "postDate": "2022-12-16T00:37:12.427Z", "content": "i need to save memory for inference too if i am using bigger size input like 2048
", "rawMarkdown": "i need to save memory for inference too if i am using bigger size input like 2048" } ] } ] }, { "id": 2062488, "postDate": "2022-12-12T06:42:25.330Z", "content": "did you use pretrained weights?
", "rawMarkdown": "did you use pretrained weights?" }, { "id": 2059564, "postDate": "2022-12-09T03:04:17.513Z", "content": "Thanks a lot for nice work. @hengck23
\nHow many epochs have you set during training?
see code at\nhttps://www.kaggle.com/datasets/hengck23/for-tpu-efficientb4-debug\ntype rubbish to prevent error :
\n\"This edit is too similar to a previous post. We prevent redundant comments to reduce spam.\"
\njnckjsakdywuienckdjfadkaslfjsdiopmcdsl,fjsklfjposdm,cplsdfm,sdmcvl;sdlifposekrmf,dsfmsdl;fkspkfdl;s,cs;kflsd;kf;dlsrkapkfl;sdfk;sdf
Hi ! I'd like to ask about the training data. Do you take any method to deal with the imbalance of the dataset? Like using Focal loss or oversampling. Or do you just put the origin data into the training? I am sincerely looking forward to your answer.😀👋
", "rawMarkdown": "Hi ! I'd like to ask about the training data. Do you take any method to deal with the imbalance of the dataset? Like using Focal loss or oversampling. Or do you just put the origin data into the training? I am sincerely looking forward to your answer.😀👋", "replies": [ { "id": 2058778, "postDate": "2022-12-08T07:55:18.720Z", "content": "see code at\nhttps://www.kaggle.com/datasets/hengck23/for-tpu-efficientb4-debug\ntype rubbish to prevent error :
\n\"This edit is too similar to a previous post. We prevent redundant comments to reduce spam.\"
\ndsfdskluremncjkzsysajbndsakljfogkdf;lgm;dfbm,;sdldjflausdsaljflkjglkngldjkglseruioejlsknflmsncvmcnhskluflsdjkfkldsjfsdlkfjlsdkursljf
Interesting topic as usual! Thank you for sharing your experience.
\na. Looking into your tables and see numbers - what is your batch size (res -> 1024)? Have you tried to change bs to see results?
\nb. Do you use ROI extracted images on resized only?
all are using batch size = 32 (including input 1024).
\ni use full image (without ROI extraction)
Have you tried to change bs to see results?
\nnot yet
Thanks for the post @hengck23. What validation are you using? StratifiedGroupKFold?
see\nhttps://www.kaggle.com/datasets/hengck23/for-tpu-efficientb4-debug\ntype rubbish to prevent error :
\n\"This edit is too similar to a previous post. We prevent redundant comments to reduce spam.\"
\nskjdslakjireutpertmgt,mgsd.,kjsapoiewopqkr;lfds./gmd/gl';vblg][ohkajkashdasdjksahdksahdksajdhsa
@hengck23 Are the 1024 models trained first at lower resolutions or do you go straight to 1024?
", "rawMarkdown": "@hengck23 Are the 1024 models trained first at lower resolutions or do you go straight to 1024?", "replies": [ { "id": 2055507, "postDate": "2022-12-05T06:00:58.243Z", "content": "i trained straight from 1024 images
", "rawMarkdown": "i trained straight from 1024 images\n", "votes": 1, "replies": [ { "id": 2069847, "postDate": "2022-12-19T11:35:13.853Z", "content": "Thanks a lot for nice work. How you choose the threshold for maximizing pfbeta when you ensemble models? Do you just take an average of thresholds of the models?
", "rawMarkdown": "Thanks a lot for nice work. How you choose the threshold for maximizing pfbeta when you ensemble models? Do you just take an average of thresholds of the models?" } ] } ] }, { "id": 2172483, "postDate": "2023-03-07T14:52:38.930Z", "content": "Thank you for the post.
", "rawMarkdown": "Thank you for the post." }, { "id": 2058718, "postDate": "2022-12-08T06:31:26.887Z", "content": "Thank you very much!
", "rawMarkdown": "Thank you very much!" } ], "comments": [ { "id": 2109056, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-21T05:12:47.273000", "content": "example of using rejection-based approach.
\nyou can improve accuracy with less computation.
LB : 0.58\n test_df0 = test_df.copy()\n probability0 = do_predict(net0, test_df0)\n test_df0.loc[:,'cancer_p0']=probability0\n\n\n #---\n t = np.percentile(probability0,65) #reject 65%\n test_df1 = test_df0[test_df0.cancer_p0>t].reset_index(drop=True)\n probability1 = do_predict(net1, test_df1)\n test_df1.loc[:,'cancer_p1']=probability1\n\n # ---\n\n\nLB : 0.59 (cv increase +0.02)\n df = test_df0.merge(test_df1[['image_id','cancer_p1']],on='image_id',how='left')\n df.loc[df.cancer_p1.isna(),'cancer_p1'] = df.cancer_p0\n probability = (df.cancer_p0.values + df.cancer_p1.values)/2\n\nnet0 and net1 are trained using same fold but different seed\n
tip:
\nnet1 can also be a network that uses higher resolution, etc
\n[1]Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
\nuse larger learning rate to learn a coarse model.
\nthen finetune using different seed (+ different hyperparameters, rate, weighing, etc)
ensemble all using the method in [1], i.e. just average the weights of the best k-models in greedy way.
", "votes": 13, "replies": [] }, { "id": 2096760, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-12T08:43:06.620000", "content": "transformer is the king!
\n
\n
Havent heard about NextVIT thanks for sharing, I do not even find it in timm :)
\nBTW I am surprised of your f1 score of 0.5 given that AUC is 0.9.
", "votes": 0, "replies": [ { "id": 2096816, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-12T09:20:57.397000", "content": "ROC-AUC is actually also not stable. you can get good F1 and AUC at CV by over sampling positive class and very strong model (e.g. imagenet top-1 0.86). But then i find that it is actually overfitting.
\nfor AUC of 0.90, F1 can varies a lot
", "votes": 1, "replies": [] }, { "id": 2096832, "author_name": "Psi", "author_url": "", "post_date": "2023-01-12T09:31:40.157000", "content": "I see slight variation but not that big.
\nWhat is ve class?
", "votes": 0, "replies": [] }, { "id": 2096835, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-12T09:32:32.033000", "content": "it is positive class
", "votes": 0, "replies": [] }, { "id": 2097902, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-13T03:44:37.150000", "content": "\n\n\nmanged to get tensorRT working!
\nperformance on local PC:
\nnum of images process 10935\n\npytorch fp16\n-----\ntime = 20 min 26 sec\nauc = 0.8925856621368753\nf1score.max() = 0.4968558969524153\n@threshold 0.3061224489795918\n\ntensorRT fp16\n-----\ntime = 9 min 05 sec\nauc = 0.8926157748322697\nf1score.max() = 0.49423766814841186\n@threshold 0.3061224489795918\nperformance on kaggle notebook to come later …
", "votes": 1, "replies": [] }, { "id": 2101298, "author_name": "nicehzj", "author_url": "", "post_date": "2023-01-15T19:43:05.330000", "content": "If oversampling positive class will cause over fitting, may I ask how did you deal with the unbalance of neg/pos? BCE loss with weight? I read your discussion about multi-view. Your neg/pos is about 7:1. Thanks!
", "votes": 0, "replies": [] }, { "id": 2108317, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2023-01-20T12:28:38.563000", "content": "\n\nBut then i find that it is actually overfitting
\n
How did you find this out?
", "votes": 0, "replies": [] }, { "id": 2108339, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-20T12:57:08.803000", "content": "\"But then i find that it is actually overfitting\"
\nHow did you find this out?
The curve below and the LB results
\nhttps://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/378521
let's do a thought experiment
\nyou should at least note the following:
\nOh, I see, I get it, thanks!
", "votes": 0, "replies": [] } ] }, { "id": 2096785, "author_name": "Kirderf", "author_url": "", "post_date": "2023-01-12T09:05:01.823000", "content": "CV of .497 👌! Is that cross-validation over how many folds? Thanks for the NextVIT architecture advice 👍
", "votes": 0, "replies": [ { "id": 2096815, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-12T09:17:43.723000", "content": "only one fold for CV and LB
", "votes": 0, "replies": [] }, { "id": 2096837, "author_name": "Psi", "author_url": "", "post_date": "2023-01-12T09:34:03.063000", "content": "okay that makes more sense then
", "votes": 0, "replies": [] }, { "id": 2096866, "author_name": "Kirderf", "author_url": "", "post_date": "2023-01-12T09:56:08.937000", "content": "Alright thanks for info.
\nFor my training and testing the diff. in scores between folds and the sensitive threshold optimizing in numbers and math, give fear for a large shakeup.
\nHope you others have a more stable solutions and scores between the folds than I have ;)
one fold nextvit-b at 1536x960 already takes 9hr to run for the submission.
\nhence i am unable to run for an ensemble of multiple folds.
now i am solving this problem. (e.g. an early rejector, faster voi_lut_apply(), tensorRT or change a faster vision transformer).
\nI also try other transformer like CoAT, PVTv2 and the results are generally good in CV (i haven't make submission for these yet)
", "votes": 0, "replies": [] }, { "id": 2098405, "author_name": "Eleftherios Fanioudakis", "author_url": "", "post_date": "2023-01-13T14:18:21.227000", "content": "thanks for the info, how does base compare to the small nextvit?
", "votes": 0, "replies": [] } ] }, { "id": 2097098, "author_name": "YYama", "author_url": "", "post_date": "2023-01-12T13:20:04.460000", "content": "Thanks again for sharing!
\nI am immediately trying out nextvit and I found that it is very difficult to train nextvit.
\nWith the same parameters as before, I had early gradient explosions, and when I reduced the learning rate, it converged but performance dropped.
Do you have any good tips for training?
", "votes": 0, "replies": [ { "id": 2097248, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-12T14:37:58.990000", "content": "the parameters for training transformer (nextvit and others) are different from cnn
\nin general, learning rate and loss (oversampling or margin or +ve weighing) determines results
i use the following:
\nimage_height = 1536\nimage_width = 960\n\npretain = '/home/titanx/hengck/share1/data/pretrain_model/nextvit_base_in1k_384.pth'\nnextvit_base()\nF.binary_cross_entropy_with_logits(cancer,batch['cancer'])\n\n\n batch_size = 8 \n ratio = 8 (1 pos sample in every 8 train sample)\n\n\n def scheduler(epoch): \n\n num_epoch = 6\n start_lr = 5e-5\n min_lr = 1e-5\n lr = (num_epoch-epoch)/num_epoch * (start_lr-min_lr) + min_lr\n lr = max(min_lr,lr)\n return lr\n\n\n optimizer = Lookahead(RAdam(filter(lambda p: p.requires_grad, net.parameters()),lr=-1), alpha=0.5, k=5)\ntrain log (validation is without TTA)
\n** start training here! **\n batch_size = 8, ratio = 8\n experiment = ['nextvit-b-1536', 'run_train_fold0.py']\n |-------------------------- VALID-------------------|---------------- TRAIN/BATCH --------\nrate iter epoch | loss auc, f1 th sen spec f1 mean | loss | time \n---------------------------------------------------------------------------------------------------------------------\n0.00e+0 00000000* 0.00 | 0.674 0.491 0.0437 0.490 0.539 0.481 0.043 | 0.000 0.000 0.000 | 0 hr 05 min\n4.33e-5 00006122* 1.00 | 0.194 0.759 0.2096 0.735 0.199 0.984 0.240 | 0.342 0.000 0.000 | 1 hr 27 min\n3.67e-5 00012244* 2.00 | 0.118 0.798 0.2516 0.531 0.249 0.984 0.349 | 0.271 0.000 0.000 | 2 hr 50 min\n3.00e-5 00018366* 3.00 | 0.091 0.815 0.3387 0.592 0.245 0.996 0.433 | 0.230 0.000 0.000 | 4 hr 13 min\n2.33e-5 00024488* 4.00 | 0.099 0.837 0.3718 0.796 0.278 0.995 0.465 | 0.179 0.000 0.000 | 5 hr 35 min\n1.67e-5 00030610* 5.00 | 0.102 0.834 0.3665 0.449 0.369 0.986 0.460 | 0.185 0.000 0.000 | 6 hr 58 min\n1.00e-5 00036732* 6.00 | 0.089 0.817 0.3350 0.122 0.320 0.987 0.452 | 0.166 0.000 0.000 | 8 hr 21 min\n1.00e-5 00042854* 7.00 | 0.094 0.819 0.3711 0.653 0.282 0.995 0.508 | 0.153 0.000 0.000 | 9 hr 44 min\n1.00e-5 00048976* 8.00 | 0.091 0.802 0.3829 0.163 0.307 0.993 0.473 | 0.154 0.000 0.000 | 11 hr 07 min\nfinally apply TTA and swa for
\ndef make_swa():\n out_file = fold_dir + f'/checkpoint/swa.model.pth' \n iteration = [\n '00048976',\n '00042854',\n '00036732',\n '00036732',\n '00030610',\n '00024488',\n ]\n state_dict = None\n for i in iteration:\n f = fold_dir + f'/checkpoint/{i}.model.pth'\n print(f)\n f = torch.load(f, map_location=lambda storage, loc: storage)\n if state_dict is None:\n state_dict = f['state_dict']\n else:\n key = list(f['state_dict'].keys())\n for k in key:\n state_dict[k] = state_dict[k] + f['state_dict'][k]\n\n for k in key:\n state_dict[k] = state_dict[k] / len(iteration)\n print('')\n\n print(out_file)\n torch.save({'state_dict': state_dict}, out_file)\ncpu augmentation is too slow. so i use kornia gpu augmentation
\nclass DataAugmentation1(nn.Module):\n def __init__(self,):\n super().__init__()\n self.flip = nn.Sequential(\n RandomHorizontalFlip(p=0.5),\n RandomVerticalFlip(p=0.5),\n )\n\n p=0.8\n self.transform_geometry = ImageSequential(\n RandomAffine(degrees=20, translate=0.1, scale=[0.8,1.2], shear=20, p=p),\n RandomThinPlateSpline(scale=0.25, p=p),\n random_apply=1, #choose 1\n )\n\n p=0.5\n self.transform_intensity = ImageSequential(\n RandomGamma(gamma=(0.5, 1.5), gain=(0.5, 1.2), p=p),\n RandomContrast(contrast=(0.8,1.2), p=p),\n RandomBrightness(brightness=(0.8,1.2), p=p),\n random_apply=1, #choose 1\n )\n\n p=0.5\n self.transform_other = ImageSequential(\n MyRoll(p=0.1), #Mosaic Augmentation using only one image, implemented by using pytorch roll , i.e. cyclic shift\n MyCutOut(num_block=5, block_size=[0.1, 0.2], fill='constant', p=0.1),\n random_apply=1, #choose 1\n )\n\n\n @torch.no_grad() # disable gradients for effiency\n def forward(self, x):\n x = self.flip(x) # BxCxHxW\n x = self.transform_geometry(x)\n x = self.transform_intensity(x)\n x = self.transform_other(x)\n return x\nThank you for being quite detailed!
\nIt's quite different from my setting, so I'm learning a lot.
can you tell what is f1 in the log above? is it non-thresholded pF1?
", "votes": 0, "replies": [] }, { "id": 2098098, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-13T08:16:50.987000", "content": "max threshold f1
", "votes": 0, "replies": [] }, { "id": 2144080, "author_name": "Neeraj Anand", "author_url": "", "post_date": "2023-02-14T18:11:39.030000", "content": "Can you please explain why lr = -1 in optimizer?
", "votes": 1, "replies": [] } ] }, { "id": 2097365, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-12T16:06:36.973000", "content": "Does SWA stands for Stochastic Weight Averaging?
\nNice score!
OK. I see …. :)
", "votes": 0, "replies": [] } ] }, { "id": 2101305, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-15T19:53:03.410000", "content": "Have you encountered any problem during training - loss NaN (I use oryginal naxtViT repo)?
", "votes": 0, "replies": [ { "id": 2101474, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-16T00:10:33.573000", "content": "maybe try use smaller learning rate. 5e-5 to 1e-5.
", "votes": 0, "replies": [] }, { "id": 2101811, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-16T07:09:42.660000", "content": "yes, I started from 5e-5 but unfortunately exploded during second epoch. I will try today to do more experiments.
", "votes": 0, "replies": [] }, { "id": 2101847, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-16T08:01:41.480000", "content": "increased pos sampling e.g. from 1 pos in batch=8 to 1 in 4.
\nuse 3e-5
I followed your suggestion (starting lr 3e-5) and no NaN appeared. Thank you so much.
\nOne more question - do you balance loss function using weight? As I can see my model is good after 1-2 epochs and then started to overfit for \"no cancer\".
Hi! I have the same question about the loss with balanced weight… Also, what's the good metric to decide a good model? AUC or f1? because in my eval, the highest AUC is around 0.975, which is the same proportion of neg/pos…
", "votes": 0, "replies": [] }, { "id": 2102225, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-16T13:31:04.297000", "content": "e.g.
\nset rate=5e-5:
\ncannot learn validation f1 is 0.05
set rate=3e-5:
\nvalidation f1 is 0.10 for some first iterations, then fall back to previous case of 0.05 in later iterations
set rate=1e-5:
\nsuccess! validation f1 improves as iterations proceeds
if it doesn't work even if you set rate=1e-6, then the model cannot be train with you current setup.
\nyou have to change oversampling, loss (intermediate aux loss, or other loss than BCE) or use a less complex version (e.g. small, tiny version, etc)
My nextvit (small) is fighting …. 😂 AUC_ROC -> 0.86 / probf1 ~0.36 (val)
\n
\n
any submission results?
\nthere could be overfitting
\nbut maybe LB results cab be good it is hard to judge.
\nyou can select a few intermediate model and submit too
may I ask what is the metric to choose best model in this competition? AUC? pf1? f1?
\nBecause sometimes I found AUC is decreasing but the pf1 is increasing.
\nThanks for your answer!
Let me do some more experiment and will submit. I will let you know.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2066638, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-15T22:58:11.623000", "content": "it turns that i have lower LB than other kagglers using efficientnet.
\ni have forgotten to set the drop path rate. Here is the fixed:
efficientnet_b2(pretrained=True, drop_rate = 0.3, drop_path_rate = 0.2)\n\nrefer to timm efficientnet source code for settings of drop_rate, drop_path_rate\nyou can use single-fold model (i.e. just one checkpoint file) to get LB >0.51. more on that later,
", "votes": 14, "replies": [] }, { "id": 2098821, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-13T21:42:00.453000", "content": "tensorRT timming is out !!!
\nas claimed in the paper, nextVIT is the fastest tensorRT vision transformer (same speed as efficientnet)
\nkaggle p100 notebook timing:\n\n--------------------------------------------------\npublic LB submission tensorRT 4 hr : LB 0.56\n\n--------------------------------------------------\nlocal cv 10939 images\n\nauc 0.8936061433377303\nf1score 0.49148140396595846\n@threshold 0.30612\n\n\ntotal (end-to-end from dashboard)\nDisk 16.4/73.1 GB\n1 hr 25 min\n\nbreakdown:\n\n1. install tensorRT, etc 5 min\n\n2. decode 10939 dicom images (keep aspect to 1539, use voi_lut-apply_32fp):\nnvjpeg2k (j2k, 5118 images) 28 min\ndicomsdl (non-j2k, 5917 images, 2 thread) 24 min\n\n3. detect breast box (resnet34 segentation) 2 min\n\n4. tensorRT fp16 nextVIT-B (1539x960)\none fold, original + hflip_TTA\n\nCPU utilisation 120%, 13/13 GB\nGPU utilisation 99% , 4.5/15 GB\n23 min 21 sec (7.80799 images per sec)\n\n\n=============================================\nreference (without tensorRT):\n\nsubmission 9hr : LB 0.56\n\n4. merged_bn fp16 (1539x960)\nCPU utilisation 108%, 13/13 GB\nGPU utilisation 100% , 8.5/15 GB\n95 min 32 sec (1.90819 images per sec)\nHow are you loading all images in 4 hours lol :)
\nThanks for sharing - transformer based architectures usually benefit the most from such compiles.
\nI am curious if anyone got Pytorch 2.0 running in kaggle kernels and have checked how close it comes to tensorrt.
", "votes": 2, "replies": [ { "id": 2098829, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-13T21:57:31.520000", "content": "voi_lut_apply_32fp() eats my time.
\nwithout it loading images is 3hr.
maybe i can improve a little by apply resize first and then the voi lut porcessing on smaller resized image.
", "votes": 0, "replies": [] }, { "id": 2098965, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-14T01:42:01.293000", "content": "\"I am curious if anyone got Pytorch 2.0 running in kaggle kernels and have checked how close it comes to tensorrt.\"
\ni would think tensorRT is better
Yes I also saw this, but reality and graphs are sometimes different. SO Im just curious if someone tried.
", "votes": 1, "replies": [] } ] }, { "id": 2098955, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-14T01:25:21.517000", "content": "tensorRT engine trt file generation code is here:
\nhttps://www.kaggle.com/code/hengck23/4hr-tensorrt-nextvit-example
i think if you edit the code for 2x T4 GPU, you will get the same LB results in 1.5hr
\ntime to try a bigger 2048 image on transformer …
2048x1280 for tensorrt NextVIT-S (smaller variant, imagenet 83.6) takes 5h.
\nhowever, results were as good?
\nlocalCV 0.4856957958 @th=0.387755102
\nLB 0.46
set dropout to 0. instead of 0
\none is float, the other is int
Many thanks! I'll try it.
", "votes": 0, "replies": [] }, { "id": 2100673, "author_name": "Guoliang", "author_url": "", "post_date": "2023-01-15T10:52:17.483000", "content": "the error still exist after setting dropout to 0. instead of 0
", "votes": 0, "replies": [] }, { "id": 2100676, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-15T10:54:21.403000", "content": "can you post the full error message
", "votes": 0, "replies": [] }, { "id": 2100684, "author_name": "Guoliang", "author_url": "", "post_date": "2023-01-15T11:00:51.820000", "content": "RuntimeError Traceback (most recent call last)
\n/tmp/ipykernel_23/1028975912.py in
\n 25 enabled_precisions={torch.half}, # Run with FP16
\n 26 workspace_size=1 << 32,\n---> 27 require_full_compilation=True,
\n 28 )
\n 29 torch.jit.save(trt_model_fp16, 'kaggle-nextvit-b-1536-gpu-aug0-01-swa.trt_fp16.ts')
/opt/conda/lib/python3.7/site-packages/torch_tensorrt/_compile.py in compile(module, ir, inputs, enabled_precisions, **kwargs)
\n 122 \"Module was provided as a torch.nn.Module, trying to script the module with torch.jit.script. In the event of a failure please preconvert your module to TorchScript\",
\n 123 )
\n--> 124 ts_mod = torch.jit.script(module)
\n 125 return torch_tensorrt.ts.compile(
\n 126 ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in script(obj, optimize, _frames_up, _rcb, example_inputs)
\n 1285 obj = call_prepare_scriptable_func(obj)
\n 1286 return torch.jit._recursive.create_script_module(
\n-> 1287 obj, torch.jit._recursive.infer_methods_to_compile
\n 1288 )
\n 1289
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module(nn_module, stubs_fn, share_types, is_tracing)
\n 456 if not is_tracing:
\n 457 AttributeTypeIsSupportedChecker().check(nn_module)
\n--> 458 return create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 459
\n 460 def create_script_module_impl(nn_module, concrete_type, stubs_fn):
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 518
\n 519 # Actually create the ScriptModule, initializing it with the function we just defined
\n--> 520 script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
\n 521
\n 522 # Compile methods if necessary
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in _construct(cpp_module, init_fn)
\n 613 \"\"\"
\n 614 script_module = RecursiveScriptModule(cpp_module)
\n--> 615 init_fn(script_module)
\n 616
\n 617 # Finalize the ScriptModule: replace the nn.Module state with our
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in init_fn(script_module)
\n 496 else:
\n 497 # always reuse the provided stubs_fn to infer the methods to compile
\n--> 498 scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
\n 499
\n 500 cpp_module.setattr(name, scripted)
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 518
\n 519 # Actually create the ScriptModule, initializing it with the function we just defined
\n--> 520 script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
\n 521
\n 522 # Compile methods if necessary
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in _construct(cpp_module, init_fn)
\n 613 \"\"\"
\n 614 script_module = RecursiveScriptModule(cpp_module)
\n--> 615 init_fn(script_module)
\n 616
\n 617 # Finalize the ScriptModule: replace the nn.Module state with our
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in init_fn(script_module)
\n 496 else:
\n 497 # always reuse the provided stubs_fn to infer the methods to compile
\n--> 498 scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
\n 499
\n 500 cpp_module.setattr(name, scripted)
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 518
\n 519 # Actually create the ScriptModule, initializing it with the function we just defined
\n--> 520 script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
\n 521
\n 522 # Compile methods if necessary
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in _construct(cpp_module, init_fn)
\n 613 \"\"\"
\n 614 script_module = RecursiveScriptModule(cpp_module)
\n--> 615 init_fn(script_module)
\n 616
\n 617 # Finalize the ScriptModule: replace the nn.Module state with our
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in init_fn(script_module)
\n 496 else:
\n 497 # always reuse the provided stubs_fn to infer the methods to compile
\n--> 498 scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
\n 499
\n 500 cpp_module.setattr(name, scripted)
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 518
\n 519 # Actually create the ScriptModule, initializing it with the function we just defined
\n--> 520 script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
\n 521
\n 522 # Compile methods if necessary
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in _construct(cpp_module, init_fn)
\n 613 \"\"\"
\n 614 script_module = RecursiveScriptModule(cpp_module)
\n--> 615 init_fn(script_module)
\n 616
\n 617 # Finalize the ScriptModule: replace the nn.Module state with our
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in init_fn(script_module)
\n 496 else:
\n 497 # always reuse the provided stubs_fn to infer the methods to compile
\n--> 498 scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
\n 499
\n 500 cpp_module.setattr(name, scripted)
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 518
\n 519 # Actually create the ScriptModule, initializing it with the function we just defined
\n--> 520 script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
\n 521
\n 522 # Compile methods if necessary
/opt/conda/lib/python3.7/site-packages/torch/jit/_script.py in _construct(cpp_module, init_fn)
\n 613 \"\"\"
\n 614 script_module = RecursiveScriptModule(cpp_module)
\n--> 615 init_fn(script_module)
\n 616
\n 617 # Finalize the ScriptModule: replace the nn.Module state with our
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in init_fn(script_module)
\n 496 else:
\n 497 # always reuse the provided stubs_fn to infer the methods to compile
\n--> 498 scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
\n 499
\n 500 cpp_module.setattr(name, scripted)
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
\n 522 # Compile methods if necessary
\n 523 if concrete_type not in concrete_type_store.methods_compiled:
\n--> 524 create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
\n 525 # Create hooks after methods to ensure no name collisions between hooks and methods.
\n 526 # If done before, hooks can overshadow methods that aren't exported.
/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
\n 373 property_rcbs = [p.resolution_callback for p in property_stubs]
\n 374
\n--> 375 concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
\n 376
\n 377 def create_hooks_from_stubs(concrete_type, hook_stubs, pre_hook_stubs):
RuntimeError: Can't redefine method: forward on class: torch.torch.nn.modules.dropout.Dropout (of Python compilation unit at: 0x5643bd433f20)
", "votes": 0, "replies": [] }, { "id": 2100711, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-15T11:30:55.010000", "content": "this error don't occurs at my side. i also try torch.jit.trace(model, x) which return successful results.
\ncheck that you are using eval()
\ni suggest the either of the followings:
there are several versions of nextvit.py file from the github. i am using the image classification one.
follow the instruction from the github (export_tensorrt_engine.py), try to create onnx file. This checks your system(version, etc), torch.onnx also calls torch jit script.
simply edit your code. replace nn.Dropout with nn.Identity. If there is problem with dropPath create afunction/class that just let input pass through. (dropout and droppath are not used in eval() mode)
Thanks for your oppions.
", "votes": 0, "replies": [] }, { "id": 2100931, "author_name": "Guoliang", "author_url": "", "post_date": "2023-01-15T13:56:41.310000", "content": "could you share a notebook for generating trt engine file of NextVitNet?
", "votes": 0, "replies": [] } ] }, { "id": 2100622, "author_name": "Guoliang", "author_url": "", "post_date": "2023-01-15T10:01:06.630000", "content": "
\nhi, I got above error when generating trt engine file. Could you tell me how to solve it?
T4x2 is King, I just switched my pipeline to it, and preprocessing is x1.24 faster. Preprocessing the training set I improve from 3h 26m to 2h 46m !!
\nprocessing the first 5118 j2k and 5917 non-j2k (to compare to your numbers), the pipeline does 9.2 minutes, and 20 minutes respectively! (time also includes yolov5 inference on 640px imgs)
\nI get this small speedup by doing all the resizing and windowing operations in different threads on EACH GPU!
\njust a note: the apply_voi_lut == apply_windowing since I believe that none of the images in the training set have a VOI lookup table.
", "votes": 0, "replies": [] } ] }, { "id": 2107692, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-20T01:01:26.237000", "content": "there is a novel method to fight rare (imbalance) class
\n[1] Background Splitting: Finding Rare Classes in a Sea of Background
\nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Mullapudi_Background_Splitting_Finding_Rare_Classes_in_a_Sea_of_Background_CVPR_2021_paper.pdf
the idea i simple randomly assign label to your background images. hence even if you sample a batch of all negative images, you will not learn to predict same class
\nhttps://www.youtube.com/watch?v=I6-8mrp99sI
That's a cool idea. I'll be very curious to hear how well it works.
", "votes": 4, "replies": [] } ] }, { "id": 2076917, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-27T02:37:49.950000", "content": "i find the treasure !!!!
\n
A amazing AUC.
", "votes": 1, "replies": [ { "id": 2076990, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-27T05:26:18.747000", "content": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8456839/
", "votes": 5, "replies": [] }, { "id": 2076997, "author_name": "ynhuhu", "author_url": "", "post_date": "2022-12-27T05:39:50.727000", "content": "Thanks you.
", "votes": 0, "replies": [] }, { "id": 2076999, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-27T05:45:35.963000", "content": "more baseline models and results later …
\nMVCCL model for ADMANI dataset
\nhttps://www.kaggle.com/code/hengck23/mvccl-model-for-admani-dataset
\"Our experiments were performed on Swinburne supercomputer OzSTAR* with a cluster of NVIDIA Tesla P100 GPUs\" .. OzSTAR comprises over 5,000 processing cores, 230 GPUs, a collective 25 Terabytes of system memory and access to over 6 Petabytes of storage.
", "votes": 0, "replies": [] }, { "id": 2077034, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-27T06:46:26.990000", "content": "this is because they have 3 million images
\nthey resolution is about 2600
\n\nthis is because they have 3 million images
\n
\n they resolution is about 2600
I am referring to the link https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8456839/
\nThis is the paper that the above screen cap came from. Thanks Remek for the cite.
\nI don't see any reference to 'millions' of images in the study. Perhaps I missed it though.
", "votes": 0, "replies": [] }, { "id": 2077041, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-27T06:57:08.987000", "content": "@ynhuhu Another thing to consider is that the AUC you see above is on the patches. Here are the top line results from the paper:
\n\n", "votes": 1, "replies": [] }, { "id": 2077053, "author_name": "ynhuhu", "author_url": "", "post_date": "2022-12-27T07:09:24.097000", "content": "Results
\n
\n Our evaluation uses the area under curve (AUC) and accuracy (ACC) for performance measurement. The best evaluation result, based on 349 test cases (930 test images), was an AUC of 0.8979 [95% confidence interval (CI) 0.873, 0.923] and ACC of 0.8178 [95% CI 0.785, 0.850].
Ye, you are right.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2069441, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-19T01:53:45.930000", "content": "
in some of the video and websites i have read, we screen mammography images by comparing left and right images side-by-side (see image above). Instead of predicting based on single image, we use stitch of of R-L image as single input. Alternatively, we can have 2 view (2x single input) and fused them later.
\nThe advantage is that we both have same breast density, so abnormality can stand out better
", "votes": 9, "replies": [ { "id": 2069467, "author_name": "", "author_url": "", "post_date": "2022-12-19T02:32:32.970000", "content": "", "votes": -10, "replies": [] } ] }, { "id": 2061348, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-11T03:12:28.180000", "content": "i see some (very) good improvement in using 16-bit png (instead of 8bit).
\nThis is because i am not applying windowing or VOI LUT in the dim data, which itself is tricky.
It is difficult to large scale experiment (especially for kaggle submission)
\nOther kagglers may want to verify this.
def read_dicom_as_image(dcm_file):\n dicom = pydicom.dcmread(dcm_file)\n image = dicom.pixel_array \n image = (image - image.min()) / (image.max() - image.min()+1e-6) #this cast to float32\n if dicom.PhotometricInterpretation == 'MONOCHROME1':\n image = 1 - image\n\n return image\n\ndef parallel_process(dcm_file):\n patient_id = dcm_file.split('/')[-2]\n image_id = dcm_file.split('/')[-1][:-4]\n image = read_dicom_as_image(dcm_file)\n image = cv2.resize(image, (image_size, image_size), interpolation=cv2.INTER_LINEAR)\n image = (image * 65535).astype(np.uint16)\n\n os.makedirs(f'{png_dir}/{patient_id}', exist_ok=True)\n cv2.imwrite(f'{png_dir}/{patient_id}/{image_id}.png',image)\n\n\nif 1:\n Parallel(n_jobs=10)(\n delayed(parallel_process)(f)\n for f in tqdm(dcm_file)\n )\ni suspect my previous improvement of 2048 is actually from the intensity improvement
\nlearnable windowing
\nPractical Window Setting Optimization for Medical Image Deep Learning
\nhttps://github.com/MGH-LMIC/windows_optimization
CT Window Trainable Neural Network for Improving Intracranial Hemorrhage Detection
\nhttps://ars.els-cdn.com/content/image/1-s2.0-S093336571930939X-gr2.jpg
I'm not sure how you saved 2048 x 2048 uint16 images. Cuz, only 13k images take >20GB space, so for total data, it would be nearly >100GB space. In that case, kaggle should throw error. Could you please share how you saved uint16 images?
You could do it off kaggle, right. Also, batching on kaggle for inference.
", "votes": 0, "replies": [] }, { "id": 2062268, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-11T23:11:24.313000", "content": "i do not save at kaggle inference.
\ni haven't tried uint16 for 2048 yet. i am running experiments for 1024 for now
@hengck23 Why it's for?
\nimage = (image * 65535).astype(np.uint16)\nSeems like image is normalized normalized beforehand (usually between 0-1), to cast uint16 you better to multiply these values by 65535 (which is the max value of uint16) before mapping them to rounded uint16 values.
", "votes": 0, "replies": [] }, { "id": 2064198, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-13T15:15:34.003000", "content": "this is to save as 16-bit png.
\nif you don't save image for inference, the casting is not required
What do you think about normalizing with 16-bit dataset mean/std? @hengck23
", "votes": 0, "replies": [] }, { "id": 2094504, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-10T19:37:10.733000", "content": "only experiment will confirms results.
\nand you don't have to stick to one processing. you can use different processing in ensemble
This is interesting. If you don't convert to 8bit, you have more information available. But do you still start from the pretrained models on 8 bit images?
", "votes": 0, "replies": [ { "id": 2094506, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-10T19:38:04.793000", "content": "no.
\nit is like if pretrain model is train on 224x224 image size, you can finetune it for larger and smaller size
", "votes": 1, "replies": [] } ] } ] }, { "id": 2067573, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-16T20:53:11.017000", "content": "the fastest you can go is slightly less than 3hr
\ninput1024, one model single-fold efficientnetb4
#share ----\ndef normalised_to_8bit(image, photometric_interpretation):\n xmin = image.min()\n xmax = image.max() \n norm = np.empty_like(image, dtype=np.uint8)\n dicomsdl.util.convert_to_uint8(image, norm, xmin, xmax)\n if photometric_interpretation == 'MONOCHROME1':\n norm = 255 - norm\n return norm\n\n\n# j2k ----\nj2k_decoder = nvjpeg2k.Decoder()\ndef process_j2k(df, dcm_dir, image_dir, image_size):\n for t, d in tqdm(df.iterrows()):\n dcm_file = f'{dcm_dir}/{d.patient_id}/{d.image_id}.dcm'\n ds = pydicom.dcmread(dcm_file)\n offset = ds.PixelData.find(b'\\x00\\x00\\x00\\x0C')\n jpeg_stream = bytearray(ds.PixelData[offset:]) \n m = j2k_decoder.decode(jpeg_stream) \n\n\n # resize and save as png\n m = normalised_to_8bit(m, ds.PhotometricInterpretation)\n m = cv2.resize(m, dsize=(image_size, image_size), interpolation=cv2.INTER_LINEAR)\n cv2.imwrite(f'{image_dir}/{d.patient_id}/{d.image_id}.png', m)\n\n...\n\n#non j2k ----\n\ndef dicomsdl_parallel_process_fn(d, dcm_dir, image_dir, image_size):\n dcm_file = f'{dcm_dir}/{d.patient_id}/{d.image_id}.dcm'\n ds = dicomsdl.open(dcm_file)\n info = ds.getPixelDataInfo() \n m = np.empty(shape=[info['Rows'], info['Cols']], dtype=info['dtype'])\n ds.copyFrameData(0, m) \n\n # resize and save as png\n m = normalised_to_8bit(m, ds.PhotometricInterpretation)\n m = cv2.resize(m, dsize=(image_size, image_size), interpolation=cv2.INTER_LINEAR)\n cv2.imwrite(f'{image_dir}/{d.patient_id}/{d.image_id}.png', m)\n\ndef process_non_j2k(df, dcm_dir, image_dir, image_size, n_jobs): \n Parallel(n_jobs=n_jobs)(\n delayed(dicomsdl_parallel_process_fn)(d, dcm_dir, image_dir, image_size)\n for t,d in tqdm(df.iterrows())\n )\nyou probably need to retrain with images generated by dicomsdl.util.convert_to_uint8()
", "votes": 8, "replies": [ { "id": 2082166, "author_name": "", "author_url": "", "post_date": "2023-01-01T07:37:53.707000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2067719, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-17T04:28:46.743000", "content": "in theory, you can stitch four 1024 images (LCC,RCC,LMLO,RMLO) into a single 2048 image. Then input this into a single network and make 2 predictions for L,R
", "votes": 7, "replies": [ { "id": 2079570, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-12-29T12:34:48.603000", "content": "Multiple-instance learning could be useful.
", "votes": 0, "replies": [] } ] }, { "id": 2059408, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-08T20:15:54.477000", "content": "[paper] Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening- Nan Wu
\nhttps://github.com/nyukat/breast_cancer_classifier
extensive experiments for multi-view prediction (over 1 million images)
\n
\n
Very nice concept. Good inspiration! 👍
", "votes": 0, "replies": [] } ] }, { "id": 2120459, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-29T16:00:55.473000", "content": "i tried many multiple images prediction method and below is the only one that works.
\nFor nextvit-B local CV improves from 0.49 (single-image predict + mean) to 0.511 (multi-image predict) for the my first experiments:
I got improvement at the first run without adjusting of hyper parameters and pipeline!
\nhere are more information:
\nPART ONE: paper review and description of method
[1] COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Prediction - A. Sriram (facebook AI), arXiv 2020
\nhttps://github.com/facebookresearch/CovidPrognosis
PART TWO: modification for kaggle breast mammography and example notebook
\ndummy code is up: https://www.kaggle.com/code/hengck23/example-of-multi-image-prediction
PART THREE: results and analysis
\nto be updated
first experiment results
\nMIP prediction\n\nget_f1score(cancer_p[site_id==1], cancer_t[site_id==1], mode='max')\nOut[10]: (0.4852269914108315, 0.4081632653061224) # (f1score, threshold)\n\nget_f1score(cancer_p[site_id==2], cancer_t[site_id==2], mode='max')\nOut[11]: (0.5681399603886713, 0.24489795918367346)\n\nget_f1score(cancer_p, cancer_t, mode='max')\nOut[12]: (0.5106197826424658, 0.2857142857142857)\nauc = 0.8344857204208718\nbce loss = 0.081253260\n
MIP trained with frozen image encoder + augmentation
\nuse age, site id, view
\n
\nunfrozen image encoder (last layer only) + augmentation : worse (overfitting observed)
unfrozen image encoder (last layer only) + no augmentation : not that bad??? … to repeat experiment again
", "votes": 2, "replies": [] }, { "id": 2123664, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-31T16:19:21.270000", "content": "Looks really good! Any sub to LB?
\nIt appeared that my bug in code had no influence on score. Still one model only 0.57. Looking for new ways to cross over 0.6 😂
Hello @remekkinas Could you tell me which image size did you use to get 0.57 LB? Thanks
", "votes": 0, "replies": [] }, { "id": 2123989, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-31T19:24:57.640000", "content": "From beggining of the competiton I use rule (SIZE, SIZE // 2). Size depends on network architecture I use. Now is (4 * 384, 2 * 384).
", "votes": 0, "replies": [] }, { "id": 2138174, "author_name": "outwrest", "author_url": "", "post_date": "2023-02-10T16:06:57.800000", "content": "@hengck23 did you ever submit this model to LB?
", "votes": 0, "replies": [] }, { "id": 2145791, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-15T11:12:01.113000", "content": "yes i did. surprsingly …. +0.03 in CV, -0.01 in LB
", "votes": 1, "replies": [] }, { "id": 2146126, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-02-15T16:16:50.693000", "content": "I had LB … CV correlation. The worse score in CV the best on LB. This is certainly joke but I am really afraid how to chose final submission. I know that my submissions are shaky… and do not count on anything.
", "votes": 4, "replies": [] }, { "id": 2146231, "author_name": "NguyenThanhNhan", "author_url": "", "post_date": "2023-02-15T17:55:37.040000", "content": "Our first 2 folds tracked public LB very closely so occassionally I would submit 2 folds' predictions to pump our public position 😂. Also, multi-image models boosted local validation pF1 at the cost of much higher optimal thresholds, which ended up in worse LB scores
", "votes": 5, "replies": [] }, { "id": 2146426, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-15T21:21:39.920000", "content": "\"multi-image models boosted local validation pF1 at the cost of much higher optimal thresholds\"
\nhigher threshold is over fitting i i i think
\nit seems that there is not enough data to do multiview prediction, it is quite a pity.
i have to remove transformer and adopted simpler fusion like mean,max, attention pool, etc …
\nalso i do not have proper augmentation for multi-view (the views are related, so are the augmentation)
\non a site note, if i perform multi-view learning on external vindr-mammo dataset with patch prediction (i.e. convert segmentation pixel label to patch label), the class activation maps do show better results on validation set.
", "votes": 1, "replies": [] }, { "id": 2146870, "author_name": "NguyenThanhNhan", "author_url": "", "post_date": "2023-02-16T08:12:27.790000", "content": "When I tried your multi-image transformer, I had to freeze 70-80% of backbone blocks and used a very small lr to make it trainable. I also applied the same augmentation to all images per side, like this (albumentations is quite convenient)
\ntransformed = augment(images=image0, images1=image1 ...)\nI agree that we simply don't have enough data for multi-view training 😅. Simpler fusion models like boosted trees, mean, max might work better.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2108173, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-20T10:10:07.493000", "content": "this is a cheap way to improve resolution and works for me at local CV
\n
Nice. As far as I understand gradient is updated from main loss?
", "votes": 0, "replies": [ { "id": 2108234, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-20T11:09:38.887000", "content": "gradient is updated from main loss?
\nno. all loss
", "votes": 3, "replies": [] }, { "id": 2108239, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-20T11:13:30.470000", "content": "Thank you for explanation.
", "votes": 0, "replies": [] }, { "id": 2108280, "author_name": "Eleftherios Fanioudakis", "author_url": "", "post_date": "2023-01-20T11:52:28.547000", "content": "did you try different weighting for these losses ?
", "votes": 0, "replies": [] }, { "id": 2108977, "author_name": "outwrest", "author_url": "", "post_date": "2023-01-21T02:16:06.237000", "content": "I think this is a mix of two good ideas for aux loss + keeping hi-rez features. I tried something similar using resnet but it didn't improve my local CV.
\n@remekkinas this reminds me of the paper (Going deeper with convolutions) from a while back, I recommend checking it out! It is also the first paper that I saw that references a meme 😄😄
\n@left13 aux loss weighting might be interesting to try out, they also do it in the paper I linked above
\n\"During training, their loss gets added to the total loss of the network with a discount weight (the losses of the auxiliary classifiers were weighted by 0.3). At inference time, these auxiliary networks are discarded.\" [Going deeper with convolutions, 6]
I think if we were training from scratch it'll be a small boost to training time but I am interested in how each part affects CV for this competition since we are only fine-tuning the model
", "votes": 2, "replies": [] } ] }, { "id": 2110137, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-22T02:08:28.810000", "content": "if you want to use tensorrt for this model, please note the pytorch resize bug
\nhttps://github.com/pytorch/TensorRT/pull/1561
\nyou will have to modify the tensorrt py file
alternatively, just convert the encoder to trt engine and use pytorch nn module for the rest
", "votes": 1, "replies": [] } ] }, { "id": 2077872, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-27T23:34:30.137000", "content": "finally, one paper that compares oversampling, undersampling and weighted class
\n
\n
[1] Comparing Techniques for Class Imbalance in Deep LearningComparing Techniques for Class Imbalance in Deep Learning
\nClassification of Breast CancerClassification of Breast Cancer
the only conclusion is that no conclusion can be made
", "votes": 5, "replies": [ { "id": 2078070, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-28T04:27:24.450000", "content": "\nI think the conclusion was generally that augmentation / pseudo labelling, especially domain specific augmentation (see the paper regarding how it does artifacting, makes a lot of sense) is effective when dealing with class imbalance. Imho, this is smart oversampling.
", "votes": 2, "replies": [] }, { "id": 2087063, "author_name": "Jia-Ming Lin", "author_url": "", "post_date": "2023-01-05T09:38:21.357000", "content": "Hi,
\nIt seems that, choosing sampling strategies should be depends on the dataset/scenarios, and should be hyperparameters.
\nBut there is one important thing the paper not answer, the sampling rate.
\nE.g. in over-sampling(ROS), higher pos. to neg. ratio would make the model overfit to pos.
\nThere is one recent paper[1], demonstrating this phenomena.
\n
\nNote that, for the 4-layer case, pos:neg = 20:80(ROS-4) is the best choice.
[1] The Effects of Data Sampling with Deep Learning and Highly Imbalanced Big Data, Information System Frontiers, Springer, 2020
", "votes": 0, "replies": [ { "id": 2087167, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-05T11:38:44.870000", "content": "
most of the paper may not be useful because results of imbalance data is very dependent on the data itself.
\nhere is how you should analyze. see the results of the validation above, the question you should ask is that \"can you draw the distribution of an unknown test LB data?
\n[1] the distribution of the neg validation is smooth and predictable. How i think the test LB neg is close to the black dotted line.
\n[2] for the pos validation, it is multi modal, unpredictable , it even have empty bins. That is why results is unpredictable.
\non a side note:
\nif you are trying to use kaggle data to create a stable CV-LB, you should ask:
\n\"choosing sampling strategies should be depends on the dataset/scenarios, and should be hyperparameters.\"
\nthis is correct. it also depends on model.
", "votes": 0, "replies": [] }, { "id": 2087382, "author_name": "Jia-Ming Lin", "author_url": "", "post_date": "2023-01-05T15:03:33.950000", "content": "Thanks, your comments are very inspiring
", "votes": 0, "replies": [] } ] } ] }, { "id": 2076911, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-27T02:03:43.957000", "content": "is the cat out of the bag …. anyone caught the cat ???
\nhttps://pubs.rsna.org/doi/pdf/10.1148/ryai.220072
\nADMANI: Annotated Digital Mammograms and Associated Non-Image Datasets
\nPublished Online:Dec 21 2022
\" A subset of 40,000 images from 10,000 episodes will be provided for the
\nRadiological Society of North America Mammography Breast Cancer Detection AI Challenge,
\nlaunching on November 28th. The challenge training dataset will be made public when the
\nchallenge is launched and will remain available to researchers when the challenge concludes.
\nThe 10,000 episodes will be randomly selected from the dataset from a three-year period.\"
https://www.rsna.org/education/ai-resources-and-training/ai-image-challenge
\n\"The dataset was contributed by mammography screening programs in Australia and the U.S. It includes detailed labels, with radiologists’ evaluations and follow-up pathology results for suspected malignancies.\"
aka site1 and site2
\nmaybe NYU + ADMANI???
\nbaseline results:
\nhere is a paper that compares results with and without NYU pretrain model:
[1] Evaluation of deep learning-based artificial intelligence techniques for breast cancer detection on mammograms: Results from a retrospective study using a BreastScreen Victoria dataset (part of ADMANI)
\n[2] Multi-view Local Co-occurrence and Global Consistency Learning Improve Mammogram Classification Generalisation
", "votes": 5, "replies": [] }, { "id": 2084837, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-03T19:36:40.053000", "content": "high quality \"Breast Micro-Calcifications Dataset with Precisely Annotated Sequential Mammograms\"
\ndataset:
\nhttps://zenodo.org/record/5036062
paper:
\n[1] Loizidou, K., Skouroumouni, G., Pitris, C. et al. Digital subtraction of temporally sequential mammograms for improved detection and classification of microcalcifications. Eur Radiol Exp 5, 40 (2021). https://doi.org/10.1186/s41747-021-00238-w
papers that uses this dataset:
\nhttps://scholar.google.com/citations?user=qadXBKAAAAAJ&hl=en
some analysis
\n
the sorted probability (red-black) graph gives you an idea how the sample prediction values fluctuate with different models.
\nthis is important when you are making ensemble and choosing threshold value in pfbeta binarization.
\nthe threshold must be stable over the fluctuation.
Thanks! I see that you are using balance sampler, would be interesting compare with bce with loss weight with the same weights.
", "votes": 0, "replies": [] }, { "id": 2054865, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-04T13:59:15.230000", "content": "
i try to understand why resolution 1024 is better than 512. I show some CAM activation results for 1024 here (but i am not sure if the model is correct)
\n
\n
I am not familiar with mammography findings, but perhaps the visibility of calcification, which is typical of malignant findings, is very different between 512 and 1024?
\nBoth of the images you posted seem to focus on the fine calcifications, which are both parapetitive.
EDIT: I don't think calcification is completely specific, as it can be seen not only in breast cancer but also in benign diseases.
", "votes": 5, "replies": [] }, { "id": 2056120, "author_name": "Sohier Dane", "author_url": "", "post_date": "2022-12-05T18:40:11.017000", "content": "I can't speak to this specific image, but to the point @yosukeyama made it's expected that the presence of some very small features (the calcifications) might favor models that use larger images. This article might provide useful context: https://radiopaedia.org/articles/breast-imaging-reporting-and-data-system-bi-rads
", "votes": 4, "replies": [] }, { "id": 2056132, "author_name": "Kirderf", "author_url": "", "post_date": "2022-12-05T19:05:44.347000", "content": "Addition to the dim. questions, can we lose some important grey level information in the png conversion from dicom and maybe it would be better using e.g. NIfTI-format, or do we have all information needed by using windowing features?
", "votes": 2, "replies": [] }, { "id": 2057734, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-07T10:11:06.943000", "content": "i show CAM heatmap for an external dataset with lesion annotation
\nhttps://vindr.ai/datasets/mammo
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
i am glad that for high predicted probability, the model (single fold 1024) performs quite well.
\nhowever t there are many misses. I think this is due to lack of kaggle train data.
use of external will play an important part in this competition
", "votes": 7, "replies": [] }, { "id": 2058004, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-07T14:37:27.680000", "content": "2048 is the king?
\n
anyone has good suggestion for a low-memory, low flop model that might work well for 2048?
", "votes": 3, "replies": [] }, { "id": 2058540, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-08T02:27:18.853000", "content": "check the distribution diagram
\nhttps://miro.medium.com/max/720/1*yF319EgJVzag9pd2ZL4D4Q.webp
it compare bce loss and soft-f1 loss
\n", "votes": 0, "replies": [] }, { "id": 2058880, "author_name": "Kirderf", "author_url": "", "post_date": "2022-12-08T09:28:06.033000", "content": "\n\nanyone has good suggestion for a low-memory, low flop model that might work well for 2048?
\n
Teacher-student architectur and knowledge distillation?
", "votes": 0, "replies": [] }, { "id": 2058958, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-08T11:00:11.143000", "content": "
i added a head to learn the best probability calibration for maximizing kaggle metric F1.
\nIn the end, it is the same as hard binary thresholding
Good explanation, well🙌
", "votes": 0, "replies": [] } ] }, { "id": 2162027, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-28T00:07:40.403000", "content": "results are out!
\n3 fold nextvit-b has private lb 0.48.
\n1 fold nextvit-b has private lb 0.45.
but unfornately, i didn't select that
\n
May I ask why you chose 1-fold over 3-fold solution?
", "votes": 0, "replies": [] } ] }, { "id": 2113008, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-24T02:53:04.297000", "content": "sometimes you want to modify timms model without messing of the original code. Here is a way to do it:
\nfrom timm.models.convnext import _create_convnext\n\ndef convnext_tiny(pretrained=False, **kwargs):\n model_args = dict(depths=(3, 3, 9, 3), dims=(96, 192, 384, 768), **kwargs)\n model = _create_convnext('convnext_tiny.in12k_ft_in1k_384', pretrained=pretrained, **model_args)\n setattr(model, 'depths', [3, 3, 9, 3])\n return model\n\n\n#modify to output all layers\nclass Encoder(nn.Module):\n def __init__(self, ):\n super(Encoder, self).__init__()\n e = convnext_tiny(pretrained=True)\n self.stem = e.stem\n self.stage1 = e.stages[ 0 : e.depths[0]]\n self.stage2 = e.stages[e.depths[0] : e.depths[1]]\n self.stage3 = e.stages[e.depths[1] : e.depths[2]]\n self.stage4 = e.stages[e.depths[2] : e.depths[3]]\n self.norm_pre = e.norm_pre\n del e\n\n def forward(self, x):\n x0 = self.stem(x)\n x1 = self.stage1(x0)\n x2 = self.stage2(x1)\n x3 = self.stage3(x2)\n x4 = self.stage4(x3)\n return [x1,x2,x3,x4]\nin order to improve the detection rate, one may want to flip some labels:
\ne.g.
\ndifficult_negative_case =1, biopsy=1 --> cancer =1
\n(since it is sent for biopsy, it should be visually close to malignant )
tried that but no improvement. might need to add density type as well
", "votes": 1, "replies": [] } ] }, { "id": 2084695, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-03T17:18:02.053000", "content": "https://github.com/nyukat/BIRADS_classifier
\nhttps://cs.nyu.edu/~kgeras/reports/datav1.0.pdf
How to map external data to kaggle label
\n\"As BI-RADS 0 and BI-RADS 1 and BI-RADS 2 should be the only BI-RADS categories used in screening mammography,
\nwe condensed all BI-RADS categories into three classes for the purposes of training our model.\"
BI-RADS 0, 4a/b/c and 5 were mapped to a new ‘BI-RADS 0’ as each indicates a possibility of malignancy.
\nBI-RADS 1 is retained at ‘BI-RADS1’.
\nBI-RADS 2 and 3 are mapped to a new ‘BI-RADS 2’, as they both indicate benign findings.
This procedure resulted in a dataset consisting of a single BI-RADS label over three classes for each of our valid screening mammography exams.\"
\nBI-RADS categories:
\n0 (‘incomplete’),
\n1 (‘negative’),
\n2 (‘benign’),
\n3 (‘probably benign’),
\n4a (‘low suspicious’),
\n4b (‘moderate sus-picious’),
\n4c (‘high suspicious’)
\n5 (‘highly suggestive of malignancy’)
\n6 (‘known biopsy with proven malignancy’)
The paper you link to is from 2019, correct? That's 3 years ago. I wonder if they've enhanced the labelling given what's occurred since then.
", "votes": 0, "replies": [] } ] }, { "id": 2082873, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-02T03:44:08.533000", "content": "results on large scale external data (vindr) is pretty much the same as kaggle data:
\n
Hey, I am a novice and I have a question, did you train on Kaggle and predict on Vindr? I am trying to figure out a good CV step because I have doubts about mine. Since we are thresholding, would holdout and N-fold CV be better to find a threshold for the competition dataset? And can we use Vindr as the holdout if it is similar as you mentioned?
\nIf the Vindr dataset is very similar to Kaggle's I am sure we can use it to validate models (I think this won't go against their license since we are not training on it).
", "votes": 2, "replies": [ { "id": 2083921, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-03T01:08:00.367000", "content": "[1] Can Vindr be used in the solution at all?
\ni haven't considered this yet. my current objective is to get a good way to understand the problem, training and metrics first. maybe it is possible that Vindr is not allowed and i will deal with that later. Vindr is currently investigated, because i can get benchmark results form paper and there are more annotation like bounding box, etc for experiments.
\n[2] did you train on Kaggle and predict on Vindr?
\nresults are based on training = Vindr, testing = Vindr. I am just repeating the paper results to confirm my pipeline is correct.
\nEven with Vindr, the metrics are not stable (both AUC and F1 score). Vindr is still not large enough. I think the reason is as follows:
\npure visual diagnostics from image mammography has its limitation. Positive predictive value (PPV), aka. precision, of radiologist in screening is not high. that is why biopsy is needed to confirm cancer cases.
that is why just only based on visual evidence, given a choice to improve either sensitivity (tp rate) or specificity (tn rate), it is \"easier\" for the model to choose specificity. You can better metrics because there are much more neg samples (imbalance) and more likely to be correct (given same detectable visual abnormality, it is more likely to be benign)
you can google for PPV for abnormality in mammography screening for more information
\nHence there is probably no good metric. and you are not likely to see good CV-LB as you would usually would in previous problems with better data.
\nYou will also see that for papers that conduct experiments on various datasets (instead of one), metrics on various datasets various. Results are more stable for those that uses millions of training images (from 1 million to 4 million).
\nsince you are a student, i strong encourage you to apply for those million image dataset like OPTIMAM, CSAW, AMAINDA and conduct experiments on those (but note that they cannot be used for kaggle solutions if you are aiming for prize). These are open to public but by need email request. You will get a bigger picture on stable CV.
\nThe next best thing is to read papers on those big dataset.
\nThen how to approach this competition?
\ndiversity:
\n\"if diverse solution agrees on diverse datasets with diverse metrics, they are probably more correct\".
", "votes": 4, "replies": [] }, { "id": 2083930, "author_name": "outwrest", "author_url": "", "post_date": "2023-01-03T01:18:52.607000", "content": "Thanks for your insights, I will continue learning!
", "votes": 0, "replies": [] }, { "id": 2084004, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-03T04:07:33.927000", "content": "
an example of repeating results. you can see how the results varies
", "votes": 1, "replies": [] } ] } ] }, { "id": 2057901, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-07T12:42:41.050000", "content": "mixed results of dicom intensity windowing
\n
Applying window operation on the fly is too expensive though. Maybe you can try stacking 2 different windowed output on channel dimension.
", "votes": 0, "replies": [] } ] }, { "id": 2065229, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-14T12:25:30.160000", "content": "how to set pos weight in loss to maximize f1 score for imbalanced class
\nhttp://ethen8181.github.io/machine-learning/model_selection/imbalanced/imbalanced_metrics.html
Balance using Sampler or loss weights … or both?
", "votes": 0, "replies": [] }, { "id": 2065446, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-14T17:02:07.163000", "content": "still experimenting. i not sure if there will be a shakeup because for the same single one-fold model + the same data that is trained differently, i can get very different lb scores of 0.39 to 0.51.
", "votes": 0, "replies": [] }, { "id": 2065958, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-15T08:34:21.677000", "content": "Do you use BalancedSampler presented in your code? If yes I think that a bit of instability in score could cause this line:
\n pos_index = np.random.choice(pos_index, self.length//self.r).reshape(-1,1)\nwhat do you think?
", "votes": 1, "replies": [] } ] }, { "id": 2059570, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-09T03:16:02.913000", "content": "validation results are complementary. This means that 2048 model is improving different samples from 1024.
\nin particular, AUC of large image is much lower (probably improving the previous low p=0 region)
\npfbeta of large model is better (probably improving the previous mid p=0.5 region)
image-wise validation results (fold-0)\n\n\neffb2-2048\nbce_loss AUC pfbeta\n0.09397 0.82092 0.19947 \n\neffb6-1024\nbce_loss AUC pfbeta\n0.106 0.7949 0.263 \nLB results of 2048
\n
when ensemble of multiple input size are used, i first convert dicom to the largest size png with cv2.resize().
\nthen, pytorch F.interpolate() function is used to create the smaller size image in net forward().
\nthis is different from training and maybe the cause of poor performance?
interesting paper
\ncvpr2022:
\nEfficient Classification of Very Large Images with Tiny Objects
\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Kong_Efficient_Classification_of_Very_Large_Images_With_Tiny_Objects_CVPR_2022_paper.pdf
Thanks for sharing @hengck23 great analysis!
", "votes": 4, "replies": [] }, { "id": 2054612, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2022-12-04T09:29:51.753000", "content": "@hengck23 I am learning a lot from these. Much appreciated.
", "votes": 4, "replies": [] }, { "id": 2149480, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-18T10:58:31.413000", "content": "some cam map thoughts:
\n
\n
\n
i suddenly have interesting question. maybe someone can take this as future research.
\nif i purposely mislabelled a negative image as positive for training, what would the CAM heatmap show? waht is the implication?
how about the reverse? mislabelled pos as neg?
", "votes": 0, "replies": [ { "id": 2152260, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-20T17:17:00.357000", "content": "if may be difficult to apply multi-view learning, but you can transfer/distill knowledge via manual annotation !!!!
\n
\n\n",
"2066638": "it turns that i have lower LB than other kagglers using efficientnet.\ni have forgotten to set the drop path rate. Here is the fixed:\n\n```\n\nefficientnet_b2(pretrained=True, drop_rate = 0.3, drop_path_rate = 0.2)\n\nrefer to timm efficientnet source code for settings of drop_rate, drop_path_rate\n```\n\nyou can use single-fold model (i.e. just one checkpoint file) to get LB >0.51. more on that later,",
"2098821": "tensorRT timming is out !!!\n\nas claimed in the paper, nextVIT is the fastest tensorRT vision transformer (same speed as efficientnet)\n\n```\nkaggle p100 notebook timing:\n\n--------------------------------------------------\npublic LB submission tensorRT 4 hr : LB 0.56\n\n--------------------------------------------------\nlocal cv 10939 images\n\nauc 0.8936061433377303\nf1score 0.49148140396595846\n@threshold 0.30612\n\n\ntotal (end-to-end from dashboard)\nDisk 16.4/73.1 GB\n1 hr 25 min\n\nbreakdown:\n\n1. install tensorRT, etc 5 min\n\n2. decode 10939 dicom images (keep aspect to 1539, use voi_lut-apply_32fp):\nnvjpeg2k (j2k, 5118 images) 28 min\ndicomsdl (non-j2k, 5917 images, 2 thread) 24 min\n\n3. detect breast box (resnet34 segentation) 2 min\n\n4. tensorRT fp16 nextVIT-B (1539x960)\none fold, original + hflip_TTA\n\nCPU utilisation 120%, 13/13 GB\nGPU utilisation 99% , 4.5/15 GB\n23 min 21 sec (7.80799 images per sec)\n\n\n=============================================\nreference (without tensorRT):\n\nsubmission 9hr : LB 0.56\n\n4. merged_bn fp16 (1539x960)\nCPU utilisation 108%, 13/13 GB\nGPU utilisation 100% , 8.5/15 GB\n95 min 32 sec (1.90819 images per sec)\n\n```",
"2107692": "there is a novel method to fight rare (imbalance) class\n[1] Background Splitting: Finding Rare Classes in a Sea of Background\nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Mullapudi_Background_Splitting_Finding_Rare_Classes_in_a_Sea_of_Background_CVPR_2021_paper.pdf\n\nthe idea i simple randomly assign label to your background images. hence even if you sample a batch of all negative images, you will not learn to predict same class\nhttps://www.youtube.com/watch?v=I6-8mrp99sI\n\n",
"2076917": "i find the treasure !!!!\n\n\n\n",
"2069441": "\n\nin some of the video and websites i have read, we screen mammography images by comparing left and right images side-by-side (see image above). Instead of predicting based on single image, we use stitch of of R-L image as single input. Alternatively, we can have 2 view (2x single input) and fused them later.\n\nThe advantage is that we both have same breast density, so abnormality can stand out better",
"2061348": "i see some (very) good improvement in using 16-bit png (instead of 8bit).\nThis is because i am not applying windowing or VOI LUT in the dim data, which itself is tricky.\n\nIt is difficult to large scale experiment (especially for kaggle submission)\nOther kagglers may want to verify this.\n\n```\ndef read_dicom_as_image(dcm_file):\n dicom = pydicom.dcmread(dcm_file)\n image = dicom.pixel_array \n image = (image - image.min()) / (image.max() - image.min()+1e-6) #this cast to float32\n if dicom.PhotometricInterpretation == 'MONOCHROME1':\n image = 1 - image\n\n return image\n\ndef parallel_process(dcm_file):\n patient_id = dcm_file.split('/')[-2]\n image_id = dcm_file.split('/')[-1][:-4]\n image = read_dicom_as_image(dcm_file)\n image = cv2.resize(image, (image_size, image_size), interpolation=cv2.INTER_LINEAR)\n image = (image * 65535).astype(np.uint16)\n\n os.makedirs(f'{png_dir}/{patient_id}', exist_ok=True)\n cv2.imwrite(f'{png_dir}/{patient_id}/{image_id}.png',image)\n\n\nif 1:\n Parallel(n_jobs=10)(\n delayed(parallel_process)(f)\n for f in tqdm(dcm_file)\n )\n\n\n```\n\ni suspect my previous improvement of 2048 is actually from the intensity improvement\n\n\n--- \nlearnable windowing\nPractical Window Setting Optimization for Medical Image Deep Learning\nhttps://github.com/MGH-LMIC/windows_optimization\n\n\nCT Window Trainable Neural Network for Improving Intracranial Hemorrhage Detection\nhttps://ars.els-cdn.com/content/image/1-s2.0-S093336571930939X-gr2.jpg",
"2067573": "the fastest you can go is slightly less than 3hr\ninput1024, one model single-fold efficientnetb4\n\n\n \n```\n#share ----\ndef normalised_to_8bit(image, photometric_interpretation):\n xmin = image.min()\n xmax = image.max() \n norm = np.empty_like(image, dtype=np.uint8)\n dicomsdl.util.convert_to_uint8(image, norm, xmin, xmax)\n if photometric_interpretation == 'MONOCHROME1':\n norm = 255 - norm\n return norm\n\n\n# j2k ----\nj2k_decoder = nvjpeg2k.Decoder()\ndef process_j2k(df, dcm_dir, image_dir, image_size):\n for t, d in tqdm(df.iterrows()):\n dcm_file = f'{dcm_dir}/{d.patient_id}/{d.image_id}.dcm'\n ds = pydicom.dcmread(dcm_file)\n offset = ds.PixelData.find(b'\\x00\\x00\\x00\\x0C')\n jpeg_stream = bytearray(ds.PixelData[offset:]) \n m = j2k_decoder.decode(jpeg_stream) \n\n\n # resize and save as png\n m = normalised_to_8bit(m, ds.PhotometricInterpretation)\n m = cv2.resize(m, dsize=(image_size, image_size), interpolation=cv2.INTER_LINEAR)\n cv2.imwrite(f'{image_dir}/{d.patient_id}/{d.image_id}.png', m)\n\n...\n\n#non j2k ----\n \ndef dicomsdl_parallel_process_fn(d, dcm_dir, image_dir, image_size):\n dcm_file = f'{dcm_dir}/{d.patient_id}/{d.image_id}.dcm'\n ds = dicomsdl.open(dcm_file)\n info = ds.getPixelDataInfo() \n m = np.empty(shape=[info['Rows'], info['Cols']], dtype=info['dtype'])\n ds.copyFrameData(0, m) \n\n # resize and save as png\n m = normalised_to_8bit(m, ds.PhotometricInterpretation)\n m = cv2.resize(m, dsize=(image_size, image_size), interpolation=cv2.INTER_LINEAR)\n cv2.imwrite(f'{image_dir}/{d.patient_id}/{d.image_id}.png', m)\n\ndef process_non_j2k(df, dcm_dir, image_dir, image_size, n_jobs): \n Parallel(n_jobs=n_jobs)(\n delayed(dicomsdl_parallel_process_fn)(d, dcm_dir, image_dir, image_size)\n for t,d in tqdm(df.iterrows())\n )\n\n```\n\nyou probably need to retrain with images generated by dicomsdl.util.convert_to_uint8()\n\n \n\n\n",
"2067719": "in theory, you can stitch four 1024 images (LCC,RCC,LMLO,RMLO) into a single 2048 image. Then input this into a single network and make 2 predictions for L,R",
"2059408": "[paper] Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening- Nan Wu\nhttps://github.com/nyukat/breast_cancer_classifier\n\nextensive experiments for multi-view prediction (over 1 million images)\n\n\n",
"2120459": "i tried many multiple images prediction method and below is the only one that works.\nFor nextvit-B local CV improves from 0.49 (single-image predict + mean) to 0.511 (multi-image predict) for the my first experiments:\n1. i used frozen nextvit-B image encoder (in future i would use finetune)\n2. i use only channel feature after global pool from image encoder (in future I can use local feature, i.e. feature at each x,y location)\n3. i did not use augmentation in multi-images training yet\n\nI got improvement at the first run without adjusting of hyper parameters and pipeline! \n\n----\n\nhere are more information:\nPART ONE: paper review and description of method\n\n![https://i.ibb.co/yy8bBGd/Selection-724.png] (https://i.ibb.co/yy8bBGd/Selection-724.png)\n\n[1] COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Prediction - A. Sriram (facebook AI), arXiv 2020\nhttps://github.com/facebookresearch/CovidPrognosis\n\n\nPART TWO: modification for kaggle breast mammography and example notebook\ndummy code is up: https://www.kaggle.com/code/hengck23/example-of-multi-image-prediction\n\n\nPART THREE: results and analysis\nto be updated",
"2108173": "this is a cheap way to improve resolution and works for me at local CV\n\n",
"2077872": "finally, one paper that compares oversampling, undersampling and weighted class\n\n\n\n[1] Comparing Techniques for Class Imbalance in Deep LearningComparing Techniques for Class Imbalance in Deep Learning\nClassification of Breast CancerClassification of Breast Cancer\n\n\nthe only conclusion is that no conclusion can be made",
"2076911": "is the cat out of the bag .... anyone caught the cat ???\n\nhttps://pubs.rsna.org/doi/pdf/10.1148/ryai.220072\nADMANI: Annotated Digital Mammograms and Associated Non-Image Datasets\nPublished Online:Dec 21 2022\n\n\n\n\" A subset of 40,000 images from 10,000 episodes will be provided for the\nRadiological Society of North America Mammography Breast Cancer Detection AI Challenge,\nlaunching on November 28th. The challenge training dataset will be made public when the\nchallenge is launched and will remain available to researchers when the challenge concludes.\nThe 10,000 episodes will be randomly selected from the dataset from a three-year period.\"\n\nhttps://www.rsna.org/education/ai-resources-and-training/ai-image-challenge\n\"The dataset was contributed by mammography screening programs in Australia and the U.S. It includes detailed labels, with radiologists’ evaluations and follow-up pathology results for suspected malignancies.\"\n\naka site1 and site2\n\nmaybe NYU + ADMANI???\n\nbaseline results:\nhere is a paper that compares results with and without NYU pretrain model:\n\n[1] Evaluation of deep learning-based artificial intelligence techniques for breast cancer detection on mammograms: Results from a retrospective study using a BreastScreen Victoria dataset (part of ADMANI)\n\n[2] Multi-view Local Co-occurrence and Global Consistency Learning Improve Mammogram Classification Generalisation\n",
"2084837": "high quality \"Breast Micro-Calcifications Dataset with Precisely Annotated Sequential Mammograms\"\n\ndataset:\nhttps://zenodo.org/record/5036062\n\npaper:\n[1] Loizidou, K., Skouroumouni, G., Pitris, C. et al. Digital subtraction of temporally sequential mammograms for improved detection and classification of microcalcifications. Eur Radiol Exp 5, 40 (2021). https://doi.org/10.1186/s41747-021-00238-w\n\npapers that uses this dataset:\nhttps://scholar.google.com/citations?user=qadXBKAAAAAJ&hl=en",
"2054402": "some analysis\n\n\nthe sorted probability (red-black) graph gives you an idea how the sample prediction values fluctuate with different models.\nthis is important when you are making ensemble and choosing threshold value in pfbeta binarization. \nthe threshold must be stable over the fluctuation.\n",
"2162027": "results are out!\n\n3 fold nextvit-b has private lb 0.48.\n1 fold nextvit-b has private lb 0.45.\n\nbut unfornately, i didn't select that\n\n",
"2113008": "sometimes you want to modify timms model without messing of the original code. Here is a way to do it:\n\n```\n\nfrom timm.models.convnext import _create_convnext\n\ndef convnext_tiny(pretrained=False, **kwargs):\n\tmodel_args = dict(depths=(3, 3, 9, 3), dims=(96, 192, 384, 768), **kwargs)\n\tmodel = _create_convnext('convnext_tiny.in12k_ft_in1k_384', pretrained=pretrained, **model_args)\n\tsetattr(model, 'depths', [3, 3, 9, 3])\n\treturn model\n\n\n#modify to output all layers\nclass Encoder(nn.Module):\n\tdef __init__(self, ):\n\t\tsuper(Encoder, self).__init__()\n\t\te = convnext_tiny(pretrained=True)\n\t\tself.stem = e.stem\n\t\tself.stage1 = e.stages[ 0 : e.depths[0]]\n\t\tself.stage2 = e.stages[e.depths[0] : e.depths[1]]\n\t\tself.stage3 = e.stages[e.depths[1] : e.depths[2]]\n\t\tself.stage4 = e.stages[e.depths[2] : e.depths[3]]\n\t\tself.norm_pre = e.norm_pre\n\t\tdel e\n\n\tdef forward(self, x):\n\t\tx0 = self.stem(x)\n\t\tx1 = self.stage1(x0)\n\t\tx2 = self.stage2(x1)\n\t\tx3 = self.stage3(x2)\n\t\tx4 = self.stage4(x3)\n\t\treturn [x1,x2,x3,x4]\n\n```\n",
"2086023": "in order to improve the detection rate, one may want to flip some labels:\ne.g. \ndifficult_negative_case =1, biopsy=1 --> cancer =1\n(since it is sent for biopsy, it should be visually close to malignant )",
"2084695": "https://github.com/nyukat/BIRADS_classifier\nhttps://cs.nyu.edu/~kgeras/reports/datav1.0.pdf\n\nHow to map external data to kaggle label\n\n\"As BI-RADS 0 and BI-RADS 1 and BI-RADS 2 should be the only BI-RADS categories used in screening mammography,\nwe condensed all BI-RADS categories into three classes for the purposes of training our model.\" \n\nBI-RADS 0, 4a/b/c and 5 were mapped to a new ‘BI-RADS 0’ as each indicates a possibility of malignancy. \nBI-RADS 1 is retained at ‘BI-RADS1’. \nBI-RADS 2 and 3 are mapped to a new ‘BI-RADS 2’, as they both indicate benign findings. \n\nThis procedure resulted in a dataset consisting of a single BI-RADS label over three classes for each of our valid screening mammography exams.\"\n\n---\n\nBI-RADS categories:\n0 (‘incomplete’), \n1 (‘negative’), \n2 (‘benign’), \n3 (‘probably benign’), \n4a (‘low suspicious’), \n4b (‘moderate sus-picious’), \n4c (‘high suspicious’) \n5 (‘highly suggestive of malignancy’)\n6 (‘known biopsy with proven malignancy’)\n\n",
"2082873": "results on large scale external data (vindr) is pretty much the same as kaggle data:\n\n",
"2057901": "mixed results of dicom intensity windowing\n\n",
"2065229": "how to set pos weight in loss to maximize f1 score for imbalanced class\nhttp://ethen8181.github.io/machine-learning/model_selection/imbalanced/imbalanced_metrics.html",
"2059570": "validation results are complementary. This means that 2048 model is improving different samples from 1024.\nin particular, AUC of large image is much lower (probably improving the previous low p=0 region)\npfbeta of large model is better (probably improving the previous mid p=0.5 region)\n\n```\n\nimage-wise validation results (fold-0)\n\n\neffb2-2048\nbce_loss\tAUC\tpfbeta\n0.09397 \t0.82092 \t0.19947 \n\neffb6-1024\nbce_loss\tAUC\tpfbeta\n0.106 0.7949 0.263 \n\n``` \n\n---\nLB results of 2048\n\n\n\nwhen ensemble of multiple input size are used, i first convert dicom to the largest size png with cv2.resize(). \nthen, pytorch F.interpolate() function is used to create the smaller size image in net forward(). \nthis is different from training and maybe the cause of poor performance?",
"2059554": "interesting paper \ncvpr2022:\nEfficient Classification of Very Large Images with Tiny Objects\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Kong_Efficient_Classification_of_Very_Large_Images_With_Tiny_Objects_CVPR_2022_paper.pdf\n",
"2055186": "Thanks for sharing @hengck23 great analysis!",
"2054612": "@hengck23 I am learning a lot from these. Much appreciated.",
"2149480": "some cam map thoughts:\n\n\n\n\n\n\n",
"2111985": "i started some experiments on multi-view. this is very tricky as you are basically reducing your train and validation data since it now become \n```\nnum of train data = num of original train data / num of view\n```",
"2104723": "i haven try this but 2 ideas work for finetunning NLP transformers may work here\n1. adversarial training (fast sign gradient method, etc)\n2. Sharpness-Aware Minimization (SAM)\n\nboth method try to flatten the valley of the resulting loss landscape to imporve robustenss and fight overfitting.\n(or in the input space sense, try to create virtual neighbours for each train samples)\n\nreference\nhttps://github.com/juntang-zhuang/GSAM \n\nDeBERTa: Decoding-enhanced BERT with Disentangled Attention\nhttps://arxiv.org/abs/2006.03654\n\"In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understanding\"\nhttps://github.com/microsoft/DeBERTa/blob/master/DeBERTa/sift/sift.py",
"2079089": "to crop or not to crop\n",
"2074114": "\ni wonder if we have enough data for transformer solution",
"2072979": "should the threshold for f1score binarisation varies with breast density (and/or BI-RADS)?",
"2146431": "i find some direct differentiable loss for AUC-PR and Fbeta score.\n(in pytorch and tensorflow)\n\nthe idea is simple.\nyou find surrogate function for FPR and TPR, which have to learn the a \"threshold parameter\" \nFPR, TPR themselves are function of a deep net, which have \"net parameter\" \n\nso the optimization become a min, max problem for \"threshold parameter\" and \"net parameter\" \nyou can solved it via sdg with weighted classification loss\n(or you can solve in globally using linear programming)\n\npaper:\nhttps://github.com/Shlomix/global_objectives_pytorch/\nhttps://github.com/facebookresearch/pytext/blob/main/pytext/loss/loss.py\n\n\n\npaper:\n[1] Scalable Learning of Non-Decomposable Objectives\nhttps://arxiv.org/abs/1608.04802\n\n\n---\n\nbut we have an issue of imbalance. so i not sure if the method would work as well.",
"2058060": "@hengck23 Your CV/LB gap seems huge. I'm having ~0.01 difference.\nAny idea why ?\n\nEDIT : Fixed a bug in my inference pipeline, I have a huge CV/LB gap as well now (0.04)",
"2056581": "Thank You for sharing @hengck23 Always Appreciated!",
"2100897": "i made a bug and notebbok gets into infinite loop at submission.\nthis eats up my GPU hours at the beginning of the week and was unable to verify the trick below.\n\n----\n\nthe reshaping of power 0.5 trick consistently improves my CV by 0.01\nKagglers may want to verify this for LB.\n\n\n```\nbefore\n valid_df.loc[:, 'cancer_p'] = cancer_p\n valid_df.loc[:, 'cancer_t'] = cancer_t\n gb = valid_df[['site_id', 'patient_id','laterality', 'cancer_t', 'cancer_p']].groupby(['patient_id', 'laterality']).mean()\n f1score, precision, recall, threshold = get_f1score(gb.cancer_p, gb.cancer_t)\n\n\nafter\n valid_df.loc[:, 'cancer_p'] = cancer_p**0.5 \n valid_df.loc[:, 'cancer_t'] = cancer_t\n gb = valid_df[['site_id', 'patient_id','laterality', 'cancer_t', 'cancer_p']].groupby(['patient_id', 'laterality']).mean()\n f1score, precision, recall, threshold = get_f1score(gb.cancer_p, gb.cancer_t)\n\n```\n\n",
"2098125": "keep only the most important patch!\nWACV2023 paper\nhttps://github.com/yueliukth/PatchDropout\n\n\n\n[1] PatchDropout: Economizing Vision Transformers Using Patch Dropout\nhttps://openaccess.thecvf.com/content/WACV2023/papers/Liu_PatchDropout_Economizing_Vision_Transformers_Using_Patch_Dropout_WACV_2023_paper.pdf\n\n",
"2085513": "seems that site1 and site 2 are biased. do we need to separate them?\n\nsee bottom of notebook\nhttps://www.kaggle.com/code/hengck23/mvccl-model-for-admani-dataset/notebook\n\n\"there is starter code of baseline pure efficient-b0 single view here: https://www.kaggle.com/datasets/hengck23/pure-effb0-single-view-starter-for-mvccl-admani\"",
"2080577": "cross-view attention and transformer\n\n\n\n\n\n",
"2150972": "For me on CV 0.32 single fold with optimal th@0.5. I was expecting a much better result on LB as I've seen alot of kagglers wrote, but got only 0.34. Anyone has a clue of what is happening ?",
"2146941": "there is something very strange about site1 and site2 data\n- strong model seems to improve site2 (e.g. over 0.62) at the expense of dropping site1 (e.g. just over 0.30)\n- mid strong model has bout 0.58 for site2 and 0.46 for site1\n\ni wonder did anyone try to train a classifier to discrminate site1 and site2. if the classiifer can differentiate them, then they are different.\n\nnote that site2 has slightly more images. So it is possible that the model will take care of the majority site and sacrify the smaller one ",
"2129867": "Thank you for your contribution. \nNoob question: (EfficientNetB2 and EfficientNetB4) are EfficientNet V1 or EfficientNetV2 ? ",
"2129810": "Thank you for your contribution 🙌\n1, Could you explain the different between 2 metrics: \"pfbeta\" and \"max pfbeta\"? \n2, In the \"aggregate by max() per patient-laterality\" part:\nLet's take an example: In my valid set, I have 1000 patients, each patients has 4 images, 2 images for each \"patient-laterality\". \n****So how pfbeta (logged to your table result) has calculated? **\n**I guess: 1000 patients * 4 = 4000 images => 4000 proba predicted => max() for each patient-laterality -> 2000 proba for 2000 (patient-laterality)s => 2000 pfbeta s for each (patient-laterality) -> then take mean of them\n\nThanks in advance ",
"2123551": "came across a question relevant to this kaggle competition\n\n\nhttps://d1.awsstatic.com/training-and-certification/docs-ml/AWS-Certified-Machine-Learning-Specialty_Sample-Questions.pdf",
"2113225": "it is noted that mean of prediction per breast is better than individual image.\nthis means that mean prediction is a relieable self supervision signal for self supevised image learning.\n\nin thoery if we apply this to online learning on hidden test data, maybe it can lead to better results?\nof course, there will resources problem",
"2110198": "i am wondering is there any CNN single fold or (single model) that can hit LB 0.58~0.60?\nI am thinking of discarding CNN model and just use vision transformer in my solution.",
"2107235": "i made a bug and fold that for one fold you can re-run with different seed and ensemble for better (and more stable?) results",
"2099444": "Great work, that's fast! What's the local score for LB 0.58, still one fold? With that speed more fold might be tested :)",
"2094574": "Hi, could you explain to me the binarised concept please?.\n\nThank you in advance",
"2082104": "how to use age information?\ninstead of consider absolute value, consider something like >30 years and <30 years, etc.\nrefer to radioloogist workflow from the web",
"2082096": "where are the cancer:\nhttps://www.youtube.com/watch?v=bH11DhzJRmA&t=2773s\nBreast Micro-calcifications : All You Need to Know | Mammography | Dr. Terry Minuk\n\nBreast Imaging: Calcifications [Basic Radiology]\nhttps://www.youtube.com/watch?v=7d3nY1ZMr9Q\n",
"2081628": "you basically can learn the relative position encoding for cross attention\n\n",
"2081227": "\"The difficulty of a problem can be indicated by how quickly a human can complete the task.\"\n\nonly 0.5 sec for screening mammogram !!!\n\n[1] Radiologists can detect the ‘gist’ of breast cancer before any overt signs of cancer appear\nhttps://www.nature.com/articles/s41598-018-26100-5\n\n\". Our findings suggest that readers can distinguish patients who were diagnosed with cancer, from individuals without breast cancer (normal category), at above-chance levels based on a half-second glimpse of the mammogram even before any lesion becomes visible on the mammogram\"\n\nhuman expert here can be interpreted as having experience (i.e. see many data before) or having natural ability to spot some image characteristics (i.e. some good network architecture that model data prior distribution well) ",
"2070378": "there is something interesting about the dicom tag ...\n\n\n\n\n\n```\nplt.plot(dicom_tag_df.ContentTime.index[dicom_tag_df.cancer==1], dicom_tag_df.ContentTime[dicom_tag_df.cancer==1],'o' )\nplt.plot(dicom_tag_df.ContentTime.index[dicom_tag_df.cancer==0], dicom_tag_df.ContentTime[dicom_tag_df.cancer==0],'.' )\n\n\n```",
"2069578": "Hi @hengck23 What is different of per image with per patient-laterality?",
"2063377": "@hengck23 I looked into your code - could you explain line \n\n```cancer = torch.nan_to_num(cancer)```\n\nin inference prart? Any issues with nan's and efficientnet?",
"2062488": "did you use pretrained weights?",
"2059564": "Thanks a lot for nice work. @hengck23 \nHow many epochs have you set during training? ",
"2058746": "Hi ! I'd like to ask about the training data. Do you take any method to deal with the imbalance of the dataset? Like using Focal loss or oversampling. Or do you just put the origin data into the training? I am sincerely looking forward to your answer.😀👋",
"2057737": "Interesting topic as usual! Thank you for sharing your experience. \na. Looking into your tables and see numbers - what is your batch size (res -> 1024)? Have you tried to change bs to see results? \nb. Do you use ROI extracted images on resized only?",
"2057219": "Thanks for the post @hengck23. What validation are you using? `StratifiedGroupKFold`?",
"2055504": "@hengck23 Are the 1024 models trained first at lower resolutions or do you go straight to 1024?",
"2172483": "Thank you for the post.",
"2058718": "Thank you very much!"
}
}