First of all, I would like to thank all the people and organizations that have made this Competition possible. In capital letters, THANK YOU to all the TEAMS that with their dedication and effort I hope contribute to improve the diagnosis of prostate cancer and thereby improve people's lives. Indeed, my most sincere congratulations to the WINNERS.
\nI am very happy as you can imagine. In a few lines I share with you a quick overview of my time in this Challenge.
\nI want to comment here what is usually included in the acknowledgments part but I reserve this special section to highlight the work of those competitors who have made my final solution better, 1) because their ability was not present in my initial knowledge or 2) because their performance improves together with the experience of mine. I mean, in no order of priority,
\nI have shared an original copy of my inference kernel without additional cleaning as well as a dataset that includes the necessary weights of each of the models that are used in obtaining the final submission,
\n\nFor all my trainings I used the free TPU resources offered by Kaggle / Google (Colaboratory). Thank you very much.
\nVery closed to Salman's training kernel I just re-ran its code to complete cross validation. I ran a fold up to 60 epochs to see the evolution of the loss and the rest down to 40 epochs.
\nIndividually the behavior of the folds was more or less similar in final loss values (mse) and in the number of times in which it stopped improving. The issue is that its merge did not improve the individual performance of some of them over LB and their performance was also uneven when they were introduced into an external ensemble.
\nThe noise of the labels is a probable cause as already discussed in the discussions or perhaps the sensitivity of the qwk metric to even small variations in mse when its jump to LB.
\nSize Image 256
\nSize Tiles 256
\nTiles 42
\nAugmentation horizonal p=0.5 and vertical p=0.5 flips
\nValidation StratifiedKFold 5 on isup grade classes
\nArch EfficientNetB0
\nConvolutional Base's Weight Imagenet trainable
\nOn Top GlobalAveragePooling2D, Dropout(0.5), Dense(1024)
\nOutput Dense(1) regression objective
\nLoss mean_squared_error
\nOptimizer Adam
\nLeaning Rate 5e-04 init
\nReduce LR decreasing 0.5 with patience 3 epochs
\nSave weights only with best validation loss epochs
\nBatch Size 64
I never tried detecting noisy labels to remove them from training data. In general I am not in favor of losing any existing information, although in principle it could be harmful by elevating the non-regular part of a data generating process. I would rather transform data than remove it.
\nI didn't try either any transformation so I thought about training the models with full dataset in order to prevent the possible existence of more noise in some folds than in others, which probably would be increasing the variability in the inference results.
\nArt Validation appears here and it is when the art of the data scientist enters and it is his instinct that determines the goodness of fit and stability of performance in generalization against new observations. Yes, this is Alchemy.
\nTiles 48
\nValidation Art Validation on instinct
\nArch EfficientNetB0, EfficientNetB1 and EfficientNetB2
\nConvolutional Base's Weight Noisy Student trainable
\nOutput Dense(5,activation='sigmoid) ordinal regression objective
\nLoss sigmoid_cross_entropy_with_logits
\nLeaning Rate custom with 5up, 3sustain, 0.8decay
\nLimits LR 1e-05min, 4e-04max
\nSave weights only with best loss epochs
\nBatch Size 32
\nEpochs 60
Arch Densenet121
\nConvolutional Base's Weight Imagenet trainable
\nEpochs 40
Of the 2 training processes shown above, the following models were available,
\nHaving re-run the Salman kernel, from the public notebooks referenced at the beginning I had,
\nTest Time Augmentation
\nType A: 5xTTA deterministic
\n1xoriginal, 1xTranspose, 1xVerticalFlip, 1xHorizontalFlip, 1xTranspose->VerticalFlip->HorizontalFlip
\nType B: 4xTTA pseudo deterministic
\n1xoriginal, 1xVerticalFlip, 2xHorizontalFlip(p=0.5)->VerticalFlip(p=0.5)
\nType C: 2xTTA random
\n2xHorizontalFlip(p=0.5)->VerticalFlip(p=0)
White Padding Tile Extraction (Qishen modes)
\n1x add zero pad and 1x add 256 pad, that is, 2 different extractions for ALL the images.
(3/10)*Public-Quishen [TTA Type A] \n(3/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/30)*EfficientNetB0-Fold0-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold2-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold4-Training One [TTA Type C] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type C] \n(1/15)*EfficientNetB1-Training Two [TTA Type C] \n(1/15)*EfficientNetB2-Training Two [TTA Type C] \n\n(1/10)*DenseNet121-Training Two [TTA Type B] \nThe random component of the TTAs was not seed (I'll be lucky) and the reproducibility of the results may vary with it. I have just re-run my inference kernel and the results are Private Score 0.92983 (0.92960 original) and Public Score 0.89443 (089352 original).
\nWith this models structure I was only able to test the last day of the competition. For example, this other ensemble got Private Score 0.93047 and Public Score 0.88889, not including random component in TTA.
\n(3.5/10)*Public-Quishen [TTA Type A] \n(3.5/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type A] \n(1/15)*EfficientNetB1-Training Two [TTA Type A] \n(1/15)*EfficientNetB2-Training Two [TTA Type A] \n\n(1/10)*DenseNet121-Training Two [TTA Type A] \nOne more, my last submission and that finished tight after the deadline got Private Score 0.93052 and Public Score 0.89110,
\n(3.5/10)*Public-Quishen [TTA Type A] \n(3.5/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/30)*EfficientNetB0-Fold0-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold2-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold4-Training One [TTA Type C] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type C] \n(1/15)*EfficientNetB1-Training Two [TTA Type C] \n(1/15)*EfficientNetB2-Training Two [TTA Type C] \nBy the way, I still tremble with fear
\nUpdate: No longer!
First of all, I would like to thank all the people and organizations that have made this Competition possible. In capital letters, THANK YOU to all the TEAMS that with their dedication and effort I hope contribute to improve the diagnosis of prostate cancer and thereby improve people's lives. Indeed, my most sincere congratulations to the WINNERS.
\nI am very happy as you can imagine. In a few lines I share with you a quick overview of my time in this Challenge.
\nI want to comment here what is usually included in the acknowledgments part but I reserve this special section to highlight the work of those competitors who have made my final solution better, 1) because their ability was not present in my initial knowledge or 2) because their performance improves together with the experience of mine. I mean, in no order of priority,
\nI have shared an original copy of my inference kernel without additional cleaning as well as a dataset that includes the necessary weights of each of the models that are used in obtaining the final submission,
\n\nFor all my trainings I used the free TPU resources offered by Kaggle / Google (Colaboratory). Thank you very much.
\nVery closed to Salman's training kernel I just re-ran its code to complete cross validation. I ran a fold up to 60 epochs to see the evolution of the loss and the rest down to 40 epochs.
\nIndividually the behavior of the folds was more or less similar in final loss values (mse) and in the number of times in which it stopped improving. The issue is that its merge did not improve the individual performance of some of them over LB and their performance was also uneven when they were introduced into an external ensemble.
\nThe noise of the labels is a probable cause as already discussed in the discussions or perhaps the sensitivity of the qwk metric to even small variations in mse when its jump to LB.
\nSize Image 256
\nSize Tiles 256
\nTiles 42
\nAugmentation horizonal p=0.5 and vertical p=0.5 flips
\nValidation StratifiedKFold 5 on isup grade classes
\nArch EfficientNetB0
\nConvolutional Base's Weight Imagenet trainable
\nOn Top GlobalAveragePooling2D, Dropout(0.5), Dense(1024)
\nOutput Dense(1) regression objective
\nLoss mean_squared_error
\nOptimizer Adam
\nLeaning Rate 5e-04 init
\nReduce LR decreasing 0.5 with patience 3 epochs
\nSave weights only with best validation loss epochs
\nBatch Size 64
I never tried detecting noisy labels to remove them from training data. In general I am not in favor of losing any existing information, although in principle it could be harmful by elevating the non-regular part of a data generating process. I would rather transform data than remove it.
\nI didn't try either any transformation so I thought about training the models with full dataset in order to prevent the possible existence of more noise in some folds than in others, which probably would be increasing the variability in the inference results.
\nArt Validation appears here and it is when the art of the data scientist enters and it is his instinct that determines the goodness of fit and stability of performance in generalization against new observations. Yes, this is Alchemy.
\nTiles 48
\nValidation Art Validation on instinct
\nArch EfficientNetB0, EfficientNetB1 and EfficientNetB2
\nConvolutional Base's Weight Noisy Student trainable
\nOutput Dense(5,activation='sigmoid) ordinal regression objective
\nLoss sigmoid_cross_entropy_with_logits
\nLeaning Rate custom with 5up, 3sustain, 0.8decay
\nLimits LR 1e-05min, 4e-04max
\nSave weights only with best loss epochs
\nBatch Size 32
\nEpochs 60
Arch Densenet121
\nConvolutional Base's Weight Imagenet trainable
\nEpochs 40
Of the 2 training processes shown above, the following models were available,
\nHaving re-run the Salman kernel, from the public notebooks referenced at the beginning I had,
\nTest Time Augmentation
\nType A: 5xTTA deterministic
\n1xoriginal, 1xTranspose, 1xVerticalFlip, 1xHorizontalFlip, 1xTranspose->VerticalFlip->HorizontalFlip
\nType B: 4xTTA pseudo deterministic
\n1xoriginal, 1xVerticalFlip, 2xHorizontalFlip(p=0.5)->VerticalFlip(p=0.5)
\nType C: 2xTTA random
\n2xHorizontalFlip(p=0.5)->VerticalFlip(p=0)
White Padding Tile Extraction (Qishen modes)
\n1x add zero pad and 1x add 256 pad, that is, 2 different extractions for ALL the images.
(3/10)*Public-Quishen [TTA Type A] \n(3/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/30)*EfficientNetB0-Fold0-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold2-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold4-Training One [TTA Type C] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type C] \n(1/15)*EfficientNetB1-Training Two [TTA Type C] \n(1/15)*EfficientNetB2-Training Two [TTA Type C] \n\n(1/10)*DenseNet121-Training Two [TTA Type B] \nThe random component of the TTAs was not seed (I'll be lucky) and the reproducibility of the results may vary with it. I have just re-run my inference kernel and the results are Private Score 0.92983 (0.92960 original) and Public Score 0.89443 (089352 original).
\nWith this models structure I was only able to test the last day of the competition. For example, this other ensemble got Private Score 0.93047 and Public Score 0.88889, not including random component in TTA.
\n(3.5/10)*Public-Quishen [TTA Type A] \n(3.5/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type A] \n(1/15)*EfficientNetB1-Training Two [TTA Type A] \n(1/15)*EfficientNetB2-Training Two [TTA Type A] \n\n(1/10)*DenseNet121-Training Two [TTA Type A] \nOne more, my last submission and that finished tight after the deadline got Private Score 0.93052 and Public Score 0.89110,
\n(3.5/10)*Public-Quishen [TTA Type A] \n(3.5/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/30)*EfficientNetB0-Fold0-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold2-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold4-Training One [TTA Type C] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type C] \n(1/15)*EfficientNetB1-Training Two [TTA Type C] \n(1/15)*EfficientNetB2-Training Two [TTA Type C] \nBy the way, I still tremble with fear
\nUpdate: No longer!
Size Image 256\nSize Tiles 256\nTiles 42\nAugmentation horizonal p=0.5 and vertical p=0.5 flips\nValidation StratifiedKFold 5 on isup grade classes\nArch EfficientNetB0\nConvolutional Base's Weight Imagenet trainable\nOn Top GlobalAveragePooling2D, Dropout(0.5), Dense(1024)\nOutput Dense(1) regression objective\nLoss mean_squared_error\nOptimizer Adam\nLeaning Rate 5e-04 init\nReduce LR decreasing 0.5 with patience 3 epochs\nSave weights only with best validation loss epochs\nBatch Size 64\n\n### Training Two: Art(Instinct) Validation\n\nI never tried detecting noisy labels to remove them from training data. In general I am not in favor of losing any existing information, although in principle it could be harmful by elevating the non-regular part of a data generating process. I would rather transform data than remove it.\n\nI didn't try either any transformation so I thought about training the models with full dataset in order to prevent the possible existence of more noise in some folds than in others, which probably would be increasing the variability in the inference results.\n\nArt Validation appears here and it is when the art of the data scientist enters and it is his instinct that determines the goodness of fit and stability of performance in generalization against new observations. Yes, this is Alchemy.\n\n- **[code-base-training-two-enets](https://www.kaggle.com/coreacasa/code-base-training-two-enets)** file, from which I trained 3 members of the EfficientNet family. Changes on training one training topics:\n\nTiles 48\nValidation Art Validation on instinct\nArch EfficientNetB0, EfficientNetB1 and EfficientNetB2\nConvolutional Base's Weight Noisy Student trainable\nOutput Dense(5,activation='sigmoid) ordinal regression objective\nLoss sigmoid_cross_entropy_with_logits\nLeaning Rate custom with 5up, 3sustain, 0.8decay\nLimits LR 1e-05min, 4e-04max\nSave weights only with best loss epochs\nBatch Size 32\nEpochs 60\n\n- **[code-base-training-two-densenet](https://www.kaggle.com/coreacasa/code-base-training-two-densenet)** file, from which I trained 1 member of the DenseNet family. Changes on training topics of the previous net family:\n\nArch Densenet121\nConvolutional Base's Weight Imagenet trainable\nEpochs 40\n\n### Inference: Diversity of Archs, nTiles and TTAs\n\nOf the 2 training processes shown above, the following models were available,\n1. EfficientNetB0 (5 skf), 42x256x256x3\n2. EfficientNetB0 (1), 48x256x256x3 \n3. EfficientNetB1 (1), 48x256x256x3 \n4. EfficientNetB2 (1), 48x256x256x3 \n5. DenseNet121 (1), 48x256x256x3 \n\nHaving re-run the Salman kernel, from the public notebooks referenced at the beginning I had,\n1. EfficientNetB0 (1 skf), 36x256x256x3 (Qishen Ha) \n2. EfficientNetB1 (1 skf), 36x256x256x3 (RAHUL SINGH INDA)\n \n- **Test Time Augmentation**\nType A: 5xTTA deterministic \n1xoriginal, 1xTranspose, 1xVerticalFlip, 1xHorizontalFlip, 1xTranspose->VerticalFlip->HorizontalFlip\nType B: 4xTTA pseudo deterministic \n1xoriginal, 1xVerticalFlip, 2xHorizontalFlip(p=0.5)->VerticalFlip(p=0.5)\nType C: 2xTTA random \n2xHorizontalFlip(p=0.5)->VerticalFlip(p=0)\n\n- **White Padding Tile Extraction (Qishen modes)**\n1x add zero pad and 1x add 256 pad, that is, 2 different extractions for ALL the images.\n\n### Model Selection and Final Ensemble\n\n(3/10)*Public-Quishen [TTA Type A] \n(3/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/30)*EfficientNetB0-Fold0-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold2-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold4-Training One [TTA Type C] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type C] \n(1/15)*EfficientNetB1-Training Two [TTA Type C] \n(1/15)*EfficientNetB2-Training Two [TTA Type C] \n\n(1/10)*DenseNet121-Training Two [TTA Type B] \n(3.5/10)*Public-Quishen [TTA Type A] \n(3.5/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type A] \n(1/15)*EfficientNetB1-Training Two [TTA Type A] \n(1/15)*EfficientNetB2-Training Two [TTA Type A] \n\n(1/10)*DenseNet121-Training Two [TTA Type A] \n(3.5/10)*Public-Quishen [TTA Type A] \n(3.5/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/30)*EfficientNetB0-Fold0-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold2-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold4-Training One [TTA Type C] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type C] \n(1/15)*EfficientNetB1-Training Two [TTA Type C] \n(1/15)*EfficientNetB2-Training Two [TTA Type C] \n@coreacasa Thanks for sharing the overview. But I couldn't understand the below code
\n`PREDS = (1/5)PREDS + (1/5)PREDS1 + (1/5)PREDS2 + (1/5)PREDS3 + (1/5)*PREDS4
FINAL = np.round( (6/10)PREDS +\n (2/60)np.array(predictions10) + (2/60)np.array(predictions12) + \n (2/60)np.array(predictions20) + (2/60)np.array(predictions22) +\n (2/60)np.array(predictions30) + (2/60)np.array(predictions32) +\n (0.5/10)np.array(predictions40) + (0.5/10)np.array(predictions42) +\n (1/60)np.array(predictions50) + (1/60)np.array(predictions52) +\n (1/60)np.array(predictions60) + (1/60)np.array(predictions62) +\n (1/60)np.array(predictions70) + (1/60)*np.array(predictions72) )
\n`
\nWhat are these numbers 6/10, 2/60,1/60,0.5/10? Would be grateful if you can elaborate on this.
Thanks
", "rawMarkdown": "@coreacasa Thanks for sharing the overview. But I couldn't understand the below code\n`PREDS = (1/5)*PREDS + (1/5)*PREDS1 + (1/5)*PREDS2 + (1/5)*PREDS3 + (1/5)*PREDS4\n\nFINAL = np.round( (6/10)*PREDS +\n (2/60)*np.array(predictions10) + (2/60)*np.array(predictions12) + \n (2/60)*np.array(predictions20) + (2/60)*np.array(predictions22) +\n (2/60)*np.array(predictions30) + (2/60)*np.array(predictions32) +\n (0.5/10)*np.array(predictions40) + (0.5/10)*np.array(predictions42) +\n (1/60)*np.array(predictions50) + (1/60)*np.array(predictions52) +\n (1/60)*np.array(predictions60) + (1/60)*np.array(predictions62) +\n (1/60)*np.array(predictions70) + (1/60)*np.array(predictions72) )\n`\nWhat are these numbers 6/10, 2/60,1/60,0.5/10? Would be grateful if you can elaborate on this.\n\nThanks", "votes": 1, "replies": [ { "id": 1006910, "postDate": "2020-09-11T16:27:32.320Z", "content": "Hi @gaur128,
\nThey are weights about the participation of each of the TTA types in the models or about each of the models in the final solution. They are based on the experience of performance on the leaderboard during the competition.
Hi @coreacasa Thanks for your reply.
\nHow did you come up with these weights? Is there some logic behind it or is it somewhat intuition+experimentation.
There is no math behind it if that is what you mean!
", "rawMarkdown": "There is no math behind it if that is what you mean!" }, { "id": 1007548, "postDate": "2020-09-12T09:02:12.967Z", "content": "Got it. Thanks
", "rawMarkdown": "Got it. Thanks" } ] }, { "id": 945513, "postDate": "2020-07-25T22:39:33.883Z", "content": "@coreacasa Congrats.
", "rawMarkdown": "@coreacasa Congrats. ", "votes": 1, "replies": [ { "id": 946115, "postDate": "2020-07-26T11:17:01.280Z", "content": "Congratulations to you and thanks a lot @micheomaano for sharing your notebooks during the competition, your code is the soul of the code files that are in this solution.
", "rawMarkdown": "Congratulations to you and thanks a lot @micheomaano for sharing your notebooks during the competition, your code is the soul of the code files that are in this solution." } ] }, { "id": 944085, "postDate": "2020-07-24T20:00:31.027Z", "content": "Thank you for sharing @coreacasa ! this is amazing. \nIt is interesting that I had some parameters almost like your code base 1. \nYou had 42 tiles for EfficientNetB0. I found 48 tiles worked best for me. but my batch size was 2. \nI was also struggling with TTA so i will check on your code. \nThank you again for sharing! this is very educational and will help us all improve! \nCheers!
", "rawMarkdown": "Thank you for sharing @coreacasa ! this is amazing. \nIt is interesting that I had some parameters almost like your code base 1. \nYou had 42 tiles for EfficientNetB0. I found 48 tiles worked best for me. but my batch size was 2. \nI was also struggling with TTA so i will check on your code. \nThank you again for sharing! this is very educational and will help us all improve! \nCheers! ", "votes": 1, "replies": [ { "id": 944887, "postDate": "2020-07-25T12:31:50.743Z", "content": "Thank you @mpsampat !!, do not hesitate to ask any questions about it.
", "rawMarkdown": "Thank you @mpsampat !!, do not hesitate to ask any questions about it.", "votes": 1 } ] }, { "id": 944181, "postDate": "2020-07-24T22:39:46.110Z", "content": "Congrats Manuel on a great solution, on solo Gold, and becoming a Competition Master!
", "rawMarkdown": "Congrats Manuel on a great solution, on solo Gold, and becoming a Competition Master!", "votes": 2, "replies": [ { "id": 944882, "postDate": "2020-07-25T12:27:48.633Z", "content": "Thank you very much @cdeotte, it is an honor to read your congratulations here!
", "rawMarkdown": "Thank you very much @cdeotte, it is an honor to read your congratulations here!", "votes": 1 } ] } ], "comments": [ { "id": 1006737, "author_name": "Gaurav Yadav", "author_url": "", "post_date": "2020-09-11T14:16:28.400000", "content": "@coreacasa Thanks for sharing the overview. But I couldn't understand the below code
\n`PREDS = (1/5)PREDS + (1/5)PREDS1 + (1/5)PREDS2 + (1/5)PREDS3 + (1/5)*PREDS4
FINAL = np.round( (6/10)PREDS +\n (2/60)np.array(predictions10) + (2/60)np.array(predictions12) + \n (2/60)np.array(predictions20) + (2/60)np.array(predictions22) +\n (2/60)np.array(predictions30) + (2/60)np.array(predictions32) +\n (0.5/10)np.array(predictions40) + (0.5/10)np.array(predictions42) +\n (1/60)np.array(predictions50) + (1/60)np.array(predictions52) +\n (1/60)np.array(predictions60) + (1/60)np.array(predictions62) +\n (1/60)np.array(predictions70) + (1/60)*np.array(predictions72) )
\n`
\nWhat are these numbers 6/10, 2/60,1/60,0.5/10? Would be grateful if you can elaborate on this.
Thanks
", "votes": 1, "replies": [ { "id": 1006910, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-09-11T16:27:32.320000", "content": "Hi @gaur128,
\nThey are weights about the participation of each of the TTA types in the models or about each of the models in the final solution. They are based on the experience of performance on the leaderboard during the competition.
Hi @coreacasa Thanks for your reply.
\nHow did you come up with these weights? Is there some logic behind it or is it somewhat intuition+experimentation.
There is no math behind it if that is what you mean!
", "votes": 0, "replies": [] }, { "id": 1007548, "author_name": "Gaurav Yadav", "author_url": "", "post_date": "2020-09-12T09:02:12.967000", "content": "Got it. Thanks
", "votes": 0, "replies": [] } ] }, { "id": 945513, "author_name": "Salman", "author_url": "", "post_date": "2020-07-25T22:39:33.883000", "content": "@coreacasa Congrats.
", "votes": 1, "replies": [ { "id": 946115, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-07-26T11:17:01.280000", "content": "Congratulations to you and thanks a lot @micheomaano for sharing your notebooks during the competition, your code is the soul of the code files that are in this solution.
", "votes": 0, "replies": [] } ] }, { "id": 944085, "author_name": "Mehul Sampat", "author_url": "", "post_date": "2020-07-24T20:00:31.027000", "content": "Thank you for sharing @coreacasa ! this is amazing. \nIt is interesting that I had some parameters almost like your code base 1. \nYou had 42 tiles for EfficientNetB0. I found 48 tiles worked best for me. but my batch size was 2. \nI was also struggling with TTA so i will check on your code. \nThank you again for sharing! this is very educational and will help us all improve! \nCheers!
", "votes": 1, "replies": [ { "id": 944887, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-07-25T12:31:50.743000", "content": "Thank you @mpsampat !!, do not hesitate to ask any questions about it.
", "votes": 1, "replies": [] } ] }, { "id": 944181, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-24T22:39:46.110000", "content": "Congrats Manuel on a great solution, on solo Gold, and becoming a Competition Master!
", "votes": 2, "replies": [ { "id": 944882, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-07-25T12:27:48.633000", "content": "Thank you very much @cdeotte, it is an honor to read your congratulations here!
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "943758": "First of all, I would like to thank all the people and organizations that have made this Competition possible. In capital letters, THANK YOU to all the TEAMS that with their dedication and effort I hope contribute to improve the diagnosis of prostate cancer and thereby improve people's lives. Indeed, my most sincere congratulations to the WINNERS.\n\nI am very happy as you can imagine. In a few lines I share with you a quick overview of my time in this Challenge.\n\n### Kaggle Learning\nI want to comment here what is usually included in the acknowledgments part but I reserve this special section to highlight the work of those competitors who have made my final solution better, 1) because their ability was not present in my initial knowledge or 2) because their performance improves together with the experience of mine. I mean, in no order of priority,\n\n- **(Salman)** @micheomaano:\n1. [Dataset tf-record-256-56-48](https://www.kaggle.com/micheomaano/tf-record-256-256-48)\n2. [TPU Training Tensorflow Iafoos Method 42x256x256x3](https://www.kaggle.com/micheomaano/tpu-training-tensorflow-iafoos-method-42x256x256x3)\n3. [Pandas 42x256x256x3 Inference](https://www.kaggle.com/micheomaano/pandas-42x256x256x3-inference)\n\n- **(Qishen Ha)** @haqishen:\n1. [Train EfficientNet-B0 w/ 36 tiles_256 [LB0.87]](https://www.kaggle.com/haqishen/train-efficientnet-b0-w-36-tiles-256-lb0-87)\n2. [PANDA Inference w/ 36 tiles_256](https://www.kaggle.com/haqishen/panda-inference-w-36-tiles-256)\n\n- **(RAHUL SINGH INDA)** @rsinda:\n1. [Panda Inference EfficientNet-b1](https://www.kaggle.com/rsinda/panda-inference-efficientnet-b1)\n\n- **(Iafoss)** @iafoss: The Best Accelerator in the Competition, ahead of TPUs.\n\n###Submission Notebook \n \nI have shared an original copy of my inference kernel without additional cleaning as well as a dataset that includes the necessary weights of each of the models that are used in obtaining the final submission,\n \n- [Quick Save Inference](https://www.kaggle.com/coreacasa/12th-place-solution-quick-save-inference)\n- [Dataset Model Weights for Inference](https://www.kaggle.com/coreacasa/pandaenetb042x256x256x3)\n \n### [TPU] Kaggle/Google(Colaboratory)\nFor all my trainings I used the free TPU resources offered by Kaggle / Google (Colaboratory). Thank you very much.\n\n### Training One: My only approach to validation\n\nVery closed to Salman's training kernel I just re-ran its code to complete cross validation. I ran a fold up to 60 epochs to see the evolution of the loss and the rest down to 40 epochs.\n\nIndividually the behavior of the folds was more or less similar in final loss values (mse) and in the number of times in which it stopped improving. The issue is that its merge did not improve the individual performance of some of them over LB and their performance was also uneven when they were introduced into an external ensemble.\n\nThe noise of the labels is a probable cause as already discussed in the discussions or perhaps the sensitivity of the qwk metric to even small variations in mse when its jump to LB.\n\n- **[code-base-training-one](https://www.kaggle.com/coreacasa/code-base-training-one)** file, training topics:\n\nSize Image 256\nSize Tiles 256\nTiles 42\nAugmentation horizonal p=0.5 and vertical p=0.5 flips\nValidation StratifiedKFold 5 on isup grade classes\nArch EfficientNetB0\nConvolutional Base's Weight Imagenet trainable\nOn Top GlobalAveragePooling2D, Dropout(0.5), Dense(1024)\nOutput Dense(1) regression objective\nLoss mean_squared_error\nOptimizer Adam\nLeaning Rate 5e-04 init\nReduce LR decreasing 0.5 with patience 3 epochs\nSave weights only with best validation loss epochs\nBatch Size 64\n\n### Training Two: Art(Instinct) Validation\n\nI never tried detecting noisy labels to remove them from training data. In general I am not in favor of losing any existing information, although in principle it could be harmful by elevating the non-regular part of a data generating process. I would rather transform data than remove it.\n\nI didn't try either any transformation so I thought about training the models with full dataset in order to prevent the possible existence of more noise in some folds than in others, which probably would be increasing the variability in the inference results.\n\nArt Validation appears here and it is when the art of the data scientist enters and it is his instinct that determines the goodness of fit and stability of performance in generalization against new observations. Yes, this is Alchemy.\n\n- **[code-base-training-two-enets](https://www.kaggle.com/coreacasa/code-base-training-two-enets)** file, from which I trained 3 members of the EfficientNet family. Changes on training one training topics:\n\nTiles 48\nValidation Art Validation on instinct\nArch EfficientNetB0, EfficientNetB1 and EfficientNetB2\nConvolutional Base's Weight Noisy Student trainable\nOutput Dense(5,activation='sigmoid) ordinal regression objective\nLoss sigmoid_cross_entropy_with_logits\nLeaning Rate custom with 5up, 3sustain, 0.8decay\nLimits LR 1e-05min, 4e-04max\nSave weights only with best loss epochs\nBatch Size 32\nEpochs 60\n\n- **[code-base-training-two-densenet](https://www.kaggle.com/coreacasa/code-base-training-two-densenet)** file, from which I trained 1 member of the DenseNet family. Changes on training topics of the previous net family:\n\nArch Densenet121\nConvolutional Base's Weight Imagenet trainable\nEpochs 40\n\n### Inference: Diversity of Archs, nTiles and TTAs\n\nOf the 2 training processes shown above, the following models were available,\n1. EfficientNetB0 (5 skf), 42x256x256x3\n2. EfficientNetB0 (1), 48x256x256x3 \n3. EfficientNetB1 (1), 48x256x256x3 \n4. EfficientNetB2 (1), 48x256x256x3 \n5. DenseNet121 (1), 48x256x256x3 \n\nHaving re-run the Salman kernel, from the public notebooks referenced at the beginning I had,\n1. EfficientNetB0 (1 skf), 36x256x256x3 (Qishen Ha) \n2. EfficientNetB1 (1 skf), 36x256x256x3 (RAHUL SINGH INDA)\n \n- **Test Time Augmentation**\nType A: 5xTTA deterministic \n1xoriginal, 1xTranspose, 1xVerticalFlip, 1xHorizontalFlip, 1xTranspose->VerticalFlip->HorizontalFlip\nType B: 4xTTA pseudo deterministic \n1xoriginal, 1xVerticalFlip, 2xHorizontalFlip(p=0.5)->VerticalFlip(p=0.5)\nType C: 2xTTA random \n2xHorizontalFlip(p=0.5)->VerticalFlip(p=0)\n\n- **White Padding Tile Extraction (Qishen modes)**\n1x add zero pad and 1x add 256 pad, that is, 2 different extractions for ALL the images.\n\n### Model Selection and Final Ensemble\n\n(3/10)*Public-Quishen [TTA Type A] \n(3/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/30)*EfficientNetB0-Fold0-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold2-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold4-Training One [TTA Type C] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type C] \n(1/15)*EfficientNetB1-Training Two [TTA Type C] \n(1/15)*EfficientNetB2-Training Two [TTA Type C] \n\n(1/10)*DenseNet121-Training Two [TTA Type B] \n(3.5/10)*Public-Quishen [TTA Type A] \n(3.5/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type A] \n(1/15)*EfficientNetB1-Training Two [TTA Type A] \n(1/15)*EfficientNetB2-Training Two [TTA Type A] \n\n(1/10)*DenseNet121-Training Two [TTA Type A] \n(3.5/10)*Public-Quishen [TTA Type A] \n(3.5/10)*Public-RAHUL SINGH INDA [TTA Type A] \n\n(1/30)*EfficientNetB0-Fold0-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold2-Training One [TTA Type C] \n(1/30)*EfficientNetB0-Fold4-Training One [TTA Type C] \n\n(1/15)*EfficientNetB0-Training Two [TTA Type C] \n(1/15)*EfficientNetB1-Training Two [TTA Type C] \n(1/15)*EfficientNetB2-Training Two [TTA Type C] \n