![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3830852%2F92b091a4351296e585ce247fbd6008c3%2FRSNA_LB063_results2.png?generation=1675771723070708&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3830852%2Fdc4e15caabae71248a9097dfeaebcc08%2FRSNA_LB063_results.png?generation=1675771667026332&alt=media\")
there is a huge score gap between my cv and my lb CV: 0.395 (after mean aggregation ) LB: 0.56 (thres=0.20, no strict tunning).
\nI am very concern about shake down.
\nHow is your score gap??
\nthanks.
my cv strategy is below. (n_splits=3)
\nI refer this notebook
strat_cols = [\n 'laterality', 'view', 'biopsy','invasive', 'BIRADS', 'age_bin',\n 'implant', 'density','machine_id', 'difficult_negative_case',\n 'cancer',\n]\n\ndf['stratify'] = ''\nfor col in strat_cols:\n df['stratify'] += df[col].astype(str)\n\nkfold = StratifiedGroupKFold(n_splits=Config['SPLITS'],random_state=Config[\"seed\"],shuffle=True)\nfor fold_, (train_idx, valid_idx) in enumerate(kfold.split(df, df['stratify'].values, df['patient_id'].values)):`\nupdate2 : CV score details (n_splits=3)
\nall data：precision=0.485 recall=0.333 pf1=0.395 thres=0.5000 roc_auc=0.873
\nsite_id=1 : precision=0.329 recall=0.287 f1=0.307 thres=0.4500 roc_auc=0.826
\nsite_id=2 : precision=0.639 recall=0.416 f1=0.504 thres=0.5000 roc_auc=0.906
there is a huge score gap between my cv and my lb CV: 0.395 (after mean aggregation ) LB: 0.56 (thres=0.20, no strict tunning).
\nI am very concern about shake down.
\nHow is your score gap??
\nthanks.
my cv strategy is below. (n_splits=3)
\nI refer this notebook
strat_cols = [\n 'laterality', 'view', 'biopsy','invasive', 'BIRADS', 'age_bin',\n 'implant', 'density','machine_id', 'difficult_negative_case',\n 'cancer',\n]\n\ndf['stratify'] = ''\nfor col in strat_cols:\n df['stratify'] += df[col].astype(str)\n\nkfold = StratifiedGroupKFold(n_splits=Config['SPLITS'],random_state=Config[\"seed\"],shuffle=True)\nfor fold_, (train_idx, valid_idx) in enumerate(kfold.split(df, df['stratify'].values, df['patient_id'].values)):`\nupdate2 : CV score details (n_splits=3)
\nall data：precision=0.485 recall=0.333 pf1=0.395 thres=0.5000 roc_auc=0.873
\nsite_id=1 : precision=0.329 recall=0.287 f1=0.307 thres=0.4500 roc_auc=0.826
\nsite_id=2 : precision=0.639 recall=0.416 f1=0.504 thres=0.5000 roc_auc=0.906
Finally my LB is more than 0.60.
\nBut my cv score is 0.42.
\nThe gap is too big !!!!!!
any post-processing?
", "rawMarkdown": "any post-processing?", "replies": [ { "id": 2131955, "postDate": "2023-02-06T14:02:18.667Z", "content": "No post-processing
", "rawMarkdown": "No post-processing" } ] }, { "id": 2132155, "postDate": "2023-02-06T16:23:15.843Z", "content": "Your LB score is very impressive. Could you tell me your image size did you use?
", "rawMarkdown": "Your LB score is very impressive. Could you tell me your image size did you use?", "replies": [ { "id": 2133133, "postDate": "2023-02-07T09:04:35.323Z", "content": "Thanks.
\ncurrent my image size is 1536x960
Amazing result! Congratulations crossing 0.6x.
\nDo you still use effnet?
BTW:
\nI have local CV: 0.54 now … and this model is only 0.51 on LB 😂
Thank you!
\nI still use the effnet.
You have a good local CV!
\nI don't still know why my CV is too low… (it bothers me)
How is the image level CV score for your model?
", "rawMarkdown": "Thank you!\nI still use the effnet.\n\nYou have a good local CV!\nI don't still know why my CV is too low... (it bothers me)\n\nHow is the image level CV score for your model?" }, { "id": 2133283, "postDate": "2023-02-07T10:44:32.670Z", "content": "For single image 0.4.
", "rawMarkdown": "For single image 0.4.", "votes": 1 }, { "id": 2133298, "postDate": "2023-02-07T10:51:47.230Z", "content": "it's a also higher score than my image level cv score 0.356.
\nthanks for your sharing.
If it is possible please share result (validator result). It would be great. Especially I am looking now on predicition dynamics (chart with probability distribution).
\nAs far as I understand you blend 3 models (3 folds) for final solution?
", "rawMarkdown": "If it is possible please share result (validator result). It would be great. Especially I am looking now on predicition dynamics (chart with probability distribution).\n\nAs far as I understand you blend 3 models (3 folds) for final solution?" }, { "id": 2133432, "postDate": "2023-02-07T12:11:27.647Z", "content": "I share you my best cv result.
\nYes, I just blended 3 models (3folds).
\n
Thank you a lot.👍
\nStats you provided are from one model or blended?
It's blended from 3 models (3folds)
", "rawMarkdown": "It's blended from 3 models (3folds)", "votes": 1 }, { "id": 2134843, "postDate": "2023-02-08T09:32:32.167Z", "content": "@taruto1215 how many epochs do you train model? Or from what epoch do you choose models from?
", "rawMarkdown": "@taruto1215 how many epochs do you train model? Or from what epoch do you choose models from? " }, { "id": 2134858, "postDate": "2023-02-08T09:39:15.480Z", "content": "@taruto1215 may i know what validation pf1,auc and thresholded pf1 you get for each fold? thanks
", "rawMarkdown": "@taruto1215 may i know what validation pf1,auc and thresholded pf1 you get for each fold? thanks" }, { "id": 2134978, "postDate": "2023-02-08T11:21:40.770Z", "content": "@remekkinas
\nI train model 5 epochs.
\nthen I choose a model has a best image-level f1 score.
\nBut almost of them has the best score at the 5 epoch.
@mobassir
\nThis is my validation results for each fold.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3830852%2F3e806521970616e694389151d535198b%2FRSNA_results_each_folds.png?generation=1675856167389336&alt=media\")
i ahve been trying another experiment. if you are training too many epoch, then the predicted distribution will be too sharp (i.e. overfitting).
\nhence one way is to stop early at, say 5 epoch.
\nwe train many of differenet models.
then we freeze them and use them as feature extractor.
\nwe use giba trick (winner of petfinder): https://developer.nvidia.com/blog/fast-fine-tuning-of-ai-transformers-using-rapids-machine-learning/
\nhttps://www.kaggle.com/c/petfinder-pawpularity-score/discussion/301686
but instead, we create ensmble for multi-view model. each image is now treated as a feature map in sequence, and trsnformer is used to fuse the feature of different view, together with other attribute like age, site id.
", "rawMarkdown": "i ahve been trying another experiment. if you are training too many epoch, then the predicted distribution will be too sharp (i.e. overfitting).\n\nhence one way is to stop early at, say 5 epoch.\nwe train many of differenet models.\n\nthen we freeze them and use them as feature extractor.\nwe use giba trick (winner of petfinder): https://developer.nvidia.com/blog/fast-fine-tuning-of-ai-transformers-using-rapids-machine-learning/\nhttps://www.kaggle.com/c/petfinder-pawpularity-score/discussion/301686\n\nbut instead, we create ensmble for multi-view model. each image is now treated as a feature map in sequence, and trsnformer is used to fuse the feature of different view, together with other attribute like age, site id.", "votes": 3 }, { "id": 2135314, "postDate": "2023-02-08T14:59:35.790Z", "content": "Thanks I’ll check it
\nBut from tomorrow I’ll be very busy at my job . I will not have more time to spend Kaggle than now hahaha😂
Is it a single fold model or an ensemble ?
", "rawMarkdown": "Is it a single fold model or an ensemble ?", "votes": 1, "replies": [ { "id": 2133145, "postDate": "2023-02-07T09:15:00.843Z", "content": "Number of folds is 3.
\nso I used 3 models for submission.
Hi! @taruto1215 Can you tell me how do you use 3 models in your inference notebook to make a submission?
", "rawMarkdown": "Hi! @taruto1215 Can you tell me how do you use 3 models in your inference notebook to make a submission?" } ] }, { "id": 2132927, "postDate": "2023-02-07T06:04:44.393Z", "content": "5 fold
\nCV:0.416
\nLB:0.58
\nMaybe public leaderboard data are simpler than train data
It's the same symptom as mine haha.
\nHow about your ROC_AUC for CV??
I guess so too for the public leaderboard data.
\nBut for the private leaderboard data …
\nI hope it's same.
ROC_AUC(group by mean) ~0.88
", "rawMarkdown": "ROC_AUC(group by mean) ~0.88", "votes": 1 }, { "id": 2133293, "postDate": "2023-02-07T10:49:20.730Z", "content": "thanks for your sharing.
\nmy ROC_AUC (group by mean) ~0.896
\nI think roc_auc scores has correlation against LB scores.
The same here - ROV_OUC for me is over 0.91 and LB is lower then yours. But I am still talking about single model in my case. All blending solutions failed for me.
", "rawMarkdown": "The same here - ROV_OUC for me is over 0.91 and LB is lower then yours. But I am still talking about single model in my case. All blending solutions failed for me.", "votes": 2 }, { "id": 2133442, "postDate": "2023-02-07T12:18:47.177Z", "content": "The ROC_AUC score is also good!
\nPerhaps I should suspect that I am overfitting to LB.
why do you talking about only single model?
\nI think we should check oof result.
To be honest? I do not know at this time. Fianally I will sub two solution: single model and blend.
", "rawMarkdown": "To be honest? I do not know at this time. Fianally I will sub two solution: single model and blend. ", "votes": 1 }, { "id": 2142767, "postDate": "2023-02-13T18:58:58.997Z", "content": "\nI want to follow up on this, my fold 0 results seem good with CV .52 (mean, LB 0.52) similar to the results you posted before but my fold 1 results CV 0.43 (mean), and fold 2 CV 0.44. Did you see similar results on your folds, it seems like your LB is still the same as your single model. Could this be just different folds (or luck as a matter of a fact) fit better on the public LB? Shake up could be possible if others are experiencing this.
", "rawMarkdown": "@remekkinas \n\nI want to follow up on this, my fold 0 results seem good with CV .52 (mean, LB 0.52) similar to the results you posted before but my fold 1 results CV 0.43 (mean), and fold 2 CV 0.44. Did you see similar results on your folds, it seems like your LB is still the same as your single model. Could this be just different folds (or luck as a matter of a fact) fit better on the public LB? Shake up could be possible if others are experiencing this." } ] } ] }, { "id": 2121189, "postDate": "2023-01-30T05:38:09.993Z", "content": "Hi @taruto1215 , how do you get your CV? I mean the LB is calculated by laterality-level, if you train and validate by images, you will get the image-level score. In my experiments, the laterality-level f1 is always much higher than the image-level f1.
", "rawMarkdown": "Hi @taruto1215 , how do you get your CV? I mean the LB is calculated by laterality-level, if you train and validate by images, you will get the image-level score. In my experiments, the laterality-level f1 is always much higher than the image-level f1.", "votes": 5, "replies": [ { "id": 2121404, "postDate": "2023-01-30T09:42:15.230Z", "content": "@forcewithme
\nThanks for your reply.
my CV was calculated by laterality-level.
\nFor the image-level F1 is ~0.35.
i am not very sure if i were correct. but for this competition there may be other ways to reduce shakeup by NOT only relying on LB vs CV.
\ne.g. my target is to make a model:
i am coming out with many such rules based on my observation.
\nit is like regularization in machine learning, when you loss is not reliable, you need to think of other \"sensible\" conditions
", "rawMarkdown": "i am not very sure if i were correct. but for this competition there may be other ways to reduce shakeup by NOT only relying on LB vs CV.\ne.g. my target is to make a model:\n1. have recall at least 0.50 for both site1 and site2\n2. have f1 of at least 0.50 for both site1 and site2 \n3. threshold not larger than 0.55\n4 . .....\n\ni am coming out with many such rules based on my observation.\n\nit is like regularization in machine learning, when you loss is not reliable, you need to think of other \"sensible\" conditions", "votes": 3, "replies": [ { "id": 2121638, "postDate": "2023-01-30T13:09:12.357Z", "content": "Thanks for sharing your ideas.
\nWe should approach this competition with that perspective for the remaining month.
beware if you aggregate by mean. it may be wrong method.
\nfor example assume
\npatient1 (cancer):
\nimage1 = 0.9
\nimage2 (occluded)=0.1
patient2 (cancer):
\nimage1 = 0.9
\nimage2 (occluded)=0.1
\nimage3 (occluded)=0.1
\nimage4 (occluded)=0.1
the score for patient 2 is divided out.
\nbut max is no better
\npatient1 (cancer):
\nimage1 = 0.7
\nimage2 =0.8
patient1 (no cancer):
\nimage1 = 0.1
\nimage2 =0.1
\nimage3 = 0.7 (noise)
\nimage4 =0.1
I put my code to aggregrate.
\nI think there is no mistake.
df_merge_groupby = df_oof[['prediction_id','site_id', 'patient_id', 'laterality', 'cancer', 'preds']].groupby(\n ['patient_id', 'laterality']).mean().reset_index()
\n", "rawMarkdown": "I put my code to aggregrate.\nI think there is no mistake.\n\n`df_merge_groupby = df_oof[['prediction_id','site_id', 'patient_id', 'laterality', 'cancer', 'preds']].groupby(\n ['patient_id', 'laterality']).mean().reset_index() `\n\npatient2 (cancer):
\n
\n image1 = 0.9
\n image2 (occluded)=0.1
\n image3 (occluded)=0.1
\n image4 (occluded)=0.1
you can try mean of top 3.
\nbut the correct way is to learn an aggregation model, which is multiple image prediction MIP at my post.
there is no definite way of mean,max or mean of top3. different aggregation works for different cases, that is why it is better to learn a function.
", "rawMarkdown": "you can try mean of top 3.\nbut the correct way is to learn an aggregation model, which is multiple image prediction MIP at my post.\n\n---\nthere is no definite way of mean,max or mean of top3. different aggregation works for different cases, that is why it is better to learn a function.\n\n\n", "votes": 1 }, { "id": 2121513, "postDate": "2023-01-30T11:27:06.137Z", "content": "you can also check @remekkinas discussion at : https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/377654#2113462
", "rawMarkdown": "you can also check @remekkinas discussion at : https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/377654#2113462", "votes": 2 }, { "id": 2121527, "postDate": "2023-01-30T11:42:51.167Z", "content": "You are right.
\nFor me, mean of top 3 boosted my CV at 0.005.
Thanks for the advice.
", "rawMarkdown": "You are right.\nFor me, mean of top 3 boosted my CV at 0.005.\n\nThanks for the advice." } ] }, { "id": 2132925, "postDate": "2023-02-07T05:54:08.740Z", "content": "Hi, I have a doubt regarding the best way to submit.
\nActually I have 4 splits, CV 0.42 and LB 0.52, for every model I have also a best_thr on the mean aggregation.
\nFor the prediction of every row I get the mean of the 4 models (trained on the 4 splits):
\np(pid, imageid) = pm = (p1+p2+p3+p4)/4\n
after that I aggregate with the mean using groupby() and after I just predict using a threshold (here the question ) that is the the mean of the 4 thresholds
\n\ntm=(t1+t2+t3+t4)/4\n
If I tried a different threshold I get better results on LB (I think that is normal beacuse it's like overfit the LB).
\nI tried the same procedure with the max aggregation instead the mean aggregation, with some improvements but depends from the models.
\nDo you choose the best threshold in this way? 🤔
\nThanks in advance
I just choose the threshold that was best for my CV, no special tunning for LB.
\nThanks.
interesting, LB0.59 is my training score, while my validation score is lower. kaggle LB score is close to my training score.
\ncould matching train score = LB score be a way to fight generalisation?
\n(of course your have to do early exit)
asuming the top kagglers did not overfit, you can train until train score is around 0.65
", "rawMarkdown": "interesting, LB0.59 is my training score, while my validation score is lower. kaggle LB score is close to my training score.\n\ncould matching train score = LB score be a way to fight generalisation?\n(of course your have to do early exit)\n\nasuming the top kagglers did not overfit, you can train until train score is around 0.65", "votes": 1, "replies": [ { "id": 2137447, "postDate": "2023-02-10T02:38:46.490Z", "content": "on a side note since the server compute probablistic fscore, it is quite easily to overfit public test or probe the public test distribution for some divided bins
\ne.g.
1. submit raw probabiliy and get LB score\n2. for th = 0.1,0.2,0.3 .....\nset probabiliy>th to random and submit, \nor set set probabiliy<th to random and submit \nor set set probabiliy<th1 and set set probabiliy>th2 to random and submit \nHello, Which model did you use?
", "rawMarkdown": "Hello, Which model did you use?", "votes": 1, "replies": [ { "id": 2123206, "postDate": "2023-01-31T11:34:16.250Z", "content": "Hello, I used the efficientnetv2_s
", "rawMarkdown": "Hello, I used the efficientnetv2_s", "votes": 2, "replies": [ { "id": 2123496, "postDate": "2023-01-31T14:51:48.270Z", "content": "Thank you! I tried efficientnet_v2 with size 1024x512 but pf1 only around 0.1-0.15
", "rawMarkdown": "Thank you! I tried efficientnet_v2 with size 1024x512 but pf1 only around 0.1-0.15", "votes": 1 } ] } ] }, { "id": 2121246, "postDate": "2023-01-30T06:47:17.847Z", "content": "site_id=1 : precision=0.329 recall=0.287 f1=0.307 thres=0.4500 roc_auc=0.637\nsite_id=2 : precision=0.639 recall=0.416 f1=0.504 thres=0.5000 roc_auc=0.706\ni am interested in the score of site1 and site2 submitted separately.
\nstable f1 if
\n1) precision is close to recall
\n2) precision is very high
stable LB if ?
\n1)site1 close to site2
high threshold : maybe overfitting
\nnote: stable means consistent (i.e. less shakeup) but not necessarily optimum
\nCV: 0.395 LB: 0.56
\n\"I am very concern about shake down.
\nHow is your score gap??\"
but even if CV close to LB doesn't guarantee no shakeup
", "rawMarkdown": "```\nsite_id=1 : precision=0.329 recall=0.287 f1=0.307 thres=0.4500 roc_auc=0.637\nsite_id=2 : precision=0.639 recall=0.416 f1=0.504 thres=0.5000 roc_auc=0.706\n```\n\ni am interested in the score of site1 and site2 submitted separately.\n\n---\nstable f1 if\n1) precision is close to recall \n2) precision is very high\n\nstable LB if ?\n1)site1 close to site2\n\nhigh threshold : maybe overfitting\n\nnote: stable means consistent (i.e. less shakeup) but not necessarily optimum\n\n---\nCV: 0.395 LB: 0.56\n\n\"I am very concern about shake down.\nHow is your score gap??\"\n\nbut even if CV close to LB doesn't guarantee no shakeup ", "votes": 1, "replies": [ { "id": 2121420, "postDate": "2023-01-30T09:55:10.267Z", "content": "Sorry my ROC_AUC was wrong (I made a fundamental code mistake). I wrote up the correct version above.
", "rawMarkdown": "Sorry my ROC_AUC was wrong (I made a fundamental code mistake). I wrote up the correct version above.\n" }, { "id": 2121431, "postDate": "2023-01-30T10:04:11.357Z", "content": "I used your code to generate results.
\nalso there is a gap between site1 and site2.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3830852%2F0d27d318170899acb50b28639a5f9c79%2FRNSA_results.png?generation=1675072875545524&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3830852%2F96c27c9cc7d78865e123a7cc3345b124%2FRNSA_results2.png?generation=1675072914143213&alt=media\")
How that possible??
\nProbably overfitted to hidden test set?
I don't think roc_auc about 0.6~0.7 is good performance.
", "rawMarkdown": "How that possible??\nProbably overfitted to hidden test set?\n\nI don't think roc_auc about 0.6~0.7 is good performance.", "votes": 1, "replies": [ { "id": 2121400, "postDate": "2023-01-30T09:40:04.787Z", "content": "I just found I made a fundamental mistakes
\nMy CV ROC_AUC is ~0.87.
\nThanks.
me too! cv:0.3 lb:0.46 threshold=0.89
", "rawMarkdown": "me too! cv:0.3 lb:0.46 threshold=0.89 ", "votes": 1, "replies": [ { "id": 2121035, "postDate": "2023-01-30T02:27:42.390Z", "content": "Thanks for your sharing
", "rawMarkdown": "Thanks for your sharing", "votes": 1 } ] }, { "id": 2121011, "postDate": "2023-01-30T01:15:29.013Z", "content": "did you try same aggregation in your cv (mean, max etc..) as you do for the pred_ids during subm?
", "rawMarkdown": "did you try same aggregation in your cv (mean, max etc..) as you do for the pred_ids during subm?", "votes": 1, "replies": [ { "id": 2121029, "postDate": "2023-01-30T02:15:46.560Z", "content": "Yes, my CV is after mean aggregation.
\nHow is your score gap? No gap?
\nThanks
CV 0.5, LB 0.53
", "rawMarkdown": "CV 0.5, LB 0.53", "votes": 2 }, { "id": 2121038, "postDate": "2023-01-30T02:31:24.297Z", "content": "Good! Almost same scores!
\nHow is your CV strategy?
\nIs it same as me?
\nThanks.
very similar, but 5 splits
", "rawMarkdown": "very similar, but 5 splits", "votes": 1 }, { "id": 2121505, "postDate": "2023-01-30T11:16:50.890Z", "content": "Huh, I maybe need more splits.
\nThanks for your sharing.
good job!
\ncurrently all team with LB score above 0.60 has good chance of wining.
\ndo spend time to:
simulate leader board results. how one or few change of pos samples can change your validation score.
\nthis enable you to interprete the LB score correctly and estimate shakeup
i still think that mean() aggreataion is not good unless your classifier has very low false positive rate.
\nso you may want to spend time to think about how to fuse results.
remember !!!!
\nit is the private hidden score that we are interested .you should try to predict the private score for every submission you make !!!
Thank you for your advice.
\nYour advices help me(us) a lot.
\nYes I also am thinking about how to aggregrate.
\n\n1 simulate leader board results. how one or few change of pos samples can change your validation score.
\n
\n this enable you to interprete the LB score correctly and estimate shakeup
You means delete a few pos samples then train model and check the influence (score) ?
\nThanks
", "rawMarkdown": "Thank you for your advice.\nYour advices help me(us) a lot.\nYes I also am thinking about how to aggregrate.\n\n>1 simulate leader board results. how one or few change of pos samples can change your validation score.\nthis enable you to interprete the LB score correctly and estimate shakeup\n\nYou means delete a few pos samples then train model and check the influence (score) ?\n\nThanks", "replies": [ { "id": 2133450, "postDate": "2023-02-07T12:25:44.813Z", "content": "somtheing like below.
\nnow you have already make prediction and predicted score and true label.
\nuse 100% of predicted score to compute cv
\nuse 90% of predicted score (repeat over a few time)
\nuse 80%
is the change stable?
\nideally, remove of one sample from the predicted score should not changes the CV metric. but is it true?
", "rawMarkdown": "somtheing like below.\n\nnow you have already make prediction and predicted score and true label.\nuse 100% of predicted score to compute cv\nuse 90% of predicted score (repeat over a few time)\nuse 80%\n\nis the change stable?\n\nideally, remove of one sample from the predicted score should not changes the CV metric. but is it true?", "votes": 1 }, { "id": 2133456, "postDate": "2023-02-07T12:32:03.093Z", "content": "I understood.
\nI'll try it.
\nThanks!
I also encounter train and validation set scores of 0.8.
\nBut the test set is only 0.02
My single image CV F1 score is 0.3596 but my LB score is only 0.38. I am using the effnetv2_s on 1536x960 images, what is your CV score for single image right now ?
", "rawMarkdown": "My single image CV F1 score is 0.3596 but my LB score is only 0.38. I am using the effnetv2_s on 1536x960 images, what is your CV score for single image right now ?", "replies": [ { "id": 2145536, "postDate": "2023-02-15T07:00:50.407Z", "content": "@namansingh2803
\nSee this reply.
\nmy image level F1 score 0.356 (CV k=3)
\nThanks.
https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198#2133432
", "rawMarkdown": "@namansingh2803 \nSee this reply.\nmy image level F1 score 0.356 (CV k=3)\nThanks.\n\nhttps://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198#2133432", "votes": 1, "replies": [ { "id": 2146305, "postDate": "2023-02-15T19:09:02.387Z", "content": "I see, could you also tell me what was your validation loss ? I observed that some of my models which have a lower CV F1 score but a lower validation loss perform better on the public LB than the models which have a higher F1 and a higher val loss.
", "rawMarkdown": "I see, could you also tell me what was your validation loss ? I observed that some of my models which have a lower CV F1 score but a lower validation loss perform better on the public LB than the models which have a higher F1 and a higher val loss." } ] } ] }, { "id": 2138057, "postDate": "2023-02-10T14:17:07.460Z", "content": "Im curious, whats your single fold LB?
", "rawMarkdown": "Im curious, whats your single fold LB? " }, { "id": 2134791, "postDate": "2023-02-08T08:57:20.073Z", "content": "@taruto1215 Sorry if this is not appropriate, could you please share your dataset you used for training? Many thanks
", "rawMarkdown": "@taruto1215 Sorry if this is not appropriate, could you please share your dataset you used for training? Many thanks", "replies": [ { "id": 2134970, "postDate": "2023-02-08T11:15:57.680Z", "content": "I am sorry.
\nit includes one of my unique solution. so I cannot share you.
I think is related to how you split the data. You can have a R-CC on train and a L-CC on validation, R-CC doesn't have cancer, L-CC has. At the same time I expect that there is a high similarity between the 2, so this makes the score on CV smaller.
", "rawMarkdown": "I think is related to how you split the data. You can have a R-CC on train and a L-CC on validation, R-CC doesn't have cancer, L-CC has. At the same time I expect that there is a high similarity between the 2, so this makes the score on CV smaller.", "replies": [ { "id": 2133163, "postDate": "2023-02-07T09:37:04.353Z", "content": "Thanks
\nI split the data as same patients are the same fold.
\nso for me I think the problem you mentioned can be ignore.
@taruto1215 Hello, Could you tell me your dataset you use? Did you use VOI_LUT in your dataset? Thanks
", "rawMarkdown": "@taruto1215 Hello, Could you tell me your dataset you use? Did you use VOI_LUT in your dataset? Thanks", "replies": [ { "id": 2131912, "postDate": "2023-02-06T13:34:55.653Z", "content": "Yes, I did
", "rawMarkdown": "Yes, I did", "votes": 2, "replies": [ { "id": 2131951, "postDate": "2023-02-06T14:01:25.017Z", "content": "I think the main reason why your CV/LB gap maybe in your dataset. Difference in training and inference?
", "rawMarkdown": "I think the main reason why your CV/LB gap maybe in your dataset. Difference in training and inference?" }, { "id": 2133184, "postDate": "2023-02-07T09:53:18.987Z", "content": "I use exactly same preprocess pipeline… Hmm
", "rawMarkdown": "I use exactly same preprocess pipeline... Hmm\n" } ] } ] }, { "id": 2126918, "postDate": "2023-02-02T15:04:29.667Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 2127714, "postDate": "2023-02-03T05:03:38.933Z", "content": "I get my cv score using this cv strategy for all train data.
\nn_splits = 3
https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198
", "rawMarkdown": "I get my cv score using this cv strategy for all train data.\nn_splits = 3\n\nhttps://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198" } ] } ], "comments": [ { "id": 2131916, "author_name": "taruto", "author_url": "", "post_date": "2023-02-06T13:37:45.833000", "content": "Finally my LB is more than 0.60.
\nBut my cv score is 0.42.
\nThe gap is too big !!!!!!
any post-processing?
", "votes": 0, "replies": [ { "id": 2131955, "author_name": "taruto", "author_url": "", "post_date": "2023-02-06T14:02:18.667000", "content": "No post-processing
", "votes": 0, "replies": [] } ] }, { "id": 2132155, "author_name": "Mad_Neil", "author_url": "", "post_date": "2023-02-06T16:23:15.843000", "content": "Your LB score is very impressive. Could you tell me your image size did you use?
", "votes": 0, "replies": [ { "id": 2133133, "author_name": "taruto", "author_url": "", "post_date": "2023-02-07T09:04:35.323000", "content": "Thanks.
\ncurrent my image size is 1536x960
Amazing result! Congratulations crossing 0.6x.
\nDo you still use effnet?
BTW:
\nI have local CV: 0.54 now … and this model is only 0.51 on LB 😂
Thank you!
\nI still use the effnet.
You have a good local CV!
\nI don't still know why my CV is too low… (it bothers me)
How is the image level CV score for your model?
", "votes": 0, "replies": [] }, { "id": 2133283, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-02-07T10:44:32.670000", "content": "For single image 0.4.
", "votes": 1, "replies": [] }, { "id": 2133298, "author_name": "taruto", "author_url": "", "post_date": "2023-02-07T10:51:47.230000", "content": "it's a also higher score than my image level cv score 0.356.
\nthanks for your sharing.
If it is possible please share result (validator result). It would be great. Especially I am looking now on predicition dynamics (chart with probability distribution).
\nAs far as I understand you blend 3 models (3 folds) for final solution?
", "votes": 0, "replies": [] }, { "id": 2133432, "author_name": "taruto", "author_url": "", "post_date": "2023-02-07T12:11:27.647000", "content": "I share you my best cv result.
\nYes, I just blended 3 models (3folds).
\n
Thank you a lot.👍
\nStats you provided are from one model or blended?
It's blended from 3 models (3folds)
", "votes": 1, "replies": [] }, { "id": 2134843, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-02-08T09:32:32.167000", "content": "@taruto1215 how many epochs do you train model? Or from what epoch do you choose models from?
", "votes": 0, "replies": [] }, { "id": 2134858, "author_name": "Mobassir", "author_url": "", "post_date": "2023-02-08T09:39:15.480000", "content": "@taruto1215 may i know what validation pf1,auc and thresholded pf1 you get for each fold? thanks
", "votes": 0, "replies": [] }, { "id": 2134978, "author_name": "taruto", "author_url": "", "post_date": "2023-02-08T11:21:40.770000", "content": "@remekkinas
\nI train model 5 epochs.
\nthen I choose a model has a best image-level f1 score.
\nBut almost of them has the best score at the 5 epoch.
@mobassir
\nThis is my validation results for each fold.
\n
i ahve been trying another experiment. if you are training too many epoch, then the predicted distribution will be too sharp (i.e. overfitting).
\nhence one way is to stop early at, say 5 epoch.
\nwe train many of differenet models.
then we freeze them and use them as feature extractor.
\nwe use giba trick (winner of petfinder): https://developer.nvidia.com/blog/fast-fine-tuning-of-ai-transformers-using-rapids-machine-learning/
\nhttps://www.kaggle.com/c/petfinder-pawpularity-score/discussion/301686
but instead, we create ensmble for multi-view model. each image is now treated as a feature map in sequence, and trsnformer is used to fuse the feature of different view, together with other attribute like age, site id.
", "votes": 3, "replies": [] }, { "id": 2135314, "author_name": "taruto", "author_url": "", "post_date": "2023-02-08T14:59:35.790000", "content": "Thanks I’ll check it
\nBut from tomorrow I’ll be very busy at my job . I will not have more time to spend Kaggle than now hahaha😂
Is it a single fold model or an ensemble ?
", "votes": 1, "replies": [ { "id": 2133145, "author_name": "taruto", "author_url": "", "post_date": "2023-02-07T09:15:00.843000", "content": "Number of folds is 3.
\nso I used 3 models for submission.
Hi! @taruto1215 Can you tell me how do you use 3 models in your inference notebook to make a submission?
", "votes": 0, "replies": [] } ] }, { "id": 2132927, "author_name": "fate", "author_url": "", "post_date": "2023-02-07T06:04:44.393000", "content": "5 fold
\nCV:0.416
\nLB:0.58
\nMaybe public leaderboard data are simpler than train data
It's the same symptom as mine haha.
\nHow about your ROC_AUC for CV??
I guess so too for the public leaderboard data.
\nBut for the private leaderboard data …
\nI hope it's same.
ROC_AUC(group by mean) ~0.88
", "votes": 1, "replies": [] }, { "id": 2133293, "author_name": "taruto", "author_url": "", "post_date": "2023-02-07T10:49:20.730000", "content": "thanks for your sharing.
\nmy ROC_AUC (group by mean) ~0.896
\nI think roc_auc scores has correlation against LB scores.
The same here - ROV_OUC for me is over 0.91 and LB is lower then yours. But I am still talking about single model in my case. All blending solutions failed for me.
", "votes": 2, "replies": [] }, { "id": 2133442, "author_name": "taruto", "author_url": "", "post_date": "2023-02-07T12:18:47.177000", "content": "The ROC_AUC score is also good!
\nPerhaps I should suspect that I am overfitting to LB.
why do you talking about only single model?
\nI think we should check oof result.
To be honest? I do not know at this time. Fianally I will sub two solution: single model and blend.
", "votes": 1, "replies": [] }, { "id": 2142767, "author_name": "outwrest", "author_url": "", "post_date": "2023-02-13T18:58:58.997000", "content": "\nI want to follow up on this, my fold 0 results seem good with CV .52 (mean, LB 0.52) similar to the results you posted before but my fold 1 results CV 0.43 (mean), and fold 2 CV 0.44. Did you see similar results on your folds, it seems like your LB is still the same as your single model. Could this be just different folds (or luck as a matter of a fact) fit better on the public LB? Shake up could be possible if others are experiencing this.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2121189, "author_name": "ForcewithMe", "author_url": "", "post_date": "2023-01-30T05:38:09.993000", "content": "Hi @taruto1215 , how do you get your CV? I mean the LB is calculated by laterality-level, if you train and validate by images, you will get the image-level score. In my experiments, the laterality-level f1 is always much higher than the image-level f1.
", "votes": 5, "replies": [ { "id": 2121404, "author_name": "taruto", "author_url": "", "post_date": "2023-01-30T09:42:15.230000", "content": "@forcewithme
\nThanks for your reply.
my CV was calculated by laterality-level.
\nFor the image-level F1 is ~0.35.
i am not very sure if i were correct. but for this competition there may be other ways to reduce shakeup by NOT only relying on LB vs CV.
\ne.g. my target is to make a model:
i am coming out with many such rules based on my observation.
\nit is like regularization in machine learning, when you loss is not reliable, you need to think of other \"sensible\" conditions
", "votes": 3, "replies": [ { "id": 2121638, "author_name": "taruto", "author_url": "", "post_date": "2023-01-30T13:09:12.357000", "content": "Thanks for sharing your ideas.
\nWe should approach this competition with that perspective for the remaining month.
beware if you aggregate by mean. it may be wrong method.
\nfor example assume
\npatient1 (cancer):
\nimage1 = 0.9
\nimage2 (occluded)=0.1
patient2 (cancer):
\nimage1 = 0.9
\nimage2 (occluded)=0.1
\nimage3 (occluded)=0.1
\nimage4 (occluded)=0.1
the score for patient 2 is divided out.
\nbut max is no better
\npatient1 (cancer):
\nimage1 = 0.7
\nimage2 =0.8
patient1 (no cancer):
\nimage1 = 0.1
\nimage2 =0.1
\nimage3 = 0.7 (noise)
\nimage4 =0.1
I put my code to aggregrate.
\nI think there is no mistake.
df_merge_groupby = df_oof[['prediction_id','site_id', 'patient_id', 'laterality', 'cancer', 'preds']].groupby(\n ['patient_id', 'laterality']).mean().reset_index()
\n", "votes": 0, "replies": [ { "id": 2121511, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-30T11:25:30.977000", "content": "patient2 (cancer):
\n
\n image1 = 0.9
\n image2 (occluded)=0.1
\n image3 (occluded)=0.1
\n image4 (occluded)=0.1
you can try mean of top 3.
\nbut the correct way is to learn an aggregation model, which is multiple image prediction MIP at my post.
there is no definite way of mean,max or mean of top3. different aggregation works for different cases, that is why it is better to learn a function.
", "votes": 1, "replies": [] }, { "id": 2121513, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-30T11:27:06.137000", "content": "you can also check @remekkinas discussion at : https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/377654#2113462
", "votes": 2, "replies": [] }, { "id": 2121527, "author_name": "taruto", "author_url": "", "post_date": "2023-01-30T11:42:51.167000", "content": "You are right.
\nFor me, mean of top 3 boosted my CV at 0.005.
Thanks for the advice.
", "votes": 0, "replies": [] } ] }, { "id": 2132925, "author_name": "AleNic", "author_url": "", "post_date": "2023-02-07T05:54:08.740000", "content": "Hi, I have a doubt regarding the best way to submit.
\nActually I have 4 splits, CV 0.42 and LB 0.52, for every model I have also a best_thr on the mean aggregation.
\nFor the prediction of every row I get the mean of the 4 models (trained on the 4 splits):
\np(pid, imageid) = pm = (p1+p2+p3+p4)/4\n
after that I aggregate with the mean using groupby() and after I just predict using a threshold (here the question ) that is the the mean of the 4 thresholds
\n\ntm=(t1+t2+t3+t4)/4\n
If I tried a different threshold I get better results on LB (I think that is normal beacuse it's like overfit the LB).
\nI tried the same procedure with the max aggregation instead the mean aggregation, with some improvements but depends from the models.
\nDo you choose the best threshold in this way? 🤔
\nThanks in advance
I just choose the threshold that was best for my CV, no special tunning for LB.
\nThanks.
interesting, LB0.59 is my training score, while my validation score is lower. kaggle LB score is close to my training score.
\ncould matching train score = LB score be a way to fight generalisation?
\n(of course your have to do early exit)
asuming the top kagglers did not overfit, you can train until train score is around 0.65
", "votes": 1, "replies": [ { "id": 2137447, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-10T02:38:46.490000", "content": "on a side note since the server compute probablistic fscore, it is quite easily to overfit public test or probe the public test distribution for some divided bins
\ne.g.
1. submit raw probabiliy and get LB score\n2. for th = 0.1,0.2,0.3 .....\nset probabiliy>th to random and submit, \nor set set probabiliy<th to random and submit \nor set set probabiliy<th1 and set set probabiliy>th2 to random and submit \nHello, Which model did you use?
", "votes": 1, "replies": [ { "id": 2123206, "author_name": "taruto", "author_url": "", "post_date": "2023-01-31T11:34:16.250000", "content": "Hello, I used the efficientnetv2_s
", "votes": 2, "replies": [ { "id": 2123496, "author_name": "Mad_Neil", "author_url": "", "post_date": "2023-01-31T14:51:48.270000", "content": "Thank you! I tried efficientnet_v2 with size 1024x512 but pf1 only around 0.1-0.15
", "votes": 1, "replies": [] } ] } ] }, { "id": 2121246, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-30T06:47:17.847000", "content": "site_id=1 : precision=0.329 recall=0.287 f1=0.307 thres=0.4500 roc_auc=0.637\nsite_id=2 : precision=0.639 recall=0.416 f1=0.504 thres=0.5000 roc_auc=0.706\ni am interested in the score of site1 and site2 submitted separately.
\nstable f1 if
\n1) precision is close to recall
\n2) precision is very high
stable LB if ?
\n1)site1 close to site2
high threshold : maybe overfitting
\nnote: stable means consistent (i.e. less shakeup) but not necessarily optimum
\nCV: 0.395 LB: 0.56
\n\"I am very concern about shake down.
\nHow is your score gap??\"
but even if CV close to LB doesn't guarantee no shakeup
", "votes": 1, "replies": [ { "id": 2121420, "author_name": "taruto", "author_url": "", "post_date": "2023-01-30T09:55:10.267000", "content": "Sorry my ROC_AUC was wrong (I made a fundamental code mistake). I wrote up the correct version above.
", "votes": 0, "replies": [] }, { "id": 2121431, "author_name": "taruto", "author_url": "", "post_date": "2023-01-30T10:04:11.357000", "content": "I used your code to generate results.
\nalso there is a gap between site1 and site2.
How that possible??
\nProbably overfitted to hidden test set?
I don't think roc_auc about 0.6~0.7 is good performance.
", "votes": 1, "replies": [ { "id": 2121400, "author_name": "taruto", "author_url": "", "post_date": "2023-01-30T09:40:04.787000", "content": "I just found I made a fundamental mistakes
\nMy CV ROC_AUC is ~0.87.
\nThanks.
me too! cv:0.3 lb:0.46 threshold=0.89
", "votes": 1, "replies": [ { "id": 2121035, "author_name": "taruto", "author_url": "", "post_date": "2023-01-30T02:27:42.390000", "content": "Thanks for your sharing
", "votes": 1, "replies": [] } ] }, { "id": 2121011, "author_name": "Eleftherios Fanioudakis", "author_url": "", "post_date": "2023-01-30T01:15:29.013000", "content": "did you try same aggregation in your cv (mean, max etc..) as you do for the pred_ids during subm?
", "votes": 1, "replies": [ { "id": 2121029, "author_name": "taruto", "author_url": "", "post_date": "2023-01-30T02:15:46.560000", "content": "Yes, my CV is after mean aggregation.
\nHow is your score gap? No gap?
\nThanks
CV 0.5, LB 0.53
", "votes": 2, "replies": [] }, { "id": 2121038, "author_name": "taruto", "author_url": "", "post_date": "2023-01-30T02:31:24.297000", "content": "Good! Almost same scores!
\nHow is your CV strategy?
\nIs it same as me?
\nThanks.
very similar, but 5 splits
", "votes": 1, "replies": [] }, { "id": 2121505, "author_name": "taruto", "author_url": "", "post_date": "2023-01-30T11:16:50.890000", "content": "Huh, I maybe need more splits.
\nThanks for your sharing.
good job!
\ncurrently all team with LB score above 0.60 has good chance of wining.
\ndo spend time to:
simulate leader board results. how one or few change of pos samples can change your validation score.
\nthis enable you to interprete the LB score correctly and estimate shakeup
i still think that mean() aggreataion is not good unless your classifier has very low false positive rate.
\nso you may want to spend time to think about how to fuse results.
remember !!!!
\nit is the private hidden score that we are interested .you should try to predict the private score for every submission you make !!!
Thank you for your advice.
\nYour advices help me(us) a lot.
\nYes I also am thinking about how to aggregrate.
\n\n1 simulate leader board results. how one or few change of pos samples can change your validation score.
\n
\n this enable you to interprete the LB score correctly and estimate shakeup
You means delete a few pos samples then train model and check the influence (score) ?
\nThanks
", "votes": 0, "replies": [ { "id": 2133450, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-07T12:25:44.813000", "content": "somtheing like below.
\nnow you have already make prediction and predicted score and true label.
\nuse 100% of predicted score to compute cv
\nuse 90% of predicted score (repeat over a few time)
\nuse 80%
is the change stable?
\nideally, remove of one sample from the predicted score should not changes the CV metric. but is it true?
", "votes": 1, "replies": [] }, { "id": 2133456, "author_name": "taruto", "author_url": "", "post_date": "2023-02-07T12:32:03.093000", "content": "I understood.
\nI'll try it.
\nThanks!
I also encounter train and validation set scores of 0.8.
\nBut the test set is only 0.02
My single image CV F1 score is 0.3596 but my LB score is only 0.38. I am using the effnetv2_s on 1536x960 images, what is your CV score for single image right now ?
", "votes": 0, "replies": [ { "id": 2145536, "author_name": "taruto", "author_url": "", "post_date": "2023-02-15T07:00:50.407000", "content": "@namansingh2803
\nSee this reply.
\nmy image level F1 score 0.356 (CV k=3)
\nThanks.
https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198#2133432
", "votes": 1, "replies": [ { "id": 2146305, "author_name": "Naman Makkar", "author_url": "", "post_date": "2023-02-15T19:09:02.387000", "content": "I see, could you also tell me what was your validation loss ? I observed that some of my models which have a lower CV F1 score but a lower validation loss perform better on the public LB than the models which have a higher F1 and a higher val loss.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2138057, "author_name": "Eleftherios Fanioudakis", "author_url": "", "post_date": "2023-02-10T14:17:07.460000", "content": "Im curious, whats your single fold LB?
", "votes": 0, "replies": [] }, { "id": 2134791, "author_name": "Mad_Neil", "author_url": "", "post_date": "2023-02-08T08:57:20.073000", "content": "@taruto1215 Sorry if this is not appropriate, could you please share your dataset you used for training? Many thanks
", "votes": 0, "replies": [ { "id": 2134970, "author_name": "taruto", "author_url": "", "post_date": "2023-02-08T11:15:57.680000", "content": "I am sorry.
\nit includes one of my unique solution. so I cannot share you.
I think is related to how you split the data. You can have a R-CC on train and a L-CC on validation, R-CC doesn't have cancer, L-CC has. At the same time I expect that there is a high similarity between the 2, so this makes the score on CV smaller.
", "votes": 0, "replies": [ { "id": 2133163, "author_name": "taruto", "author_url": "", "post_date": "2023-02-07T09:37:04.353000", "content": "Thanks
\nI split the data as same patients are the same fold.
\nso for me I think the problem you mentioned can be ignore.
@taruto1215 Hello, Could you tell me your dataset you use? Did you use VOI_LUT in your dataset? Thanks
", "votes": 0, "replies": [ { "id": 2131912, "author_name": "taruto", "author_url": "", "post_date": "2023-02-06T13:34:55.653000", "content": "Yes, I did
", "votes": 2, "replies": [ { "id": 2131951, "author_name": "Ơ con lừa!", "author_url": "", "post_date": "2023-02-06T14:01:25.017000", "content": "I think the main reason why your CV/LB gap maybe in your dataset. Difference in training and inference?
", "votes": 0, "replies": [] }, { "id": 2133184, "author_name": "taruto", "author_url": "", "post_date": "2023-02-07T09:53:18.987000", "content": "I use exactly same preprocess pipeline… Hmm
", "votes": 0, "replies": [] } ] } ] }, { "id": 2126918, "author_name": "", "author_url": "", "post_date": "2023-02-02T15:04:29.667000", "content": "", "votes": 0, "replies": [ { "id": 2127714, "author_name": "taruto", "author_url": "", "post_date": "2023-02-03T05:03:38.933000", "content": "I get my cv score using this cv strategy for all train data.
\nn_splits = 3
https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2121006": "there is a huge score gap between my cv and my lb CV: 0.395 (after mean aggregation ) LB: 0.56 (thres=0.20, no strict tunning).\nI am very concern about shake down.\nHow is your score gap??\nthanks.\n\nmy cv strategy is below. (n_splits=3)\nI refer [this notebook](https://www.kaggle.com/code/awsaf49/rsna-bcd-efficientnet-tf-tpu-1vm-train)\n\n \n strat_cols = [\n 'laterality', 'view', 'biopsy','invasive', 'BIRADS', 'age_bin',\n 'implant', 'density','machine_id', 'difficult_negative_case',\n 'cancer',\n ]\n\n df['stratify'] = ''\n for col in strat_cols:\n df['stratify'] += df[col].astype(str)\n \n kfold = StratifiedGroupKFold(n_splits=Config['SPLITS'],random_state=Config[\"seed\"],shuffle=True)\n for fold_, (train_idx, valid_idx) in enumerate(kfold.split(df, df['stratify'].values, df['patient_id'].values)):`\n\n**update2 : CV score details (n_splits=3)**\n**all data：precision=0.485 recall=0.333 pf1=0.395 thres=0.5000 roc_auc=0.873**\n**site_id=1 : precision=0.329 recall=0.287 f1=0.307 thres=0.4500 roc_auc=0.826**\n**site_id=2 : precision=0.639 recall=0.416 f1=0.504 thres=0.5000 roc_auc=0.906**", "2131916": "Finally my LB is more than 0.60.\nBut my cv score is 0.42.\nThe gap is too big !!!!!!", "2121189": "Hi @taruto1215 , how do you get your CV? I mean the LB is calculated by laterality-level, if you train and validate by images, you will get the image-level score. In my experiments, the laterality-level f1 is always much higher than the image-level f1.", "2121526": "i am not very sure if i were correct. but for this competition there may be other ways to reduce shakeup by NOT only relying on LB vs CV.\ne.g. my target is to make a model:\n1. have recall at least 0.50 for both site1 and site2\n2. have f1 of at least 0.50 for both site1 and site2 \n3. threshold not larger than 0.55\n4 . .....\n\ni am coming out with many such rules based on my observation.\n\nit is like regularization in machine learning, when you loss is not reliable, you need to think of other \"sensible\" conditions", "2121239": "beware if you aggregate by mean. it may be wrong method.\n\nfor example assume\n\npatient1 (cancer):\nimage1 = 0.9\nimage2 (occluded)=0.1\n\npatient2 (cancer):\nimage1 = 0.9\nimage2 (occluded)=0.1\nimage3 (occluded)=0.1\nimage4 (occluded)=0.1\n\n\nthe score for patient 2 is divided out.\n\n---\nbut max is no better\npatient1 (cancer):\nimage1 = 0.7\nimage2 =0.8\n\npatient1 (no cancer):\nimage1 = 0.1\nimage2 =0.1\nimage3 = 0.7 (noise)\nimage4 =0.1\n\n\n", "2136783": "interesting, LB0.59 is my training score, while my validation score is lower. kaggle LB score is close to my training score.\n\ncould matching train score = LB score be a way to fight generalisation?\n(of course your have to do early exit)\n\nasuming the top kagglers did not overfit, you can train until train score is around 0.65", "2122638": "Hello, Which model did you use?", "2121246": "```\nsite_id=1 : precision=0.329 recall=0.287 f1=0.307 thres=0.4500 roc_auc=0.637\nsite_id=2 : precision=0.639 recall=0.416 f1=0.504 thres=0.5000 roc_auc=0.706\n```\n\ni am interested in the score of site1 and site2 submitted separately.\n\n---\nstable f1 if\n1) precision is close to recall \n2) precision is very high\n\nstable LB if ?\n1)site1 close to site2\n\nhigh threshold : maybe overfitting\n\nnote: stable means consistent (i.e. less shakeup) but not necessarily optimum\n\n---\nCV: 0.395 LB: 0.56\n\n\"I am very concern about shake down.\nHow is your score gap??\"\n\nbut even if CV close to LB doesn't guarantee no shakeup ", "2121192": "How that possible??\nProbably overfitted to hidden test set?\n\nI don't think roc_auc about 0.6~0.7 is good performance.", "2121017": "me too! cv:0.3 lb:0.46 threshold=0.89 ", "2121011": "did you try same aggregation in your cv (mean, max etc..) as you do for the pred_ids during subm?", "2132637": "good job!\n\ncurrently all team with LB score above 0.60 has good chance of wining.\ndo spend time to:\n\n1. simulate leader board results. how one or few change of pos samples can change your validation score.\nthis enable you to interprete the LB score correctly and estimate shakeup\n\n2. i still think that mean() aggreataion is not good unless your classifier has very low false positive rate.\nso you may want to spend time to think about how to fuse results.\n\nremember !!!!\nit is the private hidden score that we are interested .you should try to predict the private score for every submission you make !!!\n", "2164140": "I also encounter train and validation set scores of 0.8.\nBut the test set is only 0.02", "2144025": "My single image CV F1 score is 0.3596 but my LB score is only 0.38. I am using the effnetv2_s on 1536x960 images, what is your CV score for single image right now ?", "2138057": "Im curious, whats your single fold LB? ", "2134791": "@taruto1215 Sorry if this is not appropriate, could you please share your dataset you used for training? Many thanks", "2132523": "I think is related to how you split the data. You can have a R-CC on train and a L-CC on validation, R-CC doesn't have cancer, L-CC has. At the same time I expect that there is a high similarity between the 2, so this makes the score on CV smaller.", "2130418": "@taruto1215 Hello, Could you tell me your dataset you use? Did you use VOI_LUT in your dataset? Thanks", "2126918": "" } }