{ "id": 337309, "title": "CV vs LB thread", "url": "/competitions/mayo-clinic-strip-ai/discussion/337309", "author_name": "RabotniKuma", "post_date": "2022-07-15T12:23:02.889000", "votes": 22, "comment_count": 13, "views": 0, "content": "

Share your results here!

\n

[My current model]
\ndataset: 8x downsampled images
\nmodel: multiple instance model (-> notebook)
\nvalidation: 5 fold stratified group (center_id) split
\nCV: logloss(no weight adjustment) = 0.56739
\nLB: 0.8 πŸ˜‘

", "messages": [ { "id": 1856555, "postDate": "2022-07-15T12:23:02.890Z", "content": "

Share your results here!

\n

[My current model]
\ndataset: 8x downsampled images
\nmodel: multiple instance model (-> notebook)
\nvalidation: 5 fold stratified group (center_id) split
\nCV: logloss(no weight adjustment) = 0.56739
\nLB: 0.8 πŸ˜‘

", "rawMarkdown": "Share your results here!\n\n[My current model]\ndataset: 8x downsampled images\nmodel: multiple instance model ([-> notebook](https://www.kaggle.com/code/analokamus/a-sample-of-multi-instance-learning-model))\nvalidation: 5 fold stratified group (center_id) split\nCV: logloss(no weight adjustment) = 0.56739\nLB: 0.8 πŸ˜‘", "votes": 22 }, { "id": 1949551, "postDate": "2022-09-21T17:51:51.110Z", "content": "

I just started the competition and here is my baseline.

\n\n
{\n  \"fold_scores\": {\n    \"fold1\": {\n      \"accuracy\": 0.7450980392156863,\n      \"roc_auc\": 0.5766981556455241,\n      \"log_loss\": 0.5583284267413071\n    },\n    \"fold2\": {\n      \"accuracy\": 0.7077922077922078,\n      \"roc_auc\": 0.5307218134818055,\n      \"log_loss\": 0.6123370449119186\n    },\n    \"fold3\": {\n      \"accuracy\": 0.7567567567567568,\n      \"roc_auc\": 0.548859126984127,\n      \"log_loss\": 0.5499128863353886\n    },\n    \"fold4\": {\n      \"accuracy\": 0.7266666666666667,\n      \"roc_auc\": 0.4799731483553367,\n      \"log_loss\": 0.5914205276966095\n    },\n    \"fold5\": {\n      \"accuracy\": 0.7114093959731543,\n      \"roc_auc\": 0.5449134199134199,\n      \"log_loss\": 0.6243360901310815\n    }\n  },\n  \"oof_scores\": {\n    \"accuracy\": 0.7294429708222812,\n    \"roc_auc\": 0.5291444771215855,\n    \"log_loss\": 0.5873347718428588\n  }\n
\n

Update

\n\n
  \"oof_scores\": {\n    \"accuracy\": 0.7360742705570292,\n    \"roc_auc\": 0.6244689964585044,\n    \"log_loss\": 0.5671889458533466\n  }\n
\n

Looks like improvement from the baseline is significant. It scored 0.8 on public leaderboard. The problem is it took 4 hours to run during the submission. I have to make an efficient way to process images. I'm currently using pyvips to read them.

", "rawMarkdown": "I just started the competition and here is my baseline.\n\n* Images are downsampled by taking every 4th pixel in them\n* I'm using a slightly different MIL model with ResNet18 backbone \n* CV is 5 stratified folds on target\n\n```\n{\n \"fold_scores\": {\n \"fold1\": {\n \"accuracy\": 0.7450980392156863,\n \"roc_auc\": 0.5766981556455241,\n \"log_loss\": 0.5583284267413071\n },\n \"fold2\": {\n \"accuracy\": 0.7077922077922078,\n \"roc_auc\": 0.5307218134818055,\n \"log_loss\": 0.6123370449119186\n },\n \"fold3\": {\n \"accuracy\": 0.7567567567567568,\n \"roc_auc\": 0.548859126984127,\n \"log_loss\": 0.5499128863353886\n },\n \"fold4\": {\n \"accuracy\": 0.7266666666666667,\n \"roc_auc\": 0.4799731483553367,\n \"log_loss\": 0.5914205276966095\n },\n \"fold5\": {\n \"accuracy\": 0.7114093959731543,\n \"roc_auc\": 0.5449134199134199,\n \"log_loss\": 0.6243360901310815\n }\n },\n \"oof_scores\": {\n \"accuracy\": 0.7294429708222812,\n \"roc_auc\": 0.5291444771215855,\n \"log_loss\": 0.5873347718428588\n }\n```\n**Update**\n\n* Changed downsampling to jpeg compression (100%)\n* Custom MIL model with EfficientNetB1 backbone\n\n```\n \"oof_scores\": {\n \"accuracy\": 0.7360742705570292,\n \"roc_auc\": 0.6244689964585044,\n \"log_loss\": 0.5671889458533466\n }\n```\n\nLooks like improvement from the baseline is significant. It scored 0.8 on public leaderboard. The problem is it took 4 hours to run during the submission. I have to make an efficient way to process images. I'm currently using pyvips to read them.", "votes": 1 }, { "id": 1949669, "postDate": "2022-09-21T19:40:55.807Z", "content": "

My CV comes to the 0.54X - 0.55X range, but my LB stays between 0.7-0.8 πŸ™

", "rawMarkdown": "My CV comes to the 0.54X - 0.55X range, but my LB stays between 0.7-0.8 πŸ™" }, { "id": 1938963, "postDate": "2022-09-14T12:58:36.047Z", "content": "

cv 0.567 is amazing! I think you should trust your cv in this competition.

", "rawMarkdown": "cv 0.567 is amazing! I think you should trust your cv in this competition.", "replies": [ { "id": 1940480, "postDate": "2022-09-15T11:39:32.073Z", "content": "

But he didn't say how he calculate the cv. I wonder what metric he used. Maybe the 0.567 is a ce loss score.

", "rawMarkdown": "But he didn't say how he calculate the cv. I wonder what metric he used. Maybe the 0.567 is a ce loss score." }, { "id": 1949666, "postDate": "2022-09-21T19:38:25.113Z", "content": "

@forcewithme Hey, by correct metric, do you mean logloss from sklearn metrics?

", "rawMarkdown": "@forcewithme Hey, by correct metric, do you mean `logloss` from sklearn metrics?" }, { "id": 1951392, "postDate": "2022-09-23T03:25:03.880Z", "content": "

Yes @yerramvarun

", "rawMarkdown": " Yes @yerramvarun ", "votes": 1 } ] }, { "id": 1877561, "postDate": "2022-07-30T19:16:59.787Z", "content": "

Sorry if this is a stupid question, but doesn't it make sense to trust the validation score more in this comp, since the test set is just less than 10 rows?

", "rawMarkdown": "Sorry if this is a stupid question, but doesn't it make sense to trust the validation score more in this comp, since the test set is just less than 10 rows? ", "replies": [ { "id": 1877721, "postDate": "2022-07-31T00:44:23.263Z", "content": "

@yuqizheng I agree with you - basically we should trust cv score more. But even with small sample sizes, leaderboard scores can provide some useful feedback.

", "rawMarkdown": "@yuqizheng I agree with you - basically we should trust cv score more. But even with small sample sizes, leaderboard scores can provide some useful feedback.", "votes": 1 }, { "id": 1877871, "postDate": "2022-07-31T03:59:25.523Z", "content": "

I would guess it would have to do with their loss function. May I ask, did you create a matching loss function in your code to do backprop on? Also, the weights are unknown I believe, so we would have to guess them.

", "rawMarkdown": "I would guess it would have to do with their loss function. May I ask, did you create a matching loss function in your code to do backprop on? Also, the weights are unknown I believe, so we would have to guess them." } ] }, { "id": 1859048, "postDate": "2022-07-17T11:35:33.343Z", "content": "

How many tiles did you use for 1 forward pass at inference and training?

", "rawMarkdown": "How many tiles did you use for 1 forward pass at inference and training?" }, { "id": 1857233, "postDate": "2022-07-16T01:17:22.463Z", "content": "

Hi @analokamus how did you submit a model? Slice test tiff into patches to predict & stitch back? Or just resize the original till by 8x and predict directly?
\nThanks!

", "rawMarkdown": "Hi @analokamus how did you submit a model? Slice test tiff into patches to predict & stitch back? Or just resize the original till by 8x and predict directly?\nThanks!", "replies": [ { "id": 1857966, "postDate": "2022-07-16T14:46:24.553Z", "content": "

Similar to the former, but with some improvements. The point is how to link multiple patches to a single target πŸ™‚

", "rawMarkdown": "Similar to the former, but with some improvements. The point is how to link multiple patches to a single target πŸ™‚" }, { "id": 1858101, "postDate": "2022-07-16T16:51:35.360Z", "content": "

thanks for your clarification, I finally get my model to train… still need to figure out how to submit though.

", "rawMarkdown": "thanks for your clarification, I finally get my model to train... still need to figure out how to submit though." } ] } ], "comments": [ { "id": 1949551, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-21T17:51:51.110000", "content": "

I just started the competition and here is my baseline.

\n\n
{\n  \"fold_scores\": {\n    \"fold1\": {\n      \"accuracy\": 0.7450980392156863,\n      \"roc_auc\": 0.5766981556455241,\n      \"log_loss\": 0.5583284267413071\n    },\n    \"fold2\": {\n      \"accuracy\": 0.7077922077922078,\n      \"roc_auc\": 0.5307218134818055,\n      \"log_loss\": 0.6123370449119186\n    },\n    \"fold3\": {\n      \"accuracy\": 0.7567567567567568,\n      \"roc_auc\": 0.548859126984127,\n      \"log_loss\": 0.5499128863353886\n    },\n    \"fold4\": {\n      \"accuracy\": 0.7266666666666667,\n      \"roc_auc\": 0.4799731483553367,\n      \"log_loss\": 0.5914205276966095\n    },\n    \"fold5\": {\n      \"accuracy\": 0.7114093959731543,\n      \"roc_auc\": 0.5449134199134199,\n      \"log_loss\": 0.6243360901310815\n    }\n  },\n  \"oof_scores\": {\n    \"accuracy\": 0.7294429708222812,\n    \"roc_auc\": 0.5291444771215855,\n    \"log_loss\": 0.5873347718428588\n  }\n
\n

Update

\n\n
  \"oof_scores\": {\n    \"accuracy\": 0.7360742705570292,\n    \"roc_auc\": 0.6244689964585044,\n    \"log_loss\": 0.5671889458533466\n  }\n
\n

Looks like improvement from the baseline is significant. It scored 0.8 on public leaderboard. The problem is it took 4 hours to run during the submission. I have to make an efficient way to process images. I'm currently using pyvips to read them.

", "votes": 1, "replies": [] }, { "id": 1949669, "author_name": "Yerram Varun", "author_url": "", "post_date": "2022-09-21T19:40:55.807000", "content": "

My CV comes to the 0.54X - 0.55X range, but my LB stays between 0.7-0.8 πŸ™

", "votes": 0, "replies": [] }, { "id": 1938963, "author_name": "KKY", "author_url": "", "post_date": "2022-09-14T12:58:36.047000", "content": "

cv 0.567 is amazing! I think you should trust your cv in this competition.

", "votes": 0, "replies": [ { "id": 1940480, "author_name": "ForcewithMe", "author_url": "", "post_date": "2022-09-15T11:39:32.073000", "content": "

But he didn't say how he calculate the cv. I wonder what metric he used. Maybe the 0.567 is a ce loss score.

", "votes": 0, "replies": [] }, { "id": 1949666, "author_name": "Yerram Varun", "author_url": "", "post_date": "2022-09-21T19:38:25.113000", "content": "

@forcewithme Hey, by correct metric, do you mean logloss from sklearn metrics?

", "votes": 0, "replies": [] }, { "id": 1951392, "author_name": "ForcewithMe", "author_url": "", "post_date": "2022-09-23T03:25:03.880000", "content": "

Yes @yerramvarun

", "votes": 1, "replies": [] } ] }, { "id": 1877561, "author_name": "yqz", "author_url": "", "post_date": "2022-07-30T19:16:59.787000", "content": "

Sorry if this is a stupid question, but doesn't it make sense to trust the validation score more in this comp, since the test set is just less than 10 rows?

", "votes": 0, "replies": [ { "id": 1877721, "author_name": "RabotniKuma", "author_url": "", "post_date": "2022-07-31T00:44:23.263000", "content": "

@yuqizheng I agree with you - basically we should trust cv score more. But even with small sample sizes, leaderboard scores can provide some useful feedback.

", "votes": 1, "replies": [] }, { "id": 1877871, "author_name": "yqz", "author_url": "", "post_date": "2022-07-31T03:59:25.523000", "content": "

I would guess it would have to do with their loss function. May I ask, did you create a matching loss function in your code to do backprop on? Also, the weights are unknown I believe, so we would have to guess them.

", "votes": 0, "replies": [] } ] }, { "id": 1859048, "author_name": "Salman Ahmed", "author_url": "", "post_date": "2022-07-17T11:35:33.343000", "content": "

How many tiles did you use for 1 forward pass at inference and training?

", "votes": 0, "replies": [] }, { "id": 1857233, "author_name": "θ±†ζŸ΄ι‡‘ι―±", "author_url": "", "post_date": "2022-07-16T01:17:22.463000", "content": "

Hi @analokamus how did you submit a model? Slice test tiff into patches to predict & stitch back? Or just resize the original till by 8x and predict directly?
\nThanks!

", "votes": 0, "replies": [ { "id": 1857966, "author_name": "RabotniKuma", "author_url": "", "post_date": "2022-07-16T14:46:24.553000", "content": "

Similar to the former, but with some improvements. The point is how to link multiple patches to a single target πŸ™‚

", "votes": 0, "replies": [] }, { "id": 1858101, "author_name": "θ±†ζŸ΄ι‡‘ι―±", "author_url": "", "post_date": "2022-07-16T16:51:35.360000", "content": "

thanks for your clarification, I finally get my model to train… still need to figure out how to submit though.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1856555": "Share your results here!\n\n[My current model]\ndataset: 8x downsampled images\nmodel: multiple instance model ([-> notebook](https://www.kaggle.com/code/analokamus/a-sample-of-multi-instance-learning-model))\nvalidation: 5 fold stratified group (center_id) split\nCV: logloss(no weight adjustment) = 0.56739\nLB: 0.8 πŸ˜‘", "1949551": "I just started the competition and here is my baseline.\n\n* Images are downsampled by taking every 4th pixel in them\n* I'm using a slightly different MIL model with ResNet18 backbone \n* CV is 5 stratified folds on target\n\n```\n{\n \"fold_scores\": {\n \"fold1\": {\n \"accuracy\": 0.7450980392156863,\n \"roc_auc\": 0.5766981556455241,\n \"log_loss\": 0.5583284267413071\n },\n \"fold2\": {\n \"accuracy\": 0.7077922077922078,\n \"roc_auc\": 0.5307218134818055,\n \"log_loss\": 0.6123370449119186\n },\n \"fold3\": {\n \"accuracy\": 0.7567567567567568,\n \"roc_auc\": 0.548859126984127,\n \"log_loss\": 0.5499128863353886\n },\n \"fold4\": {\n \"accuracy\": 0.7266666666666667,\n \"roc_auc\": 0.4799731483553367,\n \"log_loss\": 0.5914205276966095\n },\n \"fold5\": {\n \"accuracy\": 0.7114093959731543,\n \"roc_auc\": 0.5449134199134199,\n \"log_loss\": 0.6243360901310815\n }\n },\n \"oof_scores\": {\n \"accuracy\": 0.7294429708222812,\n \"roc_auc\": 0.5291444771215855,\n \"log_loss\": 0.5873347718428588\n }\n```\n**Update**\n\n* Changed downsampling to jpeg compression (100%)\n* Custom MIL model with EfficientNetB1 backbone\n\n```\n \"oof_scores\": {\n \"accuracy\": 0.7360742705570292,\n \"roc_auc\": 0.6244689964585044,\n \"log_loss\": 0.5671889458533466\n }\n```\n\nLooks like improvement from the baseline is significant. It scored 0.8 on public leaderboard. The problem is it took 4 hours to run during the submission. I have to make an efficient way to process images. I'm currently using pyvips to read them.", "1949669": "My CV comes to the 0.54X - 0.55X range, but my LB stays between 0.7-0.8 πŸ™", "1938963": "cv 0.567 is amazing! I think you should trust your cv in this competition.", "1877561": "Sorry if this is a stupid question, but doesn't it make sense to trust the validation score more in this comp, since the test set is just less than 10 rows? ", "1859048": "How many tiles did you use for 1 forward pass at inference and training?", "1857233": "Hi @analokamus how did you submit a model? Slice test tiff into patches to predict & stitch back? Or just resize the original till by 8x and predict directly?\nThanks!" } }