I usually tend to solve my own problems but I have joined this competition late and have been losing my sanity, so I appeal to the greater wisdom of kaggle.
\nI have trained, as a base line, a model using efficientnet B4. It showed validation score of about 0.7 and pf1 on the validation set also in this area.
\nI submitted it, it got 0.1
\nto check my model, i ran it on CBIS dataset, which is completely external. it got about 0.5 pf1
\nHow can that be?
Here is a very simple inference using my model: https://www.kaggle.com/moshel/test-mammo
\nif DEBUG is set to True, it will run on the validation set
\nif debug is set to False, it will run on test.csv
I must be doing wrong in the submission, or? I just ran out of ideas.
\nAny help will be greatly appreciated!
", "messages": [ { "id": 2149401, "postDate": "2023-02-18T09:10:17.133Z", "content": "I usually tend to solve my own problems but I have joined this competition late and have been losing my sanity, so I appeal to the greater wisdom of kaggle.
\nI have trained, as a base line, a model using efficientnet B4. It showed validation score of about 0.7 and pf1 on the validation set also in this area.
\nI submitted it, it got 0.1
\nto check my model, i ran it on CBIS dataset, which is completely external. it got about 0.5 pf1
\nHow can that be?
Here is a very simple inference using my model: https://www.kaggle.com/moshel/test-mammo
\nif DEBUG is set to True, it will run on the validation set
\nif debug is set to False, it will run on test.csv
I must be doing wrong in the submission, or? I just ran out of ideas.
\nAny help will be greatly appreciated!
", "rawMarkdown": "I usually tend to solve my own problems but I have joined this competition late and have been losing my sanity, so I appeal to the greater wisdom of kaggle.\nI have trained, as a base line, a model using efficientnet B4. It showed validation score of about 0.7 and pf1 on the validation set also in this area.\nI submitted it, it got 0.1\nto check my model, i ran it on CBIS dataset, which is completely external. it got about 0.5 pf1\nHow can that be?\n\nHere is a very simple inference using my model: https://www.kaggle.com/moshel/test-mammo\nif DEBUG is set to True, it will run on the validation set\nif debug is set to False, it will run on test.csv\n\nI must be doing wrong in the submission, or? I just ran out of ideas.\n\nAny help will be greatly appreciated!", "votes": 4 }, { "id": 2150135, "postDate": "2023-02-19T01:17:18.363Z", "content": "yep, you were right. imbalance throws it off. I was sure CBIS was unbalanced, should have checked.
\nThank you everyone, much appreciated!
The data imbalance maybe the reason….
", "rawMarkdown": "The data imbalance maybe the reason....", "votes": 1, "replies": [ { "id": 2150039, "postDate": "2023-02-18T21:54:00.730Z", "content": "Cbis is as unbalanced, but will try to add all the unused negatives and check
", "rawMarkdown": "Cbis is as unbalanced, but will try to add all the unused negatives and check " } ] }, { "id": 2149903, "postDate": "2023-02-18T19:18:26.380Z", "content": "By any chance, are you upsampling the positive cases in your validation set?
", "rawMarkdown": "By any chance, are you upsampling the positive cases in your validation set?", "votes": 1, "replies": [ { "id": 2150038, "postDate": "2023-02-18T21:53:09.610Z", "content": "No, the sets were balanced
", "rawMarkdown": "No, the sets were balanced", "replies": [ { "id": 2150059, "postDate": "2023-02-18T22:40:26.390Z", "content": "Sorry I wasn't clear. When you say sets were balanced, does that mean the frequency of positive cancer rates in the validation set were the same as it is in expected to be in the test dataset (2-4%)?
\nOnly reason I ask is I noticed that the upsampling during training tends to skew the output probabilities. I could counteract it a bit by adjusting my threshold (sigmoid output). Unfortunately, I haven't been able to spend much time in this competition, I would've loved to play around with it more.
", "rawMarkdown": "Sorry I wasn't clear. When you say sets were balanced, does that mean the frequency of positive cancer rates in the validation set were the same as it is in expected to be in the test dataset (2-4%)?\n\nOnly reason I ask is I noticed that the upsampling during training tends to skew the output probabilities. I could counteract it a bit by adjusting my threshold (sigmoid output). Unfortunately, I haven't been able to spend much time in this competition, I would've loved to play around with it more.", "votes": 1 }, { "id": 2150065, "postDate": "2023-02-18T22:47:25.533Z", "content": "My apologies, my answer was not clear. As this is just a baseline, i took all the positives and added the same amount of negatives to create a balanced dataset. Usually this works reasonably and then I start with focal loss, weights, etc.
\nSo, in the validation set there is the same number of positive and negative.
Would be a good check to have run a validation set with 96-98% negative cases (similar to training set distribution) and see the score. Looking forward to see if that had any effect.
", "rawMarkdown": "Would be a good check to have run a validation set with 96-98% negative cases (similar to training set distribution) and see the score. Looking forward to see if that had any effect. " }, { "id": 2150136, "postDate": "2023-02-19T01:17:36.690Z", "content": "Correct me if I misunderstood @moshel. If you upsample the validation data, you are going to get copies hence can lead to misleading high performance.
", "rawMarkdown": "Correct me if I misunderstood @moshel. If you upsample the validation data, you are going to get copies hence can lead to misleading high performance.", "votes": 1 }, { "id": 2150164, "postDate": "2023-02-19T02:07:00.110Z", "content": "I generally tend not to upsample, but its a valid method. If you crop and augment and carful with your upsample it can work ok. I usually try more focal loss and weights
", "rawMarkdown": "I generally tend not to upsample, but its a valid method. If you crop and augment and carful with your upsample it can work ok. I usually try more focal loss and weights " } ] } ] }, { "id": 2149403, "postDate": "2023-02-18T09:12:50.860Z", "content": "The pixel intensity distribution is different in CBIS and RSNA datasets. Try to transform the distributions and you will get higher scores…
", "rawMarkdown": "The pixel intensity distribution is different in CBIS and RSNA datasets. Try to transform the distributions and you will get higher scores...", "votes": 1, "replies": [ { "id": 2149413, "postDate": "2023-02-18T09:31:04.927Z", "content": "Thanks for your prompt answer. The model was trained only on the rsna dataset. I tested it on cbis just to see if it generalize well. It did.
\nI'll try using different distribution but from my experience this can account to 0.1 percent at most. What i am seeing here is 5x times in the metric.
yep, you were right. imbalance throws it off. I was sure CBIS was unbalanced, should have checked.
\nThank you everyone, much appreciated!
The data imbalance maybe the reason….
", "votes": 1, "replies": [ { "id": 2150039, "author_name": "Moshel", "author_url": "", "post_date": "2023-02-18T21:54:00.730000", "content": "Cbis is as unbalanced, but will try to add all the unused negatives and check
", "votes": 0, "replies": [] } ] }, { "id": 2149903, "author_name": "Michael Bolton", "author_url": "", "post_date": "2023-02-18T19:18:26.380000", "content": "By any chance, are you upsampling the positive cases in your validation set?
", "votes": 1, "replies": [ { "id": 2150038, "author_name": "Moshel", "author_url": "", "post_date": "2023-02-18T21:53:09.610000", "content": "No, the sets were balanced
", "votes": 0, "replies": [ { "id": 2150059, "author_name": "Michael Bolton", "author_url": "", "post_date": "2023-02-18T22:40:26.390000", "content": "Sorry I wasn't clear. When you say sets were balanced, does that mean the frequency of positive cancer rates in the validation set were the same as it is in expected to be in the test dataset (2-4%)?
\nOnly reason I ask is I noticed that the upsampling during training tends to skew the output probabilities. I could counteract it a bit by adjusting my threshold (sigmoid output). Unfortunately, I haven't been able to spend much time in this competition, I would've loved to play around with it more.
", "votes": 1, "replies": [] }, { "id": 2150065, "author_name": "Moshel", "author_url": "", "post_date": "2023-02-18T22:47:25.533000", "content": "My apologies, my answer was not clear. As this is just a baseline, i took all the positives and added the same amount of negatives to create a balanced dataset. Usually this works reasonably and then I start with focal loss, weights, etc.
\nSo, in the validation set there is the same number of positive and negative.
Would be a good check to have run a validation set with 96-98% negative cases (similar to training set distribution) and see the score. Looking forward to see if that had any effect.
", "votes": 0, "replies": [] }, { "id": 2150136, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2023-02-19T01:17:36.690000", "content": "Correct me if I misunderstood @moshel. If you upsample the validation data, you are going to get copies hence can lead to misleading high performance.
", "votes": 1, "replies": [] }, { "id": 2150164, "author_name": "Moshel", "author_url": "", "post_date": "2023-02-19T02:07:00.110000", "content": "I generally tend not to upsample, but its a valid method. If you crop and augment and carful with your upsample it can work ok. I usually try more focal loss and weights
", "votes": 0, "replies": [] } ] } ] }, { "id": 2149403, "author_name": "Rasoul Mojtahedzadeh", "author_url": "", "post_date": "2023-02-18T09:12:50.860000", "content": "The pixel intensity distribution is different in CBIS and RSNA datasets. Try to transform the distributions and you will get higher scores…
", "votes": 1, "replies": [ { "id": 2149413, "author_name": "Moshel", "author_url": "", "post_date": "2023-02-18T09:31:04.927000", "content": "Thanks for your prompt answer. The model was trained only on the rsna dataset. I tested it on cbis just to see if it generalize well. It did.
\nI'll try using different distribution but from my experience this can account to 0.1 percent at most. What i am seeing here is 5x times in the metric.