{
  "id": 391133,
  "title": "16th place solution : segmentation & meta-classifier",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/391133",
  "author_name": "HyeongChan Kim",
  "post_date": "2023-02-28T13:34:46.587000",
  "votes": 24,
  "comment_count": 16,
  "views": 0,
  "content": "<p>Hello everyone!</p>\n<p>First, thanks to Kaggle team and the organizers for hosting a competition! Also, congratulations to all the winners!</p>\n<h2>Data</h2>\n<h3>Preprocessing</h3>\n<p>My preprocessing code heavily depends on the public notebooks (eg. remove letters, crop breast via contour).</p>\n<ol>\n<li>decode <code>.jpeg</code> with <code>dicomsdl</code> &amp; <code>nvjpeg2000</code>.</li>\n<li>crop edge (margin pixel 10)</li>\n<li>extract breast with <code>opencv2</code> (contour based)</li>\n<li>resize to 1536x960. (I roughly guess that resizing into a 1.5 ~ 2.0 aspect ratio is fine.)</li>\n</ol>\n<p>In my experiment, windowing doesn't affect the score positively, so I decide not to use it.</p>\n<h3>Augmentation</h3>\n<p>Heavy augmentation works well. Light augmentation tends to overfit.</p>\n<ul>\n<li>v/hflip</li>\n<li>scale / rotate</li>\n<li>brightness / contrast</li>\n<li>cutout (coarse dropout with large patch size)</li>\n<li>mixup</li>\n</ul>\n<h2>Architecture</h2>\n<p>I couldn't spend much time running various experiments due to a lack of time &amp; computing resources. So, I only tested few backbones &amp; training recipes. (about 70% of my submissions are runtime errors &amp; mistakes lol)</p>\n<p>Here's a full pipeline.</p>\n<ol>\n<li>pre-train segmentation model with the <code>cbis-ddsm</code> &amp; <code>vindr</code> datasets.<ul>\n<li>segment: provided RoI image.</li>\n<li>label: <code>malignant</code> to cancer / <code>BIRADS 5</code> to cancer.<br>\n% Of course, the label doesn't perfectly correlate with the competition standards. But, I roughly think that maybe it could help train the model in some ways.</li></ul></li>\n<li>train with competition data (initialize the weight with the pre-trained model)<ul>\n<li>segment: inferred with the pre-trained model.</li>\n<li>auxiliary: given meta-features (total 11 features).</li></ul></li>\n<li>re-label the external data with the <code>step 2</code> model.</li>\n<li>re-train with competition data (initialize with <code>step 3</code> model)</li>\n<li>train a meta-classifier (oof + meta-features (e.g. laterality, age, …))</li>\n</ol>\n<p>For a baseline, I run step 1 ~ 2, 5 and achieve CV 0.4885 LB 0.59 (PB 0.46). Also, I test only with the <code>cbis-ddsm</code> dataset for pre-training, and there were about 0.02 drops on CV &amp; LB, but the same score on PB (CV 0.4656 LB 0.57 PB 0.46).</p>\n<p>A week before the deadline, I finished up to step ~ 5 and got CV 0.5012 LB 0.55 (PB 0.51). Sadly, I didn't choose it as a final submission : (</p>\n<p>Last day of the competition, I ensembled <code>effnet_v2_s</code> backbone and got CV 0.5063 LB 0.56 (PB 0.49).</p>\n<p>Lastly, I choose the best LB &amp; CV for the final submission.</p>\n<h3>Meta-Classifier</h3>\n<p>I built a meta-classifier with meta-features like age, laterality, and the (per-breast) statistics of the predictions. Usually, It gives ~ 0.02 improvements on the CV &amp; LB (also PB).</p>\n<p>I'm worried about overfitting into some meta-features (eg. machine id, (predicted) density, …), so I decided to use only 3 auxiliary features (age, site_id, laterality) to train the model.</p>\n<ul>\n<li>feature: age, site_id, laterality, (mean, std, min, max) of the predictions.</li>\n<li>cv: stratified k fold (5 folds)</li>\n<li>model: CatBoost</li>\n</ul>\n<h2>Works</h2>\n<ul>\n<li>higher resolution (1536x768 ~ 1024) is good.</li>\n<li>external data<ul>\n<li>it gives about +0.02 boosts.</li></ul></li>\n<li>architecture<ul>\n<li>encoder: backbone: <code>effnet-b3</code> works best</li>\n<li>decoder: u-net++</li></ul></li>\n<li>augmentation</li>\n<li>mixup (alpha 1.0)</li>\n<li>loss<ul>\n<li>0.6 * cls_loss (cross_entropy) + 0.4 * seg_loss (dice) + 0.1 * aux_loss (cross-entropy)</li></ul></li>\n<li>stratified group k fold (4 folds)</li>\n<li>meta-classifier</li>\n<li>TTA</li>\n</ul>\n<p>thanks for reading! I hope this could help you :)</p>",
  "messages": [
    {
      "id": 2162937,
      "postDate": "2023-02-28T13:34:46.587Z",
      "content": "<p>Hello everyone!</p>\n<p>First, thanks to Kaggle team and the organizers for hosting a competition! Also, congratulations to all the winners!</p>\n<h2>Data</h2>\n<h3>Preprocessing</h3>\n<p>My preprocessing code heavily depends on the public notebooks (eg. remove letters, crop breast via contour).</p>\n<ol>\n<li>decode <code>.jpeg</code> with <code>dicomsdl</code> &amp; <code>nvjpeg2000</code>.</li>\n<li>crop edge (margin pixel 10)</li>\n<li>extract breast with <code>opencv2</code> (contour based)</li>\n<li>resize to 1536x960. (I roughly guess that resizing into a 1.5 ~ 2.0 aspect ratio is fine.)</li>\n</ol>\n<p>In my experiment, windowing doesn't affect the score positively, so I decide not to use it.</p>\n<h3>Augmentation</h3>\n<p>Heavy augmentation works well. Light augmentation tends to overfit.</p>\n<ul>\n<li>v/hflip</li>\n<li>scale / rotate</li>\n<li>brightness / contrast</li>\n<li>cutout (coarse dropout with large patch size)</li>\n<li>mixup</li>\n</ul>\n<h2>Architecture</h2>\n<p>I couldn't spend much time running various experiments due to a lack of time &amp; computing resources. So, I only tested few backbones &amp; training recipes. (about 70% of my submissions are runtime errors &amp; mistakes lol)</p>\n<p>Here's a full pipeline.</p>\n<ol>\n<li>pre-train segmentation model with the <code>cbis-ddsm</code> &amp; <code>vindr</code> datasets.<ul>\n<li>segment: provided RoI image.</li>\n<li>label: <code>malignant</code> to cancer / <code>BIRADS 5</code> to cancer.<br>\n% Of course, the label doesn't perfectly correlate with the competition standards. But, I roughly think that maybe it could help train the model in some ways.</li></ul></li>\n<li>train with competition data (initialize the weight with the pre-trained model)<ul>\n<li>segment: inferred with the pre-trained model.</li>\n<li>auxiliary: given meta-features (total 11 features).</li></ul></li>\n<li>re-label the external data with the <code>step 2</code> model.</li>\n<li>re-train with competition data (initialize with <code>step 3</code> model)</li>\n<li>train a meta-classifier (oof + meta-features (e.g. laterality, age, …))</li>\n</ol>\n<p>For a baseline, I run step 1 ~ 2, 5 and achieve CV 0.4885 LB 0.59 (PB 0.46). Also, I test only with the <code>cbis-ddsm</code> dataset for pre-training, and there were about 0.02 drops on CV &amp; LB, but the same score on PB (CV 0.4656 LB 0.57 PB 0.46).</p>\n<p>A week before the deadline, I finished up to step ~ 5 and got CV 0.5012 LB 0.55 (PB 0.51). Sadly, I didn't choose it as a final submission : (</p>\n<p>Last day of the competition, I ensembled <code>effnet_v2_s</code> backbone and got CV 0.5063 LB 0.56 (PB 0.49).</p>\n<p>Lastly, I choose the best LB &amp; CV for the final submission.</p>\n<h3>Meta-Classifier</h3>\n<p>I built a meta-classifier with meta-features like age, laterality, and the (per-breast) statistics of the predictions. Usually, It gives ~ 0.02 improvements on the CV &amp; LB (also PB).</p>\n<p>I'm worried about overfitting into some meta-features (eg. machine id, (predicted) density, …), so I decided to use only 3 auxiliary features (age, site_id, laterality) to train the model.</p>\n<ul>\n<li>feature: age, site_id, laterality, (mean, std, min, max) of the predictions.</li>\n<li>cv: stratified k fold (5 folds)</li>\n<li>model: CatBoost</li>\n</ul>\n<h2>Works</h2>\n<ul>\n<li>higher resolution (1536x768 ~ 1024) is good.</li>\n<li>external data<ul>\n<li>it gives about +0.02 boosts.</li></ul></li>\n<li>architecture<ul>\n<li>encoder: backbone: <code>effnet-b3</code> works best</li>\n<li>decoder: u-net++</li></ul></li>\n<li>augmentation</li>\n<li>mixup (alpha 1.0)</li>\n<li>loss<ul>\n<li>0.6 * cls_loss (cross_entropy) + 0.4 * seg_loss (dice) + 0.1 * aux_loss (cross-entropy)</li></ul></li>\n<li>stratified group k fold (4 folds)</li>\n<li>meta-classifier</li>\n<li>TTA</li>\n</ul>\n<p>thanks for reading! I hope this could help you :)</p>",
      "rawMarkdown": "Hello everyone!\n\nFirst, thanks to Kaggle team and the organizers for hosting a competition! Also, congratulations to all the winners!\n\n## Data\n\n### Preprocessing\n\nMy preprocessing code heavily depends on the public notebooks (eg. remove letters, crop breast via contour).\n\n1. decode `.jpeg` with `dicomsdl` & `nvjpeg2000`.\n2. crop edge (margin pixel 10)\n3. extract breast with `opencv2` (contour based)\n4. resize to 1536x960. (I roughly guess that resizing into a 1.5 ~ 2.0 aspect ratio is fine.)\n\nIn my experiment, windowing doesn't affect the score positively, so I decide not to use it.\n\n### Augmentation\n\nHeavy augmentation works well. Light augmentation tends to overfit.\n\n* v/hflip\n* scale / rotate\n* brightness / contrast\n* cutout (coarse dropout with large patch size)\n* mixup\n\n## Architecture\n\nI couldn't spend much time running various experiments due to a lack of time & computing resources. So, I only tested few backbones & training recipes. (about 70% of my submissions are runtime errors & mistakes lol)\n\nHere's a full pipeline.\n\n1. pre-train segmentation model with the `cbis-ddsm` & `vindr` datasets.\n    * segment: provided RoI image.\n    * label: `malignant` to cancer / `BIRADS 5` to cancer.\n    % Of course, the label doesn't perfectly correlate with the competition standards. But, I roughly think that maybe it could help train the model in some ways.\n2. train with competition data (initialize the weight with the pre-trained model)\n   * segment: inferred with the pre-trained model.\n   * auxiliary: given meta-features (total 11 features).\n3. re-label the external data with the `step 2` model.\n4. re-train with competition data (initialize with `step 3` model)\n5. train a meta-classifier (oof + meta-features (e.g. laterality, age, ...))\n\nFor a baseline, I run step 1 ~ 2, 5 and achieve CV 0.4885 LB 0.59 (PB 0.46). Also, I test only with the `cbis-ddsm` dataset for pre-training, and there were about 0.02 drops on CV & LB, but the same score on PB (CV 0.4656 LB 0.57 PB 0.46).\n\nA week before the deadline, I finished up to step ~ 5 and got CV 0.5012 LB 0.55 (PB 0.51). Sadly, I didn't choose it as a final submission : (\n\nLast day of the competition, I ensembled `effnet_v2_s` backbone and got CV 0.5063 LB 0.56 (PB 0.49).\n\nLastly, I choose the best LB & CV for the final submission.\n\n### Meta-Classifier\n\nI built a meta-classifier with meta-features like age, laterality, and the (per-breast) statistics of the predictions. Usually, It gives ~ 0.02 improvements on the CV & LB (also PB).\n\nI'm worried about overfitting into some meta-features (eg. machine id, (predicted) density, ...), so I decided to use only 3 auxiliary features (age, site_id, laterality) to train the model.\n\n* feature: age, site_id, laterality, (mean, std, min, max) of the predictions.\n* cv: stratified k fold (5 folds)\n* model: CatBoost\n\n## Works\n\n* higher resolution (1536x768 ~ 1024) is good.\n* external data\n  * it gives about +0.02 boosts.\n* architecture\n  * encoder: backbone: `effnet-b3` works best\n  * decoder: u-net++\n* augmentation\n* mixup (alpha 1.0)\n* loss\n  * 0.6 * cls_loss (cross_entropy) + 0.4 * seg_loss (dice) + 0.1 * aux_loss (cross-entropy)\n* stratified group k fold (4 folds)\n* meta-classifier\n* TTA\n\nthanks for reading! I hope this could help you :)",
      "votes": 24
    },
    {
      "id": 2166771,
      "postDate": "2023-03-03T02:05:37.557Z",
      "content": "<p>Thank you for sharing the inspring solution!<br>\nI have two questions.</p>\n<blockquote>\n  <ol>\n  <li>train with competition data (initialize the weight with the pre-trained model)<br>\n  segment: inferred with the pre-trained model.<br>\n  auxiliary: given meta-features (total 11 features).</li>\n  </ol>\n</blockquote>\n<p>I guess the pre-trained model at this stage cannot predict correct region of all the competition images . Some images would be predicted wrongly and others would not be predicted at all.<br>\nHow many images could you obtain the annotated images here and how did you select them?</p>\n<blockquote>\n  <p>feature: age, site_id, laterality, (mean, std, min, max) of the predictions.</p>\n</blockquote>\n<p>\"(mean, std, min, max) of the predictions \" means the outputs from classification head?  Which stage did you switch the model from classification to segmentation? (or both?)</p>",
      "rawMarkdown": "Thank you for sharing the inspring solution!\nI have two questions.\n\n>2. train with competition data (initialize the weight with the pre-trained model)\nsegment: inferred with the pre-trained model.\nauxiliary: given meta-features (total 11 features).\n\nI guess the pre-trained model at this stage cannot predict correct region of all the competition images . Some images would be predicted wrongly and others would not be predicted at all.\nHow many images could you obtain the annotated images here and how did you select them?\n\n\n>feature: age, site_id, laterality, (mean, std, min, max) of the predictions.\n\n\"(mean, std, min, max) of the predictions \" means the outputs from classification head?  Which stage did you switch the model from classification to segmentation? (or both?)\n",
      "votes": 1,
      "replies": [
        {
          "id": 2166905,
          "postDate": "2023-03-03T05:39:57.893Z",
          "content": "<p>Q) <code>pre-trained model at this stage cannot predict correct region ~</code></p>\n<p>yes. i agree with you! But, I think that although (predicted) region is not perfect, it still can guide a model easier than without. Also, I checked segmentation boosts the cv score (I forgot the exact score), so I decided to use it.</p>\n<p>I don't remember the exact number of RoI images of the cbis &amp; vindr datasets, maybe about 20K images i guess.</p>\n<p>Q) <code>Which stage did you switch ~</code></p>\n<p>The model is segmentation model (which also has classification head too), but auxiliary head is added at the main stage!</p>\n<ul>\n<li>In the pre-training stage, the model has two heads (cls, seg)</li>\n<li>In the main stage (training with competition data), the model has three heads (cls, seg, aux)</li>\n</ul>\n<p>Q) <code>means the outputs from classification head?</code></p>\n<p>Yes, you are right and sorry for the confusion. To be more specific, the model predicts with image-level while the meta-classifier is trained with breast-level (combination of patient_id &amp; laterality). So, when i'm aggregating the prediction (to make a breast-level probability), I computes the statistics (mean, max, min, std) instead of only mean.</p>\n<p>I hope this helps : ) Thank you!</p>",
          "rawMarkdown": "Q) `pre-trained model at this stage cannot predict correct region ~`\n\nyes. i agree with you! But, I think that although (predicted) region is not perfect, it still can guide a model easier than without. Also, I checked segmentation boosts the cv score (I forgot the exact score), so I decided to use it.\n\nI don't remember the exact number of RoI images of the cbis & vindr datasets, maybe about 20K images i guess.\n\nQ) `Which stage did you switch ~`\n\nThe model is segmentation model (which also has classification head too), but auxiliary head is added at the main stage!\n\n* In the pre-training stage, the model has two heads (cls, seg)\n* In the main stage (training with competition data), the model has three heads (cls, seg, aux)\n\nQ) `means the outputs from classification head?`\n\nYes, you are right and sorry for the confusion. To be more specific, the model predicts with image-level while the meta-classifier is trained with breast-level (combination of patient_id & laterality). So, when i'm aggregating the prediction (to make a breast-level probability), I computes the statistics (mean, max, min, std) instead of only mean.\n\nI hope this helps : ) Thank you!",
          "votes": 1,
          "replies": [
            {
              "id": 2167080,
              "postDate": "2023-03-03T08:52:46.633Z",
              "content": "<p>Thank you for the reply!</p>\n<p>I see. Then,</p>\n<blockquote>\n  <p>But, I think that although (predicted) region is not perfect, it still can guide a model easier than without. Also, I checked segmentation boosts the cv score (I forgot the exact score), so I decided to use it.</p>\n</blockquote>\n<p>means you use all predicted negative/cancer images? (some cancer images don't have any cancer annotation and some negative images have cancer annotation).<br>\nAlso, could you share training settings for the segmentaiton model?:)<br>\nThanks.</p>",
              "rawMarkdown": "Thank you for the reply!\n\nI see. Then,\n>But, I think that although (predicted) region is not perfect, it still can guide a model easier than without. Also, I checked segmentation boosts the cv score (I forgot the exact score), so I decided to use it.\n\nmeans you use all predicted negative/cancer images? (some cancer images don't have any cancer annotation and some negative images have cancer annotation).\nAlso, could you share training settings for the segmentaiton model?:)\nThanks.",
              "votes": 1
            },
            {
              "id": 2167104,
              "postDate": "2023-03-03T09:11:59.413Z",
              "content": "<p>you use all predicted negative/cancer images? -&gt; yeap, I just used them all. as i remembered, maybe 6~70%  of the negative samples have an empty segment. Also, I have planned to clean &amp; remove the segments, but i got not enough time &amp; resources to do it : (</p>\n<p>for the brief settings,</p>\n<ul>\n<li>arch : effnet-v3 + u-net++</li>\n<li>input : 1x1536x960 (grayscale image)</li>\n<li>output : 1x1536x960 (segment), (1,) (label)</li>\n<li>bs : 16</li>\n<li>opt : adamw</li>\n<li>lr : cosine annealing, 1e-4 ~ 1e-6</li>\n<li>loss : dice + cross entropy</li>\n<li>mixed precision</li>\n</ul>",
              "rawMarkdown": "you use all predicted negative/cancer images? -> yeap, I just used them all. as i remembered, maybe 6~70%  of the negative samples have an empty segment. Also, I have planned to clean & remove the segments, but i got not enough time & resources to do it : (\n\nfor the brief settings,\n* arch : effnet-v3 + u-net++\n* input : 1x1536x960 (grayscale image)\n* output : 1x1536x960 (segment), (1,) (label)\n* bs : 16\n* opt : adamw\n* lr : cosine annealing, 1e-4 ~ 1e-6\n* loss : dice + cross entropy\n* mixed precision",
              "votes": 1
            },
            {
              "id": 2167375,
              "postDate": "2023-03-03T12:59:21.167Z",
              "content": "<p>I understood. Thanks! :)</p>",
              "rawMarkdown": "I understood. Thanks! :)\n"
            }
          ]
        }
      ]
    },
    {
      "id": 2165071,
      "postDate": "2023-03-02T01:05:52.200Z",
      "content": "<p>The segmentation model using external data is the model that I also planned. But I didn't because I wasn't sure if I could make the right model. It's amazing that you actually used it and did well. Congratulations and I look forward to your good solution in another competitions.</p>",
      "rawMarkdown": "The segmentation model using external data is the model that I also planned. But I didn't because I wasn't sure if I could make the right model. It's amazing that you actually used it and did well. Congratulations and I look forward to your good solution in another competitions.",
      "votes": 1,
      "replies": [
        {
          "id": 2165371,
          "postDate": "2023-03-02T06:42:44.010Z",
          "content": "<p>congrat you too to get a medal! thank you : )</p>",
          "rawMarkdown": "congrat you too to get a medal! thank you : )",
          "votes": 1
        }
      ]
    },
    {
      "id": 2163032,
      "postDate": "2023-02-28T14:45:18.093Z",
      "content": "<blockquote>\n  <p>A week before the deadline, I finished up to step ~ 5</p>\n</blockquote>\n<p>That was quite a well planned. 😃 Did you have your own model for the predicted density?</p>",
      "rawMarkdown": ">A week before the deadline, I finished up to step ~ 5\n\nThat was quite a well planned. 😃 Did you have your own model for the predicted density?",
      "votes": 1,
      "replies": [
        {
          "id": 2163069,
          "postDate": "2023-02-28T15:09:25.973Z",
          "content": "<blockquote>\n  <p>Did you have your own model for the predicted density?</p>\n</blockquote>\n<p>yes! the model is also trained with predicting the auxiliary features (eg. BIRADS, density, age, …). But, I didn't check the accuracy of each feature.</p>",
          "rawMarkdown": "> Did you have your own model for the predicted density?\n\nyes! the model is also trained with predicting the auxiliary features (eg. BIRADS, density, age, ...). But, I didn't check the accuracy of each feature.",
          "votes": 1,
          "replies": [
            {
              "id": 2163073,
              "postDate": "2023-02-28T15:11:55.150Z",
              "content": "<p>Rough estimate can be a good one, if it is consistent. 👍</p>",
              "rawMarkdown": "Rough estimate can be a good one, if it is consistent. 👍",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2163006,
      "postDate": "2023-02-28T14:28:33.800Z",
      "content": "<p>congratulations !!! Can you share your github?</p>",
      "rawMarkdown": "congratulations !!! Can you share your github?",
      "votes": 1,
      "replies": [
        {
          "id": 2163060,
          "postDate": "2023-02-28T15:04:48.700Z",
          "content": "<p>thank you! my experiment codes are messy and unorganized, maybe it'll take some time to clean up. I'll try!</p>",
          "rawMarkdown": "thank you! my experiment codes are messy and unorganized, maybe it'll take some time to clean up. I'll try!",
          "votes": 1,
          "replies": [
            {
              "id": 2163936,
              "postDate": "2023-03-01T07:27:19.433Z",
              "content": "<p>Can you explain why applying segmentation to this task can improve the score? I have just studied.Thanks</p>",
              "rawMarkdown": "Can you explain why applying segmentation to this task can improve the score? I have just studied.Thanks"
            },
            {
              "id": 2165368,
              "postDate": "2023-03-02T06:39:38.010Z",
              "content": "<p>I initially think the training with the RoI (Region of Interest) image (of the external datasets) would help better convergence than training only with the label (cancer). In my experiment, model trained only with label tends to fit a wrong region (eg. edge or sth else) in a few cases.</p>",
              "rawMarkdown": "I initially think the training with the RoI (Region of Interest) image (of the external datasets) would help better convergence than training only with the label (cancer). In my experiment, model trained only with label tends to fit a wrong region (eg. edge or sth else) in a few cases.",
              "votes": 1
            },
            {
              "id": 2166015,
              "postDate": "2023-03-02T15:26:29.257Z",
              "content": "<p>Perhaps the edges are the parts which explain the model decision in those particular cases? And discriminate those anomalies from others.</p>",
              "rawMarkdown": "Perhaps the edges are the parts which explain the model decision in those particular cases? And discriminate those anomalies from others."
            },
            {
              "id": 2167042,
              "postDate": "2023-03-03T08:21:05.383Z",
              "content": "<p>Really thanks for your help 👍</p>",
              "rawMarkdown": "Really thanks for your help 👍"
            }
          ]
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2166771,
      "author_name": "opusen",
      "author_url": "",
      "post_date": "2023-03-03T02:05:37.557000",
      "content": "<p>Thank you for sharing the inspring solution!<br>\nI have two questions.</p>\n<blockquote>\n  <ol>\n  <li>train with competition data (initialize the weight with the pre-trained model)<br>\n  segment: inferred with the pre-trained model.<br>\n  auxiliary: given meta-features (total 11 features).</li>\n  </ol>\n</blockquote>\n<p>I guess the pre-trained model at this stage cannot predict correct region of all the competition images . Some images would be predicted wrongly and others would not be predicted at all.<br>\nHow many images could you obtain the annotated images here and how did you select them?</p>\n<blockquote>\n  <p>feature: age, site_id, laterality, (mean, std, min, max) of the predictions.</p>\n</blockquote>\n<p>\"(mean, std, min, max) of the predictions \" means the outputs from classification head?  Which stage did you switch the model from classification to segmentation? (or both?)</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2166905,
          "author_name": "HyeongChan Kim",
          "author_url": "",
          "post_date": "2023-03-03T05:39:57.893000",
          "content": "<p>Q) <code>pre-trained model at this stage cannot predict correct region ~</code></p>\n<p>yes. i agree with you! But, I think that although (predicted) region is not perfect, it still can guide a model easier than without. Also, I checked segmentation boosts the cv score (I forgot the exact score), so I decided to use it.</p>\n<p>I don't remember the exact number of RoI images of the cbis &amp; vindr datasets, maybe about 20K images i guess.</p>\n<p>Q) <code>Which stage did you switch ~</code></p>\n<p>The model is segmentation model (which also has classification head too), but auxiliary head is added at the main stage!</p>\n<ul>\n<li>In the pre-training stage, the model has two heads (cls, seg)</li>\n<li>In the main stage (training with competition data), the model has three heads (cls, seg, aux)</li>\n</ul>\n<p>Q) <code>means the outputs from classification head?</code></p>\n<p>Yes, you are right and sorry for the confusion. To be more specific, the model predicts with image-level while the meta-classifier is trained with breast-level (combination of patient_id &amp; laterality). So, when i'm aggregating the prediction (to make a breast-level probability), I computes the statistics (mean, max, min, std) instead of only mean.</p>\n<p>I hope this helps : ) Thank you!</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2167080,
              "author_name": "opusen",
              "author_url": "",
              "post_date": "2023-03-03T08:52:46.633000",
              "content": "<p>Thank you for the reply!</p>\n<p>I see. Then,</p>\n<blockquote>\n  <p>But, I think that although (predicted) region is not perfect, it still can guide a model easier than without. Also, I checked segmentation boosts the cv score (I forgot the exact score), so I decided to use it.</p>\n</blockquote>\n<p>means you use all predicted negative/cancer images? (some cancer images don't have any cancer annotation and some negative images have cancer annotation).<br>\nAlso, could you share training settings for the segmentaiton model?:)<br>\nThanks.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2167104,
              "author_name": "HyeongChan Kim",
              "author_url": "",
              "post_date": "2023-03-03T09:11:59.413000",
              "content": "<p>you use all predicted negative/cancer images? -&gt; yeap, I just used them all. as i remembered, maybe 6~70%  of the negative samples have an empty segment. Also, I have planned to clean &amp; remove the segments, but i got not enough time &amp; resources to do it : (</p>\n<p>for the brief settings,</p>\n<ul>\n<li>arch : effnet-v3 + u-net++</li>\n<li>input : 1x1536x960 (grayscale image)</li>\n<li>output : 1x1536x960 (segment), (1,) (label)</li>\n<li>bs : 16</li>\n<li>opt : adamw</li>\n<li>lr : cosine annealing, 1e-4 ~ 1e-6</li>\n<li>loss : dice + cross entropy</li>\n<li>mixed precision</li>\n</ul>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2167375,
              "author_name": "opusen",
              "author_url": "",
              "post_date": "2023-03-03T12:59:21.167000",
              "content": "<p>I understood. Thanks! :)</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2165071,
      "author_name": "olivepicker",
      "author_url": "",
      "post_date": "2023-03-02T01:05:52.200000",
      "content": "<p>The segmentation model using external data is the model that I also planned. But I didn't because I wasn't sure if I could make the right model. It's amazing that you actually used it and did well. Congratulations and I look forward to your good solution in another competitions.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2165371,
          "author_name": "HyeongChan Kim",
          "author_url": "",
          "post_date": "2023-03-02T06:42:44.010000",
          "content": "<p>congrat you too to get a medal! thank you : )</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2163032,
      "author_name": "Antti Isosalo",
      "author_url": "",
      "post_date": "2023-02-28T14:45:18.093000",
      "content": "<blockquote>\n  <p>A week before the deadline, I finished up to step ~ 5</p>\n</blockquote>\n<p>That was quite a well planned. 😃 Did you have your own model for the predicted density?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2163069,
          "author_name": "HyeongChan Kim",
          "author_url": "",
          "post_date": "2023-02-28T15:09:25.973000",
          "content": "<blockquote>\n  <p>Did you have your own model for the predicted density?</p>\n</blockquote>\n<p>yes! the model is also trained with predicting the auxiliary features (eg. BIRADS, density, age, …). But, I didn't check the accuracy of each feature.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2163073,
              "author_name": "Antti Isosalo",
              "author_url": "",
              "post_date": "2023-02-28T15:11:55.150000",
              "content": "<p>Rough estimate can be a good one, if it is consistent. 👍</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2163006,
      "author_name": "Khang Duong",
      "author_url": "",
      "post_date": "2023-02-28T14:28:33.800000",
      "content": "<p>congratulations !!! Can you share your github?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2163060,
          "author_name": "HyeongChan Kim",
          "author_url": "",
          "post_date": "2023-02-28T15:04:48.700000",
          "content": "<p>thank you! my experiment codes are messy and unorganized, maybe it'll take some time to clean up. I'll try!</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2163936,
              "author_name": "Khang Duong",
              "author_url": "",
              "post_date": "2023-03-01T07:27:19.433000",
              "content": "<p>Can you explain why applying segmentation to this task can improve the score? I have just studied.Thanks</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2165368,
              "author_name": "HyeongChan Kim",
              "author_url": "",
              "post_date": "2023-03-02T06:39:38.010000",
              "content": "<p>I initially think the training with the RoI (Region of Interest) image (of the external datasets) would help better convergence than training only with the label (cancer). In my experiment, model trained only with label tends to fit a wrong region (eg. edge or sth else) in a few cases.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2166015,
              "author_name": "Antti Isosalo",
              "author_url": "",
              "post_date": "2023-03-02T15:26:29.257000",
              "content": "<p>Perhaps the edges are the parts which explain the model decision in those particular cases? And discriminate those anomalies from others.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2167042,
              "author_name": "Khang Duong",
              "author_url": "",
              "post_date": "2023-03-03T08:21:05.383000",
              "content": "<p>Really thanks for your help 👍</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2162937": "Hello everyone!\n\nFirst, thanks to Kaggle team and the organizers for hosting a competition! Also, congratulations to all the winners!\n\n## Data\n\n### Preprocessing\n\nMy preprocessing code heavily depends on the public notebooks (eg. remove letters, crop breast via contour).\n\n1. decode `.jpeg` with `dicomsdl` & `nvjpeg2000`.\n2. crop edge (margin pixel 10)\n3. extract breast with `opencv2` (contour based)\n4. resize to 1536x960. (I roughly guess that resizing into a 1.5 ~ 2.0 aspect ratio is fine.)\n\nIn my experiment, windowing doesn't affect the score positively, so I decide not to use it.\n\n### Augmentation\n\nHeavy augmentation works well. Light augmentation tends to overfit.\n\n* v/hflip\n* scale / rotate\n* brightness / contrast\n* cutout (coarse dropout with large patch size)\n* mixup\n\n## Architecture\n\nI couldn't spend much time running various experiments due to a lack of time & computing resources. So, I only tested few backbones & training recipes. (about 70% of my submissions are runtime errors & mistakes lol)\n\nHere's a full pipeline.\n\n1. pre-train segmentation model with the `cbis-ddsm` & `vindr` datasets.\n    * segment: provided RoI image.\n    * label: `malignant` to cancer / `BIRADS 5` to cancer.\n    % Of course, the label doesn't perfectly correlate with the competition standards. But, I roughly think that maybe it could help train the model in some ways.\n2. train with competition data (initialize the weight with the pre-trained model)\n   * segment: inferred with the pre-trained model.\n   * auxiliary: given meta-features (total 11 features).\n3. re-label the external data with the `step 2` model.\n4. re-train with competition data (initialize with `step 3` model)\n5. train a meta-classifier (oof + meta-features (e.g. laterality, age, ...))\n\nFor a baseline, I run step 1 ~ 2, 5 and achieve CV 0.4885 LB 0.59 (PB 0.46). Also, I test only with the `cbis-ddsm` dataset for pre-training, and there were about 0.02 drops on CV & LB, but the same score on PB (CV 0.4656 LB 0.57 PB 0.46).\n\nA week before the deadline, I finished up to step ~ 5 and got CV 0.5012 LB 0.55 (PB 0.51). Sadly, I didn't choose it as a final submission : (\n\nLast day of the competition, I ensembled `effnet_v2_s` backbone and got CV 0.5063 LB 0.56 (PB 0.49).\n\nLastly, I choose the best LB & CV for the final submission.\n\n### Meta-Classifier\n\nI built a meta-classifier with meta-features like age, laterality, and the (per-breast) statistics of the predictions. Usually, It gives ~ 0.02 improvements on the CV & LB (also PB).\n\nI'm worried about overfitting into some meta-features (eg. machine id, (predicted) density, ...), so I decided to use only 3 auxiliary features (age, site_id, laterality) to train the model.\n\n* feature: age, site_id, laterality, (mean, std, min, max) of the predictions.\n* cv: stratified k fold (5 folds)\n* model: CatBoost\n\n## Works\n\n* higher resolution (1536x768 ~ 1024) is good.\n* external data\n  * it gives about +0.02 boosts.\n* architecture\n  * encoder: backbone: `effnet-b3` works best\n  * decoder: u-net++\n* augmentation\n* mixup (alpha 1.0)\n* loss\n  * 0.6 * cls_loss (cross_entropy) + 0.4 * seg_loss (dice) + 0.1 * aux_loss (cross-entropy)\n* stratified group k fold (4 folds)\n* meta-classifier\n* TTA\n\nthanks for reading! I hope this could help you :)",
    "2166771": "Thank you for sharing the inspring solution!\nI have two questions.\n\n>2. train with competition data (initialize the weight with the pre-trained model)\nsegment: inferred with the pre-trained model.\nauxiliary: given meta-features (total 11 features).\n\nI guess the pre-trained model at this stage cannot predict correct region of all the competition images . Some images would be predicted wrongly and others would not be predicted at all.\nHow many images could you obtain the annotated images here and how did you select them?\n\n\n>feature: age, site_id, laterality, (mean, std, min, max) of the predictions.\n\n\"(mean, std, min, max) of the predictions \" means the outputs from classification head?  Which stage did you switch the model from classification to segmentation? (or both?)\n",
    "2165071": "The segmentation model using external data is the model that I also planned. But I didn't because I wasn't sure if I could make the right model. It's amazing that you actually used it and did well. Congratulations and I look forward to your good solution in another competitions.",
    "2163032": ">A week before the deadline, I finished up to step ~ 5\n\nThat was quite a well planned. 😃 Did you have your own model for the predicted density?",
    "2163006": "congratulations !!! Can you share your github?"
  }
}