{
  "id": 360658,
  "title": "Federated Learning to extend Training Time",
  "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/360658",
  "author_name": "Anouk Stein, MD",
  "post_date": "2022-10-17T16:30:37.411000",
  "votes": 15,
  "comment_count": 5,
  "views": 0,
  "content": "<p>In running a model on Kaggle, I exceeded the time limit, so in order to train the entire dataset I implemented a Federated Learning approach.</p>\n<ol>\n<li>First, the model was trained separately on the first and second half of the data.</li>\n<li>The two model weights were averaged using code from <a href=\"https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008\" target=\"_blank\">https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008</a></li>\n<li>The new model was trained on the mid one half of the data.</li>\n</ol>\n<h4>Fastai load_learner from exported models</h4>\n<p>learnA = load_learner('../input/models/model_first_ half.pkl', cpu = False)<br>\nlearnB = load_learner('../input/models/model_second_ half.pkl', cpu = False)</p>\n<p>modelA = learnA.model<br>\nmodelB = learnB.model</p>\n<p>sdA = modelA.state_dict()<br>\nsdB = modelB.state_dict()</p>\n<h4>Average all parameters</h4>\n<p>for key in sdA:<br>\n    sdB[key] = (sdB[key] + sdA[key]) / 2.</p>\n<h4>Recreate model and load averaged state_dict (or use modelA/B)</h4>\n<p>model_fed = MyModel()<br>\nmodel_fed.load_state_dict(sdB)</p>\n<h4>Train new model on middle of  dataset eg. train_df.iloc[500:1500]</h4>",
  "messages": [
    {
      "id": 1992330,
      "postDate": "2022-10-17T16:30:37.410Z",
      "content": "<p>In running a model on Kaggle, I exceeded the time limit, so in order to train the entire dataset I implemented a Federated Learning approach.</p>\n<ol>\n<li>First, the model was trained separately on the first and second half of the data.</li>\n<li>The two model weights were averaged using code from <a href=\"https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008\" target=\"_blank\">https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008</a></li>\n<li>The new model was trained on the mid one half of the data.</li>\n</ol>\n<h4>Fastai load_learner from exported models</h4>\n<p>learnA = load_learner('../input/models/model_first_ half.pkl', cpu = False)<br>\nlearnB = load_learner('../input/models/model_second_ half.pkl', cpu = False)</p>\n<p>modelA = learnA.model<br>\nmodelB = learnB.model</p>\n<p>sdA = modelA.state_dict()<br>\nsdB = modelB.state_dict()</p>\n<h4>Average all parameters</h4>\n<p>for key in sdA:<br>\n    sdB[key] = (sdB[key] + sdA[key]) / 2.</p>\n<h4>Recreate model and load averaged state_dict (or use modelA/B)</h4>\n<p>model_fed = MyModel()<br>\nmodel_fed.load_state_dict(sdB)</p>\n<h4>Train new model on middle of  dataset eg. train_df.iloc[500:1500]</h4>",
      "rawMarkdown": "In running a model on Kaggle, I exceeded the time limit, so in order to train the entire dataset I implemented a Federated Learning approach.\n\n1. First, the model was trained separately on the first and second half of the data.\n2. The two model weights were averaged using code from https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008\n3. The new model was trained on the mid one half of the data.\n\n#### Fastai load_learner from exported models\nlearnA = load_learner('../input/models/model_first_ half.pkl', cpu = False)\nlearnB = load_learner('../input/models/model_second_ half.pkl', cpu = False)\n\nmodelA = learnA.model\nmodelB = learnB.model\n\nsdA = modelA.state_dict()\nsdB = modelB.state_dict()\n\n#### Average all parameters\nfor key in sdA:\n    sdB[key] = (sdB[key] + sdA[key]) / 2.\n\n#### Recreate model and load averaged state_dict (or use modelA/B)\nmodel_fed = MyModel()\nmodel_fed.load_state_dict(sdB)\n\n#### Train new model on middle of  dataset eg. train_df.iloc[500:1500]",
      "votes": 15
    },
    {
      "id": 1992831,
      "postDate": "2022-10-18T01:04:00.163Z",
      "content": "<p>I think your approach is amazing and could be deployed for real-time data! Please do review my work and give your feedback to whenever you get time 😊</p>",
      "rawMarkdown": "I think your approach is amazing and could be deployed for real-time data! Please do review my work and give your feedback to whenever you get time 😊",
      "votes": -1
    },
    {
      "id": 1993054,
      "postDate": "2022-10-18T05:12:58.597Z",
      "content": "<p>I may not choose to manipulate with weights directly. Have you ever tried ensembled models or training with teacher-student networks?</p>",
      "rawMarkdown": " I may not choose to manipulate with weights directly. Have you ever tried ensembled models or training with teacher-student networks?",
      "replies": [
        {
          "id": 1993800,
          "postDate": "2022-10-18T15:43:30.977Z",
          "content": "<p>I have not tried either and I'm actually struggling to get past Submission Not Found errors not matter what I try. Thank you for your comment!</p>",
          "rawMarkdown": "I have not tried either and I'm actually struggling to get past Submission Not Found errors not matter what I try. Thank you for your comment!"
        },
        {
          "id": 1997487,
          "postDate": "2022-10-21T00:17:09.110Z",
          "content": "<p>You may read <a href=\"https://www.kaggle.com/code/fx6300/kaggle-error\" target=\"_blank\">this post</a>. It shows a proper way to how to generate a submission file.</p>",
          "rawMarkdown": "You may read [this post](https://www.kaggle.com/code/fx6300/kaggle-error). It shows a proper way to how to generate a submission file.",
          "votes": 1
        },
        {
          "id": 1997500,
          "postDate": "2022-10-21T00:37:59.630Z",
          "content": "<p>Yes that notebook was very helpful!! I tried saving the jpg to kaggle/ rather than kaggle/working and it got past the sticking point :-)</p>",
          "rawMarkdown": "Yes that notebook was very helpful!! I tried saving the jpg to kaggle/ rather than kaggle/working and it got past the sticking point :-)",
          "votes": 1
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 1992831,
      "author_name": "Muhammad Ammar Jamshed",
      "author_url": "",
      "post_date": "2022-10-18T01:04:00.163000",
      "content": "<p>I think your approach is amazing and could be deployed for real-time data! Please do review my work and give your feedback to whenever you get time 😊</p>",
      "votes": -1,
      "replies": []
    },
    {
      "id": 1993054,
      "author_name": "Chenjie",
      "author_url": "",
      "post_date": "2022-10-18T05:12:58.597000",
      "content": "<p>I may not choose to manipulate with weights directly. Have you ever tried ensembled models or training with teacher-student networks?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1993800,
          "author_name": "Anouk Stein, MD",
          "author_url": "",
          "post_date": "2022-10-18T15:43:30.977000",
          "content": "<p>I have not tried either and I'm actually struggling to get past Submission Not Found errors not matter what I try. Thank you for your comment!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1997487,
          "author_name": "Chenjie",
          "author_url": "",
          "post_date": "2022-10-21T00:17:09.110000",
          "content": "<p>You may read <a href=\"https://www.kaggle.com/code/fx6300/kaggle-error\" target=\"_blank\">this post</a>. It shows a proper way to how to generate a submission file.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1997500,
          "author_name": "Anouk Stein, MD",
          "author_url": "",
          "post_date": "2022-10-21T00:37:59.630000",
          "content": "<p>Yes that notebook was very helpful!! I tried saving the jpg to kaggle/ rather than kaggle/working and it got past the sticking point :-)</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1992330": "In running a model on Kaggle, I exceeded the time limit, so in order to train the entire dataset I implemented a Federated Learning approach.\n\n1. First, the model was trained separately on the first and second half of the data.\n2. The two model weights were averaged using code from https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008\n3. The new model was trained on the mid one half of the data.\n\n#### Fastai load_learner from exported models\nlearnA = load_learner('../input/models/model_first_ half.pkl', cpu = False)\nlearnB = load_learner('../input/models/model_second_ half.pkl', cpu = False)\n\nmodelA = learnA.model\nmodelB = learnB.model\n\nsdA = modelA.state_dict()\nsdB = modelB.state_dict()\n\n#### Average all parameters\nfor key in sdA:\n    sdB[key] = (sdB[key] + sdA[key]) / 2.\n\n#### Recreate model and load averaged state_dict (or use modelA/B)\nmodel_fed = MyModel()\nmodel_fed.load_state_dict(sdB)\n\n#### Train new model on middle of  dataset eg. train_df.iloc[500:1500]",
    "1992831": "I think your approach is amazing and could be deployed for real-time data! Please do review my work and give your feedback to whenever you get time 😊",
    "1993054": " I may not choose to manipulate with weights directly. Have you ever tried ensembled models or training with teacher-student networks?"
  }
}