{
  "id": 357208,
  "title": "Submission Scoring Error",
  "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/357208",
  "author_name": "MatejGazda",
  "post_date": "2022-10-03T14:48:28.471000",
  "votes": 0,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Dear community,</p>\n<p>I am trying to submit 3D model and unfortunately, I am receiving Submission Scoring Error all the time. The problem is that </p>\n<p>When I use diff on file from successfuly submitted  <a href=\"https://www.kaggle.com/code/vslaykovsky/infer-pytorch-effnetv2-single-model-lb-0-49\" target=\"_blank\">https://www.kaggle.com/code/vslaykovsky/infer-pytorch-effnetv2-single-model-lb-0-49</a> I got no substantial differences. As you can see below, the files are identical except the predicted values.</p>\n<pre><code>(base) ➜  Downloads diff 'submission (5).csv' 'submission (4).csv'\n2,4c2,4\n&lt; 1.2.826.0.1.3680043.22327_C1,0.03217704966664314\n&lt; 1.2.826.0.1.3680043.25399_C1,0.07394803315401077\n&lt; 1.2.826.0.1.3680043.5876_C1,0.34954309463500977\n---\n&gt; 1.2.826.0.1.3680043.22327_C1,0.14709191\n&gt; 1.2.826.0.1.3680043.25399_C1,0.13945813\n&gt; 1.2.826.0.1.3680043.5876_C1,0.6463179\n</code></pre>\n<p>How to debug such case? </p>\n<pre><code>dataset = RSNADataset(test_images_path, transforms = Compose([ScaleIntensity(0, 1)]))\ntest_dataloader = DataLoader(dataset, batch_size=1)\n\ntrainer = pl.Trainer(accelerator=\"gpu\", devices=1)\nmodel = MonaiModel.load_from_checkpoint(model_path)\npreds = trainer.predict(model, test_dataloader)\npredictions = []\nsubmission = pd.DataFrame(columns=[\"row_id\", \"fractured\"])\n\nfor prediction in preds:\n    name = prediction['name'][0]\n    predictions.append({\n        \"name\": prediction['name'][0],\n        \"patient_overall\": prediction['prediction'][0][0].cpu().detach().numpy(),\n        \"C1\": prediction['prediction'][0][1].cpu().detach().numpy(),\n        \"C2\": prediction['prediction'][0][2].cpu().detach().numpy(),\n        \"C3\": prediction['prediction'][0][3].cpu().detach().numpy(),\n        \"C4\": prediction['prediction'][0][4].cpu().detach().numpy(),\n        \"C5\": prediction['prediction'][0][5].cpu().detach().numpy(),\n        \"C6\": prediction['prediction'][0][6].cpu().detach().numpy(),\n        \"C7\": prediction['prediction'][0][7].cpu().detach().numpy(),\n        \"prediction_type\": metadata.loc[metadata['StudyInstanceUID'] == name]['prediction_type'].iloc[0],\n        \"row_id\": metadata.loc[metadata['StudyInstanceUID'] == name]['row_id'].iloc[0],\n    })\n\nprediction_df = pd.DataFrame(predictions)\nprediction_df['fractured'] = prediction_df.apply(lambda r: r[r.prediction_type], axis=1)\nprediction_df[['row_id', 'fractured']].to_csv('submission.csv', index=False)\nprint(prediction_df)\n</code></pre>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F413177%2F8064b8aee8a05f2a94cbcd4cc2d0088a%2F1664808767.png?generation=1664808793499308&amp;alt=media\" alt=\"\"></p>\n<p>Thanks,</p>\n<p>Matej</p>",
  "messages": [
    {
      "id": 1972818,
      "postDate": "2022-10-05T10:53:42.377Z",
      "content": "<p>unfortunately, kaggle system hasn't been really informative. I had a break and redid everything from scratch and now it works. </p>",
      "rawMarkdown": "unfortunately, kaggle system hasn't been really informative. I had a break and redid everything from scratch and now it works. "
    },
    {
      "id": 1969555,
      "postDate": "2022-10-03T14:48:28.473Z",
      "content": "<p>Dear community,</p>\n<p>I am trying to submit 3D model and unfortunately, I am receiving Submission Scoring Error all the time. The problem is that </p>\n<p>When I use diff on file from successfuly submitted  <a href=\"https://www.kaggle.com/code/vslaykovsky/infer-pytorch-effnetv2-single-model-lb-0-49\" target=\"_blank\">https://www.kaggle.com/code/vslaykovsky/infer-pytorch-effnetv2-single-model-lb-0-49</a> I got no substantial differences. As you can see below, the files are identical except the predicted values.</p>\n<pre><code>(base) ➜  Downloads diff 'submission (5).csv' 'submission (4).csv'\n2,4c2,4\n&lt; 1.2.826.0.1.3680043.22327_C1,0.03217704966664314\n&lt; 1.2.826.0.1.3680043.25399_C1,0.07394803315401077\n&lt; 1.2.826.0.1.3680043.5876_C1,0.34954309463500977\n---\n&gt; 1.2.826.0.1.3680043.22327_C1,0.14709191\n&gt; 1.2.826.0.1.3680043.25399_C1,0.13945813\n&gt; 1.2.826.0.1.3680043.5876_C1,0.6463179\n</code></pre>\n<p>How to debug such case? </p>\n<pre><code>dataset = RSNADataset(test_images_path, transforms = Compose([ScaleIntensity(0, 1)]))\ntest_dataloader = DataLoader(dataset, batch_size=1)\n\ntrainer = pl.Trainer(accelerator=\"gpu\", devices=1)\nmodel = MonaiModel.load_from_checkpoint(model_path)\npreds = trainer.predict(model, test_dataloader)\npredictions = []\nsubmission = pd.DataFrame(columns=[\"row_id\", \"fractured\"])\n\nfor prediction in preds:\n    name = prediction['name'][0]\n    predictions.append({\n        \"name\": prediction['name'][0],\n        \"patient_overall\": prediction['prediction'][0][0].cpu().detach().numpy(),\n        \"C1\": prediction['prediction'][0][1].cpu().detach().numpy(),\n        \"C2\": prediction['prediction'][0][2].cpu().detach().numpy(),\n        \"C3\": prediction['prediction'][0][3].cpu().detach().numpy(),\n        \"C4\": prediction['prediction'][0][4].cpu().detach().numpy(),\n        \"C5\": prediction['prediction'][0][5].cpu().detach().numpy(),\n        \"C6\": prediction['prediction'][0][6].cpu().detach().numpy(),\n        \"C7\": prediction['prediction'][0][7].cpu().detach().numpy(),\n        \"prediction_type\": metadata.loc[metadata['StudyInstanceUID'] == name]['prediction_type'].iloc[0],\n        \"row_id\": metadata.loc[metadata['StudyInstanceUID'] == name]['row_id'].iloc[0],\n    })\n\nprediction_df = pd.DataFrame(predictions)\nprediction_df['fractured'] = prediction_df.apply(lambda r: r[r.prediction_type], axis=1)\nprediction_df[['row_id', 'fractured']].to_csv('submission.csv', index=False)\nprint(prediction_df)\n</code></pre>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F413177%2F8064b8aee8a05f2a94cbcd4cc2d0088a%2F1664808767.png?generation=1664808793499308&amp;alt=media\" alt=\"\"></p>\n<p>Thanks,</p>\n<p>Matej</p>",
      "rawMarkdown": "Dear community,\n\nI am trying to submit 3D model and unfortunately, I am receiving Submission Scoring Error all the time. The problem is that \n\nWhen I use diff on file from successfuly submitted  https://www.kaggle.com/code/vslaykovsky/infer-pytorch-effnetv2-single-model-lb-0-49 I got no substantial differences. As you can see below, the files are identical except the predicted values.\n\n```\n(base) ➜  Downloads diff 'submission (5).csv' 'submission (4).csv'\n2,4c2,4\n< 1.2.826.0.1.3680043.22327_C1,0.03217704966664314\n< 1.2.826.0.1.3680043.25399_C1,0.07394803315401077\n< 1.2.826.0.1.3680043.5876_C1,0.34954309463500977\n---\n> 1.2.826.0.1.3680043.22327_C1,0.14709191\n> 1.2.826.0.1.3680043.25399_C1,0.13945813\n> 1.2.826.0.1.3680043.5876_C1,0.6463179\n```\n\nHow to debug such case? \n\n```\ndataset = RSNADataset(test_images_path, transforms = Compose([ScaleIntensity(0, 1)]))\ntest_dataloader = DataLoader(dataset, batch_size=1)\n\ntrainer = pl.Trainer(accelerator=\"gpu\", devices=1)\nmodel = MonaiModel.load_from_checkpoint(model_path)\npreds = trainer.predict(model, test_dataloader)\npredictions = []\nsubmission = pd.DataFrame(columns=[\"row_id\", \"fractured\"])\n\nfor prediction in preds:\n    name = prediction['name'][0]\n    predictions.append({\n        \"name\": prediction['name'][0],\n        \"patient_overall\": prediction['prediction'][0][0].cpu().detach().numpy(),\n        \"C1\": prediction['prediction'][0][1].cpu().detach().numpy(),\n        \"C2\": prediction['prediction'][0][2].cpu().detach().numpy(),\n        \"C3\": prediction['prediction'][0][3].cpu().detach().numpy(),\n        \"C4\": prediction['prediction'][0][4].cpu().detach().numpy(),\n        \"C5\": prediction['prediction'][0][5].cpu().detach().numpy(),\n        \"C6\": prediction['prediction'][0][6].cpu().detach().numpy(),\n        \"C7\": prediction['prediction'][0][7].cpu().detach().numpy(),\n        \"prediction_type\": metadata.loc[metadata['StudyInstanceUID'] == name]['prediction_type'].iloc[0],\n        \"row_id\": metadata.loc[metadata['StudyInstanceUID'] == name]['row_id'].iloc[0],\n    })\n\nprediction_df = pd.DataFrame(predictions)\nprediction_df['fractured'] = prediction_df.apply(lambda r: r[r.prediction_type], axis=1)\nprediction_df[['row_id', 'fractured']].to_csv('submission.csv', index=False)\nprint(prediction_df)\n\n```\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F413177%2F8064b8aee8a05f2a94cbcd4cc2d0088a%2F1664808767.png?generation=1664808793499308&alt=media)\n\n\n\n\nThanks,\n\nMatej"
    }
  ],
  "comments": [
    {
      "id": 1972818,
      "author_name": "MatejGazda",
      "author_url": "",
      "post_date": "2022-10-05T10:53:42.377000",
      "content": "<p>unfortunately, kaggle system hasn't been really informative. I had a break and redid everything from scratch and now it works. </p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1972818": "unfortunately, kaggle system hasn't been really informative. I had a break and redid everything from scratch and now it works. ",
    "1969555": "Dear community,\n\nI am trying to submit 3D model and unfortunately, I am receiving Submission Scoring Error all the time. The problem is that \n\nWhen I use diff on file from successfuly submitted  https://www.kaggle.com/code/vslaykovsky/infer-pytorch-effnetv2-single-model-lb-0-49 I got no substantial differences. As you can see below, the files are identical except the predicted values.\n\n```\n(base) ➜  Downloads diff 'submission (5).csv' 'submission (4).csv'\n2,4c2,4\n< 1.2.826.0.1.3680043.22327_C1,0.03217704966664314\n< 1.2.826.0.1.3680043.25399_C1,0.07394803315401077\n< 1.2.826.0.1.3680043.5876_C1,0.34954309463500977\n---\n> 1.2.826.0.1.3680043.22327_C1,0.14709191\n> 1.2.826.0.1.3680043.25399_C1,0.13945813\n> 1.2.826.0.1.3680043.5876_C1,0.6463179\n```\n\nHow to debug such case? \n\n```\ndataset = RSNADataset(test_images_path, transforms = Compose([ScaleIntensity(0, 1)]))\ntest_dataloader = DataLoader(dataset, batch_size=1)\n\ntrainer = pl.Trainer(accelerator=\"gpu\", devices=1)\nmodel = MonaiModel.load_from_checkpoint(model_path)\npreds = trainer.predict(model, test_dataloader)\npredictions = []\nsubmission = pd.DataFrame(columns=[\"row_id\", \"fractured\"])\n\nfor prediction in preds:\n    name = prediction['name'][0]\n    predictions.append({\n        \"name\": prediction['name'][0],\n        \"patient_overall\": prediction['prediction'][0][0].cpu().detach().numpy(),\n        \"C1\": prediction['prediction'][0][1].cpu().detach().numpy(),\n        \"C2\": prediction['prediction'][0][2].cpu().detach().numpy(),\n        \"C3\": prediction['prediction'][0][3].cpu().detach().numpy(),\n        \"C4\": prediction['prediction'][0][4].cpu().detach().numpy(),\n        \"C5\": prediction['prediction'][0][5].cpu().detach().numpy(),\n        \"C6\": prediction['prediction'][0][6].cpu().detach().numpy(),\n        \"C7\": prediction['prediction'][0][7].cpu().detach().numpy(),\n        \"prediction_type\": metadata.loc[metadata['StudyInstanceUID'] == name]['prediction_type'].iloc[0],\n        \"row_id\": metadata.loc[metadata['StudyInstanceUID'] == name]['row_id'].iloc[0],\n    })\n\nprediction_df = pd.DataFrame(predictions)\nprediction_df['fractured'] = prediction_df.apply(lambda r: r[r.prediction_type], axis=1)\nprediction_df[['row_id', 'fractured']].to_csv('submission.csv', index=False)\nprint(prediction_df)\n\n```\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F413177%2F8064b8aee8a05f2a94cbcd4cc2d0088a%2F1664808767.png?generation=1664808793499308&alt=media)\n\n\n\n\nThanks,\n\nMatej"
  }
}