{ "id": 339590, "title": "What's wrong with this submission.csv ?", "url": "/competitions/mayo-clinic-strip-ai/discussion/339590", "author_name": "Pathik Ghugare", "post_date": "2022-07-25T15:06:16.393000", "votes": 8, "comment_count": 45, "views": 0, "content": "

I had created two notebooks,

\n
    \n
  1. Training model
    2. \n
  2. Submitting csv file
    3. \n
\n

In the 2nd notebook, I've internet turned off with the following submission.csv:
\n\"image\"

\n

which is the same format as mentioned in given sample_submission.csv
\n\"image\"

\n

but on submission of the same I got Submission Scoring Error so what should I do?
\nare there any other requirements which I need to follow?

\n

Please let me know,
\nThanks

\n

Edit :
\nDo we need to have our model in the same notebook as submission notebook?
\nI tried submitting submission.csv in a separate notebook and it's always causing an error

", "messages": [ { "id": 1870463, "postDate": "2022-07-25T15:06:16.393Z", "content": "

I had created two notebooks,

\n
    \n
  1. Training model
    2. \n
  2. Submitting csv file
    3. \n
\n

In the 2nd notebook, I've internet turned off with the following submission.csv:
\n\"image\"

\n

which is the same format as mentioned in given sample_submission.csv
\n\"image\"

\n

but on submission of the same I got Submission Scoring Error so what should I do?
\nare there any other requirements which I need to follow?

\n

Please let me know,
\nThanks

\n

Edit :
\nDo we need to have our model in the same notebook as submission notebook?
\nI tried submitting submission.csv in a separate notebook and it's always causing an error

", "rawMarkdown": "I had created two notebooks,\n1. Training model\n2. Submitting csv file\n\nIn the 2nd notebook, I've **internet** turned **off** with the following `submission.csv`:\n\"image\"\n\nwhich is the same format as mentioned in given `sample_submission.csv`\n\"image\"\n\nbut on submission of the same I got `Submission Scoring Error` so what should I do?\nare there any other requirements which I need to follow?\n\nPlease let me know,\nThanks\n\nEdit :\nDo we need to have our model in the same notebook as `submission` notebook?\nI tried submitting `submission.csv` in a separate notebook and it's always causing an error", "votes": 8 }, { "id": 1870880, "postDate": "2022-07-25T23:43:07.580Z", "content": "

Some possible issues are that the score is computed per patient_id so you must ensure you have one unique prediction per patient and not a prediction per image. Other things to check are that the data types are correctly.

", "rawMarkdown": "Some possible issues are that the score is computed per patient_id so you must ensure you have one unique prediction per patient and not a prediction per image. Other things to check are that the data types are correctly.", "votes": 5, "replies": [ { "id": 1875401, "postDate": "2022-07-29T02:05:06.623Z", "content": "

This is the right way to eradicate scoring errors.

", "rawMarkdown": "This is the right way to eradicate scoring errors. " } ] }, { "id": 1876677, "postDate": "2022-07-30T02:34:55.410Z", "content": "

Thank you so much for opening the subject. The comments/links have been very useful. However, I still cannot make a successful submission. I keep getting 'Notebook Exceeded Allowed Compute'. Any suggestions please? Thanks! I'm just using an old model and applying to the test images. Notebook:

\n

https://www.kaggle.com/code/rodrigocarrillo/import-model-trained-locally-20220715?kernelSessionId=102081774

", "rawMarkdown": "Thank you so much for opening the subject. The comments/links have been very useful. However, I still cannot make a successful submission. I keep getting 'Notebook Exceeded Allowed Compute'. Any suggestions please? Thanks! I'm just using an old model and applying to the test images. Notebook:\n\nhttps://www.kaggle.com/code/rodrigocarrillo/import-model-trained-locally-20220715?kernelSessionId=102081774\n", "votes": 3, "replies": [ { "id": 1876690, "postDate": "2022-07-30T02:49:24.607Z", "content": "

I was getting this error when I was trying to read all test images as is during scoring run… looks like some images might be very big size and create a 'exceeded allowed compute' error. I fixed this by limiting the size of the image allowed to be read from file using this image loading code block. Since then this error is gone, I have gone upto 1.2 Gb file size without problem (my guess is that up to 2Gb file size should be fine but I'm yet to confirm that)

", "rawMarkdown": "I was getting this error when I was trying to read all test images as is during scoring run... looks like some images might be very big size and create a 'exceeded allowed compute' error. I fixed this by limiting the size of the image allowed to be read from file using [this](https://www.kaggle.com/code/realneuralnetwork/cnn-strip-ai-inference/notebook?scriptVersionId=100707854) image loading code block. Since then this error is gone, I have gone upto 1.2 Gb file size without problem (my guess is that up to 2Gb file size should be fine but I'm yet to confirm that)", "votes": 1 }, { "id": 1876737, "postDate": "2022-07-30T03:55:39.450Z", "content": "

Thank you very much.
\nDo you only refer to the 3rd cell, that one with:
\ntry:
\n os.mkdir(\"../test/\")
\nexcept:
\n?

", "rawMarkdown": "Thank you very much.\nDo you only refer to the 3rd cell, that one with:\ntry:\n os.mkdir(\"../test/\")\nexcept:\n?\n\n" }, { "id": 1876739, "postDate": "2022-07-30T04:00:01.953Z", "content": "

yes… that's the one

", "rawMarkdown": "yes... that's the one", "votes": 1 }, { "id": 1879324, "postDate": "2022-08-01T02:14:25.600Z", "content": "

Thanks! It worked :)

", "rawMarkdown": "Thanks! It worked :)" } ] }, { "id": 1873991, "postDate": "2022-07-28T02:16:51.980Z", "content": "

For every patient, sum of CA and LAA should be equal to 1

", "rawMarkdown": "For every patient, sum of CA and LAA should be equal to 1", "votes": 1, "replies": [ { "id": 1878839, "postDate": "2022-07-31T17:01:37.817Z", "content": "

In the Evaluation tab of the Overview tab where it talks about the loss function, it states the following,

\n
\n

The submitted probabilities for a given image are not required to sum to one because they are rescaled prior to being scored (each row is divided by the row sum).

\n
\n

So I don't think that this is the case, but I could be wrong.

", "rawMarkdown": "In the Evaluation tab of the Overview tab where it talks about the loss function, it states the following,\n> The submitted probabilities for a given image are not required to sum to one because they are rescaled prior to being scored (each row is divided by the row sum).\n\nSo I don't think that this is the case, but I could be wrong." }, { "id": 1879717, "postDate": "2022-08-01T08:04:16.207Z", "content": "

You are right. I also get 17.6, an error score. It make me confused.

", "rawMarkdown": "You are right. I also get 17.6, an error score. It make me confused." } ] }, { "id": 1870667, "postDate": "2022-07-25T18:01:15.120Z", "content": "

I have the same problem !!! I guess we should truncate the floating point precision to a few digits (like two or three!!?)!

\n

My today's submission is wasted. If you tried that or found the problem, please let me know. Also, ensure you don't have duplicate patient_ids in your submission.

", "rawMarkdown": "I have the same problem !!! I guess we should truncate the floating point precision to a few digits (like two or three!!?)! \n\nMy today's submission is wasted. If you tried that or found the problem, please let me know. Also, ensure you don't have duplicate patient_ids in your submission. ", "votes": 1, "replies": [ { "id": 1870716, "postDate": "2022-07-25T19:04:33.203Z", "content": "

I wasted my two submissions just like that :(
\nIn one, I mistakenly used image_id and in the recent one this problem occurred.

\n

I'll try what you suggested maybe tomorrow, but let me know if you know found anything useful on this

", "rawMarkdown": "I wasted my two submissions just like that :(\nIn one, I mistakenly used `image_id` and in the recent one this problem occurred.\n\nI'll try what you suggested maybe tomorrow, but let me know if you know found anything useful on this " }, { "id": 1871032, "postDate": "2022-07-26T03:12:30.567Z", "content": "

I tried that till 3 decimals
\nAnd still got the same error today 😔

", "rawMarkdown": "I tried that till 3 decimals \nAnd still got the same error today 😔", "votes": 1 }, { "id": 1871533, "postDate": "2022-07-26T10:15:59.510Z", "content": "

Today I tried filling NaNs in my data frame, but that also didn't work!

", "rawMarkdown": "Today I tried filling NaNs in my data frame, but that also didn't work!\n" } ] }, { "id": 1876842, "postDate": "2022-07-30T05:50:20.167Z", "content": "

Anyone has any solution for this issue? I have done what is being told here already but I am still getting the issue \"Notebook Exceeded Allowed Compute\" Why is it still failing? Can anyone please help?

\n
    submission = pd.read_csv(\"../input/mayo-clinic-strip-ai/sample_submission.csv\")\n    submission.CE = preds.CE.to_list()\n    submission.LAA = preds.LAA.to_list()\n    submission = submission.groupby(\"patient_id\").mean()\n    submission = submission[[\"CE\", \"LAA\"]].round(6).reset_index()\n    submission.to_csv(\"submission.csv\", index = False)\n
\n

PS: I have failed 2 submissions to the same error. My notebook runs for ~120sec and my submissions fail after ~25 minutes of runtime. 1 submission is literally too low to know what is wrong here. Note, I am submitting my notebook for commit and then I use the submit button on CSV to make the competition submission.

\n

Can hosts help us here?

", "rawMarkdown": "Anyone has any solution for this issue? I have done what is being told here already but I am still getting the issue \"**Notebook Exceeded Allowed Compute**\" Why is it still failing? Can anyone please help?\n\n```\n submission = pd.read_csv(\"../input/mayo-clinic-strip-ai/sample_submission.csv\")\n submission.CE = preds.CE.to_list()\n submission.LAA = preds.LAA.to_list()\n submission = submission.groupby(\"patient_id\").mean()\n submission = submission[[\"CE\", \"LAA\"]].round(6).reset_index()\n submission.to_csv(\"submission.csv\", index = False)\n```\n\nPS: I have failed 2 submissions to the same error. My notebook runs for ~120sec and my submissions fail after ~25 minutes of runtime. 1 submission is literally too low to know what is wrong here. **Note, I am submitting my notebook for commit and then I use the submit button on CSV to make the competition submission**.\n\nCan hosts help us here?", "replies": [ { "id": 1877422, "postDate": "2022-07-30T16:16:55.087Z", "content": "

I had two submissions fail at 25 min runtime with the same error… please check my explanation in reply to @rodrigocarrillo below for the solution that worked for me… it is not related to csv submission.

", "rawMarkdown": "I had two submissions fail at 25 min runtime with the same error... please check my explanation in reply to @rodrigocarrillo below for the solution that worked for me... it is not related to csv submission." }, { "id": 1877492, "postDate": "2022-07-30T17:35:10.830Z", "content": "

I see, but why it doesn't fail during data generation/train where we have over 1.5GB single file. It's so weird. Looking at the code requirement's there are no clauses regarding it as it says <= 9 hours of runtime. It will also make our score bad if we ignore few images (could be the whole dataset full of big images).

\n

Thank you, really appreciate your help. I will give it on my tomorrow's submission.

", "rawMarkdown": "I see, but why it doesn't fail during data generation/train where we have over 1.5GB single file. It's so weird. Looking at the code requirement's there are no clauses regarding it as it says <= 9 hours of runtime. It will also make our score bad if we ignore few images (could be the whole dataset full of big images). \n\nThank you, really appreciate your help. I will give it on my tomorrow's submission." }, { "id": 1877511, "postDate": "2022-07-30T18:08:19.713Z", "content": "

Since we don't know the exact file size of test files for scoring, we will not find out where the problem is… I think we can run up to max size image that training data has, since that gets loaded without error… and yeah, leaving out test data for scoring and predicting dummy data is not ideal, but there's little we can do at this point unless someone tries something wild and lets us know :)

", "rawMarkdown": "Since we don't know the exact file size of test files for scoring, we will not find out where the problem is... I think we can run up to max size image that training data has, since that gets loaded without error... and yeah, leaving out test data for scoring and predicting dummy data is not ideal, but there's little we can do at this point unless someone tries something wild and lets us know :)" }, { "id": 1877563, "postDate": "2022-07-30T19:18:36.760Z", "content": "

Yea, that's for sure

", "rawMarkdown": "Yea, that's for sure" }, { "id": 1880393, "postDate": "2022-08-01T17:16:54.313Z", "content": "

I tried your solution but now I am getting \"Notebook Threw Exception\" on my submission. I am super confused. My notebook passes on commit and generates the CSV but it is failing on \"Private Submission Run\".

", "rawMarkdown": "I tried your solution but now I am getting \"Notebook Threw Exception\" on my submission. I am super confused. My notebook passes on commit and generates the CSV but it is failing on \"Private Submission Run\". " }, { "id": 1880508, "postDate": "2022-08-01T19:15:05.817Z", "content": "

Did you check the log? the exception might be recorded there.

", "rawMarkdown": "Did you check the log? the exception might be recorded there." }, { "id": 1880526, "postDate": "2022-08-01T19:37:21.937Z", "content": "

How to check the log?

", "rawMarkdown": "How to check the log?" }, { "id": 1880547, "postDate": "2022-08-01T19:53:37.300Z", "content": "

go to your submissions on the competition page, select the one with the 'notebook threw exception' error, it will go to notebook viewer page, there you can check the log tab for runtime errors

", "rawMarkdown": "go to your submissions on the competition page, select the one with the 'notebook threw exception' error, it will go to notebook viewer page, there you can check the log tab for runtime errors " }, { "id": 1880963, "postDate": "2022-08-02T06:47:22.687Z", "content": "

Well, it's weird cause the log is perfect.

", "rawMarkdown": "Well, it's weird cause the log is perfect." }, { "id": 1881001, "postDate": "2022-08-02T07:17:48.260Z", "content": "

That's the thing, so my commit is succesful with perfect log. It fails on that \"submission\" run. :\"(

", "rawMarkdown": "That's the thing, so my commit is succesful with perfect log. It fails on that \"submission\" run. :\"( ", "votes": 1 }, { "id": 1881067, "postDate": "2022-08-02T08:27:27.330Z", "content": "

Yeah, the same problem.

", "rawMarkdown": "Yeah, the same problem." }, { "id": 1881553, "postDate": "2022-08-02T15:58:07.197Z", "content": "

My bad, I missed that your initial commit was successful… how long does the notebook run before giving this error? Here is my experience:

\n

Around 25 min -- some OOM error is expected if file size filtering is not applied
\nNotebook runs for about 1 hour for me when successful -- means no problem in prediction step
\nError beyond that time -- csv data assigning and writing error likely in the final steps (could be longer depending on how much time the prediction steps take, my initial commit took about 120-140 sec for reference)

\n

Hope this helps to figure out where the error is.

\n

'Notebook Threw Exception' is most likely a runtime error, not a memory error.. also if submission csv is wrongly written there are two other errors ('Submission CSV not found' and 'Submission score error') for that… so checking your code for potential issues against shared kernels on the leaderboard would be a starting point

", "rawMarkdown": "My bad, I missed that your initial commit was successful... how long does the notebook run before giving this error? Here is my experience:\n\nAround 25 min -- some OOM error is expected if file size filtering is not applied\nNotebook runs for about 1 hour for me when successful -- means no problem in prediction step\nError beyond that time -- csv data assigning and writing error likely in the final steps (could be longer depending on how much time the prediction steps take, my initial commit took about 120-140 sec for reference)\n\nHope this helps to figure out where the error is.\n\n'Notebook Threw Exception' is most likely a runtime error, not a memory error.. also if submission csv is wrongly written there are two other errors ('Submission CSV not found' and 'Submission score error') for that... so checking your code for potential issues against shared kernels on the leaderboard would be a starting point" }, { "id": 1883681, "postDate": "2022-08-04T02:46:00.407Z", "content": "

I tried to fix my code and refactor it. I used 1.53 GB as my limit but it still failed today. I will reduce it to 1.2 in tomorrow's submission. I am still getting the same error \"Notebook Exceeded Allowed Compute\". Though, my commit completes in 206 seconds on GPU but my notebook fails in 30 minutes (private run/submission one)

\n
def is_huge_fsize(filepath):\n    max_sz = 1180**3 # 1.53gb\n    try:\n        sz = os.path.getsize(filepath)\n    except:\n        sz = max_sz + 1\n\n    if (sz > max_sz):\n        return True\n\n    return False\n
", "rawMarkdown": "I tried to fix my code and refactor it. I used 1.53 GB as my limit but it still failed today. I will reduce it to 1.2 in tomorrow's submission. I am still getting the same error \"Notebook Exceeded Allowed Compute\". Though, my commit completes in 206 seconds on GPU but my notebook **fails** in 30 minutes (private run/submission one)\n\n```\ndef is_huge_fsize(filepath):\n max_sz = 1180**3 # 1.53gb\n try:\n sz = os.path.getsize(filepath)\n except:\n sz = max_sz + 1\n \n if (sz > max_sz):\n return True\n \n return False\n```" }, { "id": 1883728, "postDate": "2022-08-04T03:55:29.153Z", "content": "

I have noticed on the training data that a file size of 1.4/1.5+ Gb will give OOM error if you use GPU where the RAM is only 13 Gb, whereas with CPU I have successfully loaded files upto 1.8 Gb during submission scoring without exceed allowed compute error.

\n

I think these might be general indicators and also dependent on how the code uses memory.. but yeah, decreasing file size will give you your code's limit for a successful submission… hope you reach there soon :)

", "rawMarkdown": "I have noticed on the training data that a file size of 1.4/1.5+ Gb will give OOM error if you use GPU where the RAM is only 13 Gb, whereas with CPU I have successfully loaded files upto 1.8 Gb during submission scoring without exceed allowed compute error.\n\nI think these might be general indicators and also dependent on how the code uses memory.. but yeah, decreasing file size will give you your code's limit for a successful submission... hope you reach there soon :)" }, { "id": 1885314, "postDate": "2022-08-05T05:02:45.810Z", "content": "

I finally scored after setting the max limit to 1.20gb. Now, I don't think this is right as I am certain that I have to provide random data or a lot of rows. But at least I got a successful submission, thanks for your wonderful help

", "rawMarkdown": "I finally scored after setting the max limit to 1.20gb. Now, I don't think this is right as I am certain that I have to provide random data or a lot of rows. But at least I got a successful submission, thanks for your wonderful help" }, { "id": 1885358, "postDate": "2022-08-05T06:03:02.497Z", "content": "

Great… yeah need to figure that part out, it's not ideal

", "rawMarkdown": "Great... yeah need to figure that part out, it's not ideal" } ] }, { "id": 1870903, "postDate": "2022-07-26T00:29:01.297Z", "content": "

\"I

", "rawMarkdown": "![I have a new (or old) problem](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8536316%2F496ed787aad726cf44f1459a266c9009%2Ferror%20-%20Copy.png?generation=1658795312314850&alt=media)", "replies": [ { "id": 1870904, "postDate": "2022-07-26T00:29:59.200Z", "content": "

I have a new (or old) problem… anyone knows what's happening? My submission notebook runs with 4 test case images in less than 200 sec, but after submission scoring continues to 25 min and then this message

", "rawMarkdown": "I have a new (or old) problem... anyone knows what's happening? My submission notebook runs with 4 test case images in less than 200 sec, but after submission scoring continues to 25 min and then this message" }, { "id": 1871198, "postDate": "2022-07-26T06:11:49.823Z", "content": "

Checkout this answer:
\nhttps://www.kaggle.com/questions-and-answers/129198

", "rawMarkdown": "Checkout this answer:\nhttps://www.kaggle.com/questions-and-answers/129198 " }, { "id": 1872275, "postDate": "2022-07-26T19:01:42.143Z", "content": "

Thanks, will try this.

", "rawMarkdown": "Thanks, will try this." } ] }, { "id": 1870810, "postDate": "2022-07-25T21:40:42.327Z", "content": "

try this

", "rawMarkdown": "[try this](https://www.kaggle.com/code/saurabhsawhney/mayo-beginners-how-to-make-a-submission/notebook?scriptVersionId=101194169)", "replies": [ { "id": 1871037, "postDate": "2022-07-26T03:20:44.600Z", "content": "

Tried this but still didn't worked :(

", "rawMarkdown": "Tried this but still didn't worked :(\n\n" }, { "id": 1871047, "postDate": "2022-07-26T03:29:30.907Z", "content": "

Hello @pathikghugare , I dont know how you obtain your submission but is there a possible scenario where a value might be Nan? For example, maybe at some point in your code there is a division by zero which yields a Nan or infinite and that results in a scoring error. I am just brain storming here

", "rawMarkdown": "Hello @pathikghugare , I dont know how you obtain your submission but is there a possible scenario where a value might be Nan? For example, maybe at some point in your code there is a division by zero which yields a Nan or infinite and that results in a scoring error. I am just brain storming here" }, { "id": 1871177, "postDate": "2022-07-26T05:33:38.340Z", "content": "

Umm but there's no such scenario
\nBut we can see that I've dataframe in the same format as mentioned
\nIn the previous reply, I've also attached output of .info(), submission_df as well and in that I don't see any problems which is conflicting with the sample_submission 🥺🤷

\n

How was your submission.csv ?
\nCan you please share that by changing some values in your submission?

", "rawMarkdown": "Umm but there's no such scenario\nBut we can see that I've dataframe in the same format as mentioned\nIn the previous reply, I've also attached output of `.info()`, `submission_df` as well and in that I don't see any problems which is conflicting with the `sample_submission` 🥺🤷\n\nHow was your `submission.csv` ?\nCan you please share that by changing some values in your submission?" }, { "id": 1875364, "postDate": "2022-07-29T01:01:11.050Z", "content": "

@pathikghugare were you able to solve your problem?

", "rawMarkdown": "@pathikghugare were you able to solve your problem? " }, { "id": 1875382, "postDate": "2022-07-29T01:25:57.303Z", "content": "

Nope :(
\nDo you have any ideas ?

", "rawMarkdown": "Nope :(\nDo you have any ideas ?" }, { "id": 1875428, "postDate": "2022-07-29T03:00:49.533Z", "content": "

Yes I think so: I submitted successfully today, but due to this thread I checked for a few common things before submitting:

\n

1) I made the rookie mistake of giving the wrong name first - 'sample_submission.csv' - but this gave a 'Submission CSV not found' error.
\n2) Then I checked many kernel shared on the leader board (at least the top 3-4 shared codes) and found that all are submitting in a particular way which includes writing the final probability scores into the sample loaded dataframe. I have done it the same way - first creating a df_predictions.groupby('patient_id').mean() to make sure repeated patient ids are taken care of in the prediction, and then assigning 'CE' and 'LAA' scores to the respective columns in dataframe loaded from sample_submission.csv. Then finally saved that dataframe to a 'submission.csv'.

\n

Try this, hope it works and we call can finally work on the real problem :)

", "rawMarkdown": "Yes I think so: I submitted successfully today, but due to this thread I checked for a few common things before submitting:\n\n1) I made the rookie mistake of giving the wrong name first - 'sample_submission.csv' - but this gave a 'Submission CSV not found' error.\n2) Then I checked many kernel shared on the leader board (at least the top 3-4 shared codes) and found that all are submitting in a particular way which includes writing the final probability scores into the sample loaded dataframe. I have done it the same way - first creating a df_predictions.groupby('patient_id').mean() to make sure repeated patient ids are taken care of in the prediction, and then assigning 'CE' and 'LAA' scores to the respective columns in dataframe loaded from sample_submission.csv. Then finally saved that dataframe to a 'submission.csv'. \n\nTry this, hope it works and we call can finally work on the real problem :)" }, { "id": 1875429, "postDate": "2022-07-29T03:01:30.540Z", "content": "

Also confirms that rounding off values is not the issue.

", "rawMarkdown": "Also confirms that rounding off values is not the issue." }, { "id": 1879522, "postDate": "2022-08-01T05:34:13.400Z", "content": "

oh okay
\nI'll try it and let you know

", "rawMarkdown": "oh okay \nI'll try it and let you know " }, { "id": 1928796, "postDate": "2022-09-06T16:11:06.487Z", "content": "

How is it going?

", "rawMarkdown": "How is it going?" } ] } ], "comments": [ { "id": 1870880, "author_name": "moth", "author_url": "", "post_date": "2022-07-25T23:43:07.580000", "content": "

Some possible issues are that the score is computed per patient_id so you must ensure you have one unique prediction per patient and not a prediction per image. Other things to check are that the data types are correctly.

", "votes": 5, "replies": [ { "id": 1875401, "author_name": "Dominador Ramos Jr", "author_url": "", "post_date": "2022-07-29T02:05:06.623000", "content": "

This is the right way to eradicate scoring errors.

", "votes": 0, "replies": [] } ] }, { "id": 1876677, "author_name": "Rodrigo M Carrillo Larco", "author_url": "", "post_date": "2022-07-30T02:34:55.410000", "content": "

Thank you so much for opening the subject. The comments/links have been very useful. However, I still cannot make a successful submission. I keep getting 'Notebook Exceeded Allowed Compute'. Any suggestions please? Thanks! I'm just using an old model and applying to the test images. Notebook:

\n

https://www.kaggle.com/code/rodrigocarrillo/import-model-trained-locally-20220715?kernelSessionId=102081774

", "votes": 3, "replies": [ { "id": 1876690, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-30T02:49:24.607000", "content": "

I was getting this error when I was trying to read all test images as is during scoring run… looks like some images might be very big size and create a 'exceeded allowed compute' error. I fixed this by limiting the size of the image allowed to be read from file using this image loading code block. Since then this error is gone, I have gone upto 1.2 Gb file size without problem (my guess is that up to 2Gb file size should be fine but I'm yet to confirm that)

", "votes": 1, "replies": [] }, { "id": 1876737, "author_name": "Rodrigo M Carrillo Larco", "author_url": "", "post_date": "2022-07-30T03:55:39.450000", "content": "

Thank you very much.
\nDo you only refer to the 3rd cell, that one with:
\ntry:
\n os.mkdir(\"../test/\")
\nexcept:
\n?

", "votes": 0, "replies": [] }, { "id": 1876739, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-30T04:00:01.953000", "content": "

yes… that's the one

", "votes": 1, "replies": [] }, { "id": 1879324, "author_name": "Rodrigo M Carrillo Larco", "author_url": "", "post_date": "2022-08-01T02:14:25.600000", "content": "

Thanks! It worked :)

", "votes": 0, "replies": [] } ] }, { "id": 1873991, "author_name": "柯慕灵", "author_url": "", "post_date": "2022-07-28T02:16:51.980000", "content": "

For every patient, sum of CA and LAA should be equal to 1

", "votes": 1, "replies": [ { "id": 1878839, "author_name": "yqz", "author_url": "", "post_date": "2022-07-31T17:01:37.817000", "content": "

In the Evaluation tab of the Overview tab where it talks about the loss function, it states the following,

\n
\n

The submitted probabilities for a given image are not required to sum to one because they are rescaled prior to being scored (each row is divided by the row sum).

\n
\n

So I don't think that this is the case, but I could be wrong.

", "votes": 0, "replies": [] }, { "id": 1879717, "author_name": "柯慕灵", "author_url": "", "post_date": "2022-08-01T08:04:16.207000", "content": "

You are right. I also get 17.6, an error score. It make me confused.

", "votes": 0, "replies": [] } ] }, { "id": 1870667, "author_name": "Alireza", "author_url": "", "post_date": "2022-07-25T18:01:15.120000", "content": "

I have the same problem !!! I guess we should truncate the floating point precision to a few digits (like two or three!!?)!

\n

My today's submission is wasted. If you tried that or found the problem, please let me know. Also, ensure you don't have duplicate patient_ids in your submission.

", "votes": 1, "replies": [ { "id": 1870716, "author_name": "Pathik Ghugare", "author_url": "", "post_date": "2022-07-25T19:04:33.203000", "content": "

I wasted my two submissions just like that :(
\nIn one, I mistakenly used image_id and in the recent one this problem occurred.

\n

I'll try what you suggested maybe tomorrow, but let me know if you know found anything useful on this

", "votes": 0, "replies": [] }, { "id": 1871032, "author_name": "Pathik Ghugare", "author_url": "", "post_date": "2022-07-26T03:12:30.567000", "content": "

I tried that till 3 decimals
\nAnd still got the same error today 😔

", "votes": 1, "replies": [] }, { "id": 1871533, "author_name": "Alireza", "author_url": "", "post_date": "2022-07-26T10:15:59.510000", "content": "

Today I tried filling NaNs in my data frame, but that also didn't work!

", "votes": 0, "replies": [] } ] }, { "id": 1876842, "author_name": "Hey24sheep", "author_url": "", "post_date": "2022-07-30T05:50:20.167000", "content": "

Anyone has any solution for this issue? I have done what is being told here already but I am still getting the issue \"Notebook Exceeded Allowed Compute\" Why is it still failing? Can anyone please help?

\n
    submission = pd.read_csv(\"../input/mayo-clinic-strip-ai/sample_submission.csv\")\n    submission.CE = preds.CE.to_list()\n    submission.LAA = preds.LAA.to_list()\n    submission = submission.groupby(\"patient_id\").mean()\n    submission = submission[[\"CE\", \"LAA\"]].round(6).reset_index()\n    submission.to_csv(\"submission.csv\", index = False)\n
\n

PS: I have failed 2 submissions to the same error. My notebook runs for ~120sec and my submissions fail after ~25 minutes of runtime. 1 submission is literally too low to know what is wrong here. Note, I am submitting my notebook for commit and then I use the submit button on CSV to make the competition submission.

\n

Can hosts help us here?

", "votes": 0, "replies": [ { "id": 1877422, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-30T16:16:55.087000", "content": "

I had two submissions fail at 25 min runtime with the same error… please check my explanation in reply to @rodrigocarrillo below for the solution that worked for me… it is not related to csv submission.

", "votes": 0, "replies": [] }, { "id": 1877492, "author_name": "Hey24sheep", "author_url": "", "post_date": "2022-07-30T17:35:10.830000", "content": "

I see, but why it doesn't fail during data generation/train where we have over 1.5GB single file. It's so weird. Looking at the code requirement's there are no clauses regarding it as it says <= 9 hours of runtime. It will also make our score bad if we ignore few images (could be the whole dataset full of big images).

\n

Thank you, really appreciate your help. I will give it on my tomorrow's submission.

", "votes": 0, "replies": [] }, { "id": 1877511, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-30T18:08:19.713000", "content": "

Since we don't know the exact file size of test files for scoring, we will not find out where the problem is… I think we can run up to max size image that training data has, since that gets loaded without error… and yeah, leaving out test data for scoring and predicting dummy data is not ideal, but there's little we can do at this point unless someone tries something wild and lets us know :)

", "votes": 0, "replies": [] }, { "id": 1877563, "author_name": "Hey24sheep", "author_url": "", "post_date": "2022-07-30T19:18:36.760000", "content": "

Yea, that's for sure

", "votes": 0, "replies": [] }, { "id": 1880393, "author_name": "Hey24sheep", "author_url": "", "post_date": "2022-08-01T17:16:54.313000", "content": "

I tried your solution but now I am getting \"Notebook Threw Exception\" on my submission. I am super confused. My notebook passes on commit and generates the CSV but it is failing on \"Private Submission Run\".

", "votes": 0, "replies": [] }, { "id": 1880508, "author_name": "tdiceman", "author_url": "", "post_date": "2022-08-01T19:15:05.817000", "content": "

Did you check the log? the exception might be recorded there.

", "votes": 0, "replies": [] }, { "id": 1880526, "author_name": "Nikita Glazunov", "author_url": "", "post_date": "2022-08-01T19:37:21.937000", "content": "

How to check the log?

", "votes": 0, "replies": [] }, { "id": 1880547, "author_name": "tdiceman", "author_url": "", "post_date": "2022-08-01T19:53:37.300000", "content": "

go to your submissions on the competition page, select the one with the 'notebook threw exception' error, it will go to notebook viewer page, there you can check the log tab for runtime errors

", "votes": 0, "replies": [] }, { "id": 1880963, "author_name": "Nikita Glazunov", "author_url": "", "post_date": "2022-08-02T06:47:22.687000", "content": "

Well, it's weird cause the log is perfect.

", "votes": 0, "replies": [] }, { "id": 1881001, "author_name": "Hey24sheep", "author_url": "", "post_date": "2022-08-02T07:17:48.260000", "content": "

That's the thing, so my commit is succesful with perfect log. It fails on that \"submission\" run. :\"(

", "votes": 1, "replies": [] }, { "id": 1881067, "author_name": "Nikita Glazunov", "author_url": "", "post_date": "2022-08-02T08:27:27.330000", "content": "

Yeah, the same problem.

", "votes": 0, "replies": [] }, { "id": 1881553, "author_name": "tdiceman", "author_url": "", "post_date": "2022-08-02T15:58:07.197000", "content": "

My bad, I missed that your initial commit was successful… how long does the notebook run before giving this error? Here is my experience:

\n

Around 25 min -- some OOM error is expected if file size filtering is not applied
\nNotebook runs for about 1 hour for me when successful -- means no problem in prediction step
\nError beyond that time -- csv data assigning and writing error likely in the final steps (could be longer depending on how much time the prediction steps take, my initial commit took about 120-140 sec for reference)

\n

Hope this helps to figure out where the error is.

\n

'Notebook Threw Exception' is most likely a runtime error, not a memory error.. also if submission csv is wrongly written there are two other errors ('Submission CSV not found' and 'Submission score error') for that… so checking your code for potential issues against shared kernels on the leaderboard would be a starting point

", "votes": 0, "replies": [] }, { "id": 1883681, "author_name": "Hey24sheep", "author_url": "", "post_date": "2022-08-04T02:46:00.407000", "content": "

I tried to fix my code and refactor it. I used 1.53 GB as my limit but it still failed today. I will reduce it to 1.2 in tomorrow's submission. I am still getting the same error \"Notebook Exceeded Allowed Compute\". Though, my commit completes in 206 seconds on GPU but my notebook fails in 30 minutes (private run/submission one)

\n
def is_huge_fsize(filepath):\n    max_sz = 1180**3 # 1.53gb\n    try:\n        sz = os.path.getsize(filepath)\n    except:\n        sz = max_sz + 1\n\n    if (sz > max_sz):\n        return True\n\n    return False\n
", "votes": 0, "replies": [] }, { "id": 1883728, "author_name": "tdiceman", "author_url": "", "post_date": "2022-08-04T03:55:29.153000", "content": "

I have noticed on the training data that a file size of 1.4/1.5+ Gb will give OOM error if you use GPU where the RAM is only 13 Gb, whereas with CPU I have successfully loaded files upto 1.8 Gb during submission scoring without exceed allowed compute error.

\n

I think these might be general indicators and also dependent on how the code uses memory.. but yeah, decreasing file size will give you your code's limit for a successful submission… hope you reach there soon :)

", "votes": 0, "replies": [] }, { "id": 1885314, "author_name": "Hey24sheep", "author_url": "", "post_date": "2022-08-05T05:02:45.810000", "content": "

I finally scored after setting the max limit to 1.20gb. Now, I don't think this is right as I am certain that I have to provide random data or a lot of rows. But at least I got a successful submission, thanks for your wonderful help

", "votes": 0, "replies": [] }, { "id": 1885358, "author_name": "tdiceman", "author_url": "", "post_date": "2022-08-05T06:03:02.497000", "content": "

Great… yeah need to figure that part out, it's not ideal

", "votes": 0, "replies": [] } ] }, { "id": 1870903, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-26T00:29:01.297000", "content": "

\"I

", "votes": 0, "replies": [ { "id": 1870904, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-26T00:29:59.200000", "content": "

I have a new (or old) problem… anyone knows what's happening? My submission notebook runs with 4 test case images in less than 200 sec, but after submission scoring continues to 25 min and then this message

", "votes": 0, "replies": [] }, { "id": 1871198, "author_name": "Pathik Ghugare", "author_url": "", "post_date": "2022-07-26T06:11:49.823000", "content": "

Checkout this answer:
\nhttps://www.kaggle.com/questions-and-answers/129198

", "votes": 0, "replies": [] }, { "id": 1872275, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-26T19:01:42.143000", "content": "

Thanks, will try this.

", "votes": 0, "replies": [] } ] }, { "id": 1870810, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-25T21:40:42.327000", "content": "

try this

", "votes": 0, "replies": [ { "id": 1871037, "author_name": "Pathik Ghugare", "author_url": "", "post_date": "2022-07-26T03:20:44.600000", "content": "

Tried this but still didn't worked :(

", "votes": 0, "replies": [] }, { "id": 1871047, "author_name": "moth", "author_url": "", "post_date": "2022-07-26T03:29:30.907000", "content": "

Hello @pathikghugare , I dont know how you obtain your submission but is there a possible scenario where a value might be Nan? For example, maybe at some point in your code there is a division by zero which yields a Nan or infinite and that results in a scoring error. I am just brain storming here

", "votes": 0, "replies": [] }, { "id": 1871177, "author_name": "Pathik Ghugare", "author_url": "", "post_date": "2022-07-26T05:33:38.340000", "content": "

Umm but there's no such scenario
\nBut we can see that I've dataframe in the same format as mentioned
\nIn the previous reply, I've also attached output of .info(), submission_df as well and in that I don't see any problems which is conflicting with the sample_submission 🥺🤷

\n

How was your submission.csv ?
\nCan you please share that by changing some values in your submission?

", "votes": 0, "replies": [] }, { "id": 1875364, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-29T01:01:11.050000", "content": "

@pathikghugare were you able to solve your problem?

", "votes": 0, "replies": [] }, { "id": 1875382, "author_name": "Pathik Ghugare", "author_url": "", "post_date": "2022-07-29T01:25:57.303000", "content": "

Nope :(
\nDo you have any ideas ?

", "votes": 0, "replies": [] }, { "id": 1875428, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-29T03:00:49.533000", "content": "

Yes I think so: I submitted successfully today, but due to this thread I checked for a few common things before submitting:

\n

1) I made the rookie mistake of giving the wrong name first - 'sample_submission.csv' - but this gave a 'Submission CSV not found' error.
\n2) Then I checked many kernel shared on the leader board (at least the top 3-4 shared codes) and found that all are submitting in a particular way which includes writing the final probability scores into the sample loaded dataframe. I have done it the same way - first creating a df_predictions.groupby('patient_id').mean() to make sure repeated patient ids are taken care of in the prediction, and then assigning 'CE' and 'LAA' scores to the respective columns in dataframe loaded from sample_submission.csv. Then finally saved that dataframe to a 'submission.csv'.

\n

Try this, hope it works and we call can finally work on the real problem :)

", "votes": 0, "replies": [] }, { "id": 1875429, "author_name": "tdiceman", "author_url": "", "post_date": "2022-07-29T03:01:30.540000", "content": "

Also confirms that rounding off values is not the issue.

", "votes": 0, "replies": [] }, { "id": 1879522, "author_name": "Pathik Ghugare", "author_url": "", "post_date": "2022-08-01T05:34:13.400000", "content": "

oh okay
\nI'll try it and let you know

", "votes": 0, "replies": [] }, { "id": 1928796, "author_name": "Zaakcii Ru", "author_url": "", "post_date": "2022-09-06T16:11:06.487000", "content": "

How is it going?

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1870463": "I had created two notebooks,\n1. Training model\n2. Submitting csv file\n\nIn the 2nd notebook, I've **internet** turned **off** with the following `submission.csv`:\n\"image\"\n\nwhich is the same format as mentioned in given `sample_submission.csv`\n\"image\"\n\nbut on submission of the same I got `Submission Scoring Error` so what should I do?\nare there any other requirements which I need to follow?\n\nPlease let me know,\nThanks\n\nEdit :\nDo we need to have our model in the same notebook as `submission` notebook?\nI tried submitting `submission.csv` in a separate notebook and it's always causing an error", "1870880": "Some possible issues are that the score is computed per patient_id so you must ensure you have one unique prediction per patient and not a prediction per image. Other things to check are that the data types are correctly.", "1876677": "Thank you so much for opening the subject. The comments/links have been very useful. However, I still cannot make a successful submission. I keep getting 'Notebook Exceeded Allowed Compute'. Any suggestions please? Thanks! I'm just using an old model and applying to the test images. Notebook:\n\nhttps://www.kaggle.com/code/rodrigocarrillo/import-model-trained-locally-20220715?kernelSessionId=102081774\n", "1873991": "For every patient, sum of CA and LAA should be equal to 1", "1870667": "I have the same problem !!! I guess we should truncate the floating point precision to a few digits (like two or three!!?)! \n\nMy today's submission is wasted. If you tried that or found the problem, please let me know. Also, ensure you don't have duplicate patient_ids in your submission. ", "1876842": "Anyone has any solution for this issue? I have done what is being told here already but I am still getting the issue \"**Notebook Exceeded Allowed Compute**\" Why is it still failing? Can anyone please help?\n\n```\n submission = pd.read_csv(\"../input/mayo-clinic-strip-ai/sample_submission.csv\")\n submission.CE = preds.CE.to_list()\n submission.LAA = preds.LAA.to_list()\n submission = submission.groupby(\"patient_id\").mean()\n submission = submission[[\"CE\", \"LAA\"]].round(6).reset_index()\n submission.to_csv(\"submission.csv\", index = False)\n```\n\nPS: I have failed 2 submissions to the same error. My notebook runs for ~120sec and my submissions fail after ~25 minutes of runtime. 1 submission is literally too low to know what is wrong here. **Note, I am submitting my notebook for commit and then I use the submit button on CSV to make the competition submission**.\n\nCan hosts help us here?", "1870903": "![I have a new (or old) problem](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8536316%2F496ed787aad726cf44f1459a266c9009%2Ferror%20-%20Copy.png?generation=1658795312314850&alt=media)", "1870810": "[try this](https://www.kaggle.com/code/saurabhsawhney/mayo-beginners-how-to-make-a-submission/notebook?scriptVersionId=101194169)" } }