\nThey are same model with previous submission. But all ran out of time today.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5600457%2F71176cb44df74ad2553a604727cdcbca%2FQQ20201026163756.png?generation=1603701501759240&alt=media\")
All my code with Notebook Timeout today, and all GPU running time increase 100% with the same code. It looks like your Leaderboard running time is also 9 hours.
\nIs that mean we cannot make any other inference anymore? Or just pray for lucky?
All my code with Notebook Timeout today, and all GPU running time increase 100% with the same code. It looks like your Leaderboard running time is also 9 hours.
\nIs that mean we cannot make any other inference anymore? Or just pray for lucky?
The commitments that were supposed to be done in a minute are still not done after 30 minutes. As it is, I am going to end up not being able to submit the culmination of a likely gold medal. Is there any way to improve this?
\n@juliaelliott
\nPlease reply.
@yujiariyasu We had some capacity issues in the past couple of hours. We shifted some capacity around to fix that now, so things should commit quickly.
\nThis did not affect commit runtime/timeouts at all, it just meant your commit might have taken longer to be assigned a machine to execute it.
", "rawMarkdown": "@yujiariyasu We had some capacity issues in the past couple of hours. We shifted some capacity around to fix that now, so things should commit quickly.\n\nThis did not affect commit runtime/timeouts at all, it just meant your commit might have taken longer to be assigned a machine to execute it.", "votes": 6 }, { "id": 1061195, "postDate": "2020-10-26T19:48:25.010Z", "content": "I didn't experienced what you mention here. Our last 2 submission were 2 versions of the same notebooks with one of them just having some code commented. That commented notebook used 1 less model and failed the other with full code passed. Which doesn't make sense at all since the one which uses more model takes actually longer. Also previously I experienced same code passing in 7 hours vs 9 hours.
", "rawMarkdown": "I didn't experienced what you mention here. Our last 2 submission were 2 versions of the same notebooks with one of them just having some code commented. That commented notebook used 1 less model and failed the other with full code passed. Which doesn't make sense at all since the one which uses more model takes actually longer. Also previously I experienced same code passing in 7 hours vs 9 hours." }, { "id": 1061217, "postDate": "2020-10-26T20:20:06.867Z", "content": "@keremt The issue I specified was with regard to wait times for GPU machines to actually execute the notebook, not anything to do with runtime/timeout once it was actually executed.
\nThat's a separate issue mentioned in https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193252#1061191
", "rawMarkdown": "@keremt The issue I specified was with regard to wait times for GPU machines to actually execute the notebook, not anything to do with runtime/timeout once it was actually executed.\n\nThat's a separate issue mentioned in https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193252#1061191", "votes": 2 } ] }, { "id": 1060759, "postDate": "2020-10-26T14:31:09.797Z", "content": "I can't commit kernel now.
", "rawMarkdown": "I can't commit kernel now. ", "votes": 5, "replies": [ { "id": 1060776, "postDate": "2020-10-26T14:41:10.283Z", "content": "Me too.
\nThe commitments are supposed to be over soon, but they never are.
It seems that cannot commit GPU kernel only.
", "rawMarkdown": "It seems that cannot commit GPU kernel only." }, { "id": 1060850, "postDate": "2020-10-26T15:14:05.637Z", "content": "Me too, only gpu.
\nI was able to commit to it over and over again.
\nI waited 19 minutes and the commit didn't start.
\nIt may be more effective to commit quickly and repeatedly
I can't commit my GPU kernel either!!! I still have 5 submissions today, I can't even use them!!!! Had so much code I just finished, very frustrating.
\noh my god….
I think hosts will extend the deadline because of this
", "rawMarkdown": "I think hosts will extend the deadline because of this", "votes": 3 }, { "id": 1060882, "postDate": "2020-10-26T15:40:30.233Z", "content": "I don't think so, it's like studying 9hrs before the exam. I used my submissions having in mind that this would happen (because it happened before), feels bad. In any case, would be cool to have extra days even for my team haha
", "rawMarkdown": "I don't think so, it's like studying 9hrs before the exam. I used my submissions having in mind that this would happen (because it happened before), feels bad. In any case, would be cool to have extra days even for my team haha", "votes": 1 }, { "id": 1060889, "postDate": "2020-10-26T15:47:04.980Z", "content": "GPU commits should be available again now. There was some capacity issues, so there was increased delays for commits to be assigned a machine to execute.
\nhttps://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193252#1060887
", "rawMarkdown": "GPU commits should be available again now. There was some capacity issues, so there was increased delays for commits to be assigned a machine to execute.\n\nhttps://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193252#1060887", "votes": 3 } ] }, { "id": 1061191, "postDate": "2020-10-26T19:44:59.603Z", "content": "Hi all!
\nThank you to those who reported extended commit or submission times. We've investigated the issue. It's common in the final 2-3 days of a competition for demand on both filestores and notebook processors surged significantly, and that's what happened here. This is exacerbated by the large file size of this competition, in particular.
\nWhile we've taken a few measures to expand capacity, you should still expect demand to potentially delay your access to machines. Some recommendations:
\nIt's a good idea to consider the final day or two of any competition as a bit of a danger zone in terms of capacity. We'll always do our best to make sure that people get the machines they need, but last-minute submissions may run into difficulties like this. We recognize that this doesn't help much now, and is definitely easier said than done, but it may be helpful in terms of future planning.
\nTo be clear, we will not be extending the deadline of the competition.
", "rawMarkdown": "Hi all!\n\nThank you to those who reported extended commit or submission times. We've investigated the issue. It's common in the final 2-3 days of a competition for demand on both filestores and notebook processors surged significantly, and that's what happened here. This is exacerbated by the large file size of this competition, in particular. \n\nWhile we've taken a few measures to expand capacity, you should still expect demand to potentially delay your access to machines. Some recommendations:\n\n1. Ensure that your submission is not timing out for code reasons (e.g. it does not run within time constraints on the substantively larger private test set). Review these common [code debugging tips](https://www.kaggle.com/code-competition-debugging).\n2. Re-commit. It's possible that previous delays in accessing a machine have since been remediated by our increased capacity measures.\n\nIt's a good idea to consider the final day or two of any competition as a bit of a danger zone in terms of capacity. We'll always do our best to make sure that people get the machines they need, but last-minute submissions may run into difficulties like this. We recognize that this doesn't help much now, and is definitely easier said than done, but it may be helpful in terms of future planning.\n\nTo be clear, we will not be extending the deadline of the competition.", "votes": 5 }, { "id": 1060750, "postDate": "2020-10-26T14:24:54.230Z", "content": "I think the increase in notebook runtime is driven primarily by I/O.
\nI just committed a notebook where it was taking 20-30 seconds to load in 1 CT exam. Previously this took about 1s.
", "rawMarkdown": "I think the increase in notebook runtime is driven primarily by I/O.\n\nI just committed a notebook where it was taking 20-30 seconds to load in 1 CT exam. Previously this took about 1s. ", "votes": 3 }, { "id": 1060644, "postDate": "2020-10-26T12:38:45.310Z", "content": "Kernels are not using the same CPU Hardware all the time. So if your kernels are CPU bottlenecked, this can be an issue.
", "rawMarkdown": "Kernels are not using the same CPU Hardware all the time. So if your kernels are CPU bottlenecked, this can be an issue. ", "votes": 3 }, { "id": 1060663, "postDate": "2020-10-26T12:57:00.090Z", "content": "Yesterday I submitted a kernel which I knew should finish within 9 hours because it was faster than other successful kernels that finished with 1-2 hours of leeway. It timed out.
\nThen I submitted the same kernel again, as well as the same kernel with fewer models. The original worked the second time, and the one with fewer models timed out.
", "rawMarkdown": "Yesterday I submitted a kernel which I knew should finish within 9 hours because it was faster than other successful kernels that finished with 1-2 hours of leeway. It timed out.\n\nThen I submitted the same kernel again, as well as the same kernel with fewer models. The original worked the second time, and the one with fewer models timed out.", "votes": 4, "replies": [ { "id": 1060678, "postDate": "2020-10-26T13:10:34.880Z", "content": "That's good news!
", "rawMarkdown": "That's good news! ", "votes": -1 } ] }, { "id": 1060572, "postDate": "2020-10-26T10:37:22.403Z", "content": "This happened in the Bird competition as well, as far as I remember all my submissions on the last day were time outs, despite me not changing my code.
\nI'm afraid there is nothing we can do about it except try not to rely too much on the last days, which is easier said than done.
", "rawMarkdown": "This happened in the Bird competition as well, as far as I remember all my submissions on the last day were time outs, despite me not changing my code.\n\nI'm afraid there is nothing we can do about it except try not to rely too much on the last days, which is easier said than done.", "votes": 4, "replies": [ { "id": 1060575, "postDate": "2020-10-26T10:39:09.343Z", "content": "I come up with an idea, just don't know is it valid: https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193285
", "rawMarkdown": "I come up with an idea, just don't know is it valid: https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193285" } ] }, { "id": 1060487, "postDate": "2020-10-26T08:38:49.827Z", "content": "Same for me…
\nThey are same model with previous submission. But all ran out of time today.
My rank has been falling every few hours so I guess many people can inference normally.
", "rawMarkdown": "My rank has been falling every few hours so I guess many people can inference normally.", "votes": 3, "replies": [ { "id": 1060546, "postDate": "2020-10-26T09:53:00.200Z", "content": "So do I, but I think they just lucky, or they have some else detect code in their notebook.
\ncause whatever they did, it will always be taking 8-9 hours for inference.
The thing happened to me, 5 submissions yesterday close to reset and 2 today. All timed out 😒
", "rawMarkdown": "The thing happened to me, 5 submissions yesterday close to reset and 2 today. All timed out 😒", "votes": 1 }, { "id": 1060747, "postDate": "2020-10-26T14:22:59.477Z", "content": "Hello,
\nI would like to report the same issues on our end. We are preprocessing the data and this step took 20-25 minutes to run yesterday. Now it's taking 1.5 hours. The CPU percent is also hovering around 5% instead of going much higher.
\nI scaled things so it would run in 2-3 hours on public test but with these speed issues that might not be possible. Is there an issue with compute for notebooks? cc: @juliaelliott @philculliton
", "rawMarkdown": "Hello,\n\nI would like to report the same issues on our end. We are preprocessing the data and this step took 20-25 minutes to run yesterday. Now it's taking 1.5 hours. The CPU percent is also hovering around 5% instead of going much higher. \n\nI scaled things so it would run in 2-3 hours on public test but with these speed issues that might not be possible. Is there an issue with compute for notebooks? cc: @juliaelliott @philculliton ", "votes": 1 }, { "id": 1060735, "postDate": "2020-10-26T14:17:12.493Z", "content": "We had the same problem, 4 submissions wasted. I thought it could be a memory problem and reported it https://www.kaggle.com/product-feedback/193238#1060664 indeed there is a bug in kaggle's memory report dashboard. Today the exactly same submission ended in 8hrs. Feels bad man.
", "rawMarkdown": "We had the same problem, 4 submissions wasted. I thought it could be a memory problem and reported it https://www.kaggle.com/product-feedback/193238#1060664 indeed there is a bug in kaggle's memory report dashboard. Today the exactly same submission ended in 8hrs. Feels bad man.", "votes": 1, "replies": [ { "id": 1060893, "postDate": "2020-10-26T15:49:30.133Z", "content": "That 'memory bug' is only in the dashboard, it does not affect runtime. It's not actually technically a bug, we just decided that we would stop including 'cached' memory in the memory we report to the UI, so you'd see the 'in use' memory more clearly.
\nBut that does not affect the execution at all, you won't get 'more memory' or anything due to this change, and it didn't affect scoring etc.
", "rawMarkdown": "That 'memory bug' is only in the dashboard, it does not affect runtime. It's not actually technically a bug, we just decided that we would stop including 'cached' memory in the memory we report to the UI, so you'd see the 'in use' memory more clearly.\n\nBut that does not affect the execution at all, you won't get 'more memory' or anything due to this change, and it didn't affect scoring etc.", "votes": 2 }, { "id": 1060897, "postDate": "2020-10-26T15:52:40.717Z", "content": "great to know, I was confused haha @herbison thanks for the effort these last hours :)
", "rawMarkdown": "great to know, I was confused haha @herbison thanks for the effort these last hours :)", "votes": 1 } ] }, { "id": 1060674, "postDate": "2020-10-26T13:04:46.743Z", "content": "\n\n\nGPU running time increase 100% with the same code.
\n
Is this only during the submission process?
", "rawMarkdown": "@dwchen \n>GPU running time increase 100% with the same code.\n\nIs this only during the submission process?", "votes": 1, "replies": [ { "id": 1060677, "postDate": "2020-10-26T13:09:16.083Z", "content": "It happens in my notebook inference process. Same code, same parameters, different run time.
\nIt increases by 1 times seconds than yesterday. 800s yesterday 2200s today.
What's wrong with kaggle server !!! I got Timeout error as well 😣
\nWe don't have enough time and it's a pity that we can't submit the new kernel.
\n@dwchen thank you for this topic.
You are welcome, We may have one more day if staff know what happens. I believe the problem is at GPU.
\nSorry, my web is broken just yet @_@
", "rawMarkdown": "You are welcome, We may have one more day if staff know what happens. I believe the problem is at GPU.\n\nSorry, my web is broken just yet @_@" } ] }, { "id": 1061073, "postDate": "2020-10-26T18:10:25.913Z", "content": "I accidentally submitted the same notebook and version, at the same time, and one of them timed out, while the other delivered a score. However, I designed both to finish in just under 9 hours. One speculation is that small differences between VMs may cause one to go over the limit over a long enough notebook run. Some supporting evidence is that when I do Save & Runs, I'll notice slight variations in the timestamps of events in the logs, even if no code has changed.
", "rawMarkdown": "I accidentally submitted the same notebook and version, at the same time, and one of them timed out, while the other delivered a score. However, I designed both to finish in just under 9 hours. One speculation is that small differences between VMs may cause one to go over the limit over a long enough notebook run. Some supporting evidence is that when I do Save & Runs, I'll notice slight variations in the timestamps of events in the logs, even if no code has changed.", "votes": 2 }, { "id": 1061049, "postDate": "2020-10-26T17:57:21.673Z", "content": "I was a little unfortunate to rely on the last two days to implement a few important ideas in the modeling pipeline.
\nIt appears that the submission time is affected since my new notebooks that should have a similar/smaller running times compared to their previous successful versions from a few days ago are now timing out on the submission stage. Some submissions are still running, but it seems likely that none of them will make it within the time limit.
", "rawMarkdown": "I was a little unfortunate to rely on the last two days to implement a few important ideas in the modeling pipeline. \n\nIt appears that the submission time is affected since my new notebooks that should have a similar/smaller running times compared to their previous successful versions from a few days ago are now timing out on the submission stage. Some submissions are still running, but it seems likely that none of them will make it within the time limit.", "votes": 2 }, { "id": 1060529, "postDate": "2020-10-26T09:29:25.380Z", "content": "There are probably a lot of people who were using fast sub until yesterday and are planning to predict private for the first time today.
\nI would like to see some sort of remedy.
hoping they can solve this problem.
", "rawMarkdown": "hoping they can solve this problem.", "votes": 1 } ] }, { "id": 1060421, "postDate": "2020-10-26T07:13:57.660Z", "content": "I have the same problem today. The submitted code should be finished by now. But they are still running…I guess all of them will exceed the time limit. And I already ran out of submissions….
", "rawMarkdown": "I have the same problem today. The submitted code should be finished by now. But they are still running...I guess all of them will exceed the time limit. And I already ran out of submissions....\n", "votes": 2, "replies": [ { "id": 1060427, "postDate": "2020-10-26T07:17:35.350Z", "content": "I spent 2000 seconds on inference code today and 800 seconds in normal. With that same code, just for a re-run. I'm pretty sure that the server has some problems.
", "rawMarkdown": "I spent 2000 seconds on inference code today and 800 seconds in normal. With that same code, just for a re-run. I'm pretty sure that the server has some problems.", "votes": 2 }, { "id": 1060431, "postDate": "2020-10-26T07:23:11.553Z", "content": "So after you re-run the code today, the time become normal? I submitted all the inference code this morning. And all of them seems will exceed the time limit. Th e point is..all the high score submissions I sent are quick submit(only public test set). Which means I don’t have score with whole testset in this situation 😂
", "rawMarkdown": "So after you re-run the code today, the time become normal? I submitted all the inference code this morning. And all of them seems will exceed the time limit. Th e point is..all the high score submissions I sent are quick submit(only public test set). Which means I don’t have score with whole testset in this situation 😂", "votes": 2 }, { "id": 1060435, "postDate": "2020-10-26T07:32:01.680Z", "content": "I ran my same GPU code twice for testing, yesterday it cost 890s (about 24 hours ago); today, it cost 2200s (about 8 hours ago). But I think the server problem start yesterday because of all my yesterday submission Timeout.
\nHence, I add quick submit method as you said, it will run just 2 mins in the notebook, but it still running for a score till now.
Well, there certainly is a problem with the server as the same submission that succeeded previously is timing out now.
", "rawMarkdown": "Well, there certainly is a problem with the server as the same submission that succeeded previously is timing out now. ", "votes": 3 }, { "id": 1060449, "postDate": "2020-10-26T07:54:37.740Z", "content": "The same thing happened.
\n submissions submitted yesterday timed out, even with the same code.
Help me to AT@ Kaggle staff, tell them this problem. We may have 1 more day : )
", "rawMarkdown": "Help me to AT@ Kaggle staff, tell them this problem. We may have 1 more day : )" }, { "id": 1060460, "postDate": "2020-10-26T08:06:10.980Z", "content": "@juliaelliott Hi Julia! I'm sorry to bother you. But here are some kagglers seems encountered some kaggle issues which might cause the notebook timeout. Can you please help to answer our questions/doubts?
\nI think there might be more people also have the same issues.
\nThank you
One thing! I try to submit my full Notebook inference code, and fill all NaN as 0.5. I got public LB quickly, within 3mins. It gives me an idea, maybe we can upload all public data inference result. And using your kernel to inference private test dataset. It will speed up!
", "rawMarkdown": "One thing! I try to submit my **full Notebook inference** code, and fill all **NaN as 0.5**. I got public LB quickly, within 3mins. It gives me an idea, maybe we can **upload all public data inference result**. And using your kernel to inference private test dataset. It will speed up!" } ], "comments": [ { "id": 1060855, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2020-10-26T15:17:16.643000", "content": "The commitments that were supposed to be done in a minute are still not done after 30 minutes. As it is, I am going to end up not being able to submit the culmination of a likely gold medal. Is there any way to improve this?
\n@juliaelliott
\nPlease reply.
@yujiariyasu We had some capacity issues in the past couple of hours. We shifted some capacity around to fix that now, so things should commit quickly.
\nThis did not affect commit runtime/timeouts at all, it just meant your commit might have taken longer to be assigned a machine to execute it.
", "votes": 6, "replies": [] }, { "id": 1061195, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-10-26T19:48:25.010000", "content": "I didn't experienced what you mention here. Our last 2 submission were 2 versions of the same notebooks with one of them just having some code commented. That commented notebook used 1 less model and failed the other with full code passed. Which doesn't make sense at all since the one which uses more model takes actually longer. Also previously I experienced same code passing in 7 hours vs 9 hours.
", "votes": 0, "replies": [] }, { "id": 1061217, "author_name": "Dustin", "author_url": "", "post_date": "2020-10-26T20:20:06.867000", "content": "@keremt The issue I specified was with regard to wait times for GPU machines to actually execute the notebook, not anything to do with runtime/timeout once it was actually executed.
\nThat's a separate issue mentioned in https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193252#1061191
", "votes": 2, "replies": [] } ] }, { "id": 1060759, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-10-26T14:31:09.797000", "content": "I can't commit kernel now.
", "votes": 5, "replies": [ { "id": 1060776, "author_name": "SHINO", "author_url": "", "post_date": "2020-10-26T14:41:10.283000", "content": "Me too.
\nThe commitments are supposed to be over soon, but they never are.
It seems that cannot commit GPU kernel only.
", "votes": 0, "replies": [] }, { "id": 1060850, "author_name": "SHINO", "author_url": "", "post_date": "2020-10-26T15:14:05.637000", "content": "Me too, only gpu.
\nI was able to commit to it over and over again.
\nI waited 19 minutes and the commit didn't start.
\nIt may be more effective to commit quickly and repeatedly
I can't commit my GPU kernel either!!! I still have 5 submissions today, I can't even use them!!!! Had so much code I just finished, very frustrating.
\noh my god….
I think hosts will extend the deadline because of this
", "votes": 3, "replies": [] }, { "id": 1060882, "author_name": "Nanashi", "author_url": "", "post_date": "2020-10-26T15:40:30.233000", "content": "I don't think so, it's like studying 9hrs before the exam. I used my submissions having in mind that this would happen (because it happened before), feels bad. In any case, would be cool to have extra days even for my team haha
", "votes": 1, "replies": [] }, { "id": 1060889, "author_name": "Dustin", "author_url": "", "post_date": "2020-10-26T15:47:04.980000", "content": "GPU commits should be available again now. There was some capacity issues, so there was increased delays for commits to be assigned a machine to execute.
\nhttps://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193252#1060887
", "votes": 3, "replies": [] } ] }, { "id": 1061191, "author_name": "Phil Culliton", "author_url": "", "post_date": "2020-10-26T19:44:59.603000", "content": "Hi all!
\nThank you to those who reported extended commit or submission times. We've investigated the issue. It's common in the final 2-3 days of a competition for demand on both filestores and notebook processors surged significantly, and that's what happened here. This is exacerbated by the large file size of this competition, in particular.
\nWhile we've taken a few measures to expand capacity, you should still expect demand to potentially delay your access to machines. Some recommendations:
\nIt's a good idea to consider the final day or two of any competition as a bit of a danger zone in terms of capacity. We'll always do our best to make sure that people get the machines they need, but last-minute submissions may run into difficulties like this. We recognize that this doesn't help much now, and is definitely easier said than done, but it may be helpful in terms of future planning.
\nTo be clear, we will not be extending the deadline of the competition.
", "votes": 5, "replies": [] }, { "id": 1060750, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-10-26T14:24:54.230000", "content": "I think the increase in notebook runtime is driven primarily by I/O.
\nI just committed a notebook where it was taking 20-30 seconds to load in 1 CT exam. Previously this took about 1s.
", "votes": 3, "replies": [] }, { "id": 1060644, "author_name": "Psi", "author_url": "", "post_date": "2020-10-26T12:38:45.310000", "content": "Kernels are not using the same CPU Hardware all the time. So if your kernels are CPU bottlenecked, this can be an issue.
", "votes": 3, "replies": [] }, { "id": 1060663, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-10-26T12:57:00.090000", "content": "Yesterday I submitted a kernel which I knew should finish within 9 hours because it was faster than other successful kernels that finished with 1-2 hours of leeway. It timed out.
\nThen I submitted the same kernel again, as well as the same kernel with fewer models. The original worked the second time, and the one with fewer models timed out.
", "votes": 4, "replies": [ { "id": 1060678, "author_name": "Dewei Chen", "author_url": "", "post_date": "2020-10-26T13:10:34.880000", "content": "That's good news!
", "votes": -1, "replies": [] } ] }, { "id": 1060572, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-10-26T10:37:22.403000", "content": "This happened in the Bird competition as well, as far as I remember all my submissions on the last day were time outs, despite me not changing my code.
\nI'm afraid there is nothing we can do about it except try not to rely too much on the last days, which is easier said than done.
", "votes": 4, "replies": [ { "id": 1060575, "author_name": "Dewei Chen", "author_url": "", "post_date": "2020-10-26T10:39:09.343000", "content": "I come up with an idea, just don't know is it valid: https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193285
", "votes": 0, "replies": [] } ] }, { "id": 1060487, "author_name": "Gzl0506", "author_url": "", "post_date": "2020-10-26T08:38:49.827000", "content": "Same for me…
\nThey are same model with previous submission. But all ran out of time today.
My rank has been falling every few hours so I guess many people can inference normally.
", "votes": 3, "replies": [ { "id": 1060546, "author_name": "Dewei Chen", "author_url": "", "post_date": "2020-10-26T09:53:00.200000", "content": "So do I, but I think they just lucky, or they have some else detect code in their notebook.
\ncause whatever they did, it will always be taking 8-9 hours for inference.
The thing happened to me, 5 submissions yesterday close to reset and 2 today. All timed out 😒
", "votes": 1, "replies": [] }, { "id": 1060747, "author_name": "aksg87", "author_url": "", "post_date": "2020-10-26T14:22:59.477000", "content": "Hello,
\nI would like to report the same issues on our end. We are preprocessing the data and this step took 20-25 minutes to run yesterday. Now it's taking 1.5 hours. The CPU percent is also hovering around 5% instead of going much higher.
\nI scaled things so it would run in 2-3 hours on public test but with these speed issues that might not be possible. Is there an issue with compute for notebooks? cc: @juliaelliott @philculliton
", "votes": 1, "replies": [] }, { "id": 1060735, "author_name": "Nanashi", "author_url": "", "post_date": "2020-10-26T14:17:12.493000", "content": "We had the same problem, 4 submissions wasted. I thought it could be a memory problem and reported it https://www.kaggle.com/product-feedback/193238#1060664 indeed there is a bug in kaggle's memory report dashboard. Today the exactly same submission ended in 8hrs. Feels bad man.
", "votes": 1, "replies": [ { "id": 1060893, "author_name": "Dustin", "author_url": "", "post_date": "2020-10-26T15:49:30.133000", "content": "That 'memory bug' is only in the dashboard, it does not affect runtime. It's not actually technically a bug, we just decided that we would stop including 'cached' memory in the memory we report to the UI, so you'd see the 'in use' memory more clearly.
\nBut that does not affect the execution at all, you won't get 'more memory' or anything due to this change, and it didn't affect scoring etc.
", "votes": 2, "replies": [] }, { "id": 1060897, "author_name": "Nanashi", "author_url": "", "post_date": "2020-10-26T15:52:40.717000", "content": "great to know, I was confused haha @herbison thanks for the effort these last hours :)
", "votes": 1, "replies": [] } ] }, { "id": 1060674, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2020-10-26T13:04:46.743000", "content": "\n\n\nGPU running time increase 100% with the same code.
\n
Is this only during the submission process?
", "votes": 1, "replies": [ { "id": 1060677, "author_name": "Dewei Chen", "author_url": "", "post_date": "2020-10-26T13:09:16.083000", "content": "It happens in my notebook inference process. Same code, same parameters, different run time.
\nIt increases by 1 times seconds than yesterday. 800s yesterday 2200s today.
What's wrong with kaggle server !!! I got Timeout error as well 😣
\nWe don't have enough time and it's a pity that we can't submit the new kernel.
\n@dwchen thank you for this topic.
You are welcome, We may have one more day if staff know what happens. I believe the problem is at GPU.
\nSorry, my web is broken just yet @_@
", "votes": 0, "replies": [] } ] }, { "id": 1061073, "author_name": "Philip Dhingra", "author_url": "", "post_date": "2020-10-26T18:10:25.913000", "content": "I accidentally submitted the same notebook and version, at the same time, and one of them timed out, while the other delivered a score. However, I designed both to finish in just under 9 hours. One speculation is that small differences between VMs may cause one to go over the limit over a long enough notebook run. Some supporting evidence is that when I do Save & Runs, I'll notice slight variations in the timestamps of events in the logs, even if no code has changed.
", "votes": 2, "replies": [] }, { "id": 1061049, "author_name": "Nikita Kozodoi", "author_url": "", "post_date": "2020-10-26T17:57:21.673000", "content": "I was a little unfortunate to rely on the last two days to implement a few important ideas in the modeling pipeline.
\nIt appears that the submission time is affected since my new notebooks that should have a similar/smaller running times compared to their previous successful versions from a few days ago are now timing out on the submission stage. Some submissions are still running, but it seems likely that none of them will make it within the time limit.
", "votes": 2, "replies": [] }, { "id": 1060529, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2020-10-26T09:29:25.380000", "content": "There are probably a lot of people who were using fast sub until yesterday and are planning to predict private for the first time today.
\nI would like to see some sort of remedy.
hoping they can solve this problem.
", "votes": 1, "replies": [] } ] }, { "id": 1060421, "author_name": "Tsai29", "author_url": "", "post_date": "2020-10-26T07:13:57.660000", "content": "I have the same problem today. The submitted code should be finished by now. But they are still running…I guess all of them will exceed the time limit. And I already ran out of submissions….
", "votes": 2, "replies": [ { "id": 1060427, "author_name": "Dewei Chen", "author_url": "", "post_date": "2020-10-26T07:17:35.350000", "content": "I spent 2000 seconds on inference code today and 800 seconds in normal. With that same code, just for a re-run. I'm pretty sure that the server has some problems.
", "votes": 2, "replies": [] }, { "id": 1060431, "author_name": "Tsai29", "author_url": "", "post_date": "2020-10-26T07:23:11.553000", "content": "So after you re-run the code today, the time become normal? I submitted all the inference code this morning. And all of them seems will exceed the time limit. Th e point is..all the high score submissions I sent are quick submit(only public test set). Which means I don’t have score with whole testset in this situation 😂
", "votes": 2, "replies": [] }, { "id": 1060435, "author_name": "Dewei Chen", "author_url": "", "post_date": "2020-10-26T07:32:01.680000", "content": "I ran my same GPU code twice for testing, yesterday it cost 890s (about 24 hours ago); today, it cost 2200s (about 8 hours ago). But I think the server problem start yesterday because of all my yesterday submission Timeout.
\nHence, I add quick submit method as you said, it will run just 2 mins in the notebook, but it still running for a score till now.
Well, there certainly is a problem with the server as the same submission that succeeded previously is timing out now.
", "votes": 3, "replies": [] }, { "id": 1060449, "author_name": "SHINO", "author_url": "", "post_date": "2020-10-26T07:54:37.740000", "content": "The same thing happened.
\n submissions submitted yesterday timed out, even with the same code.
Help me to AT@ Kaggle staff, tell them this problem. We may have 1 more day : )
", "votes": 0, "replies": [] }, { "id": 1060460, "author_name": "Tsai29", "author_url": "", "post_date": "2020-10-26T08:06:10.980000", "content": "@juliaelliott Hi Julia! I'm sorry to bother you. But here are some kagglers seems encountered some kaggle issues which might cause the notebook timeout. Can you please help to answer our questions/doubts?
\nI think there might be more people also have the same issues.
\nThank you
One thing! I try to submit my full Notebook inference code, and fill all NaN as 0.5. I got public LB quickly, within 3mins. It gives me an idea, maybe we can upload all public data inference result. And using your kernel to inference private test dataset. It will speed up!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1060363": "All my code with **Notebook Timeout** today, and all GPU running time increase **100%** with the same code. It looks like your Leaderboard running time is also 9 hours.\nIs that mean we cannot make any other inference anymore? Or just **pray** for lucky?\n\n\n@juliaelliott ", "1060855": "The commitments that were supposed to be done in a minute are still not done after 30 minutes. As it is, I am going to end up not being able to submit the culmination of a likely gold medal. Is there any way to improve this?\n@juliaelliott \nPlease reply.", "1060759": "I can't commit kernel now. ", "1061191": "Hi all!\n\nThank you to those who reported extended commit or submission times. We've investigated the issue. It's common in the final 2-3 days of a competition for demand on both filestores and notebook processors surged significantly, and that's what happened here. This is exacerbated by the large file size of this competition, in particular. \n\nWhile we've taken a few measures to expand capacity, you should still expect demand to potentially delay your access to machines. Some recommendations:\n\n1. Ensure that your submission is not timing out for code reasons (e.g. it does not run within time constraints on the substantively larger private test set). Review these common [code debugging tips](https://www.kaggle.com/code-competition-debugging).\n2. Re-commit. It's possible that previous delays in accessing a machine have since been remediated by our increased capacity measures.\n\nIt's a good idea to consider the final day or two of any competition as a bit of a danger zone in terms of capacity. We'll always do our best to make sure that people get the machines they need, but last-minute submissions may run into difficulties like this. We recognize that this doesn't help much now, and is definitely easier said than done, but it may be helpful in terms of future planning.\n\nTo be clear, we will not be extending the deadline of the competition.", "1060750": "I think the increase in notebook runtime is driven primarily by I/O.\n\nI just committed a notebook where it was taking 20-30 seconds to load in 1 CT exam. Previously this took about 1s. ", "1060644": "Kernels are not using the same CPU Hardware all the time. So if your kernels are CPU bottlenecked, this can be an issue. ", "1060663": "Yesterday I submitted a kernel which I knew should finish within 9 hours because it was faster than other successful kernels that finished with 1-2 hours of leeway. It timed out.\n\nThen I submitted the same kernel again, as well as the same kernel with fewer models. The original worked the second time, and the one with fewer models timed out.", "1060572": "This happened in the Bird competition as well, as far as I remember all my submissions on the last day were time outs, despite me not changing my code.\n\nI'm afraid there is nothing we can do about it except try not to rely too much on the last days, which is easier said than done.", "1060487": "Same for me...\nThey are same model with previous submission. But all ran out of time today. ", "1060543": "My rank has been falling every few hours so I guess many people can inference normally.", "1060823": "The thing happened to me, 5 submissions yesterday close to reset and 2 today. All timed out 😒", "1060747": "Hello,\n\nI would like to report the same issues on our end. We are preprocessing the data and this step took 20-25 minutes to run yesterday. Now it's taking 1.5 hours. The CPU percent is also hovering around 5% instead of going much higher. \n\nI scaled things so it would run in 2-3 hours on public test but with these speed issues that might not be possible. Is there an issue with compute for notebooks? cc: @juliaelliott @philculliton ", "1060735": "We had the same problem, 4 submissions wasted. I thought it could be a memory problem and reported it https://www.kaggle.com/product-feedback/193238#1060664 indeed there is a bug in kaggle's memory report dashboard. Today the exactly same submission ended in 8hrs. Feels bad man.", "1060674": "@dwchen \n>GPU running time increase 100% with the same code.\n\nIs this only during the submission process?", "1060517": "What's wrong with kaggle server !!! I got Timeout error as well 😣\nWe don't have enough time and it's a pity that we can't submit the new kernel.\n@dwchen thank you for this topic.", "1061073": "I accidentally submitted the same notebook and version, at the same time, and one of them timed out, while the other delivered a score. However, I designed both to finish in just under 9 hours. One speculation is that small differences between VMs may cause one to go over the limit over a long enough notebook run. Some supporting evidence is that when I do Save & Runs, I'll notice slight variations in the timestamps of events in the logs, even if no code has changed.", "1061049": "I was a little unfortunate to rely on the last two days to implement a few important ideas in the modeling pipeline. \n\nIt appears that the submission time is affected since my new notebooks that should have a similar/smaller running times compared to their previous successful versions from a few days ago are now timing out on the submission stage. Some submissions are still running, but it seems likely that none of them will make it within the time limit.", "1060529": "There are probably a lot of people who were using fast sub until yesterday and are planning to predict private for the first time today.\nI would like to see some sort of remedy.", "1060421": "I have the same problem today. The submitted code should be finished by now. But they are still running...I guess all of them will exceed the time limit. And I already ran out of submissions....\n", "1060551": "One thing! I try to submit my **full Notebook inference** code, and fill all **NaN as 0.5**. I got public LB quickly, within 3mins. It gives me an idea, maybe we can **upload all public data inference result**. And using your kernel to inference private test dataset. It will speed up!" } }