![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4310004%2Ffb9c857e74d45ae655e6538dbe13bd17%2Fimage.png?generation=1760484774853845&alt=media\")
During the last 24 hours, our submissions took less time than before on the sample test set, but all of them timed out on the hidden test set. Has the same issue occurred again?
", "messages": [ { "id": 3300873, "postDate": "2025-10-11T20:58:15.427Z", "content": "During the last 24 hours, our submissions took less time than before on the sample test set, but all of them timed out on the hidden test set. Has the same issue occurred again?
", "rawMarkdown": "During the last 24 hours, our submissions took less time than before on the sample test set, but all of them timed out on the hidden test set. Has the same issue occurred again?", "votes": 33 }, { "id": 3301349, "postDate": "2025-10-13T07:38:51.193Z", "content": "Made a sub yesterday with 3 folds which timed out, same with 5 folds ran ok before… @ryanholbrook
\nIs there something that can be done to prevent unfair random deadline ?
", "rawMarkdown": "Made a sub yesterday with 3 folds which timed out, same with 5 folds ran ok before... @ryanholbrook\n\nIs there something that can be done to prevent unfair random deadline ?", "votes": 9, "replies": [ { "id": 3301457, "postDate": "2025-10-13T12:33:29.320Z", "content": "", "rawMarkdown": "@evancalabrese ?", "votes": 3, "replies": [ { "id": 3301529, "postDate": "2025-10-13T15:20:41.140Z", "content": "@ryanholbrook can we hope for increased capacities on the last day to finalize our submissions ?
", "rawMarkdown": "@ryanholbrook can we hope for increased capacities on the last day to finalize our submissions ?", "votes": 4 }, { "id": 3301665, "postDate": "2025-10-13T22:42:24.270Z", "content": "@operedoud2 that's so sad man.
", "rawMarkdown": "@operedoud2 that's so sad man.", "votes": 1 } ] } ] }, { "id": 3302014, "postDate": "2025-10-14T19:05:20.957Z", "content": "Hi everyone,
\nWe're investigating the reports of slowdowns in submission times. There are (potentially) two separate issues at play here.
\nThe first is a delay in submission start times due to GPU queueing. During times of high demand, we can run out of GPU capacity. Submissions will then enter a queue until the next GPU is available. This delay doesn't count against the overall submission time limit and wouldn't itself be a cause of submission failures.
\nThe second possible problem is submissions themselves running more slowly than they have in the past. We haven't been able to determine whether this is actually happening in a systematic way due to a fault in our evaluation backend (because of a problem with the machines running the submissions, or a database issue, say), or whether, when it happens, it's due instead to some kind of variance introduced by stochastic submission code. It's not clear that, at least on average, submissions are running more slowly, so the problem is difficult to diagnose.
\nWe will continue to investigate the issue, and if it proves that there is a fault in Kaggle's evaluation system, we will ensure that the affected submissions are rerun.
", "rawMarkdown": "Hi everyone,\n\nWe're investigating the reports of slowdowns in submission times. There are (potentially) two separate issues at play here.\n\nThe first is a delay in submission start times due to GPU queueing. During times of high demand, we can run out of GPU capacity. Submissions will then enter a queue until the next GPU is available. This delay doesn't count against the overall submission time limit and wouldn't itself be a cause of submission failures.\n\nThe second possible problem is submissions themselves running more slowly than they have in the past. We haven't been able to determine whether this is actually happening in a systematic way due to a fault in our evaluation backend (because of a problem with the machines running the submissions, or a database issue, say), or whether, when it happens, it's due instead to some kind of variance introduced by stochastic submission code. It's not clear that, at least on average, submissions are running more slowly, so the problem is difficult to diagnose.\n\nWe will continue to investigate the issue, and if it proves that there is a fault in Kaggle's evaluation system, we will ensure that the affected submissions are rerun.", "votes": 9, "isPinned": true, "replies": [ { "id": 3302016, "postDate": "2025-10-14T19:16:01.937Z", "content": "Thank you for taking a look at this!
\nYou’ll find an example in my latest subs of a 100% deterministic code which timed out 2 days ago and finished within time limit yesterday.
\nI think there are plenty of competitors with similar examples!
", "rawMarkdown": "Thank you for taking a look at this!\n\nYou’ll find an example in my latest subs of a 100% deterministic code which timed out 2 days ago and finished within time limit yesterday.\n\nI think there are plenty of competitors with similar examples!", "votes": 5, "replies": [ { "id": 3302084, "postDate": "2025-10-15T01:15:05.917Z", "content": "Thank you! M
", "rawMarkdown": "Thank you! M" } ] }, { "id": 3302017, "postDate": "2025-10-14T19:18:49.007Z", "content": "@ryanholbrook There are various mentions from kagglers where the same (or more or less the same…) submission suddenly takes a lot longer or just times out. It looks like the second issue but understandable that can be difficult to analyze.
\nIf required I can provide with some submission with more models finished, other with less models timed out. Likely other Kagglers can also provide you with details if required.
\nGood luck anyway.
", "rawMarkdown": "@ryanholbrook There are various mentions from kagglers where the same (or more or less the same...) submission suddenly takes a lot longer or just times out. It looks like the second issue but understandable that can be difficult to analyze.\n\nIf required I can provide with some submission with more models finished, other with less models timed out. Likely other Kagglers can also provide you with details if required.\n\nGood luck anyway.", "votes": 4 }, { "id": 3302020, "postDate": "2025-10-14T19:29:05.300Z", "content": "All the best.
", "rawMarkdown": "All the best." }, { "id": 3302023, "postDate": "2025-10-14T19:42:58.697Z", "content": "Please extend the deadline just a few days
", "rawMarkdown": "Please extend the deadline just a few days", "votes": 1 }, { "id": 3302031, "postDate": "2025-10-14T20:36:40.260Z", "content": "What about submissions made more than 12h before deadline which are still queuing after deadline?
", "rawMarkdown": "What about submissions made more than 12h before deadline which are still queuing after deadline?" }, { "id": 3302065, "postDate": "2025-10-14T23:23:29.337Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3302067, "postDate": "2025-10-14T23:24:28.087Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3302072, "postDate": "2025-10-14T23:32:56.103Z", "content": "@ryanholbrook this is our last few submissions.
\n
I might be naive but the situation is quite unfair, imo. Now that also the private LB is open and all submissions also revealed the private dataset score, I don't really see a way of how to bring it back to a fair state again, even if the affected submissions would be rerun… Such a pity that such a nice competition ends like this…
", "rawMarkdown": "I might be naive but the situation is quite unfair, imo. Now that also the private LB is open and all submissions also revealed the private dataset score, I don't really see a way of how to bring it back to a fair state again, even if the affected submissions would be rerun… Such a pity that such a nice competition ends like this…", "votes": 2, "replies": [ { "id": 3302187, "postDate": "2025-10-15T08:56:35.427Z", "content": "@st3v3d Agreed. Everyone can now see the private score, which means you can no longer allow participants to make another selection or automatically choose the best private score. Also, I’m really curious to see how well our full 5-fold approach would perform, since our submitted winning solution used only 2 folds due to the server slowdown.
", "rawMarkdown": "@st3v3d Agreed. Everyone can now see the private score, which means you can no longer allow participants to make another selection or automatically choose the best private score. Also, I’m really curious to see how well our full 5-fold approach would perform, since our submitted winning solution used only 2 folds due to the server slowdown.", "votes": 3 }, { "id": 3302320, "postDate": "2025-10-15T14:49:25.990Z", "content": "I completely agree.
\nNow that the private leaderboard scores have been revealed together with the public ones, it’s no longer possible to make fair re-selections of submissions — everyone already knows the correlation between the two. Even in the unrealistic case that the organizers collected a new external test set, it would still be impossible to restore true fairness at this point.
For future competitions, I think it’s essential to explicitly warn participants to finalize their submissions early, since from my experience, heavy server load and timeouts near the deadline happen quite frequently. The rule should clearly state that deadlines will not be extended and that it’s each participant’s responsibility to complete submissions in advance.
\nI’d really appreciate it if @ryanholbrook could kindly consider this for future competitions.
", "rawMarkdown": "I completely agree.\nNow that the private leaderboard scores have been revealed together with the public ones, it’s no longer possible to make fair re-selections of submissions — everyone already knows the correlation between the two. Even in the unrealistic case that the organizers collected a new external test set, it would still be impossible to restore true fairness at this point.\n\nFor future competitions, I think it’s essential to explicitly warn participants to finalize their submissions early, since from my experience, heavy server load and timeouts near the deadline happen quite frequently. The rule should clearly state that deadlines will not be extended and that it’s each participant’s responsibility to complete submissions in advance.\n\nI’d really appreciate it if @ryanholbrook could kindly consider this for future competitions.", "votes": 4 } ] } ] }, { "id": 3301260, "postDate": "2025-10-13T01:21:21.703Z", "content": "@ryanholbrook, any confirmation that the servers are running slower? Will this continue until the end of the competition? We tried to verify by submitting an old notebook that previously finished in ~10 hours, but now it’s timing out.
", "rawMarkdown": "@ryanholbrook, any confirmation that the servers are running slower? Will this continue until the end of the competition? We tried to verify by submitting an old notebook that previously finished in ~10 hours, but now it’s timing out.", "votes": 5, "replies": [ { "id": 3301262, "postDate": "2025-10-13T01:23:34.713Z", "content": "I've got 3 timeouts on previously finished submissions, just weights changed… Sad, I've discovered a bug that could explain my huge CV to test score drop. Reducing ensemble charge to minium at the moment. I hope I can at least check my bug fix.
\nEDIT: Would be nice if confirmed then timeouts would be rerun just for fairness.
", "rawMarkdown": "I've got 3 timeouts on previously finished submissions, just weights changed... Sad, I've discovered a bug that could explain my huge CV to test score drop. Reducing ensemble charge to minium at the moment. I hope I can at least check my bug fix.\n\nEDIT: Would be nice if confirmed then timeouts would be rerun just for fairness.", "votes": 5 } ] }, { "id": 3301990, "postDate": "2025-10-14T17:36:25.333Z", "content": "T.b.h.
\nI'am not sure what is worse…the submission issues about which kagglers complain already for at least 2 days…
\nOr the seemingly total lack of response from the Kaggle hosts ….
\n@inversion @ryanholbrook @sohier anybody?
", "rawMarkdown": "T.b.h.\n\nI'am not sure what is worse...the submission issues about which kagglers complain already for at least 2 days...\n\nOr the seemingly total lack of response from the Kaggle hosts ....\n\n@inversion @ryanholbrook @sohier anybody?", "votes": 6 }, { "id": 3301578, "postDate": "2025-10-13T17:04:17.937Z", "content": "Are there any updates regarding this?
", "rawMarkdown": "Are there any updates regarding this?", "votes": 3 }, { "id": 3301324, "postDate": "2025-10-13T06:29:55.540Z", "content": "Hi, we are facing the same issue. Submissions that previously took 8–9 hours are now running into timeouts. This is very unfortunate.
", "rawMarkdown": "Hi, we are facing the same issue. Submissions that previously took 8–9 hours are now running into timeouts. This is very unfortunate.", "votes": 3, "replies": [ { "id": 3301343, "postDate": "2025-10-13T07:18:02.090Z", "content": "@st3v3d It seems this competition ended two days early.
", "rawMarkdown": "@st3v3d It seems this competition ended two days early.", "votes": 2, "replies": [ { "id": 3301420, "postDate": "2025-10-13T10:39:23.483Z", "content": "That would be such a pity🥲
", "rawMarkdown": "That would be such a pity🥲", "votes": 2 } ] } ] }, { "id": 3301548, "postDate": "2025-10-13T15:52:11.427Z", "content": "Same here…very slow…
", "rawMarkdown": "Same here...very slow...", "votes": 4 }, { "id": 3301277, "postDate": "2025-10-13T02:55:20.393Z", "content": "@ryanholbrook anything on this? Will submissions be rerun or any timeline changes @evancalabrese. Thank you for your info.
", "rawMarkdown": "@ryanholbrook anything on this? Will submissions be rerun or any timeline changes @evancalabrese. Thank you for your info.", "votes": 4 }, { "id": 3301019, "postDate": "2025-10-12T09:14:24.623Z", "content": "Seems so. 2 days, submissions took around 2-3 hours. Made 2 submissions last night, one's been running for ~ 7 hours.
", "rawMarkdown": "Seems so. 2 days, submissions took around 2-3 hours. Made 2 submissions last night, one's been running for ~ 7 hours.", "votes": 4, "replies": [ { "id": 3301072, "postDate": "2025-10-12T12:10:49.493Z", "content": "Is known that submission time is ofuscated. It will show you randomly larger execution times.
", "rawMarkdown": "Is known that submission time is ofuscated. It will show you randomly larger execution times.", "votes": 1 } ] }, { "id": 3302080, "postDate": "2025-10-15T01:00:34.820Z", "content": "I want to share that in the last minute before deadline, one 21hr sub finished (best result), and I select it. One of our submission is still running now.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4310004%2F4e277ba6d3e34d7f3b0c8e65920d80cb%2Fd3053606.jpg?generation=1760490030867019&alt=media\")
Yeah, we are facing the same issue.
", "rawMarkdown": "Yeah, we are facing the same issue.", "votes": 1 }, { "id": 3301762, "postDate": "2025-10-14T06:59:21.870Z", "content": "Don't you think it's because we are ALL trying to do it at the same time, last minute panic and all that jazz… ?
", "rawMarkdown": "Don't you think it's because we are ALL trying to do it at the same time, last minute panic and all that jazz... ?", "votes": -6, "replies": [ { "id": 3301788, "postDate": "2025-10-14T08:31:11.473Z", "content": "Yes, of course, we are just trying to see if there is any feasible solution to this
", "rawMarkdown": "Yes, of course, we are just trying to see if there is any feasible solution to this", "votes": 2 } ] }, { "id": 3302332, "postDate": "2025-10-15T15:29:18.930Z", "content": "Hi Kaggle team,
\nI submitted my notebooks before the deadline, but they kept running and I only received the scores after the deadline. These were my best submissions, and I couldn’t select them in time. I believe the same issue affected many participants. Had we received the scores before, we'd have selected those. Given how rankings are clustered around 0.69, it’s likely others also made improvements but couldn’t select their final submissions due to issues beyond their control. Would it be possible to consider such submissions internally so no one is unfairly penalized for server delays?
", "rawMarkdown": "Hi Kaggle team,\n\nI submitted my notebooks before the deadline, but they kept running and I only received the scores after the deadline. These were my best submissions, and I couldn’t select them in time. I believe the same issue affected many participants. Had we received the scores before, we'd have selected those. Given how rankings are clustered around 0.69, it’s likely others also made improvements but couldn’t select their final submissions due to issues beyond their control. Would it be possible to consider such submissions internally so no one is unfairly penalized for server delays? " }, { "id": 3301922, "postDate": "2025-10-14T14:42:49.070Z", "content": "Oh man, it's been running for last 11 hours. I'd to run some final experiments, looks like all came down to 0. Shouldn't have waited for the last day. 🙇♀️
", "rawMarkdown": "Oh man, it's been running for last 11 hours. I'd to run some final experiments, looks like all came down to 0. Shouldn't have waited for the last day. 🙇♀️" }, { "id": 3301821, "postDate": "2025-10-14T10:50:23.493Z", "content": "What if the deadline has passed but my submission is still scoring? Will it still count? It's a shame to say that our submission wasn't successful at any time, and our Final version is still in training.😭
", "rawMarkdown": "What if the deadline has passed but my submission is still scoring? Will it still count? It's a shame to say that our submission wasn't successful at any time, and our Final version is still in training.😭", "replies": [ { "id": 3301823, "postDate": "2025-10-14T10:57:29.540Z", "content": "As far as I know the deadline is sharp, since the private leaderboard is published just straightaway after the deadline. If you want to do some final submission that takes almost the limit of 12h, I would recommend to do it already now. It is quite unfortunate…
", "rawMarkdown": "As far as I know the deadline is sharp, since the private leaderboard is published just straightaway after the deadline. If you want to do some final submission that takes almost the limit of 12h, I would recommend to do it already now. It is quite unfortunate...", "replies": [ { "id": 3301830, "postDate": "2025-10-14T11:21:54.077Z", "content": "OK, thanks for your response. I wil try hard🤧
", "rawMarkdown": "OK, thanks for your response. I wil try hard🤧" } ] }, { "id": 3301846, "postDate": "2025-10-14T11:46:44.603Z", "content": "I tried that once, your late submission would not count for public leaderboard. Because you can select two notebook to get scored and you can’t select the ones scoring. Plus, if you let the system choose the best scoring one, the choices are finalized before final leaderboard show up.
", "rawMarkdown": "I tried that once, your late submission would not count for public leaderboard. Because you can select two notebook to get scored and you can’t select the ones scoring. Plus, if you let the system choose the best scoring one, the choices are finalized before final leaderboard show up.", "votes": 2, "replies": [ { "id": 3301968, "postDate": "2025-10-14T16:54:05.517Z", "content": "Thanks for your remind. But we just knew that our submit was failed. We got a Kaggle Error again and still don't know why, don't know how to fix. I think it's over. I'm glad to join this competition. Unlucky but learned a lot.🥲
", "rawMarkdown": "Thanks for your remind. But we just knew that our submit was failed. We got a Kaggle Error again and still don't know why, don't know how to fix. I think it's over. I'm glad to join this competition. Unlucky but learned a lot.🥲" } ] } ] }, { "id": 3301464, "postDate": "2025-10-13T12:43:52.550Z", "content": "I don't remember where but I'm pretty sure I've read execution time would be ofuscated by show randomly larger executions times. I assume that mechanic never would produce time out by itself…
", "rawMarkdown": "I don't remember where but I'm pretty sure I've read execution time would be ofuscated by show randomly larger executions times. I assume that mechanic never would produce time out by itself..." }, { "id": 3301074, "postDate": "2025-10-12T12:21:56.903Z", "content": "I just don't know, why same submission finished within 11hr yesterday but needs to run nearly 12hr today.
", "rawMarkdown": "I just don't know, why same submission finished within 11hr yesterday but needs to run nearly 12hr today.", "replies": [ { "id": 3301236, "postDate": "2025-10-12T21:50:43.837Z", "content": "Towards the end of competitions, sometimes the submissions run slower because more people are submitting and this probably slows down the servers.
", "rawMarkdown": "Towards the end of competitions, sometimes the submissions run slower because more people are submitting and this probably slows down the servers. ", "votes": 4, "replies": [ { "id": 3301239, "postDate": "2025-10-12T22:22:07.460Z", "content": "@vaillantali queue requires a day
", "rawMarkdown": "@vaillantali queue requires a day" } ] } ] } ], "comments": [ { "id": 3301349, "author_name": "Optimo", "author_url": "", "post_date": "2025-10-13T07:38:51.193000", "content": "Made a sub yesterday with 3 folds which timed out, same with 5 folds ran ok before… @ryanholbrook
\nIs there something that can be done to prevent unfair random deadline ?
", "votes": 9, "replies": [ { "id": 3301457, "author_name": "Optimo", "author_url": "", "post_date": "2025-10-13T12:33:29.320000", "content": "", "votes": 3, "replies": [ { "id": 3301529, "author_name": "Optimo", "author_url": "", "post_date": "2025-10-13T15:20:41.140000", "content": "@ryanholbrook can we hope for increased capacities on the last day to finalize our submissions ?
", "votes": 4, "replies": [] }, { "id": 3301665, "author_name": "Tom", "author_url": "", "post_date": "2025-10-13T22:42:24.270000", "content": "@operedoud2 that's so sad man.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3302014, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2025-10-14T19:05:20.957000", "content": "Hi everyone,
\nWe're investigating the reports of slowdowns in submission times. There are (potentially) two separate issues at play here.
\nThe first is a delay in submission start times due to GPU queueing. During times of high demand, we can run out of GPU capacity. Submissions will then enter a queue until the next GPU is available. This delay doesn't count against the overall submission time limit and wouldn't itself be a cause of submission failures.
\nThe second possible problem is submissions themselves running more slowly than they have in the past. We haven't been able to determine whether this is actually happening in a systematic way due to a fault in our evaluation backend (because of a problem with the machines running the submissions, or a database issue, say), or whether, when it happens, it's due instead to some kind of variance introduced by stochastic submission code. It's not clear that, at least on average, submissions are running more slowly, so the problem is difficult to diagnose.
\nWe will continue to investigate the issue, and if it proves that there is a fault in Kaggle's evaluation system, we will ensure that the affected submissions are rerun.
", "votes": 9, "replies": [ { "id": 3302016, "author_name": "Optimo", "author_url": "", "post_date": "2025-10-14T19:16:01.937000", "content": "Thank you for taking a look at this!
\nYou’ll find an example in my latest subs of a 100% deterministic code which timed out 2 days ago and finished within time limit yesterday.
\nI think there are plenty of competitors with similar examples!
", "votes": 5, "replies": [ { "id": 3302084, "author_name": "MD The Dramatist", "author_url": "", "post_date": "2025-10-15T01:15:05.917000", "content": "Thank you! M
", "votes": 0, "replies": [] } ] }, { "id": 3302017, "author_name": "Robin Smits", "author_url": "", "post_date": "2025-10-14T19:18:49.007000", "content": "@ryanholbrook There are various mentions from kagglers where the same (or more or less the same…) submission suddenly takes a lot longer or just times out. It looks like the second issue but understandable that can be difficult to analyze.
\nIf required I can provide with some submission with more models finished, other with less models timed out. Likely other Kagglers can also provide you with details if required.
\nGood luck anyway.
", "votes": 4, "replies": [] }, { "id": 3302020, "author_name": "Tom", "author_url": "", "post_date": "2025-10-14T19:29:05.300000", "content": "All the best.
", "votes": 0, "replies": [] }, { "id": 3302023, "author_name": "Cyrus", "author_url": "", "post_date": "2025-10-14T19:42:58.697000", "content": "Please extend the deadline just a few days
", "votes": 1, "replies": [] }, { "id": 3302031, "author_name": "Optimo", "author_url": "", "post_date": "2025-10-14T20:36:40.260000", "content": "What about submissions made more than 12h before deadline which are still queuing after deadline?
", "votes": 0, "replies": [] }, { "id": 3302065, "author_name": "", "author_url": "", "post_date": "2025-10-14T23:23:29.337000", "content": "", "votes": 0, "replies": [] }, { "id": 3302067, "author_name": "", "author_url": "", "post_date": "2025-10-14T23:24:28.087000", "content": "", "votes": 0, "replies": [] }, { "id": 3302072, "author_name": "Tom", "author_url": "", "post_date": "2025-10-14T23:32:56.103000", "content": "@ryanholbrook this is our last few submissions.
\n
I might be naive but the situation is quite unfair, imo. Now that also the private LB is open and all submissions also revealed the private dataset score, I don't really see a way of how to bring it back to a fair state again, even if the affected submissions would be rerun… Such a pity that such a nice competition ends like this…
", "votes": 2, "replies": [ { "id": 3302187, "author_name": "Tom", "author_url": "", "post_date": "2025-10-15T08:56:35.427000", "content": "@st3v3d Agreed. Everyone can now see the private score, which means you can no longer allow participants to make another selection or automatically choose the best private score. Also, I’m really curious to see how well our full 5-fold approach would perform, since our submitted winning solution used only 2 folds due to the server slowdown.
", "votes": 3, "replies": [] }, { "id": 3302320, "author_name": "YYama", "author_url": "", "post_date": "2025-10-15T14:49:25.990000", "content": "I completely agree.
\nNow that the private leaderboard scores have been revealed together with the public ones, it’s no longer possible to make fair re-selections of submissions — everyone already knows the correlation between the two. Even in the unrealistic case that the organizers collected a new external test set, it would still be impossible to restore true fairness at this point.
For future competitions, I think it’s essential to explicitly warn participants to finalize their submissions early, since from my experience, heavy server load and timeouts near the deadline happen quite frequently. The rule should clearly state that deadlines will not be extended and that it’s each participant’s responsibility to complete submissions in advance.
\nI’d really appreciate it if @ryanholbrook could kindly consider this for future competitions.
", "votes": 4, "replies": [] } ] } ] }, { "id": 3301260, "author_name": "Sergio Alvarez", "author_url": "", "post_date": "2025-10-13T01:21:21.703000", "content": "@ryanholbrook, any confirmation that the servers are running slower? Will this continue until the end of the competition? We tried to verify by submitting an old notebook that previously finished in ~10 hours, but now it’s timing out.
", "votes": 5, "replies": [ { "id": 3301262, "author_name": "Ángel Jacinto Sánchez Ruiz", "author_url": "", "post_date": "2025-10-13T01:23:34.713000", "content": "I've got 3 timeouts on previously finished submissions, just weights changed… Sad, I've discovered a bug that could explain my huge CV to test score drop. Reducing ensemble charge to minium at the moment. I hope I can at least check my bug fix.
\nEDIT: Would be nice if confirmed then timeouts would be rerun just for fairness.
", "votes": 5, "replies": [] } ] }, { "id": 3301990, "author_name": "Robin Smits", "author_url": "", "post_date": "2025-10-14T17:36:25.333000", "content": "T.b.h.
\nI'am not sure what is worse…the submission issues about which kagglers complain already for at least 2 days…
\nOr the seemingly total lack of response from the Kaggle hosts ….
\n@inversion @ryanholbrook @sohier anybody?
", "votes": 6, "replies": [] }, { "id": 3301578, "author_name": "AndresMartinezMora", "author_url": "", "post_date": "2025-10-13T17:04:17.937000", "content": "Are there any updates regarding this?
", "votes": 3, "replies": [] }, { "id": 3301324, "author_name": "Stefan Denner", "author_url": "", "post_date": "2025-10-13T06:29:55.540000", "content": "Hi, we are facing the same issue. Submissions that previously took 8–9 hours are now running into timeouts. This is very unfortunate.
", "votes": 3, "replies": [ { "id": 3301343, "author_name": "Tom", "author_url": "", "post_date": "2025-10-13T07:18:02.090000", "content": "@st3v3d It seems this competition ended two days early.
", "votes": 2, "replies": [ { "id": 3301420, "author_name": "Stefan Denner", "author_url": "", "post_date": "2025-10-13T10:39:23.483000", "content": "That would be such a pity🥲
", "votes": 2, "replies": [] } ] } ] }, { "id": 3301548, "author_name": "Robin Smits", "author_url": "", "post_date": "2025-10-13T15:52:11.427000", "content": "Same here…very slow…
", "votes": 4, "replies": [] }, { "id": 3301277, "author_name": "Abhishek Jelloji", "author_url": "", "post_date": "2025-10-13T02:55:20.393000", "content": "@ryanholbrook anything on this? Will submissions be rerun or any timeline changes @evancalabrese. Thank you for your info.
", "votes": 4, "replies": [] }, { "id": 3301019, "author_name": "Eddie", "author_url": "", "post_date": "2025-10-12T09:14:24.623000", "content": "Seems so. 2 days, submissions took around 2-3 hours. Made 2 submissions last night, one's been running for ~ 7 hours.
", "votes": 4, "replies": [ { "id": 3301072, "author_name": "Ángel Jacinto Sánchez Ruiz", "author_url": "", "post_date": "2025-10-12T12:10:49.493000", "content": "Is known that submission time is ofuscated. It will show you randomly larger execution times.
", "votes": 1, "replies": [] } ] }, { "id": 3302080, "author_name": "Tom", "author_url": "", "post_date": "2025-10-15T01:00:34.820000", "content": "I want to share that in the last minute before deadline, one 21hr sub finished (best result), and I select it. One of our submission is still running now.
\n
Yeah, we are facing the same issue.
", "votes": 1, "replies": [] }, { "id": 3301762, "author_name": "MD The Dramatist", "author_url": "", "post_date": "2025-10-14T06:59:21.870000", "content": "Don't you think it's because we are ALL trying to do it at the same time, last minute panic and all that jazz… ?
", "votes": -6, "replies": [ { "id": 3301788, "author_name": "AndresMartinezMora", "author_url": "", "post_date": "2025-10-14T08:31:11.473000", "content": "Yes, of course, we are just trying to see if there is any feasible solution to this
", "votes": 2, "replies": [] } ] }, { "id": 3302332, "author_name": "AC", "author_url": "", "post_date": "2025-10-15T15:29:18.930000", "content": "Hi Kaggle team,
\nI submitted my notebooks before the deadline, but they kept running and I only received the scores after the deadline. These were my best submissions, and I couldn’t select them in time. I believe the same issue affected many participants. Had we received the scores before, we'd have selected those. Given how rankings are clustered around 0.69, it’s likely others also made improvements but couldn’t select their final submissions due to issues beyond their control. Would it be possible to consider such submissions internally so no one is unfairly penalized for server delays?
", "votes": 0, "replies": [] }, { "id": 3301922, "author_name": "AC", "author_url": "", "post_date": "2025-10-14T14:42:49.070000", "content": "Oh man, it's been running for last 11 hours. I'd to run some final experiments, looks like all came down to 0. Shouldn't have waited for the last day. 🙇♀️
", "votes": 0, "replies": [] }, { "id": 3301821, "author_name": "Sheng-Kai Ku", "author_url": "", "post_date": "2025-10-14T10:50:23.493000", "content": "What if the deadline has passed but my submission is still scoring? Will it still count? It's a shame to say that our submission wasn't successful at any time, and our Final version is still in training.😭
", "votes": 0, "replies": [ { "id": 3301823, "author_name": "AndresMartinezMora", "author_url": "", "post_date": "2025-10-14T10:57:29.540000", "content": "As far as I know the deadline is sharp, since the private leaderboard is published just straightaway after the deadline. If you want to do some final submission that takes almost the limit of 12h, I would recommend to do it already now. It is quite unfortunate…
", "votes": 0, "replies": [ { "id": 3301830, "author_name": "Sheng-Kai Ku", "author_url": "", "post_date": "2025-10-14T11:21:54.077000", "content": "OK, thanks for your response. I wil try hard🤧
", "votes": 0, "replies": [] } ] }, { "id": 3301846, "author_name": "Mao", "author_url": "", "post_date": "2025-10-14T11:46:44.603000", "content": "I tried that once, your late submission would not count for public leaderboard. Because you can select two notebook to get scored and you can’t select the ones scoring. Plus, if you let the system choose the best scoring one, the choices are finalized before final leaderboard show up.
", "votes": 2, "replies": [ { "id": 3301968, "author_name": "Sheng-Kai Ku", "author_url": "", "post_date": "2025-10-14T16:54:05.517000", "content": "Thanks for your remind. But we just knew that our submit was failed. We got a Kaggle Error again and still don't know why, don't know how to fix. I think it's over. I'm glad to join this competition. Unlucky but learned a lot.🥲
", "votes": 0, "replies": [] } ] } ] }, { "id": 3301464, "author_name": "Ángel Jacinto Sánchez Ruiz", "author_url": "", "post_date": "2025-10-13T12:43:52.550000", "content": "I don't remember where but I'm pretty sure I've read execution time would be ofuscated by show randomly larger executions times. I assume that mechanic never would produce time out by itself…
", "votes": 0, "replies": [] }, { "id": 3301074, "author_name": "Tom", "author_url": "", "post_date": "2025-10-12T12:21:56.903000", "content": "I just don't know, why same submission finished within 11hr yesterday but needs to run nearly 12hr today.
", "votes": 0, "replies": [ { "id": 3301236, "author_name": "Ian Pan", "author_url": "", "post_date": "2025-10-12T21:50:43.837000", "content": "Towards the end of competitions, sometimes the submissions run slower because more people are submitting and this probably slows down the servers.
", "votes": 4, "replies": [ { "id": 3301239, "author_name": "Tom", "author_url": "", "post_date": "2025-10-12T22:22:07.460000", "content": "@vaillantali queue requires a day
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3300873": "During the last 24 hours, our submissions took less time than before on the sample test set, but all of them timed out on the hidden test set. Has the same issue occurred again?", "3301349": "Made a sub yesterday with 3 folds which timed out, same with 5 folds ran ok before... @ryanholbrook\n\nIs there something that can be done to prevent unfair random deadline ?", "3302014": "Hi everyone,\n\nWe're investigating the reports of slowdowns in submission times. There are (potentially) two separate issues at play here.\n\nThe first is a delay in submission start times due to GPU queueing. During times of high demand, we can run out of GPU capacity. Submissions will then enter a queue until the next GPU is available. This delay doesn't count against the overall submission time limit and wouldn't itself be a cause of submission failures.\n\nThe second possible problem is submissions themselves running more slowly than they have in the past. We haven't been able to determine whether this is actually happening in a systematic way due to a fault in our evaluation backend (because of a problem with the machines running the submissions, or a database issue, say), or whether, when it happens, it's due instead to some kind of variance introduced by stochastic submission code. It's not clear that, at least on average, submissions are running more slowly, so the problem is difficult to diagnose.\n\nWe will continue to investigate the issue, and if it proves that there is a fault in Kaggle's evaluation system, we will ensure that the affected submissions are rerun.", "3301260": "@ryanholbrook, any confirmation that the servers are running slower? Will this continue until the end of the competition? We tried to verify by submitting an old notebook that previously finished in ~10 hours, but now it’s timing out.", "3301990": "T.b.h.\n\nI'am not sure what is worse...the submission issues about which kagglers complain already for at least 2 days...\n\nOr the seemingly total lack of response from the Kaggle hosts ....\n\n@inversion @ryanholbrook @sohier anybody?", "3301578": "Are there any updates regarding this?", "3301324": "Hi, we are facing the same issue. Submissions that previously took 8–9 hours are now running into timeouts. This is very unfortunate.", "3301548": "Same here...very slow...", "3301277": "@ryanholbrook anything on this? Will submissions be rerun or any timeline changes @evancalabrese. Thank you for your info.", "3301019": "Seems so. 2 days, submissions took around 2-3 hours. Made 2 submissions last night, one's been running for ~ 7 hours.", "3302080": "I want to share that in the last minute before deadline, one 21hr sub finished (best result), and I select it. One of our submission is still running now.\n\n", "3301504": "Yeah, we are facing the same issue.", "3301762": "Don't you think it's because we are ALL trying to do it at the same time, last minute panic and all that jazz... ?", "3302332": "Hi Kaggle team,\n\nI submitted my notebooks before the deadline, but they kept running and I only received the scores after the deadline. These were my best submissions, and I couldn’t select them in time. I believe the same issue affected many participants. Had we received the scores before, we'd have selected those. Given how rankings are clustered around 0.69, it’s likely others also made improvements but couldn’t select their final submissions due to issues beyond their control. Would it be possible to consider such submissions internally so no one is unfairly penalized for server delays? ", "3301922": "Oh man, it's been running for last 11 hours. I'd to run some final experiments, looks like all came down to 0. Shouldn't have waited for the last day. 🙇♀️", "3301821": "What if the deadline has passed but my submission is still scoring? Will it still count? It's a shame to say that our submission wasn't successful at any time, and our Final version is still in training.😭", "3301464": "I don't remember where but I'm pretty sure I've read execution time would be ofuscated by show randomly larger executions times. I assume that mechanic never would produce time out by itself...", "3301074": "I just don't know, why same submission finished within 11hr yesterday but needs to run nearly 12hr today." } }