Hi,
\nThroughout the challenge we were struggling a lot with the 12h time limit. Apart from the problems now towards the end with the high server load, the main issue was the strict predict() implementation in the submission notebook.
\nThis interface enforces sequential processing of the entire series end-to-end.
We wanted to split the test set across two GPUs (each GPU processes a disjoint subset of series), but the required predict() signature and its invocation pattern prevented this.
\nWe eventually used both GPUs by splitting patches across devices inside a single process, but this was awkward, added a lot of boilerplate, and still blocked parallel data loading / preprocessing / model inference at the series level.
\nPractically, this design reduced our achievable throughput and contributed to timeouts for otherwise working solutions.
\nI may be missing the rationale here, but I don’t understand why the interface mandates a single predict() that streams series one-by-one. What’s the advantage of enforcing that signature instead of giving participants freedom to orchestrate test-time execution (e.g., multi-process, multi-GPU, series sharding), as long as the outputs and timing/resource constraints are respected?
\nBest,
\nStefan
Hi,
\nThroughout the challenge we were struggling a lot with the 12h time limit. Apart from the problems now towards the end with the high server load, the main issue was the strict predict() implementation in the submission notebook.
\nThis interface enforces sequential processing of the entire series end-to-end.
We wanted to split the test set across two GPUs (each GPU processes a disjoint subset of series), but the required predict() signature and its invocation pattern prevented this.
\nWe eventually used both GPUs by splitting patches across devices inside a single process, but this was awkward, added a lot of boilerplate, and still blocked parallel data loading / preprocessing / model inference at the series level.
\nPractically, this design reduced our achievable throughput and contributed to timeouts for otherwise working solutions.
\nI may be missing the rationale here, but I don’t understand why the interface mandates a single predict() that streams series one-by-one. What’s the advantage of enforcing that signature instead of giving participants freedom to orchestrate test-time execution (e.g., multi-process, multi-GPU, series sharding), as long as the outputs and timing/resource constraints are respected?
\nBest,
\nStefan
Hi @st3v3d,
\nFor most competitions, we provide the entire test set at once so that you are free to orchestrate the test-time execution as you describe. For this competition, there were strong correlations among groups of series that, if fully exploited, would have made the task much easier. The desire however was for each series to be treated independently, motivated both by the application and also the need for a larger effective sample size.
\nI agree though that with a dual GPU setup enabling more parallelism would have been helpful. It's possible we could have streamed series in batches of four, say, and still achieved our goals. I appreciate the feedback, and we'll consider it in the design of future competitions.
\nAnd congrats on the gold!
", "rawMarkdown": "Hi @st3v3d,\n\nFor most competitions, we provide the entire test set at once so that you are free to orchestrate the test-time execution as you describe. For this competition, there were strong correlations among groups of series that, if fully exploited, would have made the task much easier. The desire however was for each series to be treated independently, motivated both by the application and also the need for a larger effective sample size.\n\nI agree though that with a dual GPU setup enabling more parallelism would have been helpful. It's possible we could have streamed series in batches of four, say, and still achieved our goals. I appreciate the feedback, and we'll consider it in the design of future competitions.\n\nAnd congrats on the gold!", "votes": 2, "replies": [ { "id": 3302857, "postDate": "2025-10-16T15:47:35.213Z", "content": "Thanks so much for the detailed explanation. Makes perfect sense! And indeed, batching would have helped us already quite a bit :)
", "rawMarkdown": "Thanks so much for the detailed explanation. Makes perfect sense! And indeed, batching would have helped us already quite a bit :) " } ] }, { "id": 3301903, "postDate": "2025-10-14T13:14:58.460Z", "content": "I second this. It's quite the bottleneck on how submissions are processed, especially when there's multiple models that require different processing for inputs.
", "rawMarkdown": "I second this. It's quite the bottleneck on how submissions are processed, especially when there's multiple models that require different processing for inputs. ", "votes": 2 } ], "comments": [ { "id": 3302793, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2025-10-16T14:18:14.733000", "content": "Hi @st3v3d,
\nFor most competitions, we provide the entire test set at once so that you are free to orchestrate the test-time execution as you describe. For this competition, there were strong correlations among groups of series that, if fully exploited, would have made the task much easier. The desire however was for each series to be treated independently, motivated both by the application and also the need for a larger effective sample size.
\nI agree though that with a dual GPU setup enabling more parallelism would have been helpful. It's possible we could have streamed series in batches of four, say, and still achieved our goals. I appreciate the feedback, and we'll consider it in the design of future competitions.
\nAnd congrats on the gold!
", "votes": 2, "replies": [ { "id": 3302857, "author_name": "Stefan Denner", "author_url": "", "post_date": "2025-10-16T15:47:35.213000", "content": "Thanks so much for the detailed explanation. Makes perfect sense! And indeed, batching would have helped us already quite a bit :)
", "votes": 0, "replies": [] } ] }, { "id": 3301903, "author_name": "Satwik", "author_url": "", "post_date": "2025-10-14T13:14:58.460000", "content": "I second this. It's quite the bottleneck on how submissions are processed, especially when there's multiple models that require different processing for inputs.
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "3301897": "Hi, \nThroughout the challenge we were struggling a lot with the 12h time limit. Apart from the problems now towards the end with the high server load, the main issue was the strict predict() implementation in the submission notebook.\nThis interface enforces sequential processing of the entire series end-to-end.\n\nWe wanted to split the test set across two GPUs (each GPU processes a disjoint subset of series), but the required predict() signature and its invocation pattern prevented this.\n\nWe eventually used both GPUs by splitting patches across devices inside a single process, but this was awkward, added a lot of boilerplate, and still blocked parallel data loading / preprocessing / model inference at the series level.\n\nPractically, this design reduced our achievable throughput and contributed to timeouts for otherwise working solutions.\n\nI may be missing the rationale here, but I don’t understand why the interface mandates a single predict() that streams series one-by-one. What’s the advantage of enforcing that signature instead of giving participants freedom to orchestrate test-time execution (e.g., multi-process, multi-GPU, series sharding), as long as the outputs and timing/resource constraints are respected?\n\nBest, \nStefan", "3302793": "Hi @st3v3d,\n\nFor most competitions, we provide the entire test set at once so that you are free to orchestrate the test-time execution as you describe. For this competition, there were strong correlations among groups of series that, if fully exploited, would have made the task much easier. The desire however was for each series to be treated independently, motivated both by the application and also the need for a larger effective sample size.\n\nI agree though that with a dual GPU setup enabling more parallelism would have been helpful. It's possible we could have streamed series in batches of four, say, and still achieved our goals. I appreciate the feedback, and we'll consider it in the design of future competitions.\n\nAnd congrats on the gold!", "3301903": "I second this. It's quite the bottleneck on how submissions are processed, especially when there's multiple models that require different processing for inputs. " } }