I saw some people already got good score on LB.
\nI think they must be figuring out how to design an appropriate architecture for this problem.
\nI wonder how much inference time you took for a submission?
\nI tried a simple baseline 3D CNN model with resnet18 backbone for submission, and it took almost 9 hours to get the result. Although I have not optimize the process, just wondering whether you also took this long for a submission.
( note : My submission used all the ct scans in test dataset)
", "messages": [ { "id": 1023823, "postDate": "2020-09-23T13:31:35.413Z", "content": "I saw some people already got good score on LB.
\nI think they must be figuring out how to design an appropriate architecture for this problem.
\nI wonder how much inference time you took for a submission?
\nI tried a simple baseline 3D CNN model with resnet18 backbone for submission, and it took almost 9 hours to get the result. Although I have not optimize the process, just wondering whether you also took this long for a submission.
( note : My submission used all the ct scans in test dataset)
", "rawMarkdown": "I saw some people already got good score on LB. \nI think they must be figuring out how to design an appropriate architecture for this problem. \nI wonder how much inference time you took for a submission?\nI tried a simple baseline 3D CNN model with resnet18 backbone for submission, and it took almost 9 hours to get the result. Although I have not optimize the process, just wondering whether you also took this long for a submission.\n\n( note : My submission used all the ct scans in test dataset)", "votes": 9 }, { "id": 1023996, "postDate": "2020-09-23T15:18:30.700Z", "content": "An option, suggested by others, is to do the inference on the \"visible\" test data and produce a \"visible\" submission file. Then add that submission file to a committed notebook that does inference on the \"hidden\" test data. Then combine the two in that notebook for submission. [It cuts down the need to inference in the 9 hour time limit from about 343,000 files (my estimate) to about 200,000 files - file counts uncertain]
\nBased on [numbers updated per maettes comment below. A little unclear to me the actual counts]
\nSee if adjusting Batch Size helps your inference (not sure if it will).
", "rawMarkdown": "An option, suggested by others, is to do the inference on the \"visible\" test data and produce a \"visible\" submission file. Then add that submission file to a committed notebook that does inference on the \"hidden\" test data. Then combine the two in that notebook for submission. [It cuts down the need to inference in the 9 hour time limit from about 343,000 files (my estimate) to about 200,000 files - file counts uncertain]\n\n Based on [numbers updated per maettes comment below. A little unclear to me the actual counts]\n- \"visible\" patients 650\n- \"hidden\" patients 1517 (which we are guessing is in addition to the visible patients)\n- Total patients 2167?\n\nSee if adjusting Batch Size helps your inference (not sure if it will).", "votes": 6, "replies": [ { "id": 1024032, "postDate": "2020-09-23T15:47:40.097Z", "content": "Thanks a lot for the advice. This is a nice idea! I might not able to increase the batch size due to the memory limit. But I will try the method you suggested. Thanks a again for the kind help!!
", "rawMarkdown": "Thanks a lot for the advice. This is a nice idea! I might not able to increase the batch size due to the memory limit. But I will try the method you suggested. Thanks a again for the kind help!!" }, { "id": 1024050, "postDate": "2020-09-23T15:58:22.097Z", "content": "That's really a good idea.
\nI understand the public/private split rather as: 650 visible, 1517 hidden, so 2167 total test patients (the data page states: \"The training set includes 7279 studies, the public set 650, and the private set has 1517\"). This would probably mean a little less speedup by the method, but definitely still worth the effort.
thanks. I updated my comment above. I'm not 100% sure when the organizers compare datasets (70 GB vs 230 GB) if the 230 GB is in addition to the 70 GB (300 GB total) or includes the 70 GB (160 GB additional). Same with study counts.
", "rawMarkdown": "thanks. I updated my comment above. I'm not 100% sure when the organizers compare datasets (70 GB vs 230 GB) if the 230 GB is in addition to the 70 GB (300 GB total) or includes the 70 GB (160 GB additional). Same with study counts." } ] } ] }, { "id": 1024400, "postDate": "2020-09-23T20:32:43.507Z", "content": "My inference time is about 4h (using all the images - no tricks).
\nFor the visible test part I get just under 1h.
yeah, with tensorflow datagenerator & pydicom, it takes basically the same (public & private). people have been saying that vtk is faster and more efficient for reading the file. I haven't conclusively tested if vtk is any faster yet apples to apples, but in a previous pipeline using vtk, the inference time took half as long also using tensorflow datagenerator and same exact model.
", "rawMarkdown": "yeah, with tensorflow datagenerator & pydicom, it takes basically the same (public & private). people have been saying that vtk is faster and more efficient for reading the file. I haven't conclusively tested if vtk is any faster yet apples to apples, but in a previous pipeline using vtk, the inference time took half as long also using tensorflow datagenerator and same exact model.", "votes": 1 }, { "id": 1024420, "postDate": "2020-09-23T21:06:00.360Z", "content": "85% of the time is the actual model running on the GPU so vtk or pydicom don't make any real difference
", "rawMarkdown": "85% of the time is the actual model running on the GPU so vtk or pydicom don't make any real difference ", "votes": 1 }, { "id": 1024497, "postDate": "2020-09-23T23:27:40.680Z", "content": "I've narrowed down the performance difference to the pre-processing window step. The difference between vtk and pydicom appear to be negligible. It seems windowing is slowing down the pipeline by almost a factor of 2.
", "rawMarkdown": "I've narrowed down the performance difference to the pre-processing window step. The difference between vtk and pydicom appear to be negligible. It seems windowing is slowing down the pipeline by almost a factor of 2. " }, { "id": 1025314, "postDate": "2020-09-24T13:26:00.183Z", "content": "Since my inference architecture is kind of complicated, so I have not figure out how to use normal datagenerator for model prediction. I guess this is why my inference time so long. But I think you are right, since my training time is way less than inference time. Although I did give up some ct scans for training though.
", "rawMarkdown": "Since my inference architecture is kind of complicated, so I have not figure out how to use normal datagenerator for model prediction. I guess this is why my inference time so long. But I think you are right, since my training time is way less than inference time. Although I did give up some ct scans for training though." } ] }, { "id": 1024236, "postDate": "2020-09-23T17:56:27.693Z", "content": "that's really helpful thank You! upvote
", "rawMarkdown": "that's really helpful thank You! upvote" } ], "comments": [ { "id": 1023996, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-09-23T15:18:30.700000", "content": "An option, suggested by others, is to do the inference on the \"visible\" test data and produce a \"visible\" submission file. Then add that submission file to a committed notebook that does inference on the \"hidden\" test data. Then combine the two in that notebook for submission. [It cuts down the need to inference in the 9 hour time limit from about 343,000 files (my estimate) to about 200,000 files - file counts uncertain]
\nBased on [numbers updated per maettes comment below. A little unclear to me the actual counts]
\nSee if adjusting Batch Size helps your inference (not sure if it will).
", "votes": 6, "replies": [ { "id": 1024032, "author_name": "Tsai29", "author_url": "", "post_date": "2020-09-23T15:47:40.097000", "content": "Thanks a lot for the advice. This is a nice idea! I might not able to increase the batch size due to the memory limit. But I will try the method you suggested. Thanks a again for the kind help!!
", "votes": 0, "replies": [] }, { "id": 1024050, "author_name": "maettes", "author_url": "", "post_date": "2020-09-23T15:58:22.097000", "content": "That's really a good idea.
\nI understand the public/private split rather as: 650 visible, 1517 hidden, so 2167 total test patients (the data page states: \"The training set includes 7279 studies, the public set 650, and the private set has 1517\"). This would probably mean a little less speedup by the method, but definitely still worth the effort.
thanks. I updated my comment above. I'm not 100% sure when the organizers compare datasets (70 GB vs 230 GB) if the 230 GB is in addition to the 70 GB (300 GB total) or includes the 70 GB (160 GB additional). Same with study counts.
", "votes": 0, "replies": [] } ] } ] }, { "id": 1024400, "author_name": "yuval reina", "author_url": "", "post_date": "2020-09-23T20:32:43.507000", "content": "My inference time is about 4h (using all the images - no tricks).
\nFor the visible test part I get just under 1h.
yeah, with tensorflow datagenerator & pydicom, it takes basically the same (public & private). people have been saying that vtk is faster and more efficient for reading the file. I haven't conclusively tested if vtk is any faster yet apples to apples, but in a previous pipeline using vtk, the inference time took half as long also using tensorflow datagenerator and same exact model.
", "votes": 1, "replies": [] }, { "id": 1024420, "author_name": "yuval reina", "author_url": "", "post_date": "2020-09-23T21:06:00.360000", "content": "85% of the time is the actual model running on the GPU so vtk or pydicom don't make any real difference
", "votes": 1, "replies": [] }, { "id": 1024497, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-09-23T23:27:40.680000", "content": "I've narrowed down the performance difference to the pre-processing window step. The difference between vtk and pydicom appear to be negligible. It seems windowing is slowing down the pipeline by almost a factor of 2.
", "votes": 0, "replies": [] }, { "id": 1025314, "author_name": "Tsai29", "author_url": "", "post_date": "2020-09-24T13:26:00.183000", "content": "Since my inference architecture is kind of complicated, so I have not figure out how to use normal datagenerator for model prediction. I guess this is why my inference time so long. But I think you are right, since my training time is way less than inference time. Although I did give up some ct scans for training though.
", "votes": 0, "replies": [] } ] }, { "id": 1024236, "author_name": "Maciej Gronczynski", "author_url": "", "post_date": "2020-09-23T17:56:27.693000", "content": "that's really helpful thank You! upvote
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1023823": "I saw some people already got good score on LB. \nI think they must be figuring out how to design an appropriate architecture for this problem. \nI wonder how much inference time you took for a submission?\nI tried a simple baseline 3D CNN model with resnet18 backbone for submission, and it took almost 9 hours to get the result. Although I have not optimize the process, just wondering whether you also took this long for a submission.\n\n( note : My submission used all the ct scans in test dataset)", "1023996": "An option, suggested by others, is to do the inference on the \"visible\" test data and produce a \"visible\" submission file. Then add that submission file to a committed notebook that does inference on the \"hidden\" test data. Then combine the two in that notebook for submission. [It cuts down the need to inference in the 9 hour time limit from about 343,000 files (my estimate) to about 200,000 files - file counts uncertain]\n\n Based on [numbers updated per maettes comment below. A little unclear to me the actual counts]\n- \"visible\" patients 650\n- \"hidden\" patients 1517 (which we are guessing is in addition to the visible patients)\n- Total patients 2167?\n\nSee if adjusting Batch Size helps your inference (not sure if it will).", "1024400": "My inference time is about 4h (using all the images - no tricks).\nFor the visible test part I get just under 1h.", "1024236": "that's really helpful thank You! upvote" } }