I attempted to segment patches on WSI images, extract features based on these patches using a pre-trained feature extraction model and classify samples using a classification model based on the extracted features. However, due to the need to temporarily store segmented patches and extracted features in the output folder, this method consumes a lot of memory during the patch segmentation and feature extraction steps. When applying this approach to test data, I encountered 'Notebook Out of Memory' and 'Notebook Timeout' issues. Has anyone tackled similar error with this approach? How can I resolve these two issues?
", "messages": [ { "id": 2534308, "postDate": "2023-11-22T14:37:36.950Z", "content": "I attempted to segment patches on WSI images, extract features based on these patches using a pre-trained feature extraction model and classify samples using a classification model based on the extracted features. However, due to the need to temporarily store segmented patches and extracted features in the output folder, this method consumes a lot of memory during the patch segmentation and feature extraction steps. When applying this approach to test data, I encountered 'Notebook Out of Memory' and 'Notebook Timeout' issues. Has anyone tackled similar error with this approach? How can I resolve these two issues?
", "rawMarkdown": "I attempted to segment patches on WSI images, extract features based on these patches using a pre-trained feature extraction model and classify samples using a classification model based on the extracted features. However, due to the need to temporarily store segmented patches and extracted features in the output folder, this method consumes a lot of memory during the patch segmentation and feature extraction steps. When applying this approach to test data, I encountered 'Notebook Out of Memory' and 'Notebook Timeout' issues. Has anyone tackled similar error with this approach? How can I resolve these two issues?", "votes": 2 }, { "id": 2535909, "postDate": "2023-11-23T18:30:49.427Z", "content": "If haven't already, you might try splitting up a single notebook into parts for data prep, training, and classification. See Jirka Borovec's notebooks for an example.
\nHere is the general workflow:
\nSee you on the leaderboard!
", "rawMarkdown": "If haven't already, you might try splitting up a single notebook into parts for data prep, training, and classification. See Jirka Borovec's notebooks for an example.\n\n- https://www.kaggle.com/code/jirkaborovec/cancer-subtype-eda-load-wsi-segm-mask?scriptVersionId=151304386\n- https://www.kaggle.com/code/jirkaborovec/cancer-subtype-tiles-masks-w-lightning-timm?scriptVersionId=151484971\n- https://www.kaggle.com/code/jirkaborovec/cancer-subtype-lit-torch-infer-tiles-parallel\n\nHere is the general workflow:\n\n- After the data prep notebook is done creating your patches, save a version so that the training notebook can add the data from the data prep notebook to use for training your model.\n- Add the data prep notebook to the data sources in your training notebook. Once your model is trained, save the model parameters or the entire model. Save a version of the training notebook.\n- Add the the training notebook to the data sources in your classification notebook and then load the model or just the model parameters (create the model but don't train it). Then, classify the test images in the test_images directory (don't assume that there is only one image) and save a submission CSV file. Save a version of the classification notebook.\n- Once a version of the classification notebook has been saved, submit it to the competition. The other notebooks aren't submitted.\n\nSee you on the leaderboard!" } ], "comments": [ { "id": 2535909, "author_name": "Russ Tokuyama", "author_url": "", "post_date": "2023-11-23T18:30:49.427000", "content": "If haven't already, you might try splitting up a single notebook into parts for data prep, training, and classification. See Jirka Borovec's notebooks for an example.
\nHere is the general workflow:
\nSee you on the leaderboard!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2534308": "I attempted to segment patches on WSI images, extract features based on these patches using a pre-trained feature extraction model and classify samples using a classification model based on the extracted features. However, due to the need to temporarily store segmented patches and extracted features in the output folder, this method consumes a lot of memory during the patch segmentation and feature extraction steps. When applying this approach to test data, I encountered 'Notebook Out of Memory' and 'Notebook Timeout' issues. Has anyone tackled similar error with this approach? How can I resolve these two issues?", "2535909": "If haven't already, you might try splitting up a single notebook into parts for data prep, training, and classification. See Jirka Borovec's notebooks for an example.\n\n- https://www.kaggle.com/code/jirkaborovec/cancer-subtype-eda-load-wsi-segm-mask?scriptVersionId=151304386\n- https://www.kaggle.com/code/jirkaborovec/cancer-subtype-tiles-masks-w-lightning-timm?scriptVersionId=151484971\n- https://www.kaggle.com/code/jirkaborovec/cancer-subtype-lit-torch-infer-tiles-parallel\n\nHere is the general workflow:\n\n- After the data prep notebook is done creating your patches, save a version so that the training notebook can add the data from the data prep notebook to use for training your model.\n- Add the data prep notebook to the data sources in your training notebook. Once your model is trained, save the model parameters or the entire model. Save a version of the training notebook.\n- Add the the training notebook to the data sources in your classification notebook and then load the model or just the model parameters (create the model but don't train it). Then, classify the test images in the test_images directory (don't assume that there is only one image) and save a submission CSV file. Save a version of the classification notebook.\n- Once a version of the classification notebook has been saved, submit it to the competition. The other notebooks aren't submitted.\n\nSee you on the leaderboard!" } }