{ "id": 451675, "title": "what will be the approach while submitting notebook? Because when i submit notebook, i am always faced the problem Notebook Out Of Memory , Can Anyone help?", "url": "/competitions/UBC-OCEAN/discussion/451675", "author_name": "sunil thite", "post_date": "2023-10-30T03:45:00.910000", "votes": 3, "comment_count": 7, "views": 0, "content": "

I had try with small batch size and taking smaller patches but i faced the notebook out of memory error while submitting.
\nI had try this parameter :
\nX_train = (45000 , 224, 224, 3)
\nY_train = (45000 ,)
\nBatch Size = 16, 8, 2, 1
\nImage Size = (224x224)
\nPlease give me solution for this problem ?

", "messages": [ { "id": 2504610, "postDate": "2023-10-30T03:45:00.910Z", "content": "

I had try with small batch size and taking smaller patches but i faced the notebook out of memory error while submitting.
\nI had try this parameter :
\nX_train = (45000 , 224, 224, 3)
\nY_train = (45000 ,)
\nBatch Size = 16, 8, 2, 1
\nImage Size = (224x224)
\nPlease give me solution for this problem ?

", "rawMarkdown": " I had try with small batch size and taking smaller patches but i faced the notebook out of memory error while submitting.\nI had try this parameter :\nX_train = (45000 , 224, 224, 3)\nY_train = (45000 ,)\nBatch Size = 16, 8, 2, 1\nImage Size = (224x224)\n\nPlease give me solution for this problem ?", "votes": 3 }, { "id": 2505378, "postDate": "2023-10-30T15:06:26.543Z", "content": "

I've also encountered this many times. Initially, I faced this issue with 8 samples in a single thread. Then, I switched from using Image.open() to cv2.imread(), and now I can process 32 samples in a single thread using 4 threads.
\nIt's particularly important to be cautious and not attempt to download the dataset and run in evel() mode directly.

", "rawMarkdown": "\nI've also encountered this many times. Initially, I faced this issue with 8 samples in a single thread. Then, I switched from using Image.open() to cv2.imread(), and now I can process 32 samples in a single thread using 4 threads.\nIt's particularly important to be cautious and not attempt to download the dataset and run in evel() mode directly.", "votes": 1 }, { "id": 2504745, "postDate": "2023-10-30T06:05:35.183Z", "content": "

I will suggest you to train elsewhere and infer on the submission kernel. This has 3 advantages-

\n
    \n
  1. Your submission pipeline will be clean and you will be able to keep track of your submitted work against the appropriate trained models well
    2. \n
  2. You are less likely to run out of memory as you will be using a small inferencing component only
    3. \n
  3. Your GPU quota will be better managed through the week.
    4. \n
\n

Best wishes @sunilthite

", "rawMarkdown": "I will suggest you to train elsewhere and infer on the submission kernel. This has 3 advantages-\n1. Your submission pipeline will be clean and you will be able to keep track of your submitted work against the appropriate trained models well\n2. You are less likely to run out of memory as you will be using a small inferencing component only\n3. Your GPU quota will be better managed through the week. \n\nBest wishes @sunilthite ", "votes": 2, "replies": [ { "id": 2504860, "postDate": "2023-10-30T07:37:35.603Z", "content": "

Ok I will try this , Thanks for sharing valuable things.

", "rawMarkdown": "Ok I will try this , Thanks for sharing valuable things.", "votes": 2 } ] }, { "id": 2504708, "postDate": "2023-10-30T05:43:54.903Z", "content": "

Don't try to save the tiles to the disk. Do the inferencing inline and cache only the model predictions.

", "rawMarkdown": "Don't try to save the tiles to the disk. Do the inferencing inline and cache only the model predictions.", "votes": 2, "replies": [ { "id": 2505119, "postDate": "2023-10-30T11:21:01.943Z", "content": "

Thank you so much for your valuable time.

", "rawMarkdown": "Thank you so much for your valuable time.", "votes": 2 } ] }, { "id": 2517098, "postDate": "2023-11-08T08:13:45.507Z", "content": "

Hello everyone, I'm having this exact problem and because my notebook can't get submitted. Please help solve this

", "rawMarkdown": "Hello everyone, I'm having this exact problem and because my notebook can't get submitted. Please help solve this", "replies": [ { "id": 2518111, "postDate": "2023-11-09T04:09:43.203Z", "content": "

Have you split your notebook into a training part and inference part for submission? If not, you should try that.
\nSee https://www.kaggle.com/competitions/UBC-OCEAN/discussion/445420#2516523 and https://www.kaggle.com/competitions/UBC-OCEAN/discussion/453584#2515631

", "rawMarkdown": "Have you split your notebook into a training part and inference part for submission? If not, you should try that.\nSee https://www.kaggle.com/competitions/UBC-OCEAN/discussion/445420#2516523 and https://www.kaggle.com/competitions/UBC-OCEAN/discussion/453584#2515631" } ] } ], "comments": [ { "id": 2505378, "author_name": "woodman718", "author_url": "", "post_date": "2023-10-30T15:06:26.543000", "content": "

I've also encountered this many times. Initially, I faced this issue with 8 samples in a single thread. Then, I switched from using Image.open() to cv2.imread(), and now I can process 32 samples in a single thread using 4 threads.
\nIt's particularly important to be cautious and not attempt to download the dataset and run in evel() mode directly.

", "votes": 1, "replies": [] }, { "id": 2504745, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2023-10-30T06:05:35.183000", "content": "

I will suggest you to train elsewhere and infer on the submission kernel. This has 3 advantages-

\n
    \n
  1. Your submission pipeline will be clean and you will be able to keep track of your submitted work against the appropriate trained models well
    2. \n
  2. You are less likely to run out of memory as you will be using a small inferencing component only
    3. \n
  3. Your GPU quota will be better managed through the week.
    4. \n
\n

Best wishes @sunilthite

", "votes": 2, "replies": [ { "id": 2504860, "author_name": "sunil thite", "author_url": "", "post_date": "2023-10-30T07:37:35.603000", "content": "

Ok I will try this , Thanks for sharing valuable things.

", "votes": 2, "replies": [] } ] }, { "id": 2504708, "author_name": "KarthiAru", "author_url": "", "post_date": "2023-10-30T05:43:54.903000", "content": "

Don't try to save the tiles to the disk. Do the inferencing inline and cache only the model predictions.

", "votes": 2, "replies": [ { "id": 2505119, "author_name": "sunil thite", "author_url": "", "post_date": "2023-10-30T11:21:01.943000", "content": "

Thank you so much for your valuable time.

", "votes": 2, "replies": [] } ] }, { "id": 2517098, "author_name": "Kamal Moha", "author_url": "", "post_date": "2023-11-08T08:13:45.507000", "content": "

Hello everyone, I'm having this exact problem and because my notebook can't get submitted. Please help solve this

", "votes": 0, "replies": [ { "id": 2518111, "author_name": "Russ Tokuyama", "author_url": "", "post_date": "2023-11-09T04:09:43.203000", "content": "

Have you split your notebook into a training part and inference part for submission? If not, you should try that.
\nSee https://www.kaggle.com/competitions/UBC-OCEAN/discussion/445420#2516523 and https://www.kaggle.com/competitions/UBC-OCEAN/discussion/453584#2515631

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2504610": " I had try with small batch size and taking smaller patches but i faced the notebook out of memory error while submitting.\nI had try this parameter :\nX_train = (45000 , 224, 224, 3)\nY_train = (45000 ,)\nBatch Size = 16, 8, 2, 1\nImage Size = (224x224)\n\nPlease give me solution for this problem ?", "2505378": "\nI've also encountered this many times. Initially, I faced this issue with 8 samples in a single thread. Then, I switched from using Image.open() to cv2.imread(), and now I can process 32 samples in a single thread using 4 threads.\nIt's particularly important to be cautious and not attempt to download the dataset and run in evel() mode directly.", "2504745": "I will suggest you to train elsewhere and infer on the submission kernel. This has 3 advantages-\n1. Your submission pipeline will be clean and you will be able to keep track of your submitted work against the appropriate trained models well\n2. You are less likely to run out of memory as you will be using a small inferencing component only\n3. Your GPU quota will be better managed through the week. \n\nBest wishes @sunilthite ", "2504708": "Don't try to save the tiles to the disk. Do the inferencing inline and cache only the model predictions.", "2517098": "Hello everyone, I'm having this exact problem and because my notebook can't get submitted. Please help solve this" } }