Training is taking too much time. It takes 1 hour to get a 15% of one epoch and got 13% accuracy and train loss is 0.2. I don't convert the image into png format. I use dicom images.
\nI don't have external GPU. I use GPU from Kaggle.
", "messages": [ { "id": 2053424, "postDate": "2022-12-03T09:00:48.847Z", "content": "Training is taking too much time. It takes 1 hour to get a 15% of one epoch and got 13% accuracy and train loss is 0.2. I don't convert the image into png format. I use dicom images.
\nI don't have external GPU. I use GPU from Kaggle.
", "rawMarkdown": "Training is taking too much time. It takes 1 hour to get a 15% of one epoch and got 13% accuracy and train loss is 0.2. I don't convert the image into png format. I use dicom images.\n\nI don't have external GPU. I use GPU from Kaggle.", "votes": 5 }, { "id": 2053480, "postDate": "2022-12-03T10:19:44.317Z", "content": "Now is a beggining of cometition - time to establish fast experimenting pipeline. My current attitude:
\nI do everything to speed up experimenting. Then (after strong pipeline discovery) will be time to train on multi gpu configuration (Lambda Cloud, Kaggle 2xGPU … or my personal machine).
", "rawMarkdown": "Now is a beggining of cometition - time to establish fast experimenting pipeline. My current attitude:\n- make images smaller (256x256)\n- chose lighter/smaller model \n- decrease amout of data (now I experimenting using only 500 images)\n- will use Colab Pro+ and A100 to increase batch size \n\nI do everything to speed up experimenting. Then (after strong pipeline discovery) will be time to train on multi gpu configuration (Lambda Cloud, Kaggle 2xGPU ... or my personal machine).", "votes": 3, "replies": [ { "id": 2053484, "postDate": "2022-12-03T10:27:01.140Z", "content": "Thanks a lot.
", "rawMarkdown": "Thanks a lot." } ] }, { "id": 2053676, "postDate": "2022-12-03T13:41:58.533Z", "content": "Plus you can train with fp16, that should speed up your training quite a bit and also help you train with larger batch sizes!
", "rawMarkdown": "Plus you can train with fp16, that should speed up your training quite a bit and also help you train with larger batch sizes!", "votes": 1 }, { "id": 2053675, "postDate": "2022-12-03T13:41:13.810Z", "content": "Based on your numbers it seems that you are training on DICOM files and are converting them on the fly.
\nThe conversion is computationally extremely expensive and should be something offloaded to before train. I would suggest you change your pipeline to train on one of the datasets that have been made publicly available (I shared one myself with resolutions of 256px, 512px, 768px and 1024px so there is plenty of sizes to chose from!)
", "rawMarkdown": "Based on your numbers it seems that you are training on DICOM files and are converting them on the fly.\n\nThe conversion is computationally extremely expensive and should be something offloaded to before train. I would suggest you change your pipeline to train on one of the datasets that have been made publicly available (I shared one myself with resolutions of 256px, 512px, 768px and 1024px so there is plenty of sizes to chose from!)", "votes": 1, "replies": [ { "id": 2053699, "postDate": "2022-12-03T14:22:04.593Z", "content": "love it. I'll try a better training pipeline.
", "rawMarkdown": "love it. I'll try a better training pipeline.", "votes": 1 }, { "id": 2080860, "postDate": "2022-12-30T15:17:38.243Z", "content": "does GPU or TPU accelerate the conversion of images to png format?
", "rawMarkdown": "does GPU or TPU accelerate the conversion of images to png format?" } ] } ], "comments": [ { "id": 2053480, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-03T10:19:44.317000", "content": "Now is a beggining of cometition - time to establish fast experimenting pipeline. My current attitude:
\nI do everything to speed up experimenting. Then (after strong pipeline discovery) will be time to train on multi gpu configuration (Lambda Cloud, Kaggle 2xGPU … or my personal machine).
", "votes": 3, "replies": [ { "id": 2053484, "author_name": "Myo Min Htet(wnp)🇲🇲", "author_url": "", "post_date": "2022-12-03T10:27:01.140000", "content": "Thanks a lot.
", "votes": 0, "replies": [] } ] }, { "id": 2053676, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-03T13:41:58.533000", "content": "Plus you can train with fp16, that should speed up your training quite a bit and also help you train with larger batch sizes!
", "votes": 1, "replies": [] }, { "id": 2053675, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-03T13:41:13.810000", "content": "Based on your numbers it seems that you are training on DICOM files and are converting them on the fly.
\nThe conversion is computationally extremely expensive and should be something offloaded to before train. I would suggest you change your pipeline to train on one of the datasets that have been made publicly available (I shared one myself with resolutions of 256px, 512px, 768px and 1024px so there is plenty of sizes to chose from!)
", "votes": 1, "replies": [ { "id": 2053699, "author_name": "Myo Min Htet(wnp)🇲🇲", "author_url": "", "post_date": "2022-12-03T14:22:04.593000", "content": "love it. I'll try a better training pipeline.
", "votes": 1, "replies": [] }, { "id": 2080860, "author_name": "Mohammad Zahrawi", "author_url": "", "post_date": "2022-12-30T15:17:38.243000", "content": "does GPU or TPU accelerate the conversion of images to png format?
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2053424": "Training is taking too much time. It takes 1 hour to get a 15% of one epoch and got 13% accuracy and train loss is 0.2. I don't convert the image into png format. I use dicom images.\n\nI don't have external GPU. I use GPU from Kaggle.", "2053480": "Now is a beggining of cometition - time to establish fast experimenting pipeline. My current attitude:\n- make images smaller (256x256)\n- chose lighter/smaller model \n- decrease amout of data (now I experimenting using only 500 images)\n- will use Colab Pro+ and A100 to increase batch size \n\nI do everything to speed up experimenting. Then (after strong pipeline discovery) will be time to train on multi gpu configuration (Lambda Cloud, Kaggle 2xGPU ... or my personal machine).", "2053676": "Plus you can train with fp16, that should speed up your training quite a bit and also help you train with larger batch sizes!", "2053675": "Based on your numbers it seems that you are training on DICOM files and are converting them on the fly.\n\nThe conversion is computationally extremely expensive and should be something offloaded to before train. I would suggest you change your pipeline to train on one of the datasets that have been made publicly available (I shared one myself with resolutions of 256px, 512px, 768px and 1024px so there is plenty of sizes to chose from!)" } }