I was wondering what hardware it takes for competitive machine learning in 2020.\nObviously its a lot of hardwork, but a good machine saves a lot of days. Whats your config.\nHere is mine:\n\n8 core AMD 1920x\n48 gb RAM\nRTX 2080Ti\n
> good machine saves a lot of days
\n\nThis is true. But there are other ways of saving time.
\n\nI read somewhere (maybe CPMP said it?) that the best Kagglers don’t necessarily have the most unique ideas or the best hardware, they just have the ability to test new ideas faster. With this in mind, I went into this competition to really focus on this idea:\n- CV is everything: Until you have a CV (cross-validation) method that correlates with the Public LB, don’t waste your time tweaking your model. It’s ok if CV doesn’t match the public LB, but they should correlate.\n- Start small and stay small: I stayed on 224x224 images and EfficientNet-B0 for as long as possible (until a week before the merger deadline). This model could do 5 folds in about 12-15 hours (on 2x2080Ti’s) which allowed relatively fast idea iteration. By the end I could get 0.06X on the stage 1 public LB with this model before I went to higher resolutions & deeper architectures. If you are GPU limited use a small sample of the data and keep using that small sample for as long as possible.\n- It’s ok to use 1-fold: In the early stages when I was testing lots of ideas, I would just let my model train for 1-fold in conjunction with the public LB to get a coarse idea if something was working (effectively test ideas 5x faster). Later, when the gains were diminishing, I would let the model run for the full 5 folds to help discriminate if something brought an improvement or not.\n- Use functional programming: write small chunks of code that can be reused (see Racheal Tatman’s recent talk on the topic). Using clean and reusable code makes it faster to iterate ideas and makes a lot of your codebase usable in future competitions (which makes those competitions faster too!)\n- Use config files: I use files in the YAML format to keep parameters that I frequently changed (e.g. windowing parameters, learning rates etc.). I used to track all my parameters in spreadsheets, but that soon got very difficult. The other benefit is that if you are away for a weekend (or using cloud computing), you can line up several configurations and have them run in batch one after another using a shell script.\n- Save everything: For each model run, my code creates a timestamped folder and copies across the config file. Save all your submission files (timestamp the filenames so make it easier to track against other submissions) and out-of-fold predictions. Save all your model checkpoints and plots in this folder too. This will make things like stacking & blending easier later.\n- Keep a logbook of your ideas: I just use a Google sheet that contains my CV/LB scores and a comment of what I changed (also a to-do list of ideas to try next). Each row is the timestamp of the folder described above.\n- Don’t use ideas from the discussion boards/notebooks/GitHub without testing first: I think this one is self-explanatory, but there are many moving parts in these models (e.g. in the preprocessing) that could make an idea work for one person but not for another.
\n\nIf you follow these steps, you'll find that the time gains with your existing hardware (you already have quite a powerful machine) will be much more than if you had 2 or even 4 GPUs.
\n\nTo answer your question (I bought this machine about a year ago):\n- CPU: 16 core AMD 2950x\n- RAM: 128GB\n- GPU: 2x2080Ti\n- Storage: 2x1TB NVMe in RAID (this was a recent upgrade for all the recent image competitions)
", "rawMarkdown": "> good machine saves a lot of days\n\nThis is true. But there are other ways of saving time.\n\nI read somewhere (maybe CPMP said it?) that the best Kagglers don’t necessarily have the most unique ideas or the best hardware, they just have the ability to test new ideas faster. With this in mind, I went into this competition to really focus on this idea:\n- **CV is everything**: Until you have a CV (cross-validation) method that correlates with the Public LB, don’t waste your time tweaking your model. It’s ok if CV doesn’t match the public LB, but they should correlate.\n- **Start small and stay small**: I stayed on 224x224 images and EfficientNet-B0 for as long as possible (until a week before the merger deadline). This model could do 5 folds in about 12-15 hours (on 2x2080Ti’s) which allowed relatively fast idea iteration. By the end I could get 0.06X on the stage 1 public LB with this model before I went to higher resolutions & deeper architectures. If you are GPU limited use a small sample of the data and keep using that small sample for as long as possible.\n- **It’s ok to use 1-fold**: In the early stages when I was testing lots of ideas, I would just let my model train for 1-fold in conjunction with the public LB to get a coarse idea if something was working (effectively test ideas 5x faster). Later, when the gains were diminishing, I would let the model run for the full 5 folds to help discriminate if something brought an improvement or not.\n- **Use functional programming**: write small chunks of code that can be reused (see Racheal Tatman’s recent talk on the topic). Using clean and reusable code makes it faster to iterate ideas and makes a lot of your codebase usable in future competitions (which makes those competitions faster too!)\n- **Use config files**: I use files in the YAML format to keep parameters that I frequently changed (e.g. windowing parameters, learning rates etc.). I used to track all my parameters in spreadsheets, but that soon got very difficult. The other benefit is that if you are away for a weekend (or using cloud computing), you can line up several configurations and have them run in batch one after another using a shell script.\n- **Save everything**: For each model run, my code creates a timestamped folder and copies across the config file. Save all your submission files (timestamp the filenames so make it easier to track against other submissions) and out-of-fold predictions. Save all your model checkpoints and plots in this folder too. This will make things like stacking & blending easier later.\n- **Keep a logbook of your ideas**: I just use a Google sheet that contains my CV/LB scores and a comment of what I changed (also a to-do list of ideas to try next). Each row is the timestamp of the folder described above.\n- **Don’t use ideas from the discussion boards/notebooks/GitHub without testing first**: I think this one is self-explanatory, but there are many moving parts in these models (e.g. in the preprocessing) that could make an idea work for one person but not for another.\n\nIf you follow these steps, you'll find that the time gains with your existing hardware (you already have quite a powerful machine) will be much more than if you had 2 or even 4 GPUs.\n\nTo answer your question (I bought this machine about a year ago):\n- CPU: 16 core AMD 2950x\n- RAM: 128GB\n- GPU: 2x2080Ti\n- Storage: 2x1TB NVMe in RAID (this was a recent upgrade for all the recent image competitions)", "votes": 17, "replies": [ { "id": 666507, "postDate": "2019-11-06T06:49:24.150Z", "content": "thanks a lot for such a great post :)
", "rawMarkdown": "thanks a lot for such a great post :) ", "votes": 1 } ] }, { "id": 666170, "postDate": "2019-11-05T20:33:13.890Z", "content": "My room heater's spec is i9-7920x + 128G RAM + Titan RTX.
", "rawMarkdown": "My room heater's spec is i9-7920x + 128G RAM + Titan RTX. ", "votes": 7 }, { "id": 666317, "postDate": "2019-11-06T01:46:56.917Z", "content": "i7-6700K\n32Gb\nRTX 2080, good, but definitely need smth better for deep nets\n1Tb SSD M.2 nvme
", "rawMarkdown": "i7-6700K\n32Gb\nRTX 2080, good, but definitely need smth better for deep nets\n1Tb SSD M.2 nvme", "votes": 1 }, { "id": 666174, "postDate": "2019-11-05T20:43:55.130Z", "content": "4 core Intel Core i5\n32 GB RAM\nTitan RTX (bought it 2 weeks ago specifically for this challenge, otherwise I can't finish all CV training. Actually, still can't finish all CV training)\n1 TB SSD harddrive
", "rawMarkdown": "4 core Intel Core i5\n32 GB RAM\nTitan RTX (bought it 2 weeks ago specifically for this challenge, otherwise I can't finish all CV training. Actually, still can't finish all CV training)\n1 TB SSD harddrive", "votes": 1 }, { "id": 666167, "postDate": "2019-11-05T20:16:30.213Z", "content": "\n6 core Intel Core i7-8086K\n64 GB RAM\nRTX 1080Ti\n
I want TITAN RTX.
", "rawMarkdown": "```\n6 core Intel Core i7-8086K\n64 GB RAM\nRTX 1080Ti\n```\n\nI want TITAN RTX.", "votes": 1 }, { "id": 666127, "postDate": "2019-11-05T18:52:14.247Z", "content": "I was wondering what hardware it takes for competitive machine learning in 2020.\nObviously its a lot of hardwork, but a good machine saves a lot of days. Whats your config.\nHere is mine:\n\n8 core AMD 1920x\n48 gb RAM\nRTX 2080Ti\n
I use my XiaoMi brand laptop to train neural networks.\nThe hardware resource is:\nCPU: Intel Core i7 8750H, 6Core mobile end CPU\nRAM: 16GB DDR4 2666MHz\nGPU: Nvidia GTX 1060 6GB\nDisk: 1 TB HDD\nThe training process is slow 😑
", "rawMarkdown": "I use my XiaoMi brand laptop to train neural networks.\nThe hardware resource is:\nCPU: Intel Core i7 8750H, 6Core mobile end CPU\nRAM: 16GB DDR4 2666MHz\nGPU: Nvidia GTX 1060 6GB\nDisk: 1 TB HDD\nThe training process is slow 😑 " }, { "id": 666444, "postDate": "2019-11-06T05:26:48.300Z", "content": "@shentao @anjum48 @dmitrylarko @darraghdog @appian @wowfattie @drhabib @roguekk007 @qiaojian @sneddy
", "rawMarkdown": "@shentao @anjum48 @dmitrylarko @darraghdog @appian @wowfattie @drhabib @roguekk007 @qiaojian @sneddy " } ], "comments": [ { "id": 666487, "author_name": "datasaurus", "author_url": "", "post_date": "2019-11-06T06:12:13.143000", "content": "> good machine saves a lot of days
\n\nThis is true. But there are other ways of saving time.
\n\nI read somewhere (maybe CPMP said it?) that the best Kagglers don’t necessarily have the most unique ideas or the best hardware, they just have the ability to test new ideas faster. With this in mind, I went into this competition to really focus on this idea:\n- CV is everything: Until you have a CV (cross-validation) method that correlates with the Public LB, don’t waste your time tweaking your model. It’s ok if CV doesn’t match the public LB, but they should correlate.\n- Start small and stay small: I stayed on 224x224 images and EfficientNet-B0 for as long as possible (until a week before the merger deadline). This model could do 5 folds in about 12-15 hours (on 2x2080Ti’s) which allowed relatively fast idea iteration. By the end I could get 0.06X on the stage 1 public LB with this model before I went to higher resolutions & deeper architectures. If you are GPU limited use a small sample of the data and keep using that small sample for as long as possible.\n- It’s ok to use 1-fold: In the early stages when I was testing lots of ideas, I would just let my model train for 1-fold in conjunction with the public LB to get a coarse idea if something was working (effectively test ideas 5x faster). Later, when the gains were diminishing, I would let the model run for the full 5 folds to help discriminate if something brought an improvement or not.\n- Use functional programming: write small chunks of code that can be reused (see Racheal Tatman’s recent talk on the topic). Using clean and reusable code makes it faster to iterate ideas and makes a lot of your codebase usable in future competitions (which makes those competitions faster too!)\n- Use config files: I use files in the YAML format to keep parameters that I frequently changed (e.g. windowing parameters, learning rates etc.). I used to track all my parameters in spreadsheets, but that soon got very difficult. The other benefit is that if you are away for a weekend (or using cloud computing), you can line up several configurations and have them run in batch one after another using a shell script.\n- Save everything: For each model run, my code creates a timestamped folder and copies across the config file. Save all your submission files (timestamp the filenames so make it easier to track against other submissions) and out-of-fold predictions. Save all your model checkpoints and plots in this folder too. This will make things like stacking & blending easier later.\n- Keep a logbook of your ideas: I just use a Google sheet that contains my CV/LB scores and a comment of what I changed (also a to-do list of ideas to try next). Each row is the timestamp of the folder described above.\n- Don’t use ideas from the discussion boards/notebooks/GitHub without testing first: I think this one is self-explanatory, but there are many moving parts in these models (e.g. in the preprocessing) that could make an idea work for one person but not for another.
\n\nIf you follow these steps, you'll find that the time gains with your existing hardware (you already have quite a powerful machine) will be much more than if you had 2 or even 4 GPUs.
\n\nTo answer your question (I bought this machine about a year ago):\n- CPU: 16 core AMD 2950x\n- RAM: 128GB\n- GPU: 2x2080Ti\n- Storage: 2x1TB NVMe in RAID (this was a recent upgrade for all the recent image competitions)
", "votes": 17, "replies": [ { "id": 666507, "author_name": "Kartik Nighania", "author_url": "", "post_date": "2019-11-06T06:49:24.150000", "content": "thanks a lot for such a great post :)
", "votes": 1, "replies": [] } ] }, { "id": 666170, "author_name": "Ren", "author_url": "", "post_date": "2019-11-05T20:33:13.890000", "content": "My room heater's spec is i9-7920x + 128G RAM + Titan RTX.
", "votes": 7, "replies": [] }, { "id": 666317, "author_name": "Alexey Kotlik", "author_url": "", "post_date": "2019-11-06T01:46:56.917000", "content": "i7-6700K\n32Gb\nRTX 2080, good, but definitely need smth better for deep nets\n1Tb SSD M.2 nvme
", "votes": 1, "replies": [] }, { "id": 666174, "author_name": "Yee Ng", "author_url": "", "post_date": "2019-11-05T20:43:55.130000", "content": "4 core Intel Core i5\n32 GB RAM\nTitan RTX (bought it 2 weeks ago specifically for this challenge, otherwise I can't finish all CV training. Actually, still can't finish all CV training)\n1 TB SSD harddrive
", "votes": 1, "replies": [] }, { "id": 666167, "author_name": "kambarakun", "author_url": "", "post_date": "2019-11-05T20:16:30.213000", "content": "\n6 core Intel Core i7-8086K\n64 GB RAM\nRTX 1080Ti\n
I want TITAN RTX.
", "votes": 1, "replies": [] }, { "id": 666403, "author_name": "Zehao Zhao", "author_url": "", "post_date": "2019-11-06T04:36:46.397000", "content": "I use my XiaoMi brand laptop to train neural networks.\nThe hardware resource is:\nCPU: Intel Core i7 8750H, 6Core mobile end CPU\nRAM: 16GB DDR4 2666MHz\nGPU: Nvidia GTX 1060 6GB\nDisk: 1 TB HDD\nThe training process is slow 😑
", "votes": 0, "replies": [] }, { "id": 666444, "author_name": "Kartik Nighania", "author_url": "", "post_date": "2019-11-06T05:26:48.300000", "content": "@shentao @anjum48 @dmitrylarko @darraghdog @appian @wowfattie @drhabib @roguekk007 @qiaojian @sneddy
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "666487": "> good machine saves a lot of days\n\nThis is true. But there are other ways of saving time.\n\nI read somewhere (maybe CPMP said it?) that the best Kagglers don’t necessarily have the most unique ideas or the best hardware, they just have the ability to test new ideas faster. With this in mind, I went into this competition to really focus on this idea:\n- **CV is everything**: Until you have a CV (cross-validation) method that correlates with the Public LB, don’t waste your time tweaking your model. It’s ok if CV doesn’t match the public LB, but they should correlate.\n- **Start small and stay small**: I stayed on 224x224 images and EfficientNet-B0 for as long as possible (until a week before the merger deadline). This model could do 5 folds in about 12-15 hours (on 2x2080Ti’s) which allowed relatively fast idea iteration. By the end I could get 0.06X on the stage 1 public LB with this model before I went to higher resolutions & deeper architectures. If you are GPU limited use a small sample of the data and keep using that small sample for as long as possible.\n- **It’s ok to use 1-fold**: In the early stages when I was testing lots of ideas, I would just let my model train for 1-fold in conjunction with the public LB to get a coarse idea if something was working (effectively test ideas 5x faster). Later, when the gains were diminishing, I would let the model run for the full 5 folds to help discriminate if something brought an improvement or not.\n- **Use functional programming**: write small chunks of code that can be reused (see Racheal Tatman’s recent talk on the topic). Using clean and reusable code makes it faster to iterate ideas and makes a lot of your codebase usable in future competitions (which makes those competitions faster too!)\n- **Use config files**: I use files in the YAML format to keep parameters that I frequently changed (e.g. windowing parameters, learning rates etc.). I used to track all my parameters in spreadsheets, but that soon got very difficult. The other benefit is that if you are away for a weekend (or using cloud computing), you can line up several configurations and have them run in batch one after another using a shell script.\n- **Save everything**: For each model run, my code creates a timestamped folder and copies across the config file. Save all your submission files (timestamp the filenames so make it easier to track against other submissions) and out-of-fold predictions. Save all your model checkpoints and plots in this folder too. This will make things like stacking & blending easier later.\n- **Keep a logbook of your ideas**: I just use a Google sheet that contains my CV/LB scores and a comment of what I changed (also a to-do list of ideas to try next). Each row is the timestamp of the folder described above.\n- **Don’t use ideas from the discussion boards/notebooks/GitHub without testing first**: I think this one is self-explanatory, but there are many moving parts in these models (e.g. in the preprocessing) that could make an idea work for one person but not for another.\n\nIf you follow these steps, you'll find that the time gains with your existing hardware (you already have quite a powerful machine) will be much more than if you had 2 or even 4 GPUs.\n\nTo answer your question (I bought this machine about a year ago):\n- CPU: 16 core AMD 2950x\n- RAM: 128GB\n- GPU: 2x2080Ti\n- Storage: 2x1TB NVMe in RAID (this was a recent upgrade for all the recent image competitions)", "666170": "My room heater's spec is i9-7920x + 128G RAM + Titan RTX. ", "666317": "i7-6700K\n32Gb\nRTX 2080, good, but definitely need smth better for deep nets\n1Tb SSD M.2 nvme", "666174": "4 core Intel Core i5\n32 GB RAM\nTitan RTX (bought it 2 weeks ago specifically for this challenge, otherwise I can't finish all CV training. Actually, still can't finish all CV training)\n1 TB SSD harddrive", "666167": "```\n6 core Intel Core i7-8086K\n64 GB RAM\nRTX 1080Ti\n```\n\nI want TITAN RTX.", "666127": "I was wondering what hardware it takes for **competitive machine learning in 2020**.\nObviously its a lot of hardwork, but a good machine saves a lot of days. Whats your config.\nHere is mine:\n```\n8 core AMD 1920x\n48 gb RAM\nRTX 2080Ti\n```", "666403": "I use my XiaoMi brand laptop to train neural networks.\nThe hardware resource is:\nCPU: Intel Core i7 8750H, 6Core mobile end CPU\nRAM: 16GB DDR4 2666MHz\nGPU: Nvidia GTX 1060 6GB\nDisk: 1 TB HDD\nThe training process is slow 😑 ", "666444": "@shentao @anjum48 @dmitrylarko @darraghdog @appian @wowfattie @drhabib @roguekk007 @qiaojian @sneddy " } }