Hi dear competitors :)\nI am currently stuck, and I need your help...
\n\nI am currently trying to train a regression model based on a EfficientNetB0 backbone. I use 12 tiles of the medium resolution images. Each of my inputs has size 224x224. \nI have added a Kappa optimizer to my code, such that I can keep track of my Kappa score. However, when I tried training it for some epochs, the loss and validation loss decreased, but my Kappa Score fluctuated around 0 (random score). I cannot quite figure out why??
\n\ni have tried the exact same settings for data preprocessing on a classification model, and that worked very well. It reach a single fold CV of 0.75 in very little time. Thus, I believe my mistake must be within either my Regression model (Maybe the SmoothL1Loss?), or with my Kappa Score calculator...
\n\nI have gone over my code several times, but couldn't find my mistake. If you don't mind giving a quick read through my code, I would be extremely grateful if you can point me out the (maybe obvious) bug/mistake I am making?
\n\nYou can find my notebook here:\nhttps://www.kaggle.com/fanconic/panda-tile-list-training-for-effnetb0-regression
\n\nI value your time for going through it a lot and appreciate any comment :)\nThanks guys.
", "messages": [ { "id": 869181, "postDate": "2020-05-31T19:50:14.410Z", "content": "Edit: after looking at it some more I see you get the following results:
\n\n```\n0.06272697101320956
\n\n[[ 0 233 345 0 0 0]\n [ 0 191 342 0 0 0]\n [ 0 87 182 0 0 0]\n [ 0 68 180 0 0 0]\n [ 0 79 171 0 0 0]\n [ 0 49 196 0 0 0]]
\n\n[0.5, 1.5, 2.5, 3.5, 4.5]\n```
\n\nI also found another one of your notebooks here: https://www.kaggle.com/fanconic/panda-training-for-effnetb0-regression
\n\nThis notebook does seem to work (outside of a bug where you had too few thresholds)
\n\nWhat is the difference between that notebook, and your current faulty one?\nIn my eyes, the only difference is the SmoothL1loss (and adding another threshold in the kappa optimizer), so my guess is that the SmoothL1loss is giving undesirable results.
\n\nYou could try debugging it by using the old notebook which still worked, run it again for a sanity check (so you know nothing else happened with data processing) , and then add the SmoothL1 loss to the old notebook to see what the difference is.
", "rawMarkdown": "**Edit:** after looking at it some more I see you get the following results:\n\n```\n0.06272697101320956 \n\n[[ 0 233 345 0 0 0]\n [ 0 191 342 0 0 0]\n [ 0 87 182 0 0 0]\n [ 0 68 180 0 0 0]\n [ 0 79 171 0 0 0]\n [ 0 49 196 0 0 0]] \n\n[0.5, 1.5, 2.5, 3.5, 4.5]\n```\n\nI also found another one of your notebooks here: https://www.kaggle.com/fanconic/panda-training-for-effnetb0-regression\n\nThis notebook does seem to work (outside of a bug where you had too few thresholds)\n\nWhat is the difference between that notebook, and your current faulty one?\nIn my eyes, the only difference is the SmoothL1loss (and adding another threshold in the kappa optimizer), so my guess is that the SmoothL1loss is giving undesirable results. \n\nYou could try debugging it by using the old notebook which still worked, run it again for a sanity check (so you know nothing else happened with data processing) , and then add the SmoothL1 loss to the old notebook to see what the difference is.", "votes": 1, "replies": [ { "id": 869221, "postDate": "2020-05-31T20:47:42.510Z", "content": "Hey @stephanovich Thank you so much for looking at it. I really appreciate that :)
\n\nExactly, that's why I am a little confused, as it seemed to learn well in the first Kernel. I only changed the data loader class, for higher resolution images. However, that class works for my classification model. Hence, I also believe that may SmoothL1Loss might not be the wisest choice. I will try to back engineer it from there.
\n\nAgain, thanks so much!
", "rawMarkdown": "Hey @stephanovich Thank you so much for looking at it. I really appreciate that :)\n\nExactly, that's why I am a little confused, as it seemed to learn well in the first Kernel. I only changed the data loader class, for higher resolution images. However, that class works for my classification model. Hence, I also believe that may SmoothL1Loss might not be the wisest choice. I will try to back engineer it from there.\n\nAgain, thanks so much!", "votes": 1 }, { "id": 869273, "postDate": "2020-05-31T22:09:38.907Z", "content": "So, I did the following things and now it's working again\n- Changed the Optimizer from RAdam to Over9000\n- Changed the loss from SmoothL1Loss to MSELossFlat\n- Changed the label format from float16 to int
\n\nI somehow believe the last point was the culprit, but I wasn't patient enough to test them all singly.\nThanks again :)
", "rawMarkdown": "So, I did the following things and now it's working again\n- Changed the Optimizer from RAdam to Over9000\n- Changed the loss from SmoothL1Loss to MSELossFlat\n- Changed the label format from float16 to int\n\nI somehow believe the last point was the culprit, but I wasn't patient enough to test them all singly.\nThanks again :)", "votes": 1 }, { "id": 869276, "postDate": "2020-05-31T22:17:25.500Z", "content": "Good to hear!
\n\nGiven the extra information that you changed the DataLoader class that might indeed have been it, but it would be strange, given that regression losses are supposed to deal with float values to start with. Maybe something strange happens in the casting? \nI'm using Tensorflow myself so I know that you have to be very careful with casting.
\n\nIn any case, I'm glad to hear you've solved it. Good luck with the rest of the competition.
", "rawMarkdown": "Good to hear!\n\nGiven the extra information that you changed the DataLoader class that might indeed have been it, but it would be strange, given that regression losses are supposed to deal with float values to start with. Maybe something strange happens in the casting? \nI'm using Tensorflow myself so I know that you have to be very careful with casting.\n\nIn any case, I'm glad to hear you've solved it. Good luck with the rest of the competition.\n\n", "votes": 1 } ] }, { "id": 868611, "postDate": "2020-05-31T11:22:17.087Z", "content": "Hi dear competitors :)\nI am currently stuck, and I need your help...
\n\nI am currently trying to train a regression model based on a EfficientNetB0 backbone. I use 12 tiles of the medium resolution images. Each of my inputs has size 224x224. \nI have added a Kappa optimizer to my code, such that I can keep track of my Kappa score. However, when I tried training it for some epochs, the loss and validation loss decreased, but my Kappa Score fluctuated around 0 (random score). I cannot quite figure out why??
\n\ni have tried the exact same settings for data preprocessing on a classification model, and that worked very well. It reach a single fold CV of 0.75 in very little time. Thus, I believe my mistake must be within either my Regression model (Maybe the SmoothL1Loss?), or with my Kappa Score calculator...
\n\nI have gone over my code several times, but couldn't find my mistake. If you don't mind giving a quick read through my code, I would be extremely grateful if you can point me out the (maybe obvious) bug/mistake I am making?
\n\nYou can find my notebook here:\nhttps://www.kaggle.com/fanconic/panda-tile-list-training-for-effnetb0-regression
\n\nI value your time for going through it a lot and appreciate any comment :)\nThanks guys.
", "rawMarkdown": "Hi dear competitors :)\nI am currently stuck, and I need your help...\n\nI am currently trying to train a regression model based on a EfficientNetB0 backbone. I use 12 tiles of the medium resolution images. Each of my inputs has size 224x224. \nI have added a Kappa optimizer to my code, such that I can keep track of my Kappa score. However, when I tried training it for some epochs, the loss and validation loss decreased, but my **Kappa Score fluctuated around 0 (random score)**. I cannot quite figure out why??\n\ni have tried the exact same settings for data preprocessing on a classification model, and that worked very well. It reach a single fold CV of 0.75 in very little time. Thus, I believe my mistake must be within either my **Regression model (Maybe the SmoothL1Loss?)**, or with my **Kappa Score calculator**...\n\nI have gone over my code several times, but couldn't find my mistake. If you don't mind giving a quick read through my code, I would be extremely grateful if you can point me out the (maybe obvious) bug/mistake I am making?\n\nYou can find my notebook here:\nhttps://www.kaggle.com/fanconic/panda-tile-list-training-for-effnetb0-regression\n\nI value your time for going through it a lot and appreciate any comment :)\nThanks guys.", "votes": 2 } ], "comments": [ { "id": 869181, "author_name": "Stephan", "author_url": "", "post_date": "2020-05-31T19:50:14.410000", "content": "Edit: after looking at it some more I see you get the following results:
\n\n```\n0.06272697101320956
\n\n[[ 0 233 345 0 0 0]\n [ 0 191 342 0 0 0]\n [ 0 87 182 0 0 0]\n [ 0 68 180 0 0 0]\n [ 0 79 171 0 0 0]\n [ 0 49 196 0 0 0]]
\n\n[0.5, 1.5, 2.5, 3.5, 4.5]\n```
\n\nI also found another one of your notebooks here: https://www.kaggle.com/fanconic/panda-training-for-effnetb0-regression
\n\nThis notebook does seem to work (outside of a bug where you had too few thresholds)
\n\nWhat is the difference between that notebook, and your current faulty one?\nIn my eyes, the only difference is the SmoothL1loss (and adding another threshold in the kappa optimizer), so my guess is that the SmoothL1loss is giving undesirable results.
\n\nYou could try debugging it by using the old notebook which still worked, run it again for a sanity check (so you know nothing else happened with data processing) , and then add the SmoothL1 loss to the old notebook to see what the difference is.
", "votes": 1, "replies": [ { "id": 869221, "author_name": "Claudio Fanconi", "author_url": "", "post_date": "2020-05-31T20:47:42.510000", "content": "Hey @stephanovich Thank you so much for looking at it. I really appreciate that :)
\n\nExactly, that's why I am a little confused, as it seemed to learn well in the first Kernel. I only changed the data loader class, for higher resolution images. However, that class works for my classification model. Hence, I also believe that may SmoothL1Loss might not be the wisest choice. I will try to back engineer it from there.
\n\nAgain, thanks so much!
", "votes": 1, "replies": [] }, { "id": 869273, "author_name": "Claudio Fanconi", "author_url": "", "post_date": "2020-05-31T22:09:38.907000", "content": "So, I did the following things and now it's working again\n- Changed the Optimizer from RAdam to Over9000\n- Changed the loss from SmoothL1Loss to MSELossFlat\n- Changed the label format from float16 to int
\n\nI somehow believe the last point was the culprit, but I wasn't patient enough to test them all singly.\nThanks again :)
", "votes": 1, "replies": [] }, { "id": 869276, "author_name": "Stephan", "author_url": "", "post_date": "2020-05-31T22:17:25.500000", "content": "Good to hear!
\n\nGiven the extra information that you changed the DataLoader class that might indeed have been it, but it would be strange, given that regression losses are supposed to deal with float values to start with. Maybe something strange happens in the casting? \nI'm using Tensorflow myself so I know that you have to be very careful with casting.
\n\nIn any case, I'm glad to hear you've solved it. Good luck with the rest of the competition.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "869181": "**Edit:** after looking at it some more I see you get the following results:\n\n```\n0.06272697101320956 \n\n[[ 0 233 345 0 0 0]\n [ 0 191 342 0 0 0]\n [ 0 87 182 0 0 0]\n [ 0 68 180 0 0 0]\n [ 0 79 171 0 0 0]\n [ 0 49 196 0 0 0]] \n\n[0.5, 1.5, 2.5, 3.5, 4.5]\n```\n\nI also found another one of your notebooks here: https://www.kaggle.com/fanconic/panda-training-for-effnetb0-regression\n\nThis notebook does seem to work (outside of a bug where you had too few thresholds)\n\nWhat is the difference between that notebook, and your current faulty one?\nIn my eyes, the only difference is the SmoothL1loss (and adding another threshold in the kappa optimizer), so my guess is that the SmoothL1loss is giving undesirable results. \n\nYou could try debugging it by using the old notebook which still worked, run it again for a sanity check (so you know nothing else happened with data processing) , and then add the SmoothL1 loss to the old notebook to see what the difference is.", "868611": "Hi dear competitors :)\nI am currently stuck, and I need your help...\n\nI am currently trying to train a regression model based on a EfficientNetB0 backbone. I use 12 tiles of the medium resolution images. Each of my inputs has size 224x224. \nI have added a Kappa optimizer to my code, such that I can keep track of my Kappa score. However, when I tried training it for some epochs, the loss and validation loss decreased, but my **Kappa Score fluctuated around 0 (random score)**. I cannot quite figure out why??\n\ni have tried the exact same settings for data preprocessing on a classification model, and that worked very well. It reach a single fold CV of 0.75 in very little time. Thus, I believe my mistake must be within either my **Regression model (Maybe the SmoothL1Loss?)**, or with my **Kappa Score calculator**...\n\nI have gone over my code several times, but couldn't find my mistake. If you don't mind giving a quick read through my code, I would be extremely grateful if you can point me out the (maybe obvious) bug/mistake I am making?\n\nYou can find my notebook here:\nhttps://www.kaggle.com/fanconic/panda-tile-list-training-for-effnetb0-regression\n\nI value your time for going through it a lot and appreciate any comment :)\nThanks guys." } }