Hi all,\n i am starting this thread to discuss on what loss function to use for this challenge. i have so far used CategoricalCrossentropy but the results are not grate . i have tried various CNN feature extractors starting from densenet to resnext but no luck . the CV caps at 67% while my LB is at 60% . \ni would like to know about what am i doing wrong and i like to learn from it. kindly share some insides.
\n\nthanks
", "messages": [ { "id": 852890, "postDate": "2020-05-18T19:03:27.120Z", "content": "Crossentropy is acceptable for now. If you are in the 60% range it's probably more a data problem (what you are feeding to the model) and using better input (like the tile method in the public kernels) will give you a good jump.
\n\nFor loss the labels are more than categorical. They are ordinal. (quadratic) Weighted Kappa penalizes more big differences in rating than small ones (contrary to say accuracy for a typical categorical problem). Crossentropy penalizes mistakes in classification the same way which therefore make it well correlated with a metric like accuracy. On the other hand Weighted Kappa is often used with regression and (root)squared error because of the better correlation between the two. The trick though is that you will have to use a threshold optimizer (if not a simple round operation). For example the regression will give you 3.56 you need now a threshold to say whether this is a 3 or a 4.
\n\nThat being said I only got a small improvement going from classification to regression so if you are below the 80 range there are more important things to fix first.
", "rawMarkdown": "Crossentropy is acceptable for now. If you are in the 60% range it's probably more a data problem (what you are feeding to the model) and using better input (like the tile method in the public kernels) will give you a good jump.\n\nFor loss the labels are more than categorical. They are ordinal. (quadratic) Weighted Kappa penalizes more big differences in rating than small ones (contrary to say accuracy for a typical categorical problem). Crossentropy penalizes mistakes in classification the same way which therefore make it well correlated with a metric like accuracy. On the other hand Weighted Kappa is often used with regression and (root)squared error because of the better correlation between the two. The trick though is that you will have to use a threshold optimizer (if not a simple round operation). For example the regression will give you 3.56 you need now a threshold to say whether this is a 3 or a 4.\n\nThat being said I only got a small improvement going from classification to regression so if you are below the 80 range there are more important things to fix first.", "votes": 4, "replies": [ { "id": 854158, "postDate": "2020-05-19T19:55:57.383Z", "content": "Thanks for sharing info about regression and RMS. my model currently is trained on tile based input from lower resolution. do you think if i move to higher resolution will it help?. i am also adding augmentation. \nbasically i glue the tiles to form a single image where i randomly augment 30% of tile. implemented lr reduction on plateau with adamw optimizer
\n\ni am not sure what new i can do on the data side. any idea or am i missing something.
", "rawMarkdown": "Thanks for sharing info about regression and RMS. my model currently is trained on tile based input from lower resolution. do you think if i move to higher resolution will it help?. i am also adding augmentation. \nbasically i glue the tiles to form a single image where i randomly augment 30% of tile. implemented lr reduction on plateau with adamw optimizer \n\ni am not sure what new i can do on the data side. any idea or am i missing something.\n" }, { "id": 854193, "postDate": "2020-05-19T20:47:33.790Z", "content": "I don't think gluing into a single image makes sense. Your model is likely using convolutions so you will have some calculations that will happen across multiple tiles trying to find a pattern there... but that pattern is 100% noise since you likely glue them in an arbitrary manner.
\n\nA CNN should only be applied onto spatially coherent data. Try applying your CNN (the backbone part of it) to individual tiles and then aggregate the results of each tile representation.
", "rawMarkdown": "I don't think gluing into a single image makes sense. Your model is likely using convolutions so you will have some calculations that will happen across multiple tiles trying to find a pattern there... but that pattern is 100% noise since you likely glue them in an arbitrary manner. \n\nA CNN should only be applied onto spatially coherent data. Try applying your CNN (the backbone part of it) to individual tiles and then aggregate the results of each tile representation.", "votes": 1 }, { "id": 854675, "postDate": "2020-05-20T08:04:34.717Z", "content": "i will try passing the patches individually and use the feature extractor for prediction. Thanks\nfor help .
", "rawMarkdown": "i will try passing the patches individually and use the feature extractor for prediction. Thanks\nfor help . " } ] }, { "id": 852882, "postDate": "2020-05-18T18:48:46.793Z", "content": "Hi all,\n i am starting this thread to discuss on what loss function to use for this challenge. i have so far used CategoricalCrossentropy but the results are not grate . i have tried various CNN feature extractors starting from densenet to resnext but no luck . the CV caps at 67% while my LB is at 60% . \ni would like to know about what am i doing wrong and i like to learn from it. kindly share some insides.
\n\nthanks
", "rawMarkdown": "Hi all,\n i am starting this thread to discuss on what loss function to use for this challenge. i have so far used CategoricalCrossentropy but the results are not grate . i have tried various CNN feature extractors starting from densenet to resnext but no luck . the CV caps at 67% while my LB is at 60% . \ni would like to know about what am i doing wrong and i like to learn from it. kindly share some insides.\n\nthanks\n", "votes": 1 } ], "comments": [ { "id": 852890, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-05-18T19:03:27.120000", "content": "Crossentropy is acceptable for now. If you are in the 60% range it's probably more a data problem (what you are feeding to the model) and using better input (like the tile method in the public kernels) will give you a good jump.
\n\nFor loss the labels are more than categorical. They are ordinal. (quadratic) Weighted Kappa penalizes more big differences in rating than small ones (contrary to say accuracy for a typical categorical problem). Crossentropy penalizes mistakes in classification the same way which therefore make it well correlated with a metric like accuracy. On the other hand Weighted Kappa is often used with regression and (root)squared error because of the better correlation between the two. The trick though is that you will have to use a threshold optimizer (if not a simple round operation). For example the regression will give you 3.56 you need now a threshold to say whether this is a 3 or a 4.
\n\nThat being said I only got a small improvement going from classification to regression so if you are below the 80 range there are more important things to fix first.
", "votes": 4, "replies": [ { "id": 854158, "author_name": "yuvaramsingh", "author_url": "", "post_date": "2020-05-19T19:55:57.383000", "content": "Thanks for sharing info about regression and RMS. my model currently is trained on tile based input from lower resolution. do you think if i move to higher resolution will it help?. i am also adding augmentation. \nbasically i glue the tiles to form a single image where i randomly augment 30% of tile. implemented lr reduction on plateau with adamw optimizer
\n\ni am not sure what new i can do on the data side. any idea or am i missing something.
", "votes": 0, "replies": [] }, { "id": 854193, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-05-19T20:47:33.790000", "content": "I don't think gluing into a single image makes sense. Your model is likely using convolutions so you will have some calculations that will happen across multiple tiles trying to find a pattern there... but that pattern is 100% noise since you likely glue them in an arbitrary manner.
\n\nA CNN should only be applied onto spatially coherent data. Try applying your CNN (the backbone part of it) to individual tiles and then aggregate the results of each tile representation.
", "votes": 1, "replies": [] }, { "id": 854675, "author_name": "yuvaramsingh", "author_url": "", "post_date": "2020-05-20T08:04:34.717000", "content": "i will try passing the patches individually and use the feature extractor for prediction. Thanks\nfor help .
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "852890": "Crossentropy is acceptable for now. If you are in the 60% range it's probably more a data problem (what you are feeding to the model) and using better input (like the tile method in the public kernels) will give you a good jump.\n\nFor loss the labels are more than categorical. They are ordinal. (quadratic) Weighted Kappa penalizes more big differences in rating than small ones (contrary to say accuracy for a typical categorical problem). Crossentropy penalizes mistakes in classification the same way which therefore make it well correlated with a metric like accuracy. On the other hand Weighted Kappa is often used with regression and (root)squared error because of the better correlation between the two. The trick though is that you will have to use a threshold optimizer (if not a simple round operation). For example the regression will give you 3.56 you need now a threshold to say whether this is a 3 or a 4.\n\nThat being said I only got a small improvement going from classification to regression so if you are below the 80 range there are more important things to fix first.", "852882": "Hi all,\n i am starting this thread to discuss on what loss function to use for this challenge. i have so far used CategoricalCrossentropy but the results are not grate . i have tried various CNN feature extractors starting from densenet to resnext but no luck . the CV caps at 67% while my LB is at 60% . \ni would like to know about what am i doing wrong and i like to learn from it. kindly share some insides.\n\nthanks\n" } }