Hi. I have a large mismatch between validation kappa score and kappa test score after submission. The score on my validation set is 0.83 while the score on the leaderboard is 0.58. The submitted notebook can be accessed here (https://www.kaggle.com/pmwaniki/finetune-resnet50?scriptVersionId=35233987). This is my first project using pytorch and my first kernel submission and it is likely i'm missing something.\nI have checked that training data is not spilling into validation data.
", "messages": [ { "id": 870158, "postDate": "2020-06-01T14:34:05.183Z", "content": "Hi. I have a large mismatch between validation kappa score and kappa test score after submission. The score on my validation set is 0.83 while the score on the leaderboard is 0.58. The submitted notebook can be accessed here (https://www.kaggle.com/pmwaniki/finetune-resnet50?scriptVersionId=35233987). This is my first project using pytorch and my first kernel submission and it is likely i'm missing something.\nI have checked that training data is not spilling into validation data.
", "rawMarkdown": "Hi. I have a large mismatch between validation kappa score and kappa test score after submission. The score on my validation set is 0.83 while the score on the leaderboard is 0.58. The submitted notebook can be accessed here (https://www.kaggle.com/pmwaniki/finetune-resnet50?scriptVersionId=35233987). This is my first project using pytorch and my first kernel submission and it is likely i'm missing something.\nI have checked that training data is not spilling into validation data.", "votes": 1 }, { "id": 870270, "postDate": "2020-06-01T15:42:57.720Z", "content": "One problem could be that your initial learning rate is too high, use a value between 1e-3 and 1e-4. Second your training images have a lot of blank tiles. Take a look at the following notebook for a tile setup.\nhttps://www.kaggle.com/iafoss/panda-16x128x128-tiles
", "rawMarkdown": "One problem could be that your initial learning rate is too high, use a value between 1e-3 and 1e-4. Second your training images have a lot of blank tiles. Take a look at the following notebook for a tile setup.\n[https://www.kaggle.com/iafoss/panda-16x128x128-tiles](https://www.kaggle.com/iafoss/panda-16x128x128-tiles)", "votes": 2, "replies": [ { "id": 871119, "postDate": "2020-06-02T06:46:23.953Z", "content": "Thanks Yovin, removing blank tiles might improve model performance. I'm using cyclic learning rate which ranges between 0.1 and 1e-5. At the end of each cycle, i'm taking a snapshot of the weights which are later used to create model ensembles. My biggest concern is the test score. The validation and test scores have significant differences. Could the kappa function i'm using be different from what kaggle is using for the test set?
", "rawMarkdown": "Thanks Yovin, removing blank tiles might improve model performance. I'm using cyclic learning rate which ranges between 0.1 and 1e-5. At the end of each cycle, i'm taking a snapshot of the weights which are later used to create model ensembles. My biggest concern is the test score. The validation and test scores have significant differences. Could the kappa function i'm using be different from what kaggle is using for the test set?" }, { "id": 871268, "postDate": "2020-06-02T08:23:56.150Z", "content": "The kappa function is correct, but your model is overfitting even on the validation data.
", "rawMarkdown": "The kappa function is correct, but your model is overfitting even on the validation data." }, { "id": 871333, "postDate": "2020-06-02T09:18:01.923Z", "content": "Thanks. i will add regularization
", "rawMarkdown": "Thanks. i will add regularization" } ] } ], "comments": [ { "id": 870270, "author_name": "Yovin Yahathugoda", "author_url": "", "post_date": "2020-06-01T15:42:57.720000", "content": "One problem could be that your initial learning rate is too high, use a value between 1e-3 and 1e-4. Second your training images have a lot of blank tiles. Take a look at the following notebook for a tile setup.\nhttps://www.kaggle.com/iafoss/panda-16x128x128-tiles
", "votes": 2, "replies": [ { "id": 871119, "author_name": "Mwaniki", "author_url": "", "post_date": "2020-06-02T06:46:23.953000", "content": "Thanks Yovin, removing blank tiles might improve model performance. I'm using cyclic learning rate which ranges between 0.1 and 1e-5. At the end of each cycle, i'm taking a snapshot of the weights which are later used to create model ensembles. My biggest concern is the test score. The validation and test scores have significant differences. Could the kappa function i'm using be different from what kaggle is using for the test set?
", "votes": 0, "replies": [] }, { "id": 871268, "author_name": "Yovin Yahathugoda", "author_url": "", "post_date": "2020-06-02T08:23:56.150000", "content": "The kappa function is correct, but your model is overfitting even on the validation data.
", "votes": 0, "replies": [] }, { "id": 871333, "author_name": "Mwaniki", "author_url": "", "post_date": "2020-06-02T09:18:01.923000", "content": "Thanks. i will add regularization
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "870158": "Hi. I have a large mismatch between validation kappa score and kappa test score after submission. The score on my validation set is 0.83 while the score on the leaderboard is 0.58. The submitted notebook can be accessed here (https://www.kaggle.com/pmwaniki/finetune-resnet50?scriptVersionId=35233987). This is my first project using pytorch and my first kernel submission and it is likely i'm missing something.\nI have checked that training data is not spilling into validation data.", "870270": "One problem could be that your initial learning rate is too high, use a value between 1e-3 and 1e-4. Second your training images have a lot of blank tiles. Take a look at the following notebook for a tile setup.\n[https://www.kaggle.com/iafoss/panda-16x128x128-tiles](https://www.kaggle.com/iafoss/panda-16x128x128-tiles)" } }