Hey, everyone.
\nEveryone asks what's your best model score, and I would like to know what's your first epoch R2 :)
\nMine starts at ca. 0.68, gets to 0.7 by epoch 3, and 0.72 by epoch 5. My total training is around 30 epochs yielding 0.746 validation and 0.75x LB. Reducing LR by 5 helps a lot around epoch 15-17.
\nI'm asking because someone said they start at 0.72 and it makes me salivate and try harder, but alas nothing helps me to get to the 0.7 zone yet.
\nCheers.
I think it's not that important to concentrate on the first epoch, because it's related to so many things like model architecture, batch size, learning rate and so on. I have a model that gets 0.44 at first epoch but finally gets 0.77.
", "rawMarkdown": "I think it's not that important to concentrate on the first epoch, because it's related to so many things like model architecture, batch size, learning rate and so on. I have a model that gets 0.44 at first epoch but finally gets 0.77.", "votes": 14 }, { "id": 2885043, "postDate": "2024-06-22T21:52:06.023Z", "content": "I also do not use dropout. I tried it, but it does not seem to help here. My best validation score after first epoch is 72.9%, and best val score of 76.6% is achieved after 15 epochs.
", "rawMarkdown": "I also do not use dropout. I tried it, but it does not seem to help here. My best validation score after first epoch is 72.9%, and best val score of 76.6% is achieved after 15 epochs.", "votes": 6 }, { "id": 2884381, "postDate": "2024-06-22T13:05:19.073Z", "content": "Hey, everyone.
\nEveryone asks what's your best model score, and I would like to know what's your first epoch R2 :)
\nMine starts at ca. 0.68, gets to 0.7 by epoch 3, and 0.72 by epoch 5. My total training is around 30 epochs yielding 0.746 validation and 0.75x LB. Reducing LR by 5 helps a lot around epoch 15-17.
\nI'm asking because someone said they start at 0.72 and it makes me salivate and try harder, but alas nothing helps me to get to the 0.7 zone yet.
\nCheers.
Lower all your dropout rate to zero, you probably will get a great first epoch.
\nBut you won't get far…
Dropout is tricky. I think we have enough data to avoid overfitting for anything but the heaviest models. I removed DO some time ago and don't see any overfitting or detrimental effect on MSE or R2.
", "rawMarkdown": "Dropout is tricky. I think we have enough data to avoid overfitting for anything but the heaviest models. I removed DO some time ago and don't see any overfitting or detrimental effect on MSE or R2.", "votes": 1 }, { "id": 2885401, "postDate": "2024-06-23T04:49:31.180Z", "content": "It also depends on lots of other things so I don't think comparing first epoch performance makes sense.
", "rawMarkdown": "It also depends on lots of other things so I don't think comparing first epoch performance makes sense.", "votes": 1, "replies": [ { "id": 2885452, "postDate": "2024-06-23T05:25:39.157Z", "content": "While the first few epochs are not indicative of the final score they are still useful to evaluate the model performance quickly without the full training. The difference between 0.67 and 0.72 is just too large to ignore.
", "rawMarkdown": "While the first few epochs are not indicative of the final score they are still useful to evaluate the model performance quickly without the full training. The difference between 0.67 and 0.72 is just too large to ignore.", "votes": 1 } ] } ] }, { "id": 2886262, "postDate": "2024-06-23T14:50:48.117Z", "content": "How much data are you using to validate?
", "rawMarkdown": "How much data are you using to validate?", "votes": 1, "replies": [ { "id": 2888580, "postDate": "2024-06-24T23:21:21.540Z", "content": "I am also interested in which cv strategy you use. As you know, it is one of the factor for Val score.
", "rawMarkdown": "I am also interested in which cv strategy you use. As you know, it is one of the factor for Val score." } ] }, { "id": 2885380, "postDate": "2024-06-23T04:33:07.427Z", "content": "How many data points did you use as the training set?
", "rawMarkdown": "How many data points did you use as the training set?", "votes": 1 }, { "id": 2885455, "postDate": "2024-06-23T05:31:15.580Z", "content": "Depends on steps / batch size too.
", "rawMarkdown": "Depends on steps / batch size too.", "votes": 2 }, { "id": 2888065, "postDate": "2024-06-24T15:12:03.063Z", "content": "My first epoch could only reach 0.52, and it ended up at 0.66 on validation. Currently, the R-squared on the training set is about 0.73+, and it can increase further with training. I suspect there is severe overfitting. During the process where the training loss decreases, the R-squared on validation does not increase accordingly. Although I have tried methods like dropout and regularization, the overfitting issue has not been alleviated. Additionally, another model I'm using is underfitting; it can reach about 0.67 on validation, but the R-squared on the training set is only 0.69.Additionally, the loss seems to have a hard time decreasing further during the training process.
", "rawMarkdown": "My first epoch could only reach 0.52, and it ended up at 0.66 on validation. Currently, the R-squared on the training set is about 0.73+, and it can increase further with training. I suspect there is severe overfitting. During the process where the training loss decreases, the R-squared on validation does not increase accordingly. Although I have tried methods like dropout and regularization, the overfitting issue has not been alleviated. Additionally, another model I'm using is underfitting; it can reach about 0.67 on validation, but the R-squared on the training set is only 0.69.Additionally, the loss seems to have a hard time decreasing further during the training process." }, { "id": 2886264, "postDate": "2024-06-23T14:52:00.363Z", "content": "Normally I validate my model in the first 4/5 epochs if the validation score is better than my previous best model, I continue training, if not, I stop.
", "rawMarkdown": "Normally I validate my model in the first 4/5 epochs if the validation score is better than my previous best model, I continue training, if not, I stop." } ], "comments": [ { "id": 2885578, "author_name": "嘴爷", "author_url": "", "post_date": "2024-06-23T06:54:44.890000", "content": "I think it's not that important to concentrate on the first epoch, because it's related to so many things like model architecture, batch size, learning rate and so on. I have a model that gets 0.44 at first epoch but finally gets 0.77.
", "votes": 14, "replies": [] }, { "id": 2885043, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2024-06-22T21:52:06.023000", "content": "I also do not use dropout. I tried it, but it does not seem to help here. My best validation score after first epoch is 72.9%, and best val score of 76.6% is achieved after 15 epochs.
", "votes": 6, "replies": [] }, { "id": 2884387, "author_name": "greySnow", "author_url": "", "post_date": "2024-06-22T13:12:41.700000", "content": "Lower all your dropout rate to zero, you probably will get a great first epoch.
\nBut you won't get far…
Dropout is tricky. I think we have enough data to avoid overfitting for anything but the heaviest models. I removed DO some time ago and don't see any overfitting or detrimental effect on MSE or R2.
", "votes": 1, "replies": [] }, { "id": 2885401, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-06-23T04:49:31.180000", "content": "It also depends on lots of other things so I don't think comparing first epoch performance makes sense.
", "votes": 1, "replies": [ { "id": 2885452, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-06-23T05:25:39.157000", "content": "While the first few epochs are not indicative of the final score they are still useful to evaluate the model performance quickly without the full training. The difference between 0.67 and 0.72 is just too large to ignore.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2886262, "author_name": "Fernando Melo", "author_url": "", "post_date": "2024-06-23T14:50:48.117000", "content": "How much data are you using to validate?
", "votes": 1, "replies": [ { "id": 2888580, "author_name": "HideBu", "author_url": "", "post_date": "2024-06-24T23:21:21.540000", "content": "I am also interested in which cv strategy you use. As you know, it is one of the factor for Val score.
", "votes": 0, "replies": [] } ] }, { "id": 2885380, "author_name": "Zhuoqun Li", "author_url": "", "post_date": "2024-06-23T04:33:07.427000", "content": "How many data points did you use as the training set?
", "votes": 1, "replies": [] }, { "id": 2885455, "author_name": "sroger", "author_url": "", "post_date": "2024-06-23T05:31:15.580000", "content": "Depends on steps / batch size too.
", "votes": 2, "replies": [] }, { "id": 2888065, "author_name": "Chengwei Yan", "author_url": "", "post_date": "2024-06-24T15:12:03.063000", "content": "My first epoch could only reach 0.52, and it ended up at 0.66 on validation. Currently, the R-squared on the training set is about 0.73+, and it can increase further with training. I suspect there is severe overfitting. During the process where the training loss decreases, the R-squared on validation does not increase accordingly. Although I have tried methods like dropout and regularization, the overfitting issue has not been alleviated. Additionally, another model I'm using is underfitting; it can reach about 0.67 on validation, but the R-squared on the training set is only 0.69.Additionally, the loss seems to have a hard time decreasing further during the training process.
", "votes": 0, "replies": [] }, { "id": 2886264, "author_name": "Fernando Melo", "author_url": "", "post_date": "2024-06-23T14:52:00.363000", "content": "Normally I validate my model in the first 4/5 epochs if the validation score is better than my previous best model, I continue training, if not, I stop.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2885578": "I think it's not that important to concentrate on the first epoch, because it's related to so many things like model architecture, batch size, learning rate and so on. I have a model that gets 0.44 at first epoch but finally gets 0.77.", "2885043": "I also do not use dropout. I tried it, but it does not seem to help here. My best validation score after first epoch is 72.9%, and best val score of 76.6% is achieved after 15 epochs.", "2884381": "Hey, everyone.\nEveryone asks what's your best model score, and I would like to know what's your first epoch R2 :)\nMine starts at ca. 0.68, gets to 0.7 by epoch 3, and 0.72 by epoch 5. My total training is around 30 epochs yielding 0.746 validation and 0.75x LB. Reducing LR by 5 helps a lot around epoch 15-17.\nI'm asking because someone said they start at 0.72 and it makes me salivate and try harder, but alas nothing helps me to get to the 0.7 zone yet.\nCheers.", "2884387": "Lower all your dropout rate to zero, you probably will get a great first epoch.\nBut you won't get far...", "2886262": "How much data are you using to validate?", "2885380": "How many data points did you use as the training set?", "2885455": "Depends on steps / batch size too.", "2888065": "My first epoch could only reach 0.52, and it ended up at 0.66 on validation. Currently, the R-squared on the training set is about 0.73+, and it can increase further with training. I suspect there is severe overfitting. During the process where the training loss decreases, the R-squared on validation does not increase accordingly. Although I have tried methods like dropout and regularization, the overfitting issue has not been alleviated. Additionally, another model I'm using is underfitting; it can reach about 0.67 on validation, but the R-squared on the training set is only 0.69.Additionally, the loss seems to have a hard time decreasing further during the training process.", "2886264": "Normally I validate my model in the first 4/5 epochs if the validation score is better than my previous best model, I continue training, if not, I stop." } }