In one submission, I didn't change predicted labels, even though probability was low.
\nIn another submission, I replaced all labels to 'Others', while using the same model.
\nOf course, these two submissions gave totally different results, but got the same score on the leaderboard.
\nWhy did it happen?
In one submission, I didn't change predicted labels, even though probability was low.
\nIn another submission, I replaced all labels to 'Others', while using the same model.
\nOf course, these two submissions gave totally different results, but got the same score on the leaderboard.
\nWhy did it happen?
Receiving the same private score for substantially different submissions in a machine learning competition, like changing all labels to \"other\" or not altering the predictions despite varying thresholds, can be perplexing. This situation could be due to several reasons:
\nEvaluation Metric Insensitivity: The scoring metric used in the competition might not be sensitive to the changes made in the submissions. Some metrics may not effectively capture the impact of certain types of changes in the predictions.
\nData Distribution in Private Leaderboard: The private leaderboard usually evaluates submissions on a different set of data. If this data has a distribution where the changes in your submission don't significantly alter the performance as per the evaluation metric, the score might remain the same.
\nOverfitting to Public Leaderboard: If the model is overfitted to the public leaderboard data, it might not generalize well to the private leaderboard data, resulting in similar scores despite changes.
\nIssues with Submission: There could be issues with how the submissions were processed or evaluated. Ensuring that the submission files were correctly formatted and contained the intended changes is important.
\nUnderstanding the specific evaluation criteria and data characteristics of the competition is crucial in these scenarios. It's also beneficial to review competition forums or reach out to the organizers for clarification.
", "rawMarkdown": "Receiving the same private score for substantially different submissions in a machine learning competition, like changing all labels to \"other\" or not altering the predictions despite varying thresholds, can be perplexing. This situation could be due to several reasons:\n\nEvaluation Metric Insensitivity: The scoring metric used in the competition might not be sensitive to the changes made in the submissions. Some metrics may not effectively capture the impact of certain types of changes in the predictions.\n\nData Distribution in Private Leaderboard: The private leaderboard usually evaluates submissions on a different set of data. If this data has a distribution where the changes in your submission don't significantly alter the performance as per the evaluation metric, the score might remain the same.\n\nOverfitting to Public Leaderboard: If the model is overfitted to the public leaderboard data, it might not generalize well to the private leaderboard data, resulting in similar scores despite changes.\n\nIssues with Submission: There could be issues with how the submissions were processed or evaluated. Ensuring that the submission files were correctly formatted and contained the intended changes is important.\n\nUnderstanding the specific evaluation criteria and data characteristics of the competition is crucial in these scenarios. It's also beneficial to review competition forums or reach out to the organizers for clarification.", "votes": 2 } ], "comments": [ { "id": 2563675, "author_name": "", "author_url": "", "post_date": "2023-12-16T13:59:10.480000", "content": "Receiving the same private score for substantially different submissions in a machine learning competition, like changing all labels to \"other\" or not altering the predictions despite varying thresholds, can be perplexing. This situation could be due to several reasons:
\nEvaluation Metric Insensitivity: The scoring metric used in the competition might not be sensitive to the changes made in the submissions. Some metrics may not effectively capture the impact of certain types of changes in the predictions.
\nData Distribution in Private Leaderboard: The private leaderboard usually evaluates submissions on a different set of data. If this data has a distribution where the changes in your submission don't significantly alter the performance as per the evaluation metric, the score might remain the same.
\nOverfitting to Public Leaderboard: If the model is overfitted to the public leaderboard data, it might not generalize well to the private leaderboard data, resulting in similar scores despite changes.
\nIssues with Submission: There could be issues with how the submissions were processed or evaluated. Ensuring that the submission files were correctly formatted and contained the intended changes is important.
\nUnderstanding the specific evaluation criteria and data characteristics of the competition is crucial in these scenarios. It's also beneficial to review competition forums or reach out to the organizers for clarification.
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2563660": "In one submission, I didn't change predicted labels, even though probability was low.\nIn another submission, I replaced all labels to 'Others', while using the same model.\nOf course, these two submissions gave totally different results, but got the same score on the leaderboard.\nWhy did it happen? ", "2563675": "Receiving the same private score for substantially different submissions in a machine learning competition, like changing all labels to \"other\" or not altering the predictions despite varying thresholds, can be perplexing. This situation could be due to several reasons:\n\nEvaluation Metric Insensitivity: The scoring metric used in the competition might not be sensitive to the changes made in the submissions. Some metrics may not effectively capture the impact of certain types of changes in the predictions.\n\nData Distribution in Private Leaderboard: The private leaderboard usually evaluates submissions on a different set of data. If this data has a distribution where the changes in your submission don't significantly alter the performance as per the evaluation metric, the score might remain the same.\n\nOverfitting to Public Leaderboard: If the model is overfitted to the public leaderboard data, it might not generalize well to the private leaderboard data, resulting in similar scores despite changes.\n\nIssues with Submission: There could be issues with how the submissions were processed or evaluated. Ensuring that the submission files were correctly formatted and contained the intended changes is important.\n\nUnderstanding the specific evaluation criteria and data characteristics of the competition is crucial in these scenarios. It's also beneficial to review competition forums or reach out to the organizers for clarification." } }