{ "id": 415764, "title": "Did the labelers know about the test set ?", "url": "/competitions/google-research-identify-contrails-reduce-global-warming/discussion/415764", "author_name": "JEANMPIA", "post_date": "2023-06-08T00:21:44.981000", "votes": 11, "comment_count": 3, "views": 0, "content": "

Hello @aaronsarna,
\nI have a question regarding the train/test split. (not train/valid, I count valid as part of train)
\nWere the labellers informed that what they were labelling was either the train or the test set ?
\nMy question might sound anecdotal but take a look:
\n\"\"
\nthe 4 first labels are layers of the human_masks and the last is the pixel_mask

\n

I don't know for sure those are con-trails, but thats really not the only image where some of the labellers didn't label some con-trails where other did, often resulting in no-contrail labels.
\nI plan on changing the pixel mask for this image to one of the ones that saw the con-trail, would that be foolish since the same amounth of care was given to the test set, or did they know that this set was going to be important so they took extra attention ? (no disrespect to the labellers of course I get that this job requires a lot of focus and I'm glad I'm not on this side of the pipeline)

", "messages": [ { "id": 2291944, "postDate": "2023-06-08T01:10:55.887Z", "content": "

The labelers did not know if a sample was going to be in train, test, or validation. The splits were determined using random sampling after the labeling was completed.

", "rawMarkdown": "The labelers did not know if a sample was going to be in train, test, or validation. The splits were determined using random sampling after the labeling was completed.", "votes": 10, "replies": [ { "id": 2291945, "postDate": "2023-06-08T01:16:55.217Z", "content": "

Thx for your fast answer ! really appreciated,
\ngood to know as well. Noted.

", "rawMarkdown": "Thx for your fast answer ! really appreciated,\ngood to know as well. Noted." } ] }, { "id": 2291920, "postDate": "2023-06-08T00:21:44.983Z", "content": "

Hello @aaronsarna,
\nI have a question regarding the train/test split. (not train/valid, I count valid as part of train)
\nWere the labellers informed that what they were labelling was either the train or the test set ?
\nMy question might sound anecdotal but take a look:
\n\"\"
\nthe 4 first labels are layers of the human_masks and the last is the pixel_mask

\n

I don't know for sure those are con-trails, but thats really not the only image where some of the labellers didn't label some con-trails where other did, often resulting in no-contrail labels.
\nI plan on changing the pixel mask for this image to one of the ones that saw the con-trail, would that be foolish since the same amounth of care was given to the test set, or did they know that this set was going to be important so they took extra attention ? (no disrespect to the labellers of course I get that this job requires a lot of focus and I'm glad I'm not on this side of the pipeline)

", "rawMarkdown": "Hello @aaronsarna,\nI have a question regarding the train/test split. (not train/valid, I count valid as part of train)\n**Were the labellers informed that what they were labelling was either the train or the test set ?**\n*My question might sound anecdotal but take a look:*\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2Fea0fa6bd03a87e1da55d1101c7fddcbd%2FScreenshot_101.jpg?generation=1686183194357307&alt=media)\n*the 4 first labels are layers of the human_masks and the last is the pixel_mask*\n\nI don't know for sure those are con-trails, but thats really not the only image where some of the labellers didn't label some con-trails where other did, often resulting in no-contrail labels.\nI plan on changing the pixel mask for this image to one of the ones that saw the con-trail, would that be foolish since the same amounth of care was given to the test set, or did they know that this set was going to be important so they took extra attention ? (no disrespect to the labellers of course I get that this job requires a lot of focus and I'm glad I'm not on this side of the pipeline)\n", "votes": 10 }, { "id": 2291923, "postDate": "2023-06-08T00:28:07.040Z", "content": "

other examples of mask that might prevent the model from training correctely:

\n

\"\"

\n

\"\"

", "rawMarkdown": "*other examples of mask that might prevent the model from training correctely:*\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F61d08133cd8eab8ea3462649d26e5263%2FScreenshot_103.jpg?generation=1686184278301327&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F458a6b124edc9c83af4440b09808ab34%2FScreenshot_102.jpg?generation=1686184063362719&alt=media)", "votes": 1 } ], "comments": [ { "id": 2291944, "author_name": "Aaron Sarna", "author_url": "", "post_date": "2023-06-08T01:10:55.887000", "content": "

The labelers did not know if a sample was going to be in train, test, or validation. The splits were determined using random sampling after the labeling was completed.

", "votes": 10, "replies": [ { "id": 2291945, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-06-08T01:16:55.217000", "content": "

Thx for your fast answer ! really appreciated,
\ngood to know as well. Noted.

", "votes": 0, "replies": [] } ] }, { "id": 2291923, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-06-08T00:28:07.040000", "content": "

other examples of mask that might prevent the model from training correctely:

\n

\"\"

\n

\"\"

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2291944": "The labelers did not know if a sample was going to be in train, test, or validation. The splits were determined using random sampling after the labeling was completed.", "2291920": "Hello @aaronsarna,\nI have a question regarding the train/test split. (not train/valid, I count valid as part of train)\n**Were the labellers informed that what they were labelling was either the train or the test set ?**\n*My question might sound anecdotal but take a look:*\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2Fea0fa6bd03a87e1da55d1101c7fddcbd%2FScreenshot_101.jpg?generation=1686183194357307&alt=media)\n*the 4 first labels are layers of the human_masks and the last is the pixel_mask*\n\nI don't know for sure those are con-trails, but thats really not the only image where some of the labellers didn't label some con-trails where other did, often resulting in no-contrail labels.\nI plan on changing the pixel mask for this image to one of the ones that saw the con-trail, would that be foolish since the same amounth of care was given to the test set, or did they know that this set was going to be important so they took extra attention ? (no disrespect to the labellers of course I get that this job requires a lot of focus and I'm glad I'm not on this side of the pipeline)\n", "2291923": "*other examples of mask that might prevent the model from training correctely:*\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F61d08133cd8eab8ea3462649d26e5263%2FScreenshot_103.jpg?generation=1686184278301327&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F458a6b124edc9c83af4440b09808ab34%2FScreenshot_102.jpg?generation=1686184063362719&alt=media)" } }