@masadia
\nIn the Dataset Description, [train/test]_thumbnails is defined as follows.
\n\n[train/test]_thumbnails A folder containing smaller .png copies of the whole slide images. Thumbnails are not provided for TMAs.
\n
If this is true, we can determine if test sample is TMA or not by checking this directory.
\nBut test.csv does not have an is_tma column. Is there a reason for this? Or do some test TMA images have thumbnail images?
\nP.S.
\nI have published a notebook for probing this.
", "messages": [ { "id": 2492907, "postDate": "2023-10-23T02:46:37.380Z", "content": "@masadia
\nIn the Dataset Description, [train/test]_thumbnails is defined as follows.
\n\n[train/test]_thumbnails A folder containing smaller .png copies of the whole slide images. Thumbnails are not provided for TMAs.
\n
If this is true, we can determine if test sample is TMA or not by checking this directory.
\nBut test.csv does not have an is_tma column. Is there a reason for this? Or do some test TMA images have thumbnail images?
\nP.S.
\nI have published a notebook for probing this.
", "rawMarkdown": "@masadia \nIn the [Dataset Description](https://www.kaggle.com/competitions/UBC-OCEAN/data), `[train/test]_thumbnails` is defined as follows.\n\n> [train/test]_thumbnails A folder containing smaller .png copies of the whole slide images. Thumbnails are not provided for TMAs.\n\nIf this is true, we can determine if test sample is TMA or not by checking this directory.\n\nBut test.csv does not have an is_tma column. Is there a reason for this? Or do some test TMA images have thumbnail images? \n\nP.S.\n\nI have published a [notebook](https://www.kaggle.com/code/yukkyo/probing-all-test-sample-have-thumbnail/notebook) for probing this.", "votes": 14 }, { "id": 2500289, "postDate": "2023-10-26T15:13:57Z", "content": "I find all the hidden test files now have thumbnails. Has there been an update to the test set?
", "rawMarkdown": "I find all the hidden test files now have thumbnails. Has there been an update to the test set?", "votes": 5, "replies": [ { "id": 2500331, "postDate": "2023-10-26T15:31:03.953Z", "content": "It was probably there from the beginning.
\nHowever, this is different from dataset description. So I would like an additional announcement from organizers.
That is correct. You can create the same is_tma column by simply doing this
df_test[] = \ndf_test.loc[df_test[].isin(test_thumbnails), ] = \nIt is also mentioned here but it's not clarified by the organizers or Kaggle team.
", "rawMarkdown": "That is correct. You can create the same `is_tma` column by simply doing this\n```python\ndf_test['is_tma'] = True\ndf_test.loc[df_test['image_id'].isin(test_thumbnails), 'is_tma'] = False\n```\n\nIt is also mentioned [here](https://www.kaggle.com/competitions/UBC-OCEAN/discussion/446102) but it's not clarified by the organizers or Kaggle team.", "votes": 5, "replies": [ { "id": 2493106, "postDate": "2023-10-23T06:53:52.660Z", "content": "I understand that.
\nMy question here is whether this rule is guaranteed even with Unseen test data.
The TMA determination by thumbnail dir will be meaningless if there is a TMA test sample with a thumbnail.
\nI throw this discussion because I was hoping that if such a sample exists, it would be announced in advance.
I also want to know that. Tagging @sohier and @masadia hope to get an answer.
", "rawMarkdown": "I also want to know that. Tagging @sohier and @masadia hope to get an answer.", "votes": 2 }, { "id": 2517867, "postDate": "2023-11-08T20:06:07.057Z", "content": "@michaeldiazstewart, Could you please provide some insight here?
\nIt would be good to know if we have access to is_tma information when evaluating against the test set.
did you figure out the TMA in thumbnail thing? If they are in thumbnail it would be so weird since they are not in the training thumbnail data. I did some testing and found images from the test dataset are in thumbnail folder. So either is no TMA in the public test data or they also have a thumbnail (and we don't know if they are resized or the original size)
", "rawMarkdown": "did you figure out the TMA in thumbnail thing? If they are in thumbnail it would be so weird since they are not in the training thumbnail data. I did some testing and found images from the test dataset are in thumbnail folder. So either is no TMA in the public test data or they also have a thumbnail (and we don't know if they are resized or the original size)", "votes": 1 }, { "id": 2562325, "postDate": "2023-12-15T08:54:35.877Z", "content": "from other discussion, the general solution to this is if image size is less then 5000px, it is TMA
", "rawMarkdown": "from other discussion, the general solution to this is if image size is less then 5000px, it is TMA", "votes": 2 } ] } ] }, { "id": 2493255, "postDate": "2023-10-23T08:52:43.970Z", "content": "Yes, you brought up the question I also wanted to ask. If TMA indeed doesn't have thumbnails, and we can determine whether it's TMA through the directory, then it seems this task has become much clearer.
", "rawMarkdown": "Yes, you brought up the question I also wanted to ask. If TMA indeed doesn't have thumbnails, and we can determine whether it's TMA through the directory, then it seems this task has become much clearer.", "votes": 1 }, { "id": 2499703, "postDate": "2023-10-26T07:31:28.243Z", "content": "You can easily create them in the inference time, see for example:
\nhttps://www.kaggle.com/code/jirkaborovec/cancer-subtype-lightning-torch-inference
I find all the hidden test files now have thumbnails. Has there been an update to the test set?
", "votes": 5, "replies": [ { "id": 2500331, "author_name": "fam_taro", "author_url": "", "post_date": "2023-10-26T15:31:03.953000", "content": "It was probably there from the beginning.
\nHowever, this is different from dataset description. So I would like an additional announcement from organizers.
That is correct. You can create the same is_tma column by simply doing this
df_test[] = \ndf_test.loc[df_test[].isin(test_thumbnails), ] = \nIt is also mentioned here but it's not clarified by the organizers or Kaggle team.
", "votes": 5, "replies": [ { "id": 2493106, "author_name": "fam_taro", "author_url": "", "post_date": "2023-10-23T06:53:52.660000", "content": "I understand that.
\nMy question here is whether this rule is guaranteed even with Unseen test data.
The TMA determination by thumbnail dir will be meaningless if there is a TMA test sample with a thumbnail.
\nI throw this discussion because I was hoping that if such a sample exists, it would be announced in advance.
I also want to know that. Tagging @sohier and @masadia hope to get an answer.
", "votes": 2, "replies": [] }, { "id": 2517867, "author_name": "Bartley", "author_url": "", "post_date": "2023-11-08T20:06:07.057000", "content": "@michaeldiazstewart, Could you please provide some insight here?
\nIt would be good to know if we have access to is_tma information when evaluating against the test set.
did you figure out the TMA in thumbnail thing? If they are in thumbnail it would be so weird since they are not in the training thumbnail data. I did some testing and found images from the test dataset are in thumbnail folder. So either is no TMA in the public test data or they also have a thumbnail (and we don't know if they are resized or the original size)
", "votes": 1, "replies": [] }, { "id": 2562325, "author_name": "Jirka", "author_url": "", "post_date": "2023-12-15T08:54:35.877000", "content": "from other discussion, the general solution to this is if image size is less then 5000px, it is TMA
", "votes": 2, "replies": [] } ] } ] }, { "id": 2493255, "author_name": "Huang Jin Feng", "author_url": "", "post_date": "2023-10-23T08:52:43.970000", "content": "Yes, you brought up the question I also wanted to ask. If TMA indeed doesn't have thumbnails, and we can determine whether it's TMA through the directory, then it seems this task has become much clearer.
", "votes": 1, "replies": [] }, { "id": 2499703, "author_name": "Jirka", "author_url": "", "post_date": "2023-10-26T07:31:28.243000", "content": "You can easily create them in the inference time, see for example:
\nhttps://www.kaggle.com/code/jirkaborovec/cancer-subtype-lightning-torch-inference