![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1315955%2Fb616a636653314c704f248857ef8b8c3%2F3123.png?generation=1591207392596413&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1315955%2Ff0d4eea1df4e7f86d4baaeeb5fc8eaaa%2F33333.png?generation=1591207393118544&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1315955%2Fc16d2b48f8d51bae38cbf6fc728795f8%2F88888.png?generation=1591207393504011&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1315955%2Ffd227cb5029bb204d735ebca8bb402d5%2F66666.png?generation=1591207393483157&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1315955%2Fe98568419857f167c2dfcf761cd2c6f1%2F666.png?generation=1591207393840149&alt=media\")
Dear kaggler's! I found many examples of duplicate images. Many in 5+3, many in 5+5 radboud and so on. \nSome examples of duplicate:
\n\n81f1706e615ff80757a041cf70d022f5\n94e121bc4a7091b881e8dd9b6c2be6b5
\n\n00951a7fad040bf7e90f32e81fc0746f\n47ebeac86bb4d9eb51f2cb5ce110a823
\n\n00951a7fad040bf7e90f32e81fc0746f\n47ebeac86bb4d9eb51f2cb5ce110a823
\n\n51b79eb617718f11e64587422e6877b0\ne739dc3b9262a94c9a5c206c39864278
\n\n0e305c4d31f47db36c44a1d55d389193\n00a26aaa82c959624d90dfb69fcf259c
\n\n07a14fa5b8f74272e4cc0a439dbc8f7f\n0da0915a236f2fc98b299d6fdefe7b8b
\n\nand many many others some of them a bit different!\n\n\n\n
Dear kaggler's! I found many examples of duplicate images. Many in 5+3, many in 5+5 radboud and so on. \nSome examples of duplicate:
\n\n81f1706e615ff80757a041cf70d022f5\n94e121bc4a7091b881e8dd9b6c2be6b5
\n\n00951a7fad040bf7e90f32e81fc0746f\n47ebeac86bb4d9eb51f2cb5ce110a823
\n\n00951a7fad040bf7e90f32e81fc0746f\n47ebeac86bb4d9eb51f2cb5ce110a823
\n\n51b79eb617718f11e64587422e6877b0\ne739dc3b9262a94c9a5c206c39864278
\n\n0e305c4d31f47db36c44a1d55d389193\n00a26aaa82c959624d90dfb69fcf259c
\n\n07a14fa5b8f74272e4cc0a439dbc8f7f\n0da0915a236f2fc98b299d6fdefe7b8b
\n\nand many many others some of them a bit different!\n\n\n\n
I think it might be the serial sections coming from the same tissue block. Both of slides looks almost same, but it seems that slightly different in some lesions.
", "rawMarkdown": "I think it might be the serial sections coming from the same tissue block. Both of slides looks almost same, but it seems that slightly different in some lesions.", "votes": 9, "replies": [ { "id": 873624, "postDate": "2020-06-04T10:03:13.640Z", "content": "Good observation, this is indeed the case for many of the \"duplicates\". In that sense they are not real duplicates but in fact a serial section of the same tissue. For the training set we included all tissue that was present on the glass slide.
", "rawMarkdown": "Good observation, this is indeed the case for many of the \"duplicates\". In that sense they are not real duplicates but in fact a serial section of the same tissue. For the training set we included all tissue that was present on the glass slide.\n", "votes": 5 }, { "id": 873640, "postDate": "2020-06-04T10:15:33.700Z", "content": "Like mr. @appian mentioned if simple algorithms like imagehash find about 700 duplicates => such images not suitable for ML and validation become not representative.
", "rawMarkdown": "Like mr. @appian mentioned if simple algorithms like imagehash find about 700 duplicates => such images not suitable for ML and validation become not representative." }, { "id": 873657, "postDate": "2020-06-04T10:31:06.260Z", "content": "You can still make use of duplicate images. As @wouterbulten mentioned, they are not really duplicates but very similar and from the same tissue. In that sense they can be used in a way similar to augmentations. Let's say image A and image B is very similar and can be grouped. During training/validation you can choose one from a similar pair.
", "rawMarkdown": "You can still make use of duplicate images. As @wouterbulten mentioned, they are not really duplicates but very similar and from the same tissue. In that sense they can be used in a way similar to augmentations. Let's say image A and image B is very similar and can be grouped. During training/validation you can choose one from a similar pair. ", "votes": 2 }, { "id": 873729, "postDate": "2020-06-04T11:41:42.717Z", "content": "Yes, we did not exclude them because they are physically different objects; It's not a rescan of the same tissue. As you can also see in the examples, differences can range from subtle changes to larger differences when tissue is missing or due to the presence of artifacts.
", "rawMarkdown": "Yes, we did not exclude them because they are physically different objects; It's not a rescan of the same tissue. As you can also see in the examples, differences can range from subtle changes to larger differences when tissue is missing or due to the presence of artifacts.", "votes": 5 }, { "id": 887173, "postDate": "2020-06-15T14:17:01.450Z", "content": "Could any of you tell a bit more about what a \"serial section\" means ?
", "rawMarkdown": "Could any of you tell a bit more about what a \"serial section\" means ?", "votes": 1 }, { "id": 887187, "postDate": "2020-06-15T14:25:39.777Z", "content": "In my understanding, it's the same tissue but different settings such as angle or how it's focused. Would be nice if someone corrects me if I'm wrong.
", "rawMarkdown": "In my understanding, it's the same tissue but different settings such as angle or how it's focused. Would be nice if someone corrects me if I'm wrong." }, { "id": 887264, "postDate": "2020-06-15T15:24:43.677Z", "content": "In my understanding they are different slices of the same tissue sample
", "rawMarkdown": "In my understanding they are different slices of the same tissue sample", "votes": 3 }, { "id": 887728, "postDate": "2020-06-15T20:31:45.003Z", "content": "Think of a sliced bread. Although slices look alike they are not the same.
\n\nSimilarly, tissue is thinly sliced and mounted on microscopic slides for detailed inspection.
", "rawMarkdown": "Think of a sliced bread. Although slices look alike they are not the same.\n\nSimilarly, tissue is thinly sliced and mounted on microscopic slides for detailed inspection. ", "votes": 5 }, { "id": 888190, "postDate": "2020-06-16T07:14:44.817Z", "content": "I think I remember learning something along those lines when I was in junior high school... should have paid more attention ^^ thank you guys !
", "rawMarkdown": "I think I remember learning something along those lines when I was in junior high school... should have paid more attention ^^ thank you guys ! ", "votes": 1 }, { "id": 892908, "postDate": "2020-06-19T08:46:17.767Z", "content": "Just to give you a confirmation - this is indeed correct. The \"bread\" is a single tissue sample extracted from the prostate with a needle, and each \"duplicate\" tissue section is a \"bread slice\". Depending on where the tumor happens to be and how large it is, the slices from the same piece of tissue may or may not show the cancer tissue. So sections from the same sample are typically correlated, but it is perfectly possible that some of them contain no cancer whereas others do.
", "rawMarkdown": "Just to give you a confirmation - this is indeed correct. The \"bread\" is a single tissue sample extracted from the prostate with a needle, and each \"duplicate\" tissue section is a \"bread slice\". Depending on where the tumor happens to be and how large it is, the slices from the same piece of tissue may or may not show the cancer tissue. So sections from the same sample are typically correlated, but it is perfectly possible that some of them contain no cancer whereas others do.", "votes": 2 }, { "id": 894835, "postDate": "2020-06-20T19:36:44.110Z", "content": "You may put in one file, like karolinska do =)
", "rawMarkdown": "You may put in one file, like karolinska do =)" } ] }, { "id": 873092, "postDate": "2020-06-03T19:30:22.370Z", "content": "There were similar issues in APTOS competition... Great Work!
", "rawMarkdown": "There were similar issues in APTOS competition... Great Work! ", "votes": 7 }, { "id": 873460, "postDate": "2020-06-04T06:59:21.460Z", "content": "It seems there are about 700 duplicate pairs. You can detect similar images using imagehash. https://github.com/JohannesBuchner/imagehash
", "rawMarkdown": "It seems there are about 700 duplicate pairs. You can detect similar images using imagehash. https://github.com/JohannesBuchner/imagehash", "votes": 5, "replies": [ { "id": 873496, "postDate": "2020-06-04T07:45:43.827Z", "content": "I thought I'll use CNN features with KNN
", "rawMarkdown": "I thought I'll use CNN features with KNN", "votes": 1 }, { "id": 873522, "postDate": "2020-06-04T08:08:55.443Z", "content": "I think there are ways to detect duplicates. Using CNN features with kNN sound promising and would be nice if you try.
\n\nI just created a quick notebook to detect duplicates using imagehash. https://www.kaggle.com/appian/panda-imagehash-to-detect-duplicate-images
", "rawMarkdown": "I think there are ways to detect duplicates. Using CNN features with kNN sound promising and would be nice if you try.\n\nI just created a quick notebook to detect duplicates using imagehash. https://www.kaggle.com/appian/panda-imagehash-to-detect-duplicate-images\n", "votes": 7 }, { "id": 876388, "postDate": "2020-06-06T16:47:01.363Z", "content": "@appian Thank you for sharing. I checked all data with image hashes. Trying different thresholds, I found that threshold less than 0.96 gave some false positives. Finally, there were ~150 \"duplicates\".
", "rawMarkdown": "@appian Thank you for sharing. I checked all data with image hashes. Trying different thresholds, I found that threshold less than 0.96 gave some false positives. Finally, there were ~150 \"duplicates\".", "votes": 1 }, { "id": 884188, "postDate": "2020-06-13T08:27:15.680Z", "content": "You get more false positives by lowering threshold but there are still many many duplicates.
", "rawMarkdown": "You get more false positives by lowering threshold but there are still many many duplicates." } ] }, { "id": 873076, "postDate": "2020-06-03T18:57:26.163Z", "content": "Nice catch! Would be interesting to hear from organizers on that.
", "rawMarkdown": "Nice catch! Would be interesting to hear from organizers on that.", "votes": 3 }, { "id": 878080, "postDate": "2020-06-08T08:29:03.867Z", "content": "Wow. This might explain why my validation score is so different from my submission score. I'm getting around 0.8 on my validation set and 0.65 on the public leaderboard. Regularization don't seem to help much. I was losing my mind thinking that there must be a mistake in my code
", "rawMarkdown": "Wow. This might explain why my validation score is so different from my submission score. I'm getting around 0.8 on my validation set and 0.65 on the public leaderboard. Regularization don't seem to help much. I was losing my mind thinking that there must be a mistake in my code", "votes": 2 }, { "id": 911984, "postDate": "2020-07-02T06:29:03.623Z", "content": "Did anyone use this information to group \"duplicates\" as same subject so train and validation could be on different subjects? Would have been good to have an Id that grouped these slices.
", "rawMarkdown": "Did anyone use this information to group \"duplicates\" as same subject so train and validation could be on different subjects? Would have been good to have an Id that grouped these slices." }, { "id": 873571, "postDate": "2020-06-04T09:19:36.100Z", "content": "Will the duplicate occur in test images too?
", "rawMarkdown": "Will the duplicate occur in test images too?", "isDeleted": true, "replies": [ { "id": 873633, "postDate": "2020-06-04T10:10:34.700Z", "content": "This only holds for the training set.
", "rawMarkdown": "This only holds for the training set.", "votes": 5 }, { "id": 873639, "postDate": "2020-06-04T10:15:07.383Z", "content": "Got it, thanks
", "rawMarkdown": "Got it, thanks", "isDeleted": true } ] } ], "comments": [ { "id": 873265, "author_name": "Kyungdoc Kim", "author_url": "", "post_date": "2020-06-04T01:31:46.227000", "content": "I think it might be the serial sections coming from the same tissue block. Both of slides looks almost same, but it seems that slightly different in some lesions.
", "votes": 9, "replies": [ { "id": 873624, "author_name": "Wouter Bulten", "author_url": "", "post_date": "2020-06-04T10:03:13.640000", "content": "Good observation, this is indeed the case for many of the \"duplicates\". In that sense they are not real duplicates but in fact a serial section of the same tissue. For the training set we included all tissue that was present on the glass slide.
", "votes": 5, "replies": [] }, { "id": 873640, "author_name": "Alex Donchuk", "author_url": "", "post_date": "2020-06-04T10:15:33.700000", "content": "Like mr. @appian mentioned if simple algorithms like imagehash find about 700 duplicates => such images not suitable for ML and validation become not representative.
", "votes": 0, "replies": [] }, { "id": 873657, "author_name": "Appian", "author_url": "", "post_date": "2020-06-04T10:31:06.260000", "content": "You can still make use of duplicate images. As @wouterbulten mentioned, they are not really duplicates but very similar and from the same tissue. In that sense they can be used in a way similar to augmentations. Let's say image A and image B is very similar and can be grouped. During training/validation you can choose one from a similar pair.
", "votes": 2, "replies": [] }, { "id": 873729, "author_name": "Wouter Bulten", "author_url": "", "post_date": "2020-06-04T11:41:42.717000", "content": "Yes, we did not exclude them because they are physically different objects; It's not a rescan of the same tissue. As you can also see in the examples, differences can range from subtle changes to larger differences when tissue is missing or due to the presence of artifacts.
", "votes": 5, "replies": [] }, { "id": 887173, "author_name": "Benjamin Dubreu", "author_url": "", "post_date": "2020-06-15T14:17:01.450000", "content": "Could any of you tell a bit more about what a \"serial section\" means ?
", "votes": 1, "replies": [] }, { "id": 887187, "author_name": "Appian", "author_url": "", "post_date": "2020-06-15T14:25:39.777000", "content": "In my understanding, it's the same tissue but different settings such as angle or how it's focused. Would be nice if someone corrects me if I'm wrong.
", "votes": 0, "replies": [] }, { "id": 887264, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2020-06-15T15:24:43.677000", "content": "In my understanding they are different slices of the same tissue sample
", "votes": 3, "replies": [] }, { "id": 887728, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-06-15T20:31:45.003000", "content": "Think of a sliced bread. Although slices look alike they are not the same.
\n\nSimilarly, tissue is thinly sliced and mounted on microscopic slides for detailed inspection.
", "votes": 5, "replies": [] }, { "id": 888190, "author_name": "Benjamin Dubreu", "author_url": "", "post_date": "2020-06-16T07:14:44.817000", "content": "I think I remember learning something along those lines when I was in junior high school... should have paid more attention ^^ thank you guys !
", "votes": 1, "replies": [] }, { "id": 892908, "author_name": "Kimmo Kartasalo", "author_url": "", "post_date": "2020-06-19T08:46:17.767000", "content": "Just to give you a confirmation - this is indeed correct. The \"bread\" is a single tissue sample extracted from the prostate with a needle, and each \"duplicate\" tissue section is a \"bread slice\". Depending on where the tumor happens to be and how large it is, the slices from the same piece of tissue may or may not show the cancer tissue. So sections from the same sample are typically correlated, but it is perfectly possible that some of them contain no cancer whereas others do.
", "votes": 2, "replies": [] }, { "id": 894835, "author_name": "Alex Donchuk", "author_url": "", "post_date": "2020-06-20T19:36:44.110000", "content": "You may put in one file, like karolinska do =)
", "votes": 0, "replies": [] } ] }, { "id": 873092, "author_name": "DrHB", "author_url": "", "post_date": "2020-06-03T19:30:22.370000", "content": "There were similar issues in APTOS competition... Great Work!
", "votes": 7, "replies": [] }, { "id": 873460, "author_name": "Appian", "author_url": "", "post_date": "2020-06-04T06:59:21.460000", "content": "It seems there are about 700 duplicate pairs. You can detect similar images using imagehash. https://github.com/JohannesBuchner/imagehash
", "votes": 5, "replies": [ { "id": 873496, "author_name": "Alex Donchuk", "author_url": "", "post_date": "2020-06-04T07:45:43.827000", "content": "I thought I'll use CNN features with KNN
", "votes": 1, "replies": [] }, { "id": 873522, "author_name": "Appian", "author_url": "", "post_date": "2020-06-04T08:08:55.443000", "content": "I think there are ways to detect duplicates. Using CNN features with kNN sound promising and would be nice if you try.
\n\nI just created a quick notebook to detect duplicates using imagehash. https://www.kaggle.com/appian/panda-imagehash-to-detect-duplicate-images
", "votes": 7, "replies": [] }, { "id": 876388, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2020-06-06T16:47:01.363000", "content": "@appian Thank you for sharing. I checked all data with image hashes. Trying different thresholds, I found that threshold less than 0.96 gave some false positives. Finally, there were ~150 \"duplicates\".
", "votes": 1, "replies": [] }, { "id": 884188, "author_name": "Appian", "author_url": "", "post_date": "2020-06-13T08:27:15.680000", "content": "You get more false positives by lowering threshold but there are still many many duplicates.
", "votes": 0, "replies": [] } ] }, { "id": 873076, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2020-06-03T18:57:26.163000", "content": "Nice catch! Would be interesting to hear from organizers on that.
", "votes": 3, "replies": [] }, { "id": 878080, "author_name": "Mwaniki", "author_url": "", "post_date": "2020-06-08T08:29:03.867000", "content": "Wow. This might explain why my validation score is so different from my submission score. I'm getting around 0.8 on my validation set and 0.65 on the public leaderboard. Regularization don't seem to help much. I was losing my mind thinking that there must be a mistake in my code
", "votes": 2, "replies": [] }, { "id": 911984, "author_name": "something4kag", "author_url": "", "post_date": "2020-07-02T06:29:03.623000", "content": "Did anyone use this information to group \"duplicates\" as same subject so train and validation could be on different subjects? Would have been good to have an Id that grouped these slices.
", "votes": 0, "replies": [] }, { "id": 873571, "author_name": "", "author_url": "", "post_date": "2020-06-04T09:19:36.100000", "content": "Will the duplicate occur in test images too?
", "votes": 0, "replies": [ { "id": 873633, "author_name": "Wouter Bulten", "author_url": "", "post_date": "2020-06-04T10:10:34.700000", "content": "This only holds for the training set.
", "votes": 5, "replies": [] }, { "id": 873639, "author_name": "", "author_url": "", "post_date": "2020-06-04T10:15:07.383000", "content": "Got it, thanks
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "873041": "Dear kaggler's! I found many examples of duplicate images. Many in 5+3, many in 5+5 radboud and so on. \nSome examples of duplicate:\n\n81f1706e615ff80757a041cf70d022f5\n94e121bc4a7091b881e8dd9b6c2be6b5\n\n00951a7fad040bf7e90f32e81fc0746f\n47ebeac86bb4d9eb51f2cb5ce110a823\n\n00951a7fad040bf7e90f32e81fc0746f\n47ebeac86bb4d9eb51f2cb5ce110a823\n\n51b79eb617718f11e64587422e6877b0\ne739dc3b9262a94c9a5c206c39864278\n\n0e305c4d31f47db36c44a1d55d389193\n00a26aaa82c959624d90dfb69fcf259c\n\n07a14fa5b8f74272e4cc0a439dbc8f7f\n0da0915a236f2fc98b299d6fdefe7b8b\n\nand many many others some of them a bit different!\n\n\n\n\n", "873265": "I think it might be the serial sections coming from the same tissue block. Both of slides looks almost same, but it seems that slightly different in some lesions.", "873092": "There were similar issues in APTOS competition... Great Work! ", "873460": "It seems there are about 700 duplicate pairs. You can detect similar images using imagehash. https://github.com/JohannesBuchner/imagehash", "873076": "Nice catch! Would be interesting to hear from organizers on that.", "878080": "Wow. This might explain why my validation score is so different from my submission score. I'm getting around 0.8 on my validation set and 0.65 on the public leaderboard. Regularization don't seem to help much. I was losing my mind thinking that there must be a mistake in my code", "911984": "Did anyone use this information to group \"duplicates\" as same subject so train and validation could be on different subjects? Would have been good to have an Id that grouped these slices.", "873571": "Will the duplicate occur in test images too?" } }