I am dealing with WSIs everyday in my work. I don't know how the WSIs scanned but probably they were in some kind of pyramidal image format (e.g tiff, svs) and later converted to png from these formats. If you can upload the original WSIs (maybe you can convert them to tiff using vips?) so that we can work on these slides without getting any memory errors.
", "messages": [ { "id": 2479491, "postDate": "2023-10-12T16:45:51.017Z", "content": "I am dealing with WSIs everyday in my work. I don't know how the WSIs scanned but probably they were in some kind of pyramidal image format (e.g tiff, svs) and later converted to png from these formats. If you can upload the original WSIs (maybe you can convert them to tiff using vips?) so that we can work on these slides without getting any memory errors.
", "rawMarkdown": "I am dealing with WSIs everyday in my work. I don't know how the WSIs scanned but probably they were in some kind of pyramidal image format (e.g tiff, svs) and later converted to png from these formats. If you can upload the original WSIs (maybe you can convert them to tiff using vips?) so that we can work on these slides without getting any memory errors.", "votes": 25 }, { "id": 2479540, "postDate": "2023-10-12T17:15:50.923Z", "content": "It's a totally fair question! The short answer is that it's my fault and that I ran out of time. The longer answer is that the raw data was indeed a variety of .tiff and .svs formats from the various imaging machines. From a competitions perspective, we essentially can't ever use raw medical images. They must be rebuilt prior to release in order to remove both personal information and (often) the actual class labels from the image metadata. I actually reprocessed the entire dataset to tiled tiff with two different libraries, including pyvips, but caught fatal flaws in the results in each case. Frankly, I ran out of time to try again before the competition launch date.
\nThe current setup is definitely suboptimal but we're hoping to provide an update early next week that will improve the situation.
", "rawMarkdown": "It's a totally fair question! The short answer is that it's my fault and that I ran out of time. The longer answer is that the raw data was indeed a variety of .tiff and .svs formats from the various imaging machines. From a competitions perspective, we essentially can't ever use raw medical images. They must be rebuilt prior to release in order to remove both personal information and (often) the actual class labels from the image metadata. I actually reprocessed the entire dataset to tiled tiff with two different libraries, including pyvips, but caught fatal flaws in the results in each case. Frankly, I ran out of time to try again before the competition launch date. \n\nThe current setup is definitely suboptimal but we're hoping to provide an update early next week that will improve the situation.", "votes": 17, "replies": [ { "id": 2479563, "postDate": "2023-10-12T17:37:12.203Z", "content": "Of course all medical data should be deidentified before released into the public, I totally understand that! Thank you for the update.
", "rawMarkdown": "Of course all medical data should be deidentified before released into the public, I totally understand that! Thank you for the update.", "votes": 1 }, { "id": 2482861, "postDate": "2023-10-15T10:29:31.600Z", "content": "Are there any plans to follow up with raw competition data (i.e. svs or tiff format)?
", "rawMarkdown": "Are there any plans to follow up with raw competition data (i.e. svs or tiff format)?", "votes": 2 }, { "id": 2489028, "postDate": "2023-10-19T16:29:21.340Z", "content": "Any updates on if the dataset is going to change @sohier?
\nJust wondering if I should start downloading the current dataset or wait for an update :)
", "rawMarkdown": "Any updates on if the dataset is going to change @sohier? \n\nJust wondering if I should start downloading the current dataset or wait for an update :)", "votes": 1 }, { "id": 2489759, "postDate": "2023-10-20T07:24:28.360Z", "content": "@sohier Yes, me too.
\nPlease let me know when you update dataset.
@sohier any update on the format issue or any timeline you can provide ? tiff or svs would be much better (and more standard too). High resolution is key for good performances in digital pathology, and keeping the png format would bias the submissions toward suboptimal solutions :/
", "rawMarkdown": "@sohier any update on the format issue or any timeline you can provide ? tiff or svs would be much better (and more standard too). High resolution is key for good performances in digital pathology, and keeping the png format would bias the submissions toward suboptimal solutions :/ ", "votes": 5, "replies": [ { "id": 2501133, "postDate": "2023-10-27T09:03:00.313Z", "content": "+1! I totally agree with @simjeg. By design, one cannot only decode parts of a large PNG image. Instead, one has to decode the whole image at once. This creates a huge overhead, likely preventing most participants from making good use of full resolution images. TIFF/SVS would be definitely better!
", "rawMarkdown": "+1! I totally agree with @simjeg. By design, one cannot only decode *parts* of a large PNG image. Instead, one has to decode *the whole image* at once. This creates a huge overhead, likely preventing most participants from making good use of full resolution images. TIFF/SVS would be definitely better!", "votes": 3 } ] } ] }, { "id": 2482640, "postDate": "2023-10-15T07:43:30.417Z", "content": "Does using raw tiff files have any advantage over lossless pngs or is it for using 3rd party software like QuPath?
", "rawMarkdown": "Does using raw tiff files have any advantage over lossless pngs or is it for using 3rd party software like QuPath?", "votes": 2, "replies": [ { "id": 2482811, "postDate": "2023-10-15T09:49:21.983Z", "content": "Yes, big images (100kx50k) is hard to deal with. Libraries like OpenSlide make it easier to read these images into tiles, they don't load all the image into the memory, so you don't encounter memory errors too. Also another advantage of raw files is that you have a pyramidal image (first layer 100kx50k second layer 50kx25k so 2x downsample) you can read your tiles in any downsample you want and create your model accordingly. You can do this with PNG's too of course, but it's easier doing it like this.
", "rawMarkdown": "Yes, big images (100kx50k) is hard to deal with. Libraries like OpenSlide make it easier to read these images into tiles, they don't load all the image into the memory, so you don't encounter memory errors too. Also another advantage of raw files is that you have a pyramidal image (first layer 100kx50k second layer 50kx25k so 2x downsample) you can read your tiles in any downsample you want and create your model accordingly. You can do this with PNG's too of course, but it's easier doing it like this. ", "votes": 14 } ] } ], "comments": [ { "id": 2479540, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-10-12T17:15:50.923000", "content": "It's a totally fair question! The short answer is that it's my fault and that I ran out of time. The longer answer is that the raw data was indeed a variety of .tiff and .svs formats from the various imaging machines. From a competitions perspective, we essentially can't ever use raw medical images. They must be rebuilt prior to release in order to remove both personal information and (often) the actual class labels from the image metadata. I actually reprocessed the entire dataset to tiled tiff with two different libraries, including pyvips, but caught fatal flaws in the results in each case. Frankly, I ran out of time to try again before the competition launch date.
\nThe current setup is definitely suboptimal but we're hoping to provide an update early next week that will improve the situation.
", "votes": 17, "replies": [ { "id": 2479563, "author_name": "Eren Tekin", "author_url": "", "post_date": "2023-10-12T17:37:12.203000", "content": "Of course all medical data should be deidentified before released into the public, I totally understand that! Thank you for the update.
", "votes": 1, "replies": [] }, { "id": 2482861, "author_name": "阳光开朗大男孩", "author_url": "", "post_date": "2023-10-15T10:29:31.600000", "content": "Are there any plans to follow up with raw competition data (i.e. svs or tiff format)?
", "votes": 2, "replies": [] }, { "id": 2489028, "author_name": "Bartley", "author_url": "", "post_date": "2023-10-19T16:29:21.340000", "content": "Any updates on if the dataset is going to change @sohier?
\nJust wondering if I should start downloading the current dataset or wait for an update :)
", "votes": 1, "replies": [] }, { "id": 2489759, "author_name": "Tensor Titan", "author_url": "", "post_date": "2023-10-20T07:24:28.360000", "content": "@sohier Yes, me too.
\nPlease let me know when you update dataset.
@sohier any update on the format issue or any timeline you can provide ? tiff or svs would be much better (and more standard too). High resolution is key for good performances in digital pathology, and keeping the png format would bias the submissions toward suboptimal solutions :/
", "votes": 5, "replies": [ { "id": 2501133, "author_name": "jibounet", "author_url": "", "post_date": "2023-10-27T09:03:00.313000", "content": "+1! I totally agree with @simjeg. By design, one cannot only decode parts of a large PNG image. Instead, one has to decode the whole image at once. This creates a huge overhead, likely preventing most participants from making good use of full resolution images. TIFF/SVS would be definitely better!
", "votes": 3, "replies": [] } ] } ] }, { "id": 2482640, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2023-10-15T07:43:30.417000", "content": "Does using raw tiff files have any advantage over lossless pngs or is it for using 3rd party software like QuPath?
", "votes": 2, "replies": [ { "id": 2482811, "author_name": "Eren Tekin", "author_url": "", "post_date": "2023-10-15T09:49:21.983000", "content": "Yes, big images (100kx50k) is hard to deal with. Libraries like OpenSlide make it easier to read these images into tiles, they don't load all the image into the memory, so you don't encounter memory errors too. Also another advantage of raw files is that you have a pyramidal image (first layer 100kx50k second layer 50kx25k so 2x downsample) you can read your tiles in any downsample you want and create your model accordingly. You can do this with PNG's too of course, but it's easier doing it like this.
", "votes": 14, "replies": [] } ] } ], "raw_markdown_by_id": { "2479491": "I am dealing with WSIs everyday in my work. I don't know how the WSIs scanned but probably they were in some kind of pyramidal image format (e.g tiff, svs) and later converted to png from these formats. If you can upload the original WSIs (maybe you can convert them to tiff using vips?) so that we can work on these slides without getting any memory errors.", "2479540": "It's a totally fair question! The short answer is that it's my fault and that I ran out of time. The longer answer is that the raw data was indeed a variety of .tiff and .svs formats from the various imaging machines. From a competitions perspective, we essentially can't ever use raw medical images. They must be rebuilt prior to release in order to remove both personal information and (often) the actual class labels from the image metadata. I actually reprocessed the entire dataset to tiled tiff with two different libraries, including pyvips, but caught fatal flaws in the results in each case. Frankly, I ran out of time to try again before the competition launch date. \n\nThe current setup is definitely suboptimal but we're hoping to provide an update early next week that will improve the situation.", "2482640": "Does using raw tiff files have any advantage over lossless pngs or is it for using 3rd party software like QuPath?" } }