Hey,
\nI can see that quite a few people have been using a dataset that I shared: RSNA Mammography PNGs 256px, 512px, 768px & 1024px so just would like to give a quick update:
\nopencv is 10%-20% faster than with PILDue to limited VM time (and us wanting to squeeze as much inference time into the pipeline as possible) we have to be very careful about how long it takes to process the images to PNGs.
\nBased on limited experiments I believe that processing using opencv can be 10%-20% faster! As such, I uploaded files processed using opencv to the dataset for resolutions 256px and 512px. Currently also working on adding 384px (halfway between 256px and 512px).
You can find the code I used to process the images here: π‘ how to process DICOM images to PNGs
\nNo one has used the dataset for successful training & submission to the LB yet as far as I know (there are multiple training notebooks that train fine but the inference part has not been made public).
\nHowever, in working quite intensely on π€ [fast.ai starter pack] train + inference π as part of the verification of why my model is not training well, I have been switching across different public datasets, looking very closely at what my model is seeing in training. And I have not observed any issues with the data. The data seems πand safe to train on.
\n(In fact, I got the best performance on the smallest size 256px x 256px with the data processed using PIL but I wouldn't read too much into those differences, my model training is unstable for now hence it is not a good idea too read too much into differences that don't seem very large)
But just wanted to let you know that extensive testing has been done on the dataset and so far everything looks okay π
", "messages": [ { "id": 2055378, "postDate": "2022-12-05T03:42:51.830Z", "content": "Hey,
\nI can see that quite a few people have been using a dataset that I shared: RSNA Mammography PNGs 256px, 512px, 768px & 1024px so just would like to give a quick update:
\nopencv is 10%-20% faster than with PILDue to limited VM time (and us wanting to squeeze as much inference time into the pipeline as possible) we have to be very careful about how long it takes to process the images to PNGs.
\nBased on limited experiments I believe that processing using opencv can be 10%-20% faster! As such, I uploaded files processed using opencv to the dataset for resolutions 256px and 512px. Currently also working on adding 384px (halfway between 256px and 512px).
You can find the code I used to process the images here: π‘ how to process DICOM images to PNGs
\nNo one has used the dataset for successful training & submission to the LB yet as far as I know (there are multiple training notebooks that train fine but the inference part has not been made public).
\nHowever, in working quite intensely on π€ [fast.ai starter pack] train + inference π as part of the verification of why my model is not training well, I have been switching across different public datasets, looking very closely at what my model is seeing in training. And I have not observed any issues with the data. The data seems πand safe to train on.
\n(In fact, I got the best performance on the smallest size 256px x 256px with the data processed using PIL but I wouldn't read too much into those differences, my model training is unstable for now hence it is not a good idea too read too much into differences that don't seem very large)
But just wanted to let you know that extensive testing has been done on the dataset and so far everything looks okay π
", "rawMarkdown": "Hey,\n\nI can see that quite a few people have been using a dataset that I shared: [RSNA Mammography PNGs 256px, 512px, 768px & 1024px](https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs) so just would like to give a quick update:\n\n### Processing using `opencv` is 10%-20% faster than with `PIL`\n\nDue to limited VM time (and us wanting to squeeze as much inference time into the pipeline as possible) we have to be very careful about how long it takes to process the images to PNGs.\n\nBased on limited experiments *I believe* that processing using `opencv` can be 10%-20% faster! As such, I uploaded files processed using `opencv` to the dataset for resolutions 256px and 512px. Currently also working on adding 384px (halfway between 256px and 512px).\n\nYou can find the code I used to process the images here: [π‘ how to process DICOM images to PNGs](https://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs)\n\n### I have not found any issues with any versions of the dataset π₯³\n\nNo one has used the dataset for successful training & submission to the LB yet as far as I know (there are multiple training notebooks that train fine but the inference part has not been made public).\n\nHowever, in working quite intensely on [π€ [fast.ai starter pack] train + inference π](https://www.kaggle.com/code/radek1/fast-ai-starter-pack-train-inference) as part of the verification of why my model is not training well, I have been switching across different public datasets, looking very closely at what my model is seeing in training. And I have not observed any issues with the data. The data seems πand safe to train on. \n\n(In fact, I got the best performance on the smallest size 256px x 256px with the data processed using `PIL` but I wouldn't read too much into those differences, my model training is unstable for now hence it is not a good idea too read too much into differences that don't seem very large)\n\nBut just wanted to let you know that extensive testing has been done on the dataset and so far everything looks okay π", "votes": 6 } ], "comments": [], "raw_markdown_by_id": { "2055378": "Hey,\n\nI can see that quite a few people have been using a dataset that I shared: [RSNA Mammography PNGs 256px, 512px, 768px & 1024px](https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs) so just would like to give a quick update:\n\n### Processing using `opencv` is 10%-20% faster than with `PIL`\n\nDue to limited VM time (and us wanting to squeeze as much inference time into the pipeline as possible) we have to be very careful about how long it takes to process the images to PNGs.\n\nBased on limited experiments *I believe* that processing using `opencv` can be 10%-20% faster! As such, I uploaded files processed using `opencv` to the dataset for resolutions 256px and 512px. Currently also working on adding 384px (halfway between 256px and 512px).\n\nYou can find the code I used to process the images here: [π‘ how to process DICOM images to PNGs](https://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs)\n\n### I have not found any issues with any versions of the dataset π₯³\n\nNo one has used the dataset for successful training & submission to the LB yet as far as I know (there are multiple training notebooks that train fine but the inference part has not been made public).\n\nHowever, in working quite intensely on [π€ [fast.ai starter pack] train + inference π](https://www.kaggle.com/code/radek1/fast-ai-starter-pack-train-inference) as part of the verification of why my model is not training well, I have been switching across different public datasets, looking very closely at what my model is seeing in training. And I have not observed any issues with the data. The data seems πand safe to train on. \n\n(In fact, I got the best performance on the smallest size 256px x 256px with the data processed using `PIL` but I wouldn't read too much into those differences, my model training is unstable for now hence it is not a good idea too read too much into differences that don't seem very large)\n\nBut just wanted to let you know that extensive testing has been done on the dataset and so far everything looks okay π" } }