Hi all,
\nWe've created TFRecords for the public train and test set, and created a notebook detailing how to load / use them. Given the complexity of the DICOM dataset, the included data is more extensive than is normally included in TFRecord format. Let us know if you have questions or feedback!
", "messages": [ { "id": 1041691, "postDate": "2020-10-07T21:53:37.613Z", "content": "Hi all,
\nWe've created TFRecords for the public train and test set, and created a notebook detailing how to load / use them. Given the complexity of the DICOM dataset, the included data is more extensive than is normally included in TFRecord format. Let us know if you have questions or feedback!
", "rawMarkdown": "Hi all,\n\nWe've created [TFRecords](https://www.kaggle.com/philculliton/rsna-pe-tfrecords-v2) for the public train and test set, and created a [notebook](https://www.kaggle.com/philculliton/rsna-pe-tfrecords-loading) detailing how to load / use them. Given the complexity of the DICOM dataset, the included data is more extensive than is normally included in TFRecord format. Let us know if you have questions or feedback!", "votes": 43 }, { "id": 1044060, "postDate": "2020-10-09T13:32:36.210Z", "content": "Trying to train the model on TPU using the tf data, following the process accordingly to the great notebook on TPU
\nhttps://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu
\nIn Cell 6 of the above notebook there is a decoding process where the image array is getting decoded according to the image type, here it's DICOM and tried applying image = tfio.image.decode_dicom_image(image_data, dtype=tf.uint16) , throwing error while running on TPU is there any compatible way to decode of DICOM image array like this way? Not sure if am missing something here.
Hi @soumya5891,
\nTry something like this, replacing decode_image function,
decode_image(image_data):
\nimage = tf.io.decode_raw(image_data, out_type = 'int16')
\nimage = tf.reshape(image, [*IMAGE_SIZE])
\nreturn image
At least for visualization it should work …
", "rawMarkdown": "Hi @soumya5891, \nTry something like this, replacing decode_image function,\n\n`decode_image(image_data):`\n`image = tf.io.decode_raw(image_data, out_type = 'int16')`\n`image = tf.reshape(image, [*IMAGE_SIZE])` \n`return image` \n\nAt least for visualization it should work ...\n\n", "votes": 1 }, { "id": 1044122, "postDate": "2020-10-09T14:34:18.633Z", "content": "Thanks @coreacasa for your response, I have tried it previously, the dataset is getting generated, though not checked the visualization part, but during training it's failing , the image dimensions are messed up
", "rawMarkdown": "Thanks @coreacasa for your response, I have tried it previously, the dataset is getting generated, though not checked the visualization part, but during training it's failing , the image dimensions are messed up", "votes": 1 }, { "id": 1044187, "postDate": "2020-10-09T15:24:50.720Z", "content": "@soumya5891,
\nYou would have to do a more complete pre-processing but something like that works in training.
decode_image(image_data):
\nimage = tf.io.decode_raw(image_data, out_type = 'int16')
\nimage = tf.repeat(image,3,axis=0)
\nimage = tf.cast(image, tf.float32) / 255.0
\nimage = tf.reshape(image, [*IMAGE_SIZE,3])
\nreturn image
Thanks again @coreacasa , I was just trying to test if training is happening properly with tf records(at least completion of 1 epoch), the snippet you have shared working fine , now can think about other stuffs
", "rawMarkdown": "Thanks again @coreacasa , I was just trying to test if training is happening properly with tf records(at least completion of 1 epoch), the snippet you have shared working fine , now can think about other stuffs", "votes": 1 } ] }, { "id": 1042649, "postDate": "2020-10-08T11:33:18.617Z", "content": "Hey @philculliton, so in the test set some of the files are hidden right. So can we use these test tfrecords for submission?
\nI mean do these tf-records contains images from those hidden files ?
Hi @rsinda - no, this is only the public set of test files, none of the hidden test files are included.
", "rawMarkdown": "Hi @rsinda - no, this is only the public set of test files, none of the hidden test files are included.", "votes": 3 }, { "id": 1046312, "postDate": "2020-10-11T14:42:50.990Z", "content": "@philculliton Thanks for sharing the dataset. does that mean for the private test set, we can't use tfrecord ?
", "rawMarkdown": "@philculliton Thanks for sharing the dataset. does that mean for the private test set, we can't use tfrecord ?", "replies": [ { "id": 1046592, "postDate": "2020-10-11T20:11:26.070Z", "content": "You can't create them ahead of time but you can create them and use them in your committed notebook. Might not be faster, could be slower but at least your code structure for training and inference stays the same
", "rawMarkdown": "You can't create them ahead of time but you can create them and use them in your committed notebook. Might not be faster, could be slower but at least your code structure for training and inference stays the same", "votes": 1 }, { "id": 1046596, "postDate": "2020-10-11T20:21:49.993Z", "content": "I see thanks !
", "rawMarkdown": "I see thanks !", "votes": 1 } ] }, { "id": 1047651, "postDate": "2020-10-12T19:53:42.870Z", "content": "Hi @philculliton,
\nI just do not understand this, what's the point of making tf-records not available for private test set?
\nDo you have in mind to release the complete data as if it were dicom?
\nThanks in advance!
I agree. It would be great if private test tfrecords in the same format would be hidden but available in the competition folder on the submission time. Not sure if this is feasible given a deadline, but it would help a lot with the inference…. Otherwise, loading and predicting 230GB images within 9 hours is quite challenging :)
", "rawMarkdown": "I agree. It would be great if private test tfrecords in the same format would be hidden but available in the competition folder on the submission time. Not sure if this is feasible given a deadline, but it would help a lot with the inference.... Otherwise, loading and predicting 230GB images within 9 hours is quite challenging :) ", "votes": 2 }, { "id": 1050347, "postDate": "2020-10-15T09:55:10.550Z", "content": "I think it's because TPU requires Internet to be turned on. And that's not allowed. Otherwise people can hack and get info from the private dataset.
", "rawMarkdown": "I think it's because TPU requires Internet to be turned on. And that's not allowed. Otherwise people can hack and get info from the private dataset. ", "votes": 1 }, { "id": 1051799, "postDate": "2020-10-16T22:24:03.683Z", "content": "@serigne I don't think that's the reason..
\n@philculliton can you answer, please?
\nOne answer is the least that we expect when we asked to competition staff… don't you think?
@serigne is correct. TPUs cannot be enabled on submission. Making TFRecords available was intended to support users in training on TPUs, but those model(s) need to be uploaded as an external data source into a submission notebook whose TPU is disabled. This pipeline is described in more detail on our TPU docs, linked through the Code Requirements for this competition. This is done, as Serigne states, because using TPUs on our current architecture require that internet be enabled, but because this competition has a private test set, we must require internet to be disabled on all submission notebooks, or the private test set can be trivially hacked.
\nAdditionally, we're unable to provide TFRecords on the private test set within the private test folder, because of the sheer size of this dataset already pushing the boundaries of what we can run comfortably within notebook constraints. So we encourage the training set (and validation set) to be used for training purposes, but for those trained models to then be applied against the existing private DICOM test set for inference/prediction. This also aligns with the host's interests in DICOM being more relevant from a radiological standpoint as the test set format.
", "rawMarkdown": "@serigne is correct. TPUs cannot be enabled on submission. Making TFRecords available was intended to support users in training on TPUs, but those model(s) need to be uploaded as an external data source into a submission notebook whose TPU is disabled. This pipeline is described in more detail on our TPU docs, linked through the [Code Requirements](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/overview/code-requirements) for this competition. This is done, as Serigne states, because using TPUs on our current architecture require that internet be enabled, but because this competition has a private test set, we must require internet to be disabled on all submission notebooks, or the private test set can be trivially hacked.\n\nAdditionally, we're unable to provide TFRecords on the private test set within the private test folder, because of the sheer size of this dataset already pushing the boundaries of what we can run comfortably within notebook constraints. So we encourage the training set (and validation set) to be used for training purposes, but for those trained models to then be applied against the existing private DICOM test set for inference/prediction. This also aligns with the host's interests in DICOM being more relevant from a radiological standpoint as the test set format.", "votes": 1 }, { "id": 1054876, "postDate": "2020-10-20T08:50:41.273Z", "content": "@juliaelliott, Thank you very much for your detailed answer.
", "rawMarkdown": "@juliaelliott, Thank you very much for your detailed answer." } ] }, { "id": 1042026, "postDate": "2020-10-08T02:57:45.167Z", "content": "Well then… There goes 20 hours of work! I guess it's good to make the competition more accessible though.
", "rawMarkdown": "Well then... There goes 20 hours of work! I guess it's good to make the competition more accessible though.", "votes": 1 }, { "id": 1041794, "postDate": "2020-10-07T23:12:55.567Z", "content": "Thank you so much for everything, it will save a lot of work … This is great!
", "rawMarkdown": "Thank you so much for everything, it will save a lot of work ... This is great!" }, { "id": 1062046, "postDate": "2020-10-27T14:28:05.950Z", "content": "Thank you for your share
", "rawMarkdown": "Thank you for your share " }, { "id": 1042128, "postDate": "2020-10-08T04:47:43.353Z", "content": "Thank for the amazing share!
", "rawMarkdown": "Thank for the amazing share!" } ], "comments": [ { "id": 1044060, "author_name": "soumya", "author_url": "", "post_date": "2020-10-09T13:32:36.210000", "content": "Trying to train the model on TPU using the tf data, following the process accordingly to the great notebook on TPU
\nhttps://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu
\nIn Cell 6 of the above notebook there is a decoding process where the image array is getting decoded according to the image type, here it's DICOM and tried applying image = tfio.image.decode_dicom_image(image_data, dtype=tf.uint16) , throwing error while running on TPU is there any compatible way to decode of DICOM image array like this way? Not sure if am missing something here.
Hi @soumya5891,
\nTry something like this, replacing decode_image function,
decode_image(image_data):
\nimage = tf.io.decode_raw(image_data, out_type = 'int16')
\nimage = tf.reshape(image, [*IMAGE_SIZE])
\nreturn image
At least for visualization it should work …
", "votes": 1, "replies": [] }, { "id": 1044122, "author_name": "soumya", "author_url": "", "post_date": "2020-10-09T14:34:18.633000", "content": "Thanks @coreacasa for your response, I have tried it previously, the dataset is getting generated, though not checked the visualization part, but during training it's failing , the image dimensions are messed up
", "votes": 1, "replies": [] }, { "id": 1044187, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-10-09T15:24:50.720000", "content": "@soumya5891,
\nYou would have to do a more complete pre-processing but something like that works in training.
decode_image(image_data):
\nimage = tf.io.decode_raw(image_data, out_type = 'int16')
\nimage = tf.repeat(image,3,axis=0)
\nimage = tf.cast(image, tf.float32) / 255.0
\nimage = tf.reshape(image, [*IMAGE_SIZE,3])
\nreturn image
Thanks again @coreacasa , I was just trying to test if training is happening properly with tf records(at least completion of 1 epoch), the snippet you have shared working fine , now can think about other stuffs
", "votes": 1, "replies": [] } ] }, { "id": 1042649, "author_name": "RAHUL SINGH INDA", "author_url": "", "post_date": "2020-10-08T11:33:18.617000", "content": "Hey @philculliton, so in the test set some of the files are hidden right. So can we use these test tfrecords for submission?
\nI mean do these tf-records contains images from those hidden files ?
Hi @rsinda - no, this is only the public set of test files, none of the hidden test files are included.
", "votes": 3, "replies": [] }, { "id": 1046312, "author_name": "Shiro", "author_url": "", "post_date": "2020-10-11T14:42:50.990000", "content": "@philculliton Thanks for sharing the dataset. does that mean for the private test set, we can't use tfrecord ?
", "votes": 0, "replies": [ { "id": 1046592, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-10-11T20:11:26.070000", "content": "You can't create them ahead of time but you can create them and use them in your committed notebook. Might not be faster, could be slower but at least your code structure for training and inference stays the same
", "votes": 1, "replies": [] }, { "id": 1046596, "author_name": "Shiro", "author_url": "", "post_date": "2020-10-11T20:21:49.993000", "content": "I see thanks !
", "votes": 1, "replies": [] } ] }, { "id": 1047651, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-10-12T19:53:42.870000", "content": "Hi @philculliton,
\nI just do not understand this, what's the point of making tf-records not available for private test set?
\nDo you have in mind to release the complete data as if it were dicom?
\nThanks in advance!
I agree. It would be great if private test tfrecords in the same format would be hidden but available in the competition folder on the submission time. Not sure if this is feasible given a deadline, but it would help a lot with the inference…. Otherwise, loading and predicting 230GB images within 9 hours is quite challenging :)
", "votes": 2, "replies": [] }, { "id": 1050347, "author_name": "Serigne ", "author_url": "", "post_date": "2020-10-15T09:55:10.550000", "content": "I think it's because TPU requires Internet to be turned on. And that's not allowed. Otherwise people can hack and get info from the private dataset.
", "votes": 1, "replies": [] }, { "id": 1051799, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-10-16T22:24:03.683000", "content": "@serigne I don't think that's the reason..
\n@philculliton can you answer, please?
\nOne answer is the least that we expect when we asked to competition staff… don't you think?
@serigne is correct. TPUs cannot be enabled on submission. Making TFRecords available was intended to support users in training on TPUs, but those model(s) need to be uploaded as an external data source into a submission notebook whose TPU is disabled. This pipeline is described in more detail on our TPU docs, linked through the Code Requirements for this competition. This is done, as Serigne states, because using TPUs on our current architecture require that internet be enabled, but because this competition has a private test set, we must require internet to be disabled on all submission notebooks, or the private test set can be trivially hacked.
\nAdditionally, we're unable to provide TFRecords on the private test set within the private test folder, because of the sheer size of this dataset already pushing the boundaries of what we can run comfortably within notebook constraints. So we encourage the training set (and validation set) to be used for training purposes, but for those trained models to then be applied against the existing private DICOM test set for inference/prediction. This also aligns with the host's interests in DICOM being more relevant from a radiological standpoint as the test set format.
", "votes": 1, "replies": [] }, { "id": 1054876, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-10-20T08:50:41.273000", "content": "@juliaelliott, Thank you very much for your detailed answer.
", "votes": 0, "replies": [] } ] }, { "id": 1042026, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-10-08T02:57:45.167000", "content": "Well then… There goes 20 hours of work! I guess it's good to make the competition more accessible though.
", "votes": 1, "replies": [] }, { "id": 1041794, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-10-07T23:12:55.567000", "content": "Thank you so much for everything, it will save a lot of work … This is great!
", "votes": 0, "replies": [] }, { "id": 1062046, "author_name": "Demetre Dzmanashvili", "author_url": "", "post_date": "2020-10-27T14:28:05.950000", "content": "Thank you for your share
", "votes": 0, "replies": [] }, { "id": 1042128, "author_name": "Ronaldo S.A. Batista", "author_url": "", "post_date": "2020-10-08T04:47:43.353000", "content": "Thank for the amazing share!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1041691": "Hi all,\n\nWe've created [TFRecords](https://www.kaggle.com/philculliton/rsna-pe-tfrecords-v2) for the public train and test set, and created a [notebook](https://www.kaggle.com/philculliton/rsna-pe-tfrecords-loading) detailing how to load / use them. Given the complexity of the DICOM dataset, the included data is more extensive than is normally included in TFRecord format. Let us know if you have questions or feedback!", "1044060": "Trying to train the model on TPU using the tf data, following the process accordingly to the great notebook on TPU\nhttps://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu\nIn Cell 6 of the above notebook there is a decoding process where the image array is getting decoded according to the image type, here it's DICOM and tried applying image = tfio.image.decode_dicom_image(image_data, dtype=tf.uint16) , throwing error while running on TPU is there any compatible way to decode of DICOM image array like this way? Not sure if am missing something here.", "1042649": "Hey @philculliton, so in the test set some of the files are hidden right. So can we use these test tfrecords for submission?\nI mean do these tf-records contains images from those hidden files ?", "1042026": "Well then... There goes 20 hours of work! I guess it's good to make the competition more accessible though.", "1041794": "Thank you so much for everything, it will save a lot of work ... This is great!", "1062046": "Thank you for your share ", "1042128": "Thank for the amazing share!" } }