Hello everybody!
\nAfter weeks of trying, I still couldn't get the model to actually work out and not just dance around the class frequencies, but I did gain some useful experience and will share it. I know that in theory I'm posting this a little early, but as I wrote above, my models don't learn to classify classes, so it obviously won't affect the problem solving or the allocation of places.
\nSo let's start:
In conclusion, the result is bad, but I learned a lot. I significantly improved my knowledge in working with data, and working with tensorflows. The next serious competition is only with pytorch.
\nAnd I think the classification models are not critical here, this problem requires proper data preprocessing, I look forward to the publication of better solutions.
\nGood luck to everyone!
\nP.S. The obvious ignoring of my letters and messages by the kaggle team when they write that \"Please let me know if you have any questions!\" seemed disrespectful to me. I think that if you do not have the opportunity to answer letters, it is better not to write such a sentence. In general, in the context of correspondence, it looked to me like you do not exist for us. Peace to all!
", "messages": [ { "id": 2478192, "postDate": "2023-10-11T19:54:46.160Z", "content": "Hello everybody!
\nAfter weeks of trying, I still couldn't get the model to actually work out and not just dance around the class frequencies, but I did gain some useful experience and will share it. I know that in theory I'm posting this a little early, but as I wrote above, my models don't learn to classify classes, so it obviously won't affect the problem solving or the allocation of places.
\nSo let's start:
In conclusion, the result is bad, but I learned a lot. I significantly improved my knowledge in working with data, and working with tensorflows. The next serious competition is only with pytorch.
\nAnd I think the classification models are not critical here, this problem requires proper data preprocessing, I look forward to the publication of better solutions.
\nGood luck to everyone!
\nP.S. The obvious ignoring of my letters and messages by the kaggle team when they write that \"Please let me know if you have any questions!\" seemed disrespectful to me. I think that if you do not have the opportunity to answer letters, it is better not to write such a sentence. In general, in the context of correspondence, it looked to me like you do not exist for us. Peace to all!
", "rawMarkdown": "Hello everybody!\nAfter weeks of trying, I still couldn't get the model to actually work out and not just dance around the class frequencies, but I did gain some useful experience and will share it. I know that in theory I'm posting this a little early, but as I wrote above, my models don't learn to classify classes, so it obviously won't affect the problem solving or the allocation of places. \nSo let's start:\n1. As a basis, I used this public notebook: https://www.kaggle.com/code/awsaf49/rsna-atd-cnn-tpu-infer and other notebooks of this branch, special thanks to the author for the work and for the errors.....\n2. I used 2D, 2.5D, 3D models, none of them worked, in the end I came up with the idea of transferring the 3D object to the model, and \"compressing it to 2.5D\". In fact, I compressed the image using something similar to an encoder, but it was immediately followed by the next layers of the model. This is a serious time saver compared to training a 3D model.\n3. Image expansion 256x256x256.\n4. Augmentation was done using volumentations-3D. Great library, but slow, I screwed it up after \"dancing with a tambourine\" to build a dataset for tensorflows. But even on TPU, it seriously slowed down work.\n5. After the tests, I realized that I need to try to highlight the areas with the location of the organs, but it did not help, I need high-quality segmentation.\n6. And today I prepared the model after segmentation, but I got a Submission Scoring Error. I don't know why, everything works on part of the training data and test public data, I don't know why it doesn't work. In a word, debugging on a private dataset is bad (I'm stating the fact, although I understand why it's done this way, because people will tweet as soon as you give them more freedom).\n\nIn conclusion, the result is bad, but I learned a lot. I significantly improved my knowledge in working with data, and working with tensorflows. The next serious competition is only with pytorch.\n\nAnd I think the classification models are not critical here, this problem requires proper data preprocessing, I look forward to the publication of better solutions. \n\nGood luck to everyone!\n\nP.S. The obvious ignoring of my letters and messages by the kaggle team when they write that \"Please let me know if you have any questions!\" seemed disrespectful to me. I think that if you do not have the opportunity to answer letters, it is better not to write such a sentence. In general, in the context of correspondence, it looked to me like you do not exist for us. Peace to all!", "votes": 8 }, { "id": 2479738, "postDate": "2023-10-12T20:32:46.067Z", "content": "is your public score 0.65 from this kind of model?
", "rawMarkdown": "is your public score 0.65 from this kind of model? ", "replies": [ { "id": 2479749, "postDate": "2023-10-12T20:41:24.880Z", "content": "To be precise, with this model I was able to slightly exceed the average values for kidney, liver and spleen. But as I wrote above, the model is meaningless, it does not separate classes, but simply returns values close to their average probabilities
", "rawMarkdown": "To be precise, with this model I was able to slightly exceed the average values for kidney, liver and spleen. But as I wrote above, the model is meaningless, it does not separate classes, but simply returns values close to their average probabilities" } ] }, { "id": 2479270, "postDate": "2023-10-12T14:16:03.477Z", "content": "!pip install volumentations-3D
\nfrom volumentations import *
\n def get_augmentation(patch_size):
\n return Compose([
\n Rotate((-15, 15), (0, 0), (0, 0), p=0.5),
\n #RandomCropFromBorders(crop_value=0.1, p=0.5),
\n ElasticTransform((0, 0.25), interpolation=2, p=0.1),
\n Resize(patch_size, interpolation=1, resize_type=0, always_apply=True, p=1.0),
\n #ColorJitter(),
\n Flip(0, p=0.5),
\n Flip(1, p=0.5),
\n Flip(2, p=0.5),
\n RandomRotate90((1, 2), p=0.5),
\n GaussianNoise(var_limit=(0, 5), p=0.2),
\n #RandomGamma(gamma_limit=(80, 120), p=0.2),
\n ], p=1.0)
\n volume3D = get_augmentation((128, 128, 128))
\n def volume3Dfn(image):
\n aug_data = volume3D(**{\"image\":image})
\n return tf.cast(aug_data[\"image\"], tf.float32)
\n def build_augmenter(with_labels=True, dim=CFG.img_size):
\n def augment(img, dim=dim):
\n # Wraps a python function and uses it as a TensorFlow op.
\n img_shape=img.shape
\n img = tf.numpy_function(func=volume3Dfn,
\n inp=[img],
\n Tout=tf.float32)
\n img.set_shape((img_shape[0], img_shape[1], img_shape[2], 1))
\n img = tf.reshape(img, [img_shape[0], img_shape[1], img_shape[2]])
\n return img
\n def augment_with_labels(img, label):
\n return augment(img), label
\n return augment_with_labels if with_labels else augment
\nThis is a sample code of how I use volumentations-3D. Note that this library runs on numpy, so you can use it in any pipeline.
\nAlso, for some reason RandomGamma does not work on TPU, it works without problems on CPU. ColorJitter sometimes generates black images, obviously you need to adjust the parameters there.
Resize and elastic transform are the bottleneck here. You should resize and save data to disk for faster training.
", "rawMarkdown": "Resize and elastic transform are the bottleneck here. You should resize and save data to disk for faster training.", "votes": 1, "replies": [ { "id": 2479408, "postDate": "2023-10-12T15:52:15.167Z", "content": "Yes, this is exactly what I do in preprocessing, after loading the dcm into numpy I compress the 3D array to 256,256,256 (for segmentation I use 128,128,128 for each organ). Here I am forced to use Resize due to the use of RandomCropFromBorders, as I understand this method resizes the image somewhat, by the way if you don't use RandomCropFromBorders you don't need to use Resize
", "rawMarkdown": "Yes, this is exactly what I do in preprocessing, after loading the dcm into numpy I compress the 3D array to 256,256,256 (for segmentation I use 128,128,128 for each organ). Here I am forced to use Resize due to the use of RandomCropFromBorders, as I understand this method resizes the image somewhat, by the way if you don't use RandomCropFromBorders you don't need to use Resize" } ] } ] } ], "comments": [ { "id": 2479738, "author_name": "Michalina Hulak", "author_url": "", "post_date": "2023-10-12T20:32:46.067000", "content": "is your public score 0.65 from this kind of model?
", "votes": 0, "replies": [ { "id": 2479749, "author_name": "Andrij", "author_url": "", "post_date": "2023-10-12T20:41:24.880000", "content": "To be precise, with this model I was able to slightly exceed the average values for kidney, liver and spleen. But as I wrote above, the model is meaningless, it does not separate classes, but simply returns values close to their average probabilities
", "votes": 0, "replies": [] } ] }, { "id": 2479270, "author_name": "Andrij", "author_url": "", "post_date": "2023-10-12T14:16:03.477000", "content": "!pip install volumentations-3D
\nfrom volumentations import *
\n def get_augmentation(patch_size):
\n return Compose([
\n Rotate((-15, 15), (0, 0), (0, 0), p=0.5),
\n #RandomCropFromBorders(crop_value=0.1, p=0.5),
\n ElasticTransform((0, 0.25), interpolation=2, p=0.1),
\n Resize(patch_size, interpolation=1, resize_type=0, always_apply=True, p=1.0),
\n #ColorJitter(),
\n Flip(0, p=0.5),
\n Flip(1, p=0.5),
\n Flip(2, p=0.5),
\n RandomRotate90((1, 2), p=0.5),
\n GaussianNoise(var_limit=(0, 5), p=0.2),
\n #RandomGamma(gamma_limit=(80, 120), p=0.2),
\n ], p=1.0)
\n volume3D = get_augmentation((128, 128, 128))
\n def volume3Dfn(image):
\n aug_data = volume3D(**{\"image\":image})
\n return tf.cast(aug_data[\"image\"], tf.float32)
\n def build_augmenter(with_labels=True, dim=CFG.img_size):
\n def augment(img, dim=dim):
\n # Wraps a python function and uses it as a TensorFlow op.
\n img_shape=img.shape
\n img = tf.numpy_function(func=volume3Dfn,
\n inp=[img],
\n Tout=tf.float32)
\n img.set_shape((img_shape[0], img_shape[1], img_shape[2], 1))
\n img = tf.reshape(img, [img_shape[0], img_shape[1], img_shape[2]])
\n return img
\n def augment_with_labels(img, label):
\n return augment(img), label
\n return augment_with_labels if with_labels else augment
\nThis is a sample code of how I use volumentations-3D. Note that this library runs on numpy, so you can use it in any pipeline.
\nAlso, for some reason RandomGamma does not work on TPU, it works without problems on CPU. ColorJitter sometimes generates black images, obviously you need to adjust the parameters there.
Resize and elastic transform are the bottleneck here. You should resize and save data to disk for faster training.
", "votes": 1, "replies": [ { "id": 2479408, "author_name": "Andrij", "author_url": "", "post_date": "2023-10-12T15:52:15.167000", "content": "Yes, this is exactly what I do in preprocessing, after loading the dcm into numpy I compress the 3D array to 256,256,256 (for segmentation I use 128,128,128 for each organ). Here I am forced to use Resize due to the use of RandomCropFromBorders, as I understand this method resizes the image somewhat, by the way if you don't use RandomCropFromBorders you don't need to use Resize
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2478192": "Hello everybody!\nAfter weeks of trying, I still couldn't get the model to actually work out and not just dance around the class frequencies, but I did gain some useful experience and will share it. I know that in theory I'm posting this a little early, but as I wrote above, my models don't learn to classify classes, so it obviously won't affect the problem solving or the allocation of places. \nSo let's start:\n1. As a basis, I used this public notebook: https://www.kaggle.com/code/awsaf49/rsna-atd-cnn-tpu-infer and other notebooks of this branch, special thanks to the author for the work and for the errors.....\n2. I used 2D, 2.5D, 3D models, none of them worked, in the end I came up with the idea of transferring the 3D object to the model, and \"compressing it to 2.5D\". In fact, I compressed the image using something similar to an encoder, but it was immediately followed by the next layers of the model. This is a serious time saver compared to training a 3D model.\n3. Image expansion 256x256x256.\n4. Augmentation was done using volumentations-3D. Great library, but slow, I screwed it up after \"dancing with a tambourine\" to build a dataset for tensorflows. But even on TPU, it seriously slowed down work.\n5. After the tests, I realized that I need to try to highlight the areas with the location of the organs, but it did not help, I need high-quality segmentation.\n6. And today I prepared the model after segmentation, but I got a Submission Scoring Error. I don't know why, everything works on part of the training data and test public data, I don't know why it doesn't work. In a word, debugging on a private dataset is bad (I'm stating the fact, although I understand why it's done this way, because people will tweet as soon as you give them more freedom).\n\nIn conclusion, the result is bad, but I learned a lot. I significantly improved my knowledge in working with data, and working with tensorflows. The next serious competition is only with pytorch.\n\nAnd I think the classification models are not critical here, this problem requires proper data preprocessing, I look forward to the publication of better solutions. \n\nGood luck to everyone!\n\nP.S. The obvious ignoring of my letters and messages by the kaggle team when they write that \"Please let me know if you have any questions!\" seemed disrespectful to me. I think that if you do not have the opportunity to answer letters, it is better not to write such a sentence. In general, in the context of correspondence, it looked to me like you do not exist for us. Peace to all!", "2479738": "is your public score 0.65 from this kind of model? ", "2479270": " !pip install volumentations-3D\n from volumentations import *\n\n\n def get_augmentation(patch_size):\n return Compose([\n Rotate((-15, 15), (0, 0), (0, 0), p=0.5),\n #RandomCropFromBorders(crop_value=0.1, p=0.5),\n ElasticTransform((0, 0.25), interpolation=2, p=0.1),\n Resize(patch_size, interpolation=1, resize_type=0, always_apply=True, p=1.0),\n #ColorJitter(),\n Flip(0, p=0.5),\n Flip(1, p=0.5),\n Flip(2, p=0.5),\n RandomRotate90((1, 2), p=0.5),\n GaussianNoise(var_limit=(0, 5), p=0.2),\n #RandomGamma(gamma_limit=(80, 120), p=0.2),\n ], p=1.0)\n\n volume3D = get_augmentation((128, 128, 128))\n\n def volume3Dfn(image): \n aug_data = volume3D(**{\"image\":image})\n return tf.cast(aug_data[\"image\"], tf.float32)\n\n def build_augmenter(with_labels=True, dim=CFG.img_size):\n\n\n def augment(img, dim=dim):\n # Wraps a python function and uses it as a TensorFlow op.\n img_shape=img.shape\n img = tf.numpy_function(func=volume3Dfn, \n inp=[img], \n Tout=tf.float32)\n img.set_shape((img_shape[0], img_shape[1], img_shape[2], 1))\n img = tf.reshape(img, [img_shape[0], img_shape[1], img_shape[2]])\n return img\n\n def augment_with_labels(img, label): \n return augment(img), label\n\n return augment_with_labels if with_labels else augment\n\nThis is a sample code of how I use volumentations-3D. Note that this library runs on numpy, so you can use it in any pipeline.\nAlso, for some reason RandomGamma does not work on TPU, it works without problems on CPU. ColorJitter sometimes generates black images, obviously you need to adjust the parameters there." } }