![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3149580%2Fdc8e06ded7852b66b596cbf210d9d52d%2Ffilter_pollution.png?generation=1593913045180801&alt=media\")
As pointed out before, the methods for extracting information from the biopsy images are important. Currently I'm doing some \"simple\"/straight forward tiling (similar to this and this). Gets me to 0.89 LB with a single EfficientNetB0/B1 (using level 1 resolution images). I've also experimented with some more complex tiling, which got me to 0.86 LB (using 960x960 'big images' in comparison with 1536x1536 'big images' of the 0.89 LB submission). I've shared a variant of the tiling method of the 0.86 LB implementation here, to stimulate the discussion a bit :-). Feel free to share me all of your secrets on how to extract information from the biopsy images.
", "messages": [ { "id": 892340, "postDate": "2020-06-18T20:11:14.620Z", "content": "As pointed out before, the methods for extracting information from the biopsy images are important. Currently I'm doing some \"simple\"/straight forward tiling (similar to this and this). Gets me to 0.89 LB with a single EfficientNetB0/B1 (using level 1 resolution images). I've also experimented with some more complex tiling, which got me to 0.86 LB (using 960x960 'big images' in comparison with 1536x1536 'big images' of the 0.89 LB submission). I've shared a variant of the tiling method of the 0.86 LB implementation here, to stimulate the discussion a bit :-). Feel free to share me all of your secrets on how to extract information from the biopsy images.
", "rawMarkdown": "As pointed out before, the methods for extracting information from the biopsy images are important. Currently I'm doing some \"simple\"/straight forward tiling (similar to [this](https://www.kaggle.com/haqishen/train-efficientnet-b0-w-36-tiles-256-lb0-87) and [this](https://www.kaggle.com/iafoss/panda-concat-tile-pooling-starter-0-79-lb)). Gets me to 0.89 LB with a single EfficientNetB0/B1 (using level 1 resolution images). I've also experimented with some more complex tiling, which got me to 0.86 LB (using 960x960 'big images' in comparison with 1536x1536 'big images' of the 0.89 LB submission). I've shared a variant of the tiling method of the 0.86 LB implementation [here](https://www.kaggle.com/akensert/panda-optimized-tiling-tf-data-dataset), to stimulate the discussion a bit :-). Feel free to share me *all of your secrets* on how to extract information from the biopsy images.", "votes": 28 }, { "id": 914994, "postDate": "2020-07-04T12:08:37.747Z", "content": "I implement a method to filter some tiles which are polluted by green/blue(the color observed by my eyes) ink, and random pick some images, find the result is better than just sort by pixel-sum.\nMy model is runing, if I can boost >= 0.01, I will hint about the filter method here😄 \n*UPDATE:*The model is not so good as what I expect in first 10 epochs... I show the picture below is the tiles after my filter method, it surely filter some polluted tiles, but the variety of tiles decline... original argsort means iafoss's method\nThe image-id from top to bottom is \n'4a3e6e77b42a7e8f8b2dec472667bdbe',\n'51ea992db967a0b925588ee2eee322ab',\n'42482ed25f9001ae44f6148432da91d7',\n'f114a15f4c02900ebed9f638b68f42cb',\n'22c2e58ea58b90066e98d86bfb8a0baf',
\n\n
I also implement this method but I notice that is not 100% of the biopsy. So how can I get the representative tiles for the different categories? The difference is that I'm trying to categorize the gleason score instead of the ISUP grade
", "rawMarkdown": "I also implement this method but I notice that is not 100% of the biopsy. So how can I get the representative tiles for the different categories? The difference is that I'm trying to categorize the gleason score instead of the ISUP grade" } ] }, { "id": 894373, "postDate": "2020-06-20T11:28:47.110Z", "content": "I use a NMS-type approach for selecting tiles. It gives a bit of an improvement over the simple approach, but nothing ground breaking. Probably in the order of 0.005 improvement on CV/LB.
", "rawMarkdown": "I use a NMS-type approach for selecting tiles. It gives a bit of an improvement over the simple approach, but nothing ground breaking. Probably in the order of 0.005 improvement on CV/LB.", "votes": 4, "replies": [ { "id": 894387, "postDate": "2020-06-20T11:43:22.390Z", "content": "I think ~0.005 is a hurge improvement. And the NMS you mentioned is refer to Non Maximum Suppression? Thanks in advance.
", "rawMarkdown": "I think ~0.005 is a hurge improvement. And the NMS you mentioned is refer to Non Maximum Suppression? Thanks in advance." }, { "id": 894392, "postDate": "2020-06-20T11:49:29.977Z", "content": "Yep, non-maximum suppression
", "rawMarkdown": "Yep, non-maximum suppression", "votes": 1 }, { "id": 894395, "postDate": "2020-06-20T11:51:01.903Z", "content": "OK, and thanks for your fast reply. :D
", "rawMarkdown": "OK, and thanks for your fast reply. :D", "votes": 1 }, { "id": 899033, "postDate": "2020-06-23T22:58:21.587Z", "content": "Anyone care to elaborate on how NMS works this competition?
", "rawMarkdown": "Anyone care to elaborate on how NMS works this competition?", "votes": 2 }, { "id": 899967, "postDate": "2020-06-24T14:52:06.950Z", "content": "Think of your proposal boxes as tiles and the information in those tiles as scores...
", "rawMarkdown": "Think of your proposal boxes as tiles and the information in those tiles as scores...", "votes": 1 }, { "id": 910543, "postDate": "2020-07-01T07:48:37.917Z", "content": "How to calculate the information score of a tile?
", "rawMarkdown": "How to calculate the information score of a tile? " }, { "id": 910559, "postDate": "2020-07-01T08:06:36.660Z", "content": "In the simplest case it's just the level of darkness of the tile. You could be a bit more intelligent about it, (maybe something useful here), but I'm not sure if there is that much value in the additional complexity.
", "rawMarkdown": "In the simplest case it's just the level of darkness of the tile. You could be a bit more intelligent about it, (maybe something useful [here](https://developer.ibm.com/technologies/data-science/articles/an-automatic-method-to-identify-tissues-from-big-whole-slide-images-pt1/)), but I'm not sure if there is that much value in the additional complexity.", "votes": 2 } ] }, { "id": 892458, "postDate": "2020-06-18T22:36:18.780Z", "content": "A quick question...\nYou got 0.86 with 36 patches from this method?
", "rawMarkdown": "A quick question...\nYou got 0.86 with 36 patches from this method?", "votes": 1, "replies": [ { "id": 892798, "postDate": "2020-06-19T06:59:31.593Z", "content": "Yes 160 patch size, 6 x 6. But in the implementation that got 0.86, I did not use any _excess_coords_filtering() I just randomly shuffled all coords that were obtained for each image (and took the first 36 from that array/list). Further, I also used heavy augmentation on the big image.
I wonder if your advanced tiling method is not skipping too much tissue. Are you using the same selection in both cases ? \nI think the task and discussion needs to be divided in two subtask:\n* The tiling. Making the list of patches.\n=> Simple method just uses a simple grid.\n* The selection. Which ones to send to the model.\n=> Simple method finds the darker ones on average.
", "rawMarkdown": "I wonder if your advanced tiling method is not skipping too much tissue. Are you using the same selection in both cases ? \nI think the task and discussion needs to be divided in two subtask:\n* The tiling. Making the list of patches.\n=> Simple method just uses a simple grid.\n* The selection. Which ones to send to the model.\n=> Simple method finds the darker ones on average.\n\n", "votes": 2, "replies": [ { "id": 893779, "postDate": "2020-06-19T22:03:27.227Z", "content": "To be honest from the notebook it looks like it's getting pretty much all the tissue. I thought the problem was that the tiles are too small. Imagine having tiny 10x10 tiles and getting all the tissue, the model would likely perform pretty bad because you'd lose too much spatial information. You would get all the tissue but would be destroying all the patterns by breaking it into \"meaningless\" tiles
", "rawMarkdown": "To be honest from the notebook it looks like it's getting pretty much all the tissue. I thought the problem was that the tiles are too small. Imagine having tiny 10x10 tiles and getting all the tissue, the model would likely perform pretty bad because you'd lose too much spatial information. You would get all the tissue but would be destroying all the patterns by breaking it into \"meaningless\" tiles" }, { "id": 893794, "postDate": "2020-06-19T22:50:56.920Z", "content": "Then just increase the patch size. \nLike as 128x128 for Level 2, Then 512x512 for Level 1. Because Level 1 image is 4 times image from Level 2......
", "rawMarkdown": "Then just increase the patch size. \nLike as 128x128 for Level 2, Then 512x512 for Level 1. Because Level 1 image is 4 times image from Level 2......", "votes": 1 }, { "id": 893800, "postDate": "2020-06-19T23:32:26.633Z", "content": "If you generate 16 tiles (as for level 2) and each one is 512x512, the total image will be 2048x2048. You'd run out of memory very quickly
", "rawMarkdown": "If you generate 16 tiles (as for level 2) and each one is 512x512, the total image will be 2048x2048. You'd run out of memory very quickly" }, { "id": 893807, "postDate": "2020-06-20T00:06:31.260Z", "content": "That is the issue. I don't even have local GPU to train 512x512.\nAnd kaggle have limited resources...\nSo I am trying to set this up in TPU by creating tfrecords of 2048x2048.\nSo we wont have memory issues.
", "rawMarkdown": "That is the issue. I don't even have local GPU to train 512x512.\nAnd kaggle have limited resources...\nSo I am trying to set this up in TPU by creating tfrecords of 2048x2048.\nSo we wont have memory issues." }, { "id": 894322, "postDate": "2020-06-20T10:36:14.450Z", "content": "Good points, thanks.
\n\nYes this problem could indeed be divided into those two categories. The current kernel is computing the tiles relatively OK I would say, but the selection of the 36 tiles that would go in to the neural net are likely far from optimal. I also recommend changing the parameters a bit and visualize it for yourselves, perhaps even change the code a bit if you have too much free-time ;-). From my experience visualizing this, the algorithm works well when the tissue is big enough and aligned nicely (not L-shaped etc.). Further, one has to be careful with the small/tiny tissues. So try to look at both small, medium and large sized tissues when visualizing the output/results, likewise for your own algorithms that you use.
", "rawMarkdown": "Good points, thanks. \n\nYes this problem could indeed be divided into those two categories. The current kernel is computing the tiles relatively OK I would say, but the selection of the 36 tiles that would go in to the neural net are likely far from optimal. I also recommend changing the parameters a bit and visualize it for yourselves, perhaps even change the code a bit if you have too much free-time ;-). From my experience visualizing this, the algorithm works well when the tissue is big enough and aligned nicely (not L-shaped etc.). Further, one has to be careful with the small/tiny tissues. So try to look at both small, medium and large sized tissues when visualizing the output/results, likewise for your own algorithms that you use." }, { "id": 897088, "postDate": "2020-06-22T15:56:10.177Z", "content": "@arroqc you mentionned a few days ago a new approach to tilling on the other thread, but isn't \" Simple method finds the darker ones on average.\" what Lafoss approach has been doing since the beginning ?
\n\nOn my part, I've been working with selecting the tiles based not on the slide, but on the mask. It didn't work well, and there is also the problem that for inference you need to come up with a segmentation model ^^
\n\nI've also tried to \"center\" the tiles (as very often the long thin line is split into two half empty tiles). My code produces the indended effect (I do get better tiles) but no improvement on CV -_-
", "rawMarkdown": "@arroqc you mentionned a few days ago a new approach to tilling on the other thread, but isn't \" Simple method finds the darker ones on average.\" what Lafoss approach has been doing since the beginning ?\n\nOn my part, I've been working with selecting the tiles based not on the slide, but on the mask. It didn't work well, and there is also the problem that for inference you need to come up with a segmentation model ^^\n\nI've also tried to \"center\" the tiles (as very often the long thin line is split into two half empty tiles). My code produces the indended effect (I do get better tiles) but no improvement on CV -_-" }, { "id": 897488, "postDate": "2020-06-22T22:22:45.743Z", "content": "@bdubreu What I use is model based but while the initial results were really good for low tile counts (9 and 16) it doesn't give me much benefit for high tile counts (despite better CV). At least I can try faster with these smaller samples... But tbh I'm getting frustrated by the bad correlation between cv and test.
", "rawMarkdown": "@bdubreu What I use is model based but while the initial results were really good for low tile counts (9 and 16) it doesn't give me much benefit for high tile counts (despite better CV). At least I can try faster with these smaller samples... But tbh I'm getting frustrated by the bad correlation between cv and test." }, { "id": 898001, "postDate": "2020-06-23T08:10:28.033Z", "content": "There's something else. Slides that have a high isup grade can contain connective tissue or healthy tissue and in the contrary begnin slides can contain cancer tissues. So we should focus on extracting tiles with cells and glands that are representative to isup grade.\nProbably when selecting more than 16 tiles with your method then you include some non-necessary tissues which skew your model. It' just a supposition, what are your thoughts about this @arroqc ?
", "rawMarkdown": "There's something else. Slides that have a high isup grade can contain connective tissue or healthy tissue and in the contrary begnin slides can contain cancer tissues. So we should focus on extracting tiles with cells and glands that are representative to isup grade.\nProbably when selecting more than 16 tiles with your method then you include some non-necessary tissues which skew your model. It' just a supposition, what are your thoughts about this @arroqc ?" }, { "id": 898526, "postDate": "2020-06-23T15:13:09.810Z", "content": "It's indeed a possibility and my model is skewed by that kind of noise. I guess I can test further if the small changes I also added are actually decreasing performance. What I find really weird is that everything is much better with lower loss and higher kappa but then the performance really degrade on LB. I don't see how there could be data leakage too :/. Anyway I prefer to keep working with 16 since I can at least fit batches of 9 in my gtx 1080.
", "rawMarkdown": "It's indeed a possibility and my model is skewed by that kind of noise. I guess I can test further if the small changes I also added are actually decreasing performance. What I find really weird is that everything is much better with lower loss and higher kappa but then the performance really degrade on LB. I don't see how there could be data leakage too :/. Anyway I prefer to keep working with 16 since I can at least fit batches of 9 in my gtx 1080.", "votes": 1 }, { "id": 898925, "postDate": "2020-06-23T20:55:35.093Z", "content": "16 of size 256x256 or 512x512?
", "rawMarkdown": "16 of size 256x256 or 512x512?" }, { "id": 898996, "postDate": "2020-06-23T21:59:27.217Z", "content": "256
", "rawMarkdown": "256" } ] }, { "id": 892496, "postDate": "2020-06-18T23:41:54.147Z", "content": "Did you try to make 1536x1536 images with the more complex method?
", "rawMarkdown": "Did you try to make 1536x1536 images with the more complex method?", "votes": 2, "replies": [ { "id": 892791, "postDate": "2020-06-19T06:55:03.960Z", "content": "Good question. I actually tried it overnight, and I didn't reach the same score as the simple method. So I would say that the simple method works better, somehow. Perhaps the complex method looks good visually (to some extent), but the performance is a bit worse. But then there might be some adjustments that could be made to improve the the complex method.
", "rawMarkdown": "Good question. I actually tried it overnight, and I didn't reach the same score as the simple method. So I would say that the simple method works better, somehow. Perhaps the complex method looks good visually (to some extent), but the performance is a bit worse. But then there might be some adjustments that could be made to improve the the complex method.", "votes": 1 }, { "id": 893232, "postDate": "2020-06-19T13:26:58.437Z", "content": "Did you keep same patch size as in simple tiling and this method?
", "rawMarkdown": "Did you keep same patch size as in simple tiling and this method?" } ] }, { "id": 923008, "postDate": "2020-07-10T13:30:09.460Z", "content": "Hey @akensert, \nAfter implementing everything you suggested and some other ideas, when i finally got to training, this happened. \nCould you please look into this? \nThanks!!
", "rawMarkdown": "Hey @akensert, \nAfter implementing everything you suggested and some other ideas, when i finally got to training, [this](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/165645) happened. \nCould you please look into this? \nThanks!! \n" }, { "id": 919982, "postDate": "2020-07-08T08:29:44.753Z", "content": "Hey @akensert, really nice work man. I was trying to experiment with your kernel and what i notice is that it takes way too much time for a single epoch on Kaggle GPU. I wanted to ask, did you use your private rig or kaggle kernel itself. Also, when i tried using TPU with your tf.data.Dataset (after incorporating the targets), TPUs don't really work with py_function. BTW, how much time did it take for you to train your model?. Any ideas or help would be much appreciated.
", "rawMarkdown": "Hey @akensert, really nice work man. I was trying to experiment with your kernel and what i notice is that it takes way too much time for a single epoch on Kaggle GPU. I wanted to ask, did you use your private rig or kaggle kernel itself. Also, when i tried using TPU with your tf.data.Dataset (after incorporating the targets), TPUs don't really work with py_function. BTW, how much time did it take for you to train your model?. Any ideas or help would be much appreciated.", "replies": [ { "id": 920061, "postDate": "2020-07-08T09:30:20.537Z", "content": "Hello @nikhilbartwal001. Yes, you would need to precompute the coordinates if you're running in the Kaggle kernels. (With >=6 CPU cores (12 threads), it would be possible to compute the coordinates on the fly, but still not ideal.). I suggest you precompute the coordinates for each example, add the coordinates (array of arrays) to a column in your pandas dataframe, and then you can input this together with the image + label (so your input would be, per example, (image, label, coords). I think that will speed it up the input pipeline significantly (especially when working the Kaggle kernels).
About TPU: Yes you are right, it doesn't seem to work with tf.py_function. You need to avoid that :-(
I guess for me, locally, it would take around 10-30 minutes per epoch using the same pipeline as the public kernel. And it depends on the size of the input (I usually vary it between 1024x1024 and 1536x1536).
\n\nSide note: Currently I use the simple approach for computing tiles/patches. Similar to the other public kernels. I haven't had much time/energy to continue experiment with the one I published.
", "rawMarkdown": "Hello @nikhilbartwal001. Yes, you would need to precompute the coordinates if you're running in the Kaggle kernels. (With >=6 CPU cores (12 threads), it would be possible to compute the coordinates on the fly, but still not ideal.). I suggest you precompute the coordinates for each example, add the coordinates (array of arrays) to a column in your pandas dataframe, and then you can input this together with the image + label (so your input would be, per example, `(image, label, coords)`. I think that will speed it up the input pipeline significantly (especially when working the Kaggle kernels).\n\nAbout TPU: Yes you are right, it doesn't seem to work with `tf.py_function`. You need to avoid that :-(\n\nI guess for me, locally, it would take around 10-30 minutes per epoch using the same pipeline as the public kernel. And it depends on the size of the input (I usually vary it between 1024x1024 and 1536x1536). \n\nSide note: Currently I use the simple approach for computing tiles/patches. Similar to the other public kernels. I haven't had much time/energy to continue experiment with the one I published.\n\n", "votes": 1 }, { "id": 920069, "postDate": "2020-07-08T09:48:40.603Z", "content": "Thanks for such a quick and insightful reply @akensert. I had some doubts which I thought you might help me clear.\nCan you tell me more about the 'simple' approach that you are talking about as I'm new to image processing and yours was one of the most detailed kernels written in tensorflow.\nAlso have you tried using TPU (with/without tfrec format) in this competition? \nAnother thing that I'm confused about is the choice of loss function and the evaluation metric as people seem to be using classification , regression, ordinal regression etc. \nIt would be really help if you could help me out.\nThanks !!
", "rawMarkdown": "Thanks for such a quick and insightful reply @akensert. I had some doubts which I thought you might help me clear.\nCan you tell me more about the 'simple' approach that you are talking about as I'm new to image processing and yours was one of the most detailed kernels written in tensorflow.\nAlso have you tried using TPU (with/without tfrec format) in this competition? \nAnother thing that I'm confused about is the choice of loss function and the evaluation metric as people seem to be using classification , regression, ordinal regression etc. \nIt would be really help if you could help me out.\nThanks !!", "votes": 1 }, { "id": 920249, "postDate": "2020-07-08T13:05:50.640Z", "content": "@nikhilbartwal001 No problem :-)
\n\nAbout the simple approach, see here, although it's in PyTorch. I rewrote it for TensorFlow (with some small modifications).
\n\nI haven't used TPU yet, however I think there should be some nice TPU + TF2 kernels around to get one started.
\n\nLoss function: I suggest either mean squared error (so you would keep your target as it is in the csv file, or ordinal regression (target = 3 would be [1, 1, 1, 0, 0] etc.). Both works equally well for me. Both makes sense because the target data have some order.
", "rawMarkdown": "@nikhilbartwal001 No problem :-)\n\nAbout the simple approach, [see here](https://www.kaggle.com/haqishen/train-efficientnet-b0-w-36-tiles-256-lb0-87), although it's in PyTorch. I rewrote it for TensorFlow (with some small modifications). \n\nI haven't used TPU yet, however I think there should be some nice TPU + TF2 kernels around to get one started. \n\nLoss function: I suggest either mean squared error (so you would keep your target as it is in the csv file, or ordinal regression (target = 3 would be [1, 1, 1, 0, 0] etc.). Both works equally well for me. Both makes sense because the target data have some order.", "votes": 1 }, { "id": 920275, "postDate": "2020-07-08T13:20:14.223Z", "content": "Hey @akensert, just one more thing that i had, I've read about QWK for the first time in this competition.\nThere is a QWK metric in sklearn as well as in tf addons, do you think it would work or would I need to write a custom metric for the sane? I don't quite get how to optimize qwk during training.
\n\nBTW, you earlier suggested pre computing the coordinates, I was thinking what if I create all the patched images before hand and store it in the output directory and then train the model in a different kernel by just reading and converting the patched image to numpy array and feeding it to the input pipeline. What are your thoughts?\nThanks for the help mate !
", "rawMarkdown": "Hey @akensert, just one more thing that i had, I've read about QWK for the first time in this competition.\nThere is a QWK metric in sklearn as well as in tf addons, do you think it would work or would I need to write a custom metric for the sane? I don't quite get how to optimize qwk during training.\n\nBTW, you earlier suggested pre computing the coordinates, I was thinking what if I create all the patched images before hand and store it in the output directory and then train the model in a different kernel by just reading and converting the patched image to numpy array and feeding it to the input pipeline. What are your thoughts?\nThanks for the help mate !" }, { "id": 921684, "postDate": "2020-07-09T13:29:03.873Z", "content": "@nikhilbartwal001 Your loss function will be MSE or BCE (or CCE). So you will not optimize for QWK directly (although you could try, but I figure it might be difficult), but rather for MSE/BCE/CCE.
\n\nSure you could do that (even faster then), but then you wouldn't have the patch-level augmentation (the random rotate/flip/transposing/(selection) of each patch before being concatenated to the 'big image'), which could be nice to have. Although you will have the 'image-level' augmentation :-)
\n\nGood luck!
", "rawMarkdown": "@nikhilbartwal001 Your loss function will be MSE or BCE (or CCE). So you will not optimize for QWK directly (although you could try, but I figure it might be difficult), but rather for MSE/BCE/CCE.\n\nSure you could do that (even faster then), but then you wouldn't have the patch-level augmentation (the random rotate/flip/transposing/(selection) of each patch before being concatenated to the 'big image'), which could be nice to have. Although you will have the 'image-level' augmentation :-)\n\nGood luck!", "votes": 1 }, { "id": 921728, "postDate": "2020-07-09T14:00:48.790Z", "content": "Hey @akensert, thanks for the help with the loss function.
\n\nAlthough i didn't understand what you meant that i won't be able to have patch level augs. Isn't it possible to to do the patch level augs before storing the patched images? I'm currently not that deep into image processing so idk about that.
", "rawMarkdown": "Hey @akensert, thanks for the help with the loss function. \n\nAlthough i didn't understand what you meant that i won't be able to have patch level augs. Isn't it possible to to do the patch level augs before storing the patched images? I'm currently not that deep into image processing so idk about that. " }, { "id": 921829, "postDate": "2020-07-09T15:31:48.380Z", "content": "@nikhilbartwal001 Sure, you could do that, but then your dataset which you're saving to disk will be huge :-) let's say that you wanna train for at least 10 epochs, then \"ideally\" you would want to have 10 versions of each instance (or 'big image'). So over 100k images saved to disk.
", "rawMarkdown": "@nikhilbartwal001 Sure, you could do that, but then your dataset which you're saving to disk will be huge :-) let's say that you wanna train for at least 10 epochs, then \"ideally\" you would want to have 10 versions of each instance (or 'big image'). So over 100k images saved to disk. " }, { "id": 922490, "postDate": "2020-07-10T06:12:52.833Z", "content": "Woah I definitely wouldn't want that:(\nI guess I'll try experimenting with the precomputed coords only.\nThanks for the help dude!!
", "rawMarkdown": "Woah I definitely wouldn't want that:(\nI guess I'll try experimenting with the precomputed coords only.\nThanks for the help dude!!" } ] } ], "comments": [ { "id": 914994, "author_name": "Shiyuan Zeng", "author_url": "", "post_date": "2020-07-04T12:08:37.747000", "content": "I implement a method to filter some tiles which are polluted by green/blue(the color observed by my eyes) ink, and random pick some images, find the result is better than just sort by pixel-sum.\nMy model is runing, if I can boost >= 0.01, I will hint about the filter method here😄 \n*UPDATE:*The model is not so good as what I expect in first 10 epochs... I show the picture below is the tiles after my filter method, it surely filter some polluted tiles, but the variety of tiles decline... original argsort means iafoss's method\nThe image-id from top to bottom is \n'4a3e6e77b42a7e8f8b2dec472667bdbe',\n'51ea992db967a0b925588ee2eee322ab',\n'42482ed25f9001ae44f6148432da91d7',\n'f114a15f4c02900ebed9f638b68f42cb',\n'22c2e58ea58b90066e98d86bfb8a0baf',
\n\n
I also implement this method but I notice that is not 100% of the biopsy. So how can I get the representative tiles for the different categories? The difference is that I'm trying to categorize the gleason score instead of the ISUP grade
", "votes": 0, "replies": [] } ] }, { "id": 894373, "author_name": "fergusoci", "author_url": "", "post_date": "2020-06-20T11:28:47.110000", "content": "I use a NMS-type approach for selecting tiles. It gives a bit of an improvement over the simple approach, but nothing ground breaking. Probably in the order of 0.005 improvement on CV/LB.
", "votes": 4, "replies": [ { "id": 894387, "author_name": "Shiyuan Zeng", "author_url": "", "post_date": "2020-06-20T11:43:22.390000", "content": "I think ~0.005 is a hurge improvement. And the NMS you mentioned is refer to Non Maximum Suppression? Thanks in advance.
", "votes": 0, "replies": [] }, { "id": 894392, "author_name": "fergusoci", "author_url": "", "post_date": "2020-06-20T11:49:29.977000", "content": "Yep, non-maximum suppression
", "votes": 1, "replies": [] }, { "id": 894395, "author_name": "Shiyuan Zeng", "author_url": "", "post_date": "2020-06-20T11:51:01.903000", "content": "OK, and thanks for your fast reply. :D
", "votes": 1, "replies": [] }, { "id": 899033, "author_name": "Shujun", "author_url": "", "post_date": "2020-06-23T22:58:21.587000", "content": "Anyone care to elaborate on how NMS works this competition?
", "votes": 2, "replies": [] }, { "id": 899967, "author_name": "fergusoci", "author_url": "", "post_date": "2020-06-24T14:52:06.950000", "content": "Think of your proposal boxes as tiles and the information in those tiles as scores...
", "votes": 1, "replies": [] }, { "id": 910543, "author_name": "Francis Chen", "author_url": "", "post_date": "2020-07-01T07:48:37.917000", "content": "How to calculate the information score of a tile?
", "votes": 0, "replies": [] }, { "id": 910559, "author_name": "fergusoci", "author_url": "", "post_date": "2020-07-01T08:06:36.660000", "content": "In the simplest case it's just the level of darkness of the tile. You could be a bit more intelligent about it, (maybe something useful here), but I'm not sure if there is that much value in the additional complexity.
", "votes": 2, "replies": [] } ] }, { "id": 892458, "author_name": "Salman", "author_url": "", "post_date": "2020-06-18T22:36:18.780000", "content": "A quick question...\nYou got 0.86 with 36 patches from this method?
", "votes": 1, "replies": [ { "id": 892798, "author_name": "Alex", "author_url": "", "post_date": "2020-06-19T06:59:31.593000", "content": "Yes 160 patch size, 6 x 6. But in the implementation that got 0.86, I did not use any _excess_coords_filtering() I just randomly shuffled all coords that were obtained for each image (and took the first 36 from that array/list). Further, I also used heavy augmentation on the big image.
I wonder if your advanced tiling method is not skipping too much tissue. Are you using the same selection in both cases ? \nI think the task and discussion needs to be divided in two subtask:\n* The tiling. Making the list of patches.\n=> Simple method just uses a simple grid.\n* The selection. Which ones to send to the model.\n=> Simple method finds the darker ones on average.
", "votes": 2, "replies": [ { "id": 893779, "author_name": "Pasquale", "author_url": "", "post_date": "2020-06-19T22:03:27.227000", "content": "To be honest from the notebook it looks like it's getting pretty much all the tissue. I thought the problem was that the tiles are too small. Imagine having tiny 10x10 tiles and getting all the tissue, the model would likely perform pretty bad because you'd lose too much spatial information. You would get all the tissue but would be destroying all the patterns by breaking it into \"meaningless\" tiles
", "votes": 0, "replies": [] }, { "id": 893794, "author_name": "Salman", "author_url": "", "post_date": "2020-06-19T22:50:56.920000", "content": "Then just increase the patch size. \nLike as 128x128 for Level 2, Then 512x512 for Level 1. Because Level 1 image is 4 times image from Level 2......
", "votes": 1, "replies": [] }, { "id": 893800, "author_name": "Pasquale", "author_url": "", "post_date": "2020-06-19T23:32:26.633000", "content": "If you generate 16 tiles (as for level 2) and each one is 512x512, the total image will be 2048x2048. You'd run out of memory very quickly
", "votes": 0, "replies": [] }, { "id": 893807, "author_name": "Salman", "author_url": "", "post_date": "2020-06-20T00:06:31.260000", "content": "That is the issue. I don't even have local GPU to train 512x512.\nAnd kaggle have limited resources...\nSo I am trying to set this up in TPU by creating tfrecords of 2048x2048.\nSo we wont have memory issues.
", "votes": 0, "replies": [] }, { "id": 894322, "author_name": "Alex", "author_url": "", "post_date": "2020-06-20T10:36:14.450000", "content": "Good points, thanks.
\n\nYes this problem could indeed be divided into those two categories. The current kernel is computing the tiles relatively OK I would say, but the selection of the 36 tiles that would go in to the neural net are likely far from optimal. I also recommend changing the parameters a bit and visualize it for yourselves, perhaps even change the code a bit if you have too much free-time ;-). From my experience visualizing this, the algorithm works well when the tissue is big enough and aligned nicely (not L-shaped etc.). Further, one has to be careful with the small/tiny tissues. So try to look at both small, medium and large sized tissues when visualizing the output/results, likewise for your own algorithms that you use.
", "votes": 0, "replies": [] }, { "id": 897088, "author_name": "Benjamin Dubreu", "author_url": "", "post_date": "2020-06-22T15:56:10.177000", "content": "@arroqc you mentionned a few days ago a new approach to tilling on the other thread, but isn't \" Simple method finds the darker ones on average.\" what Lafoss approach has been doing since the beginning ?
\n\nOn my part, I've been working with selecting the tiles based not on the slide, but on the mask. It didn't work well, and there is also the problem that for inference you need to come up with a segmentation model ^^
\n\nI've also tried to \"center\" the tiles (as very often the long thin line is split into two half empty tiles). My code produces the indended effect (I do get better tiles) but no improvement on CV -_-
", "votes": 0, "replies": [] }, { "id": 897488, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-06-22T22:22:45.743000", "content": "@bdubreu What I use is model based but while the initial results were really good for low tile counts (9 and 16) it doesn't give me much benefit for high tile counts (despite better CV). At least I can try faster with these smaller samples... But tbh I'm getting frustrated by the bad correlation between cv and test.
", "votes": 0, "replies": [] }, { "id": 898001, "author_name": "Alex", "author_url": "", "post_date": "2020-06-23T08:10:28.033000", "content": "There's something else. Slides that have a high isup grade can contain connective tissue or healthy tissue and in the contrary begnin slides can contain cancer tissues. So we should focus on extracting tiles with cells and glands that are representative to isup grade.\nProbably when selecting more than 16 tiles with your method then you include some non-necessary tissues which skew your model. It' just a supposition, what are your thoughts about this @arroqc ?
", "votes": 0, "replies": [] }, { "id": 898526, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-06-23T15:13:09.810000", "content": "It's indeed a possibility and my model is skewed by that kind of noise. I guess I can test further if the small changes I also added are actually decreasing performance. What I find really weird is that everything is much better with lower loss and higher kappa but then the performance really degrade on LB. I don't see how there could be data leakage too :/. Anyway I prefer to keep working with 16 since I can at least fit batches of 9 in my gtx 1080.
", "votes": 1, "replies": [] }, { "id": 898925, "author_name": "Salman", "author_url": "", "post_date": "2020-06-23T20:55:35.093000", "content": "16 of size 256x256 or 512x512?
", "votes": 0, "replies": [] }, { "id": 898996, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-06-23T21:59:27.217000", "content": "256
", "votes": 0, "replies": [] } ] }, { "id": 892496, "author_name": "Pasquale", "author_url": "", "post_date": "2020-06-18T23:41:54.147000", "content": "Did you try to make 1536x1536 images with the more complex method?
", "votes": 2, "replies": [ { "id": 892791, "author_name": "Alex", "author_url": "", "post_date": "2020-06-19T06:55:03.960000", "content": "Good question. I actually tried it overnight, and I didn't reach the same score as the simple method. So I would say that the simple method works better, somehow. Perhaps the complex method looks good visually (to some extent), but the performance is a bit worse. But then there might be some adjustments that could be made to improve the the complex method.
", "votes": 1, "replies": [] }, { "id": 893232, "author_name": "Salman", "author_url": "", "post_date": "2020-06-19T13:26:58.437000", "content": "Did you keep same patch size as in simple tiling and this method?
", "votes": 0, "replies": [] } ] }, { "id": 923008, "author_name": "Nikhil Bartwal", "author_url": "", "post_date": "2020-07-10T13:30:09.460000", "content": "Hey @akensert, \nAfter implementing everything you suggested and some other ideas, when i finally got to training, this happened. \nCould you please look into this? \nThanks!!
", "votes": 0, "replies": [] }, { "id": 919982, "author_name": "Nikhil Bartwal", "author_url": "", "post_date": "2020-07-08T08:29:44.753000", "content": "Hey @akensert, really nice work man. I was trying to experiment with your kernel and what i notice is that it takes way too much time for a single epoch on Kaggle GPU. I wanted to ask, did you use your private rig or kaggle kernel itself. Also, when i tried using TPU with your tf.data.Dataset (after incorporating the targets), TPUs don't really work with py_function. BTW, how much time did it take for you to train your model?. Any ideas or help would be much appreciated.
", "votes": 0, "replies": [ { "id": 920061, "author_name": "Alex", "author_url": "", "post_date": "2020-07-08T09:30:20.537000", "content": "Hello @nikhilbartwal001. Yes, you would need to precompute the coordinates if you're running in the Kaggle kernels. (With >=6 CPU cores (12 threads), it would be possible to compute the coordinates on the fly, but still not ideal.). I suggest you precompute the coordinates for each example, add the coordinates (array of arrays) to a column in your pandas dataframe, and then you can input this together with the image + label (so your input would be, per example, (image, label, coords). I think that will speed it up the input pipeline significantly (especially when working the Kaggle kernels).
About TPU: Yes you are right, it doesn't seem to work with tf.py_function. You need to avoid that :-(
I guess for me, locally, it would take around 10-30 minutes per epoch using the same pipeline as the public kernel. And it depends on the size of the input (I usually vary it between 1024x1024 and 1536x1536).
\n\nSide note: Currently I use the simple approach for computing tiles/patches. Similar to the other public kernels. I haven't had much time/energy to continue experiment with the one I published.
", "votes": 1, "replies": [] }, { "id": 920069, "author_name": "Nikhil Bartwal", "author_url": "", "post_date": "2020-07-08T09:48:40.603000", "content": "Thanks for such a quick and insightful reply @akensert. I had some doubts which I thought you might help me clear.\nCan you tell me more about the 'simple' approach that you are talking about as I'm new to image processing and yours was one of the most detailed kernels written in tensorflow.\nAlso have you tried using TPU (with/without tfrec format) in this competition? \nAnother thing that I'm confused about is the choice of loss function and the evaluation metric as people seem to be using classification , regression, ordinal regression etc. \nIt would be really help if you could help me out.\nThanks !!
", "votes": 1, "replies": [] }, { "id": 920249, "author_name": "Alex", "author_url": "", "post_date": "2020-07-08T13:05:50.640000", "content": "@nikhilbartwal001 No problem :-)
\n\nAbout the simple approach, see here, although it's in PyTorch. I rewrote it for TensorFlow (with some small modifications).
\n\nI haven't used TPU yet, however I think there should be some nice TPU + TF2 kernels around to get one started.
\n\nLoss function: I suggest either mean squared error (so you would keep your target as it is in the csv file, or ordinal regression (target = 3 would be [1, 1, 1, 0, 0] etc.). Both works equally well for me. Both makes sense because the target data have some order.
", "votes": 1, "replies": [] }, { "id": 920275, "author_name": "Nikhil Bartwal", "author_url": "", "post_date": "2020-07-08T13:20:14.223000", "content": "Hey @akensert, just one more thing that i had, I've read about QWK for the first time in this competition.\nThere is a QWK metric in sklearn as well as in tf addons, do you think it would work or would I need to write a custom metric for the sane? I don't quite get how to optimize qwk during training.
\n\nBTW, you earlier suggested pre computing the coordinates, I was thinking what if I create all the patched images before hand and store it in the output directory and then train the model in a different kernel by just reading and converting the patched image to numpy array and feeding it to the input pipeline. What are your thoughts?\nThanks for the help mate !
", "votes": 0, "replies": [] }, { "id": 921684, "author_name": "Alex", "author_url": "", "post_date": "2020-07-09T13:29:03.873000", "content": "@nikhilbartwal001 Your loss function will be MSE or BCE (or CCE). So you will not optimize for QWK directly (although you could try, but I figure it might be difficult), but rather for MSE/BCE/CCE.
\n\nSure you could do that (even faster then), but then you wouldn't have the patch-level augmentation (the random rotate/flip/transposing/(selection) of each patch before being concatenated to the 'big image'), which could be nice to have. Although you will have the 'image-level' augmentation :-)
\n\nGood luck!
", "votes": 1, "replies": [] }, { "id": 921728, "author_name": "Nikhil Bartwal", "author_url": "", "post_date": "2020-07-09T14:00:48.790000", "content": "Hey @akensert, thanks for the help with the loss function.
\n\nAlthough i didn't understand what you meant that i won't be able to have patch level augs. Isn't it possible to to do the patch level augs before storing the patched images? I'm currently not that deep into image processing so idk about that.
", "votes": 0, "replies": [] }, { "id": 921829, "author_name": "Alex", "author_url": "", "post_date": "2020-07-09T15:31:48.380000", "content": "@nikhilbartwal001 Sure, you could do that, but then your dataset which you're saving to disk will be huge :-) let's say that you wanna train for at least 10 epochs, then \"ideally\" you would want to have 10 versions of each instance (or 'big image'). So over 100k images saved to disk.
", "votes": 0, "replies": [] }, { "id": 922490, "author_name": "Nikhil Bartwal", "author_url": "", "post_date": "2020-07-10T06:12:52.833000", "content": "Woah I definitely wouldn't want that:(\nI guess I'll try experimenting with the precomputed coords only.\nThanks for the help dude!!
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "892340": "As pointed out before, the methods for extracting information from the biopsy images are important. Currently I'm doing some \"simple\"/straight forward tiling (similar to [this](https://www.kaggle.com/haqishen/train-efficientnet-b0-w-36-tiles-256-lb0-87) and [this](https://www.kaggle.com/iafoss/panda-concat-tile-pooling-starter-0-79-lb)). Gets me to 0.89 LB with a single EfficientNetB0/B1 (using level 1 resolution images). I've also experimented with some more complex tiling, which got me to 0.86 LB (using 960x960 'big images' in comparison with 1536x1536 'big images' of the 0.89 LB submission). I've shared a variant of the tiling method of the 0.86 LB implementation [here](https://www.kaggle.com/akensert/panda-optimized-tiling-tf-data-dataset), to stimulate the discussion a bit :-). Feel free to share me *all of your secrets* on how to extract information from the biopsy images.", "914994": "I implement a method to filter some tiles which are polluted by green/blue(the color observed by my eyes) ink, and random pick some images, find the result is better than just sort by pixel-sum.\nMy model is runing, if I can boost >= 0.01, I will hint about the filter method here😄 \n**UPDATE:**The model is not so good as what I expect in first 10 epochs... I show the picture below is the tiles after my filter method, it surely filter some polluted tiles, but the variety of tiles decline... original argsort means [iafoss's method](https://www.kaggle.com/iafoss/panda-16x128x128-tiles)\nThe image-id from top to bottom is \n'4a3e6e77b42a7e8f8b2dec472667bdbe',\n'51ea992db967a0b925588ee2eee322ab',\n'42482ed25f9001ae44f6148432da91d7',\n'f114a15f4c02900ebed9f638b68f42cb',\n'22c2e58ea58b90066e98d86bfb8a0baf',\n\n\n", "894373": "I use a NMS-type approach for selecting tiles. It gives a bit of an improvement over the simple approach, but nothing ground breaking. Probably in the order of 0.005 improvement on CV/LB.", "892458": "A quick question...\nYou got 0.86 with 36 patches from this method?", "893744": "I wonder if your advanced tiling method is not skipping too much tissue. Are you using the same selection in both cases ? \nI think the task and discussion needs to be divided in two subtask:\n* The tiling. Making the list of patches.\n=> Simple method just uses a simple grid.\n* The selection. Which ones to send to the model.\n=> Simple method finds the darker ones on average.\n\n", "892496": "Did you try to make 1536x1536 images with the more complex method?", "923008": "Hey @akensert, \nAfter implementing everything you suggested and some other ideas, when i finally got to training, [this](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/165645) happened. \nCould you please look into this? \nThanks!! \n", "919982": "Hey @akensert, really nice work man. I was trying to experiment with your kernel and what i notice is that it takes way too much time for a single epoch on Kaggle GPU. I wanted to ask, did you use your private rig or kaggle kernel itself. Also, when i tried using TPU with your tf.data.Dataset (after incorporating the targets), TPUs don't really work with py_function. BTW, how much time did it take for you to train your model?. Any ideas or help would be much appreciated." } }