For this competition, I have several thoughts and findings, some confusion and some ideas, I hope to get the discussion and opinions of other contestants:
\n| Differences between different specification models | \n
|---|
| - Try to use a large model and a small model to process the same parameters on the image, and the processing results are almost no different | \n
| Differences between different input sizes | \n
|---|
| - Try feeding images of different scales into the model for analysis and processing (for img_size in [256,512,768,1024]), and the training results are not too different | \n
| - I think such images may need some detailed features as support. - However, common sizes such as [256,512,768,1024] are still too small for the model, and too large sizes cannot be directly fed into the neural network due to the limitation of computing resources | \n
| Try to add attention module | \n
|---|
| - did not get very good results | \n
| Some Different | \n
|---|
| - When the learning rate drops to a certain position, the relevant Loss falls into the basin and is difficult to change. I think there are some things to think about here - lr = 0.0001 train_acc=0.74 val_acc= 0.76 | \n
| - The approximate value is like this, from 1E-4 to 1E-6 completely unchanged, the value is almost completely unchanged, for example, although the learning rate of multiple epochs training changes, but the output is always 0.7675, 0.7845; Like this: Epoch =0: TARin_ACC = 0.6475, VAL_ACC =0.5978 LR =0.001 Epoch =1: TARin_ACC = 0.7645, VAL_ACC =0.7331 LR =0.0001 Epoch =2: TARin_ACC = 0.7645, VAL_ACC =0.7331 LR =0.0001 Epoch =3: TARin_ACC = 0.7645, VAL_ACC =0.7331 LR =0.00001 Epoch =4: TARin_ACC = 0.7645, VAL_ACC =0.7331 LR =0.000001 Different models and parameters may fall into different basins; | \n
| One last thought | \n
|---|
| - In fact, there are also problems in the overall data processing. - The official data storage space is only 20 gigabytes. - If the data set is processed with too many details, the computer will crash during the actual reasoning. - Therefore, I think it is actually difficult to optimize the performance of the overall project by keeping more details of the data. - Perhaps the only way is to use less data detail to improve the score based on the network structure | \n
For this competition, I have several thoughts and findings, some confusion and some ideas, I hope to get the discussion and opinions of other contestants:
\n| Differences between different specification models | \n
|---|
| - Try to use a large model and a small model to process the same parameters on the image, and the processing results are almost no different | \n
| Differences between different input sizes | \n
|---|
| - Try feeding images of different scales into the model for analysis and processing (for img_size in [256,512,768,1024]), and the training results are not too different | \n
| - I think such images may need some detailed features as support. - However, common sizes such as [256,512,768,1024] are still too small for the model, and too large sizes cannot be directly fed into the neural network due to the limitation of computing resources | \n
| Try to add attention module | \n
|---|
| - did not get very good results | \n
| Some Different | \n
|---|
| - When the learning rate drops to a certain position, the relevant Loss falls into the basin and is difficult to change. I think there are some things to think about here - lr = 0.0001 train_acc=0.74 val_acc= 0.76 | \n
| - The approximate value is like this, from 1E-4 to 1E-6 completely unchanged, the value is almost completely unchanged, for example, although the learning rate of multiple epochs training changes, but the output is always 0.7675, 0.7845; Like this: Epoch =0: TARin_ACC = 0.6475, VAL_ACC =0.5978 LR =0.001 Epoch =1: TARin_ACC = 0.7645, VAL_ACC =0.7331 LR =0.0001 Epoch =2: TARin_ACC = 0.7645, VAL_ACC =0.7331 LR =0.0001 Epoch =3: TARin_ACC = 0.7645, VAL_ACC =0.7331 LR =0.00001 Epoch =4: TARin_ACC = 0.7645, VAL_ACC =0.7331 LR =0.000001 Different models and parameters may fall into different basins; | \n
| One last thought | \n
|---|
| - In fact, there are also problems in the overall data processing. - The official data storage space is only 20 gigabytes. - If the data set is processed with too many details, the computer will crash during the actual reasoning. - Therefore, I think it is actually difficult to optimize the performance of the overall project by keeping more details of the data. - Perhaps the only way is to use less data detail to improve the score based on the network structure | \n
Thank you so much for sharing your discoveries so far in training a model for this competition. I've got a question regarding your choice of accuracy as an evaluation metric. Based on @abhishek's book, Approach Almost Any Machine Learning Problem, I used the AUC score. The decision is based on the following quote from his book,
\n\n\nAs a data doctor would say: this is a classic case of skewed binary classification. Therefore, we choose the evaluation metric to be AUC and go for a stratified k-fold cross-validation scheme.
\n
Here are two links that discuss the topic in blog posts:
\nhttps://machinelearningmastery.com/failure-of-accuracy-for-imbalanced-class-distributions/
\nhttps://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification/
My question then is, have you tried other metrics? I've been basing my iteration of models on the improvement of the AUC score. I've been quite frustrated with transformers due to the AUC score being worse for ViT in comparison to something like random forest (AUC=0.51, 0.52).
", "rawMarkdown": "Thank you so much for sharing your discoveries so far in training a model for this competition. I've got a question regarding your choice of accuracy as an evaluation metric. Based on @abhishek's book, Approach Almost Any Machine Learning Problem, I used the AUC score. The decision is based on the following quote from his book,\n\n> As a data doctor would say: this is a classic case of skewed binary classification. Therefore, we choose the evaluation metric to be AUC and go for a stratified k-fold cross-validation scheme.\n\nHere are two links that discuss the topic in blog posts:\n[https://machinelearningmastery.com/failure-of-accuracy-for-imbalanced-class-distributions/](https://machinelearningmastery.com/failure-of-accuracy-for-imbalanced-class-distributions/)\n[https://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification/](https://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification/)\n\nMy question then is, have you tried other metrics? I've been basing my iteration of models on the improvement of the AUC score. I've been quite frustrated with transformers due to the AUC score being worse for ViT in comparison to something like random forest (AUC=0.51, 0.52).", "votes": 1, "replies": [ { "id": 1903729, "postDate": "2022-08-17T15:54:25.507Z", "content": "| Leon' s Reply | \n
|---|
| - In fact, I personally think that CNN and TF should have stronger performance in this respect than forest; | \n
| - But oddly enough, there are a number of these forest-like practices in Kaggle competitions that are more powerful in terms of accuracy and speed; | \n
| - And I personally don't think it's overfitting or anything like that. | \n
| - In terms of indicators, I don't really have a lot of ideas. | \n
I'm guessing one stand-out thing is that RF is probably much simpler to train in comparison to a neural network with its many architectures. I still think that there are certain models with a specific tuning out there that have the potential to do just fine, but I lack the skills myself to build and run such designs.
", "rawMarkdown": "I'm guessing one stand-out thing is that RF is probably much simpler to train in comparison to a neural network with its many architectures. I still think that there are certain models with a specific tuning out there that have the potential to do just fine, but I lack the skills myself to build and run such designs. " }, { "id": 1914674, "postDate": "2022-08-26T09:46:29.327Z", "content": "- In fact , i have tried many ways to get a loss-function can solve us question ; Firstly i tried many relative fc in the package-sklearn , but the scores were similar to the original function ; Than i make some fc by myself based on the competition, the score is also below before.\nI am not entirely sure where to ask this question: How do you know how many artificial neurons to put in a layer? Is there a formula?
", "rawMarkdown": "I am not entirely sure where to ask this question: How do you know how many artificial neurons to put in a layer? Is there a formula?" }, { "id": 1893700, "postDate": "2022-08-11T02:26:43.870Z", "content": "The input images are too large to apply ML DL methods directly.
\nI think essential to pre-process the images and extract (crop, cut)
\nessential image elements before using ML.
\nHas anyone succeeded in such pre-processing/ image elements extraction
\nto generate more trainable data?
For some sample images, you can extract more patches than others. You can then random sample among those patches to generate a different series of patches for a single gigantic image.
", "rawMarkdown": "For some sample images, you can extract more patches than others. You can then random sample among those patches to generate a different series of patches for a single gigantic image." }, { "id": 1894381, "postDate": "2022-08-11T12:58:03.493Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1894911, "postDate": "2022-08-11T19:35:53.697Z", "content": "Yes, that's what I was thinking.
\nLet's not worry about submission for now.
\nHow big was your image patches and how many image patches did you generate per slide?
\nI would imagine using many small image patches like 224x224 to feed into DL pipeline…
\nDid you save image patches on the disk before ML processing?
| Leon' s Reply | \n
|---|
| - I'm creating patches on the image | \n
| - However, in order to maintain the original randomness, my patch generation is indeed relatively large, basically within the range of size in [3000,30000]; | \n
| - But by infer, I can't infer the hardware ； I don't think it should break through the computer hardware | \n
Did you try creating much more image samples but smaller size of image, to massively pass through the ML process?
\nI would think about 100k small patch images.
\nI would think images needs to be saved offload on the disk temporarily as the won't fit in main memory.
Thank you so much for sharing your discoveries so far in training a model for this competition. I've got a question regarding your choice of accuracy as an evaluation metric. Based on @abhishek's book, Approach Almost Any Machine Learning Problem, I used the AUC score. The decision is based on the following quote from his book,
\n\n\nAs a data doctor would say: this is a classic case of skewed binary classification. Therefore, we choose the evaluation metric to be AUC and go for a stratified k-fold cross-validation scheme.
\n
Here are two links that discuss the topic in blog posts:
\nhttps://machinelearningmastery.com/failure-of-accuracy-for-imbalanced-class-distributions/
\nhttps://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification/
My question then is, have you tried other metrics? I've been basing my iteration of models on the improvement of the AUC score. I've been quite frustrated with transformers due to the AUC score being worse for ViT in comparison to something like random forest (AUC=0.51, 0.52).
", "votes": 1, "replies": [ { "id": 1903729, "author_name": "Leon", "author_url": "", "post_date": "2022-08-17T15:54:25.507000", "content": "| Leon' s Reply | \n
|---|
| - In fact, I personally think that CNN and TF should have stronger performance in this respect than forest; | \n
| - But oddly enough, there are a number of these forest-like practices in Kaggle competitions that are more powerful in terms of accuracy and speed; | \n
| - And I personally don't think it's overfitting or anything like that. | \n
| - In terms of indicators, I don't really have a lot of ideas. | \n
I'm guessing one stand-out thing is that RF is probably much simpler to train in comparison to a neural network with its many architectures. I still think that there are certain models with a specific tuning out there that have the potential to do just fine, but I lack the skills myself to build and run such designs.
", "votes": 0, "replies": [] }, { "id": 1914674, "author_name": "Leon", "author_url": "", "post_date": "2022-08-26T09:46:29.327000", "content": "- In fact , i have tried many ways to get a loss-function can solve us question ; Firstly i tried many relative fc in the package-sklearn , but the scores were similar to the original function ; Than i make some fc by myself based on the competition, the score is also below before.\nI am not entirely sure where to ask this question: How do you know how many artificial neurons to put in a layer? Is there a formula?
", "votes": 0, "replies": [] }, { "id": 1893700, "author_name": "Igiku124", "author_url": "", "post_date": "2022-08-11T02:26:43.870000", "content": "The input images are too large to apply ML DL methods directly.
\nI think essential to pre-process the images and extract (crop, cut)
\nessential image elements before using ML.
\nHas anyone succeeded in such pre-processing/ image elements extraction
\nto generate more trainable data?
For some sample images, you can extract more patches than others. You can then random sample among those patches to generate a different series of patches for a single gigantic image.
", "votes": 0, "replies": [] }, { "id": 1894381, "author_name": "", "author_url": "", "post_date": "2022-08-11T12:58:03.493000", "content": "", "votes": 1, "replies": [] }, { "id": 1894911, "author_name": "Igiku124", "author_url": "", "post_date": "2022-08-11T19:35:53.697000", "content": "Yes, that's what I was thinking.
\nLet's not worry about submission for now.
\nHow big was your image patches and how many image patches did you generate per slide?
\nI would imagine using many small image patches like 224x224 to feed into DL pipeline…
\nDid you save image patches on the disk before ML processing?
| Leon' s Reply | \n
|---|
| - I'm creating patches on the image | \n
| - However, in order to maintain the original randomness, my patch generation is indeed relatively large, basically within the range of size in [3000,30000]; | \n
| - But by infer, I can't infer the hardware ； I don't think it should break through the computer hardware | \n
Did you try creating much more image samples but smaller size of image, to massively pass through the ML process?
\nI would think about 100k small patch images.
\nI would think images needs to be saved offload on the disk temporarily as the won't fit in main memory.