Hi! I'm a complete beginner at the Kaggle competition, and I'm confused about choosing the right transformation method:
\nI have seen 2 types of normalization:
\nSo my question is: If I use this converted dataset, could I use normalization type 1(I listed above) in my custom transform function in my training code?
\nI guess there is no robust answer, but I hope to hear about others' experiences.
\nI really appreciate any help you can provide.
Hey! You can normalize data however you see fit (in fact, that is something worth experimenting with! 🙂)
\nYou just need to make sure that you normalize the data in the same way in training as in test. As in, you can read in the data from whatever format, you just need to make sure that once the data reaches your model, that it has been preprocessed in the same way (that is, you can construct a custom preprocessing pipeline that sits between reading the data and it being shown to your model for training/inference)
\nAll the best, hope this helps! 🙌
", "rawMarkdown": "Hey! You can normalize data however you see fit (in fact, that is something worth experimenting with! 🙂)\n\nYou just need to make sure that you normalize the data in the same way in training as in test. As in, you can read in the data from whatever format, you just need to make sure that once the data reaches your model, that it has been preprocessed in the same way (that is, you can construct a custom preprocessing pipeline that sits between reading the data and it being shown to your model for training/inference)\n\nAll the best, hope this helps! 🙌", "votes": 1 }, { "id": 2077574, "postDate": "2022-12-27T17:55:15.327Z", "content": "What you refer to as Type1 is data standardization. You can rescale images by Type2 first, then calculate mean and std of the normalized dataset, and finally standardize data. It might help the model to generalize better.
", "rawMarkdown": "What you refer to as Type1 is data standardization. You can rescale images by Type2 first, then calculate mean and std of the normalized dataset, and finally standardize data. It might help the model to generalize better.", "votes": 1 }, { "id": 2077489, "postDate": "2022-12-27T16:22:41.997Z", "content": "Hi! I'm a complete beginner at the Kaggle competition, and I'm confused about choosing the right transformation method:
\nI have seen 2 types of normalization:
\nSo my question is: If I use this converted dataset, could I use normalization type 1(I listed above) in my custom transform function in my training code?
\nI guess there is no robust answer, but I hope to hear about others' experiences.
\nI really appreciate any help you can provide.
Hey! You can normalize data however you see fit (in fact, that is something worth experimenting with! 🙂)
\nYou just need to make sure that you normalize the data in the same way in training as in test. As in, you can read in the data from whatever format, you just need to make sure that once the data reaches your model, that it has been preprocessed in the same way (that is, you can construct a custom preprocessing pipeline that sits between reading the data and it being shown to your model for training/inference)
\nAll the best, hope this helps! 🙌
", "votes": 1, "replies": [] }, { "id": 2077574, "author_name": "Rasoul Mojtahedzadeh", "author_url": "", "post_date": "2022-12-27T17:55:15.327000", "content": "What you refer to as Type1 is data standardization. You can rescale images by Type2 first, then calculate mean and std of the normalized dataset, and finally standardize data. It might help the model to generalize better.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2077903": "Hey! You can normalize data however you see fit (in fact, that is something worth experimenting with! 🙂)\n\nYou just need to make sure that you normalize the data in the same way in training as in test. As in, you can read in the data from whatever format, you just need to make sure that once the data reaches your model, that it has been preprocessed in the same way (that is, you can construct a custom preprocessing pipeline that sits between reading the data and it being shown to your model for training/inference)\n\nAll the best, hope this helps! 🙌", "2077574": "What you refer to as Type1 is data standardization. You can rescale images by Type2 first, then calculate mean and std of the normalized dataset, and finally standardize data. It might help the model to generalize better.", "2077489": "Hi! I'm a complete beginner at the Kaggle competition, and I'm confused about choosing the right transformation method:\n\nI have seen 2 types of normalization:\n\n1. to subtract the mean of the training set and divide by the standard deviation\n2. scales the data to the range [0, 1] by subtracting min and then dividing by (max - min) \nIn this convert dataset from theoviel [Dicom -> Resized PNG/JPG\n](https://www.kaggle.com/code/theoviel/dicom-resized-png-jpg) , I noticed that the original image had been normalized by type 2 (I listed above).\n\nSo my question is: If I use this converted dataset, could I use normalization type 1(I listed above) in my custom transform function in my training code?\n\nI guess there is no robust answer, but I hope to hear about others' experiences.\nI really appreciate any help you can provide. " } }