I am not used to fastai so sorry for any rudimentary question. It may not be the content to be discussed in this discussion.
\n\nIf you use the pre-trained models of ImageNet, I think we use DataBunch.normalize(), but \n reproducibility cannot be ensured. I thik this is caused by the effect applied to each batch.
\nI fix numpy random seed. Is it the influence of the seed related to GPU? Or is it a problem caused by multiple cpu_workers?
```
\n\nnp.random.seed(0)\nimg_list = ImageList.from_df(df_train, path_train)\ndata = (img_list.split_by_idxs(index_train, index_valid)\n .label_from_df(label_delim=' ')\n .add_test(ImageList.from_df(df_test, path=path_test))\n .databunch(bs=256, num_workers=os.cpu_count())\n .normalize())\n...\nreturn learn.get_preds(DatasetType.Valid)\n```
", "messages": [ { "id": 638413, "postDate": "2019-10-01T21:46:03.160Z", "content": "this should help
\n\n\ndef seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\nSEED = 1\nseed_everything(SEED)\n
Thanks @valanm for your reply! I tried in my local machine with 1080ti, but I got different results...\nI think this caused by multiprocessing or other processes or my GPU-seed trouble.
", "rawMarkdown": "Thanks @valanm for your reply! I tried in my local machine with 1080ti, but I got different results...\nI think this caused by multiprocessing or other processes or my GPU-seed trouble." } ] }, { "id": 639963, "postDate": "2019-10-03T18:13:30.473Z", "content": "See Val's post for setting appropriate seeds, though he missed torch.backends.cudnn.benchmark = False which is in the PyTorch guide on this.
Passing no options to normalize will cause it to use the mean/std of a couple of batches and so cause non-determinism. Passing stats to use will avoid this.\nHowever the imagenet stats are not really appropriate. They are the mean and std of imagenet images and so will normalize them appropriately to create values of the appropriate range for pretrained models (mean of 0, std of 1, i.e. Z-scores). You want to use the mean and std of your images so they are scaled to the same range the saved model was trained on.\nSo you want to calculate the actual mean and std of your training data, or in this case probably a subset given the large size, and then pass that to normalize. This post on the fastai forums contains a link to code I wrote for collecting stats (to do a subset just do something like collect_stats(data.train_ds.x[:1000])).
Thx. Fixed as suggested
", "rawMarkdown": "Thx. Fixed as suggested " }, { "id": 642323, "postDate": "2019-10-05T22:08:44.223Z", "content": "It works fine, thanks a lot!!
", "rawMarkdown": "It works fine, thanks a lot!!" } ] }, { "id": 638123, "postDate": "2019-10-01T14:52:11.290Z", "content": "I am not used to fastai so sorry for any rudimentary question. It may not be the content to be discussed in this discussion.
\n\nIf you use the pre-trained models of ImageNet, I think we use DataBunch.normalize(), but \n reproducibility cannot be ensured. I thik this is caused by the effect applied to each batch.
\nI fix numpy random seed. Is it the influence of the seed related to GPU? Or is it a problem caused by multiple cpu_workers?
```
\n\nnp.random.seed(0)\nimg_list = ImageList.from_df(df_train, path_train)\ndata = (img_list.split_by_idxs(index_train, index_valid)\n .label_from_df(label_delim=' ')\n .add_test(ImageList.from_df(df_test, path=path_test))\n .databunch(bs=256, num_workers=os.cpu_count())\n .normalize())\n...\nreturn learn.get_preds(DatasetType.Valid)\n```
", "rawMarkdown": "**I am not used to fastai so sorry for any rudimentary question. It may not be the content to be discussed in this discussion.** \n\nIf you use the pre-trained models of ImageNet, I think we use `DataBunch.normalize()`, but \n reproducibility cannot be ensured. I thik this is caused by the effect applied to each batch. \nI fix numpy random seed. Is it the influence of the seed related to GPU? Or is it a problem caused by multiple cpu_workers?\n\n```\n# Pseudo code block\n# I run these lines, everytime preds are different.\n# I try delete normalize(), so preds are same.\n\nnp.random.seed(0)\nimg_list = ImageList.from_df(df_train, path_train)\ndata = (img_list.split_by_idxs(index_train, index_valid)\n .label_from_df(label_delim=' ')\n .add_test(ImageList.from_df(df_test, path=path_test))\n .databunch(bs=256, num_workers=os.cpu_count())\n .normalize())\n...\nreturn learn.get_preds(DatasetType.Valid)\n```" }, { "id": 638279, "postDate": "2019-10-01T17:21:47.553Z", "content": "Please, have a look here: https://docs.fast.ai/vision.data.html#ImageDataBunch.normalize\n \nAdd normalize transform using stats (defaults to DataBunch.batch_stats).\nIn the fast.ai library we have imagenet_stats, ... so we can add normalization easily with any of these datasets.\n\nSo .normalize(imagenet_stats) should do the trick!
Thank you @micpie!
\nAre there loss differences between no-normalize and normalize() and normalize(imagenet_stats) in your model?
this should help
\n\n\ndef seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\nSEED = 1\nseed_everything(SEED)\n
Thanks @valanm for your reply! I tried in my local machine with 1080ti, but I got different results...\nI think this caused by multiprocessing or other processes or my GPU-seed trouble.
", "votes": 0, "replies": [] } ] }, { "id": 639963, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2019-10-03T18:13:30.473000", "content": "See Val's post for setting appropriate seeds, though he missed torch.backends.cudnn.benchmark = False which is in the PyTorch guide on this.
Passing no options to normalize will cause it to use the mean/std of a couple of batches and so cause non-determinism. Passing stats to use will avoid this.\nHowever the imagenet stats are not really appropriate. They are the mean and std of imagenet images and so will normalize them appropriately to create values of the appropriate range for pretrained models (mean of 0, std of 1, i.e. Z-scores). You want to use the mean and std of your images so they are scaled to the same range the saved model was trained on.\nSo you want to calculate the actual mean and std of your training data, or in this case probably a subset given the large size, and then pass that to normalize. This post on the fastai forums contains a link to code I wrote for collecting stats (to do a subset just do something like collect_stats(data.train_ds.x[:1000])).
Thx. Fixed as suggested
", "votes": 0, "replies": [] }, { "id": 642323, "author_name": "kambarakun", "author_url": "", "post_date": "2019-10-05T22:08:44.223000", "content": "It works fine, thanks a lot!!
", "votes": 0, "replies": [] } ] }, { "id": 638279, "author_name": "Michael Pieler", "author_url": "", "post_date": "2019-10-01T17:21:47.553000", "content": "Please, have a look here: https://docs.fast.ai/vision.data.html#ImageDataBunch.normalize\n \nAdd normalize transform using stats (defaults to DataBunch.batch_stats).\nIn the fast.ai library we have imagenet_stats, ... so we can add normalization easily with any of these datasets.\n\nSo .normalize(imagenet_stats) should do the trick!
Thank you @micpie!
\nAre there loss differences between no-normalize and normalize() and normalize(imagenet_stats) in your model?