{ "id": 457249, "title": "Using External Data for Validation", "url": "/competitions/UBC-OCEAN/discussion/457249", "author_name": "Ivan Aerlic", "post_date": "2023-11-23T18:01:57.294000", "votes": 20, "comment_count": 8, "views": 0, "content": "

@masadia Hi, this is a question for the competition hosts.

\n

I was wondering whether we can use any external data for validating models. I already know that only open license datasets can be used for training and that they need to be accessible to all participants. But using data for validation is a different story. In GUIE this link we were allowed to use commercially licensed datasets for validation but not training. I believe it should be fine here as well.

\n

I know that MIT license datasets can be used anywhere. If you could add to this list that would be great, just to let us know which licenses are permissible.

\n

Please reply with your answer. This is an important question, because validating data in this competition is very difficult.

\n

Also, are datasets used for validation purposes required to be shared on the forum. I already know that datasets used for training purposes are required to be shared.

", "messages": [ { "id": 2535886, "postDate": "2023-11-23T18:01:57.293Z", "content": "

@masadia Hi, this is a question for the competition hosts.

\n

I was wondering whether we can use any external data for validating models. I already know that only open license datasets can be used for training and that they need to be accessible to all participants. But using data for validation is a different story. In GUIE this link we were allowed to use commercially licensed datasets for validation but not training. I believe it should be fine here as well.

\n

I know that MIT license datasets can be used anywhere. If you could add to this list that would be great, just to let us know which licenses are permissible.

\n

Please reply with your answer. This is an important question, because validating data in this competition is very difficult.

\n

Also, are datasets used for validation purposes required to be shared on the forum. I already know that datasets used for training purposes are required to be shared.

", "rawMarkdown": "@masadia Hi, this is a question for the competition hosts. \n\nI was wondering whether we can use any external data for validating models. I already know that only open license datasets can be used for training and that they need to be accessible to all participants. But using data for validation is a different story. In GUIE [this link](https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/337384) we were allowed to use commercially licensed datasets for validation but not training. I believe it should be fine here as well. \n\nI know that MIT license datasets can be used anywhere. If you could add to this list that would be great, just to let us know which licenses are permissible. \n\nPlease reply with your answer. This is an important question, because validating data in this competition is very difficult. \n\nAlso, are datasets used for validation purposes required to be shared on the forum. I already know that datasets used for training purposes are required to be shared. ", "votes": 20 }, { "id": 2543229, "postDate": "2023-11-29T21:56:37.703Z", "content": "

As a general guideline, to explore different model architectures locally, you can use any public dataset available to all and then use the best architecture on the competition data. However, please note that usage of any dataset that affects the weights of the model is subject to Section 7C of the competition rules.

", "rawMarkdown": "As a general guideline, to explore different model architectures locally, you can use any public dataset available to all and then use the best architecture on the competition data. However, please note that usage of any dataset that affects the weights of the model is subject to Section 7C of the competition rules.", "votes": 4, "replies": [ { "id": 2543236, "postDate": "2023-11-29T22:08:34.090Z", "content": "

I would argue that validation is intrinsically linked to the training process since hyperparameter adjustments, based on validation scores, significantly impact the final model weights. This intertwining raises concerns about fairness, especially if the validation data isn't uniformly accessible to all competitors.

", "rawMarkdown": "I would argue that validation is intrinsically linked to the training process since hyperparameter adjustments, based on validation scores, significantly impact the final model weights. This intertwining raises concerns about fairness, especially if the validation data isn't uniformly accessible to all competitors.", "votes": 1, "replies": [ { "id": 2543261, "postDate": "2023-11-29T23:04:36.580Z", "content": "

Thank you for raising this. We just amended the comment above. This should address your concerns.

", "rawMarkdown": "Thank you for raising this. We just amended the comment above. This should address your concerns.", "votes": 1 }, { "id": 2543266, "postDate": "2023-11-29T23:47:15.053Z", "content": "

@jonasgjerris Great point. The concern here is, can we use data for validation that is accessible to all participants but has a private license. This way the hosts can still legally use it, but participants can have better validation data.

\n

This has been done in past competitions, such as GUIE.

\n

Unfortunately, this ship has now sailed.

", "rawMarkdown": "@jonasgjerris Great point. The concern here is, can we use data for validation that is accessible to all participants but has a private license. This way the hosts can still legally use it, but participants can have better validation data.\n\nThis has been done in past competitions, such as GUIE. \n\nUnfortunately, this ship has now sailed. " } ] }, { "id": 2543263, "postDate": "2023-11-29T23:34:21.393Z", "content": "

Thanks Hossein, this clears things up.

", "rawMarkdown": "Thanks Hossein, this clears things up." }, { "id": 2544594, "postDate": "2023-12-01T00:29:52.647Z", "content": "

@hsfarahani Do you have any external data source recommendations?

", "rawMarkdown": "@hsfarahani Do you have any external data source recommendations?", "votes": 1 } ] }, { "id": 2536078, "postDate": "2023-11-23T22:30:26.923Z", "content": "

@masadia could you please also clarify this :

\n

1 . datasets used for training purposes are required to be shared

\n

I mean if we for example use datasets from https://portal.gdc.cancer.gov/ mentioned in this discussion does this mean that we can consider that ovarian cancer slides images from this site 'are shared' because participants have link to the source and can download this data? I ask because ovarian cancer dataset from this site is very big (about 1000 slides) and there is no sence and no possibility to upload it to Kaggle site (because if external dataset's size is for example 1 Tb than model would be trained locally not in Kaggle notebook and uploading 1 Tb dataset on Kaggle site would take enormous time just 'to share' the same dataset as in original source all participant have a link).

\n

2 . Can we use datasets which have CC BY 3.0 and CC BY 4.0 licenses? Can we use datasets from educational sites with copyright terms are like 'The contents of this website are intended for personal, noncommercial use. You may view or display these materials for personal, educational use.' ?

\n

To summarize: do I understand correctly that

\n\n

Thank you in advance!

", "rawMarkdown": "@masadia could you please also clarify this :\n\n1 . `datasets used for training purposes are required to be shared`\n\nI mean if we for example use datasets from https://portal.gdc.cancer.gov/ mentioned in [this discussion](https://www.kaggle.com/competitions/UBC-OCEAN/discussion/445470) does this mean that we can consider that ovarian cancer slides images from this site 'are shared' because participants have link to the source and can download this data? I ask because ovarian cancer dataset from this site is very big (about 1000 slides) and there is no sence and no possibility to upload it to Kaggle site (because if external dataset's size is for example 1 Tb than model would be trained locally not in Kaggle notebook and uploading 1 Tb dataset on Kaggle site would take enormous time just 'to share' the same dataset as in original source all participant have a link).\n\n2 . Can we use datasets which have CC BY 3.0 and CC BY 4.0 licenses? Can we use datasets from educational sites with copyright terms are like 'The contents of this website are intended for personal, noncommercial use. You may view or display these materials for personal, educational use.' ? \n\nTo summarize: do I understand correctly that \n- if we use external dataset for training than this is enough to share the link to the source of dataset in Kaggle site and this link should allow to download dataset for free (such situation was for example in the recent Bengali competition where links to external datasets were shared [in this discussion](https://www.kaggle.com/competitions/bengaliai-speech/discussion/440512)) \n- if we use any external dataset for validation we must not share anything ?\n\nThank you in advance!", "votes": 4 }, { "id": 2539392, "postDate": "2023-11-27T03:20:19.130Z", "content": "

@michaeldiazstewart Can you please confirm this. Also, can we have confirmation that we can use external data with CC BY 3.0 and CC BY 4.0 license for training.

", "rawMarkdown": "@michaeldiazstewart Can you please confirm this. Also, can we have confirmation that we can use external data with CC BY 3.0 and CC BY 4.0 license for training. ", "votes": 1 } ], "comments": [ { "id": 2543229, "author_name": "Hossein", "author_url": "", "post_date": "2023-11-29T21:56:37.703000", "content": "

As a general guideline, to explore different model architectures locally, you can use any public dataset available to all and then use the best architecture on the competition data. However, please note that usage of any dataset that affects the weights of the model is subject to Section 7C of the competition rules.

", "votes": 4, "replies": [ { "id": 2543236, "author_name": "Jonas Gjerris", "author_url": "", "post_date": "2023-11-29T22:08:34.090000", "content": "

I would argue that validation is intrinsically linked to the training process since hyperparameter adjustments, based on validation scores, significantly impact the final model weights. This intertwining raises concerns about fairness, especially if the validation data isn't uniformly accessible to all competitors.

", "votes": 1, "replies": [ { "id": 2543261, "author_name": "Hossein", "author_url": "", "post_date": "2023-11-29T23:04:36.580000", "content": "

Thank you for raising this. We just amended the comment above. This should address your concerns.

", "votes": 1, "replies": [] }, { "id": 2543266, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-11-29T23:47:15.053000", "content": "

@jonasgjerris Great point. The concern here is, can we use data for validation that is accessible to all participants but has a private license. This way the hosts can still legally use it, but participants can have better validation data.

\n

This has been done in past competitions, such as GUIE.

\n

Unfortunately, this ship has now sailed.

", "votes": 0, "replies": [] } ] }, { "id": 2543263, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-11-29T23:34:21.393000", "content": "

Thanks Hossein, this clears things up.

", "votes": 0, "replies": [] }, { "id": 2544594, "author_name": "Bartley", "author_url": "", "post_date": "2023-12-01T00:29:52.647000", "content": "

@hsfarahani Do you have any external data source recommendations?

", "votes": 1, "replies": [] } ] }, { "id": 2536078, "author_name": "Ivan Isaev", "author_url": "", "post_date": "2023-11-23T22:30:26.923000", "content": "

@masadia could you please also clarify this :

\n

1 . datasets used for training purposes are required to be shared

\n

I mean if we for example use datasets from https://portal.gdc.cancer.gov/ mentioned in this discussion does this mean that we can consider that ovarian cancer slides images from this site 'are shared' because participants have link to the source and can download this data? I ask because ovarian cancer dataset from this site is very big (about 1000 slides) and there is no sence and no possibility to upload it to Kaggle site (because if external dataset's size is for example 1 Tb than model would be trained locally not in Kaggle notebook and uploading 1 Tb dataset on Kaggle site would take enormous time just 'to share' the same dataset as in original source all participant have a link).

\n

2 . Can we use datasets which have CC BY 3.0 and CC BY 4.0 licenses? Can we use datasets from educational sites with copyright terms are like 'The contents of this website are intended for personal, noncommercial use. You may view or display these materials for personal, educational use.' ?

\n

To summarize: do I understand correctly that

\n\n

Thank you in advance!

", "votes": 4, "replies": [] }, { "id": 2539392, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-11-27T03:20:19.130000", "content": "

@michaeldiazstewart Can you please confirm this. Also, can we have confirmation that we can use external data with CC BY 3.0 and CC BY 4.0 license for training.

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2535886": "@masadia Hi, this is a question for the competition hosts. \n\nI was wondering whether we can use any external data for validating models. I already know that only open license datasets can be used for training and that they need to be accessible to all participants. But using data for validation is a different story. In GUIE [this link](https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/337384) we were allowed to use commercially licensed datasets for validation but not training. I believe it should be fine here as well. \n\nI know that MIT license datasets can be used anywhere. If you could add to this list that would be great, just to let us know which licenses are permissible. \n\nPlease reply with your answer. This is an important question, because validating data in this competition is very difficult. \n\nAlso, are datasets used for validation purposes required to be shared on the forum. I already know that datasets used for training purposes are required to be shared. ", "2543229": "As a general guideline, to explore different model architectures locally, you can use any public dataset available to all and then use the best architecture on the competition data. However, please note that usage of any dataset that affects the weights of the model is subject to Section 7C of the competition rules.", "2536078": "@masadia could you please also clarify this :\n\n1 . `datasets used for training purposes are required to be shared`\n\nI mean if we for example use datasets from https://portal.gdc.cancer.gov/ mentioned in [this discussion](https://www.kaggle.com/competitions/UBC-OCEAN/discussion/445470) does this mean that we can consider that ovarian cancer slides images from this site 'are shared' because participants have link to the source and can download this data? I ask because ovarian cancer dataset from this site is very big (about 1000 slides) and there is no sence and no possibility to upload it to Kaggle site (because if external dataset's size is for example 1 Tb than model would be trained locally not in Kaggle notebook and uploading 1 Tb dataset on Kaggle site would take enormous time just 'to share' the same dataset as in original source all participant have a link).\n\n2 . Can we use datasets which have CC BY 3.0 and CC BY 4.0 licenses? Can we use datasets from educational sites with copyright terms are like 'The contents of this website are intended for personal, noncommercial use. You may view or display these materials for personal, educational use.' ? \n\nTo summarize: do I understand correctly that \n- if we use external dataset for training than this is enough to share the link to the source of dataset in Kaggle site and this link should allow to download dataset for free (such situation was for example in the recent Bengali competition where links to external datasets were shared [in this discussion](https://www.kaggle.com/competitions/bengaliai-speech/discussion/440512)) \n- if we use any external dataset for validation we must not share anything ?\n\nThank you in advance!", "2539392": "@michaeldiazstewart Can you please confirm this. Also, can we have confirmation that we can use external data with CC BY 3.0 and CC BY 4.0 license for training. " } }