{ "id": 495301, "title": "Many columns with one unique value in the training set.", "url": "/competitions/leap-atmospheric-physics-ai-climsim/discussion/495301", "author_name": "Catadanna", "post_date": "2024-04-20T13:38:53.858000", "votes": 11, "comment_count": 7, "views": 0, "content": "

It seems that in the training set there are a lot of features with only one unique value.
\nI made a notebook with an example code which illustrates that for one feature.
\nYou can find it here.

\n

These columns should be deleted for training in order to get a better result.

\n

UPDATE

\n

I add here a notebook which selects the columns with only one unique value.

", "messages": [ { "id": 2763345, "postDate": "2024-04-20T13:38:53.860Z", "content": "

It seems that in the training set there are a lot of features with only one unique value.
\nI made a notebook with an example code which illustrates that for one feature.
\nYou can find it here.

\n

These columns should be deleted for training in order to get a better result.

\n

UPDATE

\n

I add here a notebook which selects the columns with only one unique value.

", "rawMarkdown": "It seems that in the training set there are a lot of features with only one unique value. \nI made a notebook with an example code which illustrates that for one feature.\nYou can find it [here](https://www.kaggle.com/code/catadanna/leap-get-features-with-one-unique-value).\n\nThese columns should be deleted for training in order to get a better result.\n\n**UPDATE**\n\nI add [here](https://www.kaggle.com/code/catadanna/columns-to-delete/notebook) a notebook which selects the columns with only one unique value.", "votes": 10 }, { "id": 2764101, "postDate": "2024-04-20T22:31:46.203Z", "content": "

i think all 60 values of each vector input should be interpreted as a series. If some elements of the series are always 0, they still should be kept for processing like CNN or RNN. Otherwise they could be dropped.

", "rawMarkdown": "i think all 60 values of each vector input should be interpreted as a series. If some elements of the series are always 0, they still should be kept for processing like CNN or RNN. Otherwise they could be dropped.", "votes": 7, "replies": [ { "id": 2764111, "postDate": "2024-04-20T23:02:40.990Z", "content": "

Good point.

", "rawMarkdown": "Good point." }, { "id": 2765506, "postDate": "2024-04-21T07:01:14.287Z", "content": "

I agree that they should be treated as series, maybe some kind of sequential modelling could be useful. I think we can still drop them if they always have the same value within different samples.

", "rawMarkdown": "I agree that they should be treated as series, maybe some kind of sequential modelling could be useful. I think we can still drop them if they always have the same value within different samples.", "votes": 2 } ] }, { "id": 2763748, "postDate": "2024-04-20T18:04:57.763Z", "content": "

I noticed that too. Maybe this is happening due to 32-bit float downcasting.

", "rawMarkdown": "I noticed that too. Maybe this is happening due to 32-bit float downcasting.", "replies": [ { "id": 2764110, "postDate": "2024-04-20T23:01:13.607Z", "content": "

Was there a downcast though?

", "rawMarkdown": "Was there a downcast though?", "replies": [ { "id": 2765284, "postDate": "2024-04-21T04:46:04.373Z", "content": "

I'm not able to load all columns as 64-bit floats so I do it myself and I assumed that she would also do that.

", "rawMarkdown": "I'm not able to load all columns as 64-bit floats so I do it myself and I assumed that she would also do that.", "votes": 1 } ] }, { "id": 2765739, "postDate": "2024-04-21T10:27:33.547Z", "content": "

In the related notebook (see the link) there is no downcasting so we have float64.

", "rawMarkdown": "In the related notebook (see the link) there is no downcasting so we have float64." } ] } ], "comments": [ { "id": 2764101, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2024-04-20T22:31:46.203000", "content": "

i think all 60 values of each vector input should be interpreted as a series. If some elements of the series are always 0, they still should be kept for processing like CNN or RNN. Otherwise they could be dropped.

", "votes": 7, "replies": [ { "id": 2764111, "author_name": "greySnow", "author_url": "", "post_date": "2024-04-20T23:02:40.990000", "content": "

Good point.

", "votes": 0, "replies": [] }, { "id": 2765506, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-04-21T07:01:14.287000", "content": "

I agree that they should be treated as series, maybe some kind of sequential modelling could be useful. I think we can still drop them if they always have the same value within different samples.

", "votes": 2, "replies": [] } ] }, { "id": 2763748, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-04-20T18:04:57.763000", "content": "

I noticed that too. Maybe this is happening due to 32-bit float downcasting.

", "votes": 0, "replies": [ { "id": 2764110, "author_name": "greySnow", "author_url": "", "post_date": "2024-04-20T23:01:13.607000", "content": "

Was there a downcast though?

", "votes": 0, "replies": [ { "id": 2765284, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-04-21T04:46:04.373000", "content": "

I'm not able to load all columns as 64-bit floats so I do it myself and I assumed that she would also do that.

", "votes": 1, "replies": [] } ] }, { "id": 2765739, "author_name": "Catadanna", "author_url": "", "post_date": "2024-04-21T10:27:33.547000", "content": "

In the related notebook (see the link) there is no downcasting so we have float64.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2763345": "It seems that in the training set there are a lot of features with only one unique value. \nI made a notebook with an example code which illustrates that for one feature.\nYou can find it [here](https://www.kaggle.com/code/catadanna/leap-get-features-with-one-unique-value).\n\nThese columns should be deleted for training in order to get a better result.\n\n**UPDATE**\n\nI add [here](https://www.kaggle.com/code/catadanna/columns-to-delete/notebook) a notebook which selects the columns with only one unique value.", "2764101": "i think all 60 values of each vector input should be interpreted as a series. If some elements of the series are always 0, they still should be kept for processing like CNN or RNN. Otherwise they could be dropped.", "2763748": "I noticed that too. Maybe this is happening due to 32-bit float downcasting." } }