{ "topic": { "id": 402648, "title": "ProteinBert - Protein Function language Model", "authorName": "Dan Ofer", "commentCount": 15, "votes": 45, "postDate": "2023-04-19T07:00:25.335000" }, "comments": [ { "id": 2239458, "authorName": "snowdrop613", "votes": 1, "postDate": "2023-04-29T14:08:29.590000", "content": "

Thanks for sending the link!

\n

Has anyone given BioBERT a go on Kaggle notebooks yet?

\n

I tried the protein-bert demo after installing it with pip, but it threw an error when running the finetune func.

\n
\n

AttributeError: 'Adam' object has no attribute 'get_weights'

\n
\n

I heard the error has already been fixed, but I can't seem to install it with pip.
\nI also attempted to clone it from GitHub, but it didn't work out.

\n

Do you have any ideas on what I should do next?

" }, { "id": 2229759, "authorName": "Alexander Chervov", "votes": 1, "postDate": "2023-04-21T16:49:01.297000", "content": "

Cool !
\nWould it be possible to generate the embeddings for the train/test here and share them with the community in the Kaggle datasets ?

\n

What is the average time for inference on sequences like we have here ?

" }, { "id": 2231515, "authorName": "Dan Ofer", "votes": 2, "postDate": "2023-04-23T11:42:55.750000", "content": "

It's very fast. I'd love to share such embeddings! If you have the capacity to run them and to share as a dataset, i'd be delighted to link to them from the protein_bert repo as well!

" }, { "id": 2231527, "authorName": "Alexander Chervov", "votes": 1, "postDate": "2023-04-23T11:57:02.310000", "content": "

Thanks for the comment !
\nKaggle itself gives unlimited capacity to share the data - just create public dataset and put <50G there.
\nIf it is more than 50G - just create several datasets. Notebook can access any number of datasets.
\nThe only technical remark is that huge Kaggle datasets - with more than say 5G on - leads to quite long notebook initialization - when Kaggle loads the dataset to the notebook environment, so sometimes it is better to have several smaller datasets rather than one big.

" }, { "id": 2229695, "authorName": "Andrew", "votes": 2, "postDate": "2023-04-21T15:56:39.923000", "content": "

Hi @danofer,

\n

Thanks for the link. I wonder if you have comparative data on this vs other protein embeddings. I've found one comparison that puts ProtT5 ahead of ProteinBert by ~2 percentage points for GO prediction across all of BP, MF, CC. But that comparison was done by the author of ProtT5 so perhaps it uses ProteinBert in a sub-optimal way or has other problems that make it misleading.

\n

Is there anything you can point to that should lead me to question those findings or does it look right to you? (It sounds like ProtT5 is much bigger than ProteinBert, which probably makes it harder to work with. So perhaps the key advantage of ProteinBert is its small size?)

\n

Thanks,
\nAndrew

" }, { "id": 2231387, "authorName": "Dan Ofer", "votes": 1, "postDate": "2023-04-23T09:01:29.510000", "content": "

They didn't compare ProteinBert there, they compared the author's own ProtBERT (which has a different architecture from us, and didn't do a GO pretraining task).

\n

PS - We were very stringent in filtering out annotations in the pretraining, but that's a seperate thing :)

" }, { "id": 2236192, "authorName": "Andrew", "votes": 1, "postDate": "2023-04-26T15:52:38.763000", "content": "

Sorry, my mistake. I had assumed ProtBert = ProteinBert. Thanks for the clarification.

" }, { "id": 2236882, "authorName": "Dan Ofer", "votes": 0, "postDate": "2023-04-27T07:00:12.283000", "content": "

Haha. It's a legit mistake. There's little in common between the two though, surprisingly ;)

" }, { "id": 2227164, "authorName": "nhgiang", "votes": 1, "postDate": "2023-04-19T14:45:14.057000", "content": "

I'm also trying this approach and did some minimal work to prepare data for finetuning ProteinBERT. Kaggle-provided environment doesn't have enough resources, though, so you'll have to run it elsewhere.

\n

See my shared notebook in this competition (I've posted the link too many times for Kaggle's spam-limit)

" }, { "id": 2229232, "authorName": "Dan Ofer", "votes": 1, "postDate": "2023-04-21T07:14:49.247000", "content": "

Kaggle's env should easily have enough resources, it's a 16M model! People ran it in other protein competitions. Where's the bottleneck for you?

" }, { "id": 2398409, "authorName": "Levi Junyan Zhang", "votes": 0, "postDate": "2023-08-19T17:02:37.890000", "content": "

really cooooool

" }, { "id": 2327818, "authorName": "", "votes": 0, "postDate": "2023-07-03T07:33:32.713000", "content": "

@danofer , hello

\n

I looking for CAFA 3 4 and cannot find them..
\nCan you help me find link to CAFA 3 CAFA 4 ?

" }, { "id": 2328015, "authorName": "Alexander Chervov", "votes": 1, "postDate": "2023-07-03T09:50:26.670000", "content": "

https://www.kaggle.com/datasets/alexandervc/cafa-protein-function-annotation-challenges

" }, { "id": 2307093, "authorName": "Maryam Mohammadi", "votes": 0, "postDate": "2023-06-17T19:50:36.550000", "content": "

oh really thanks for sharing the github link of this great project

" }, { "id": 2259380, "authorName": "Abish Pius", "votes": 0, "postDate": "2023-05-15T00:21:05.937000", "content": "

Could I get some help with running the inference using the pre-trained model, not sure where to start to be hones.

" } ], "index": { "id": "402648", "title": "ProteinBert - Protein Function language Model", "authorName": "", "commentCount": "15", "votes": "45", "postDate": "2023-04-19 07:00:25.335000" }, "competition": "cafa-5-protein-function-prediction" }