{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"colab":{"provenance":[],"gpuType":"T4"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":41875,"databundleVersionId":5521661,"sourceType":"competition"},{"sourceId":5499219,"sourceType":"datasetVersion","datasetId":3167603},{"sourceId":8051704,"sourceType":"datasetVersion","datasetId":4748350},{"sourceId":8174341,"sourceType":"datasetVersion","datasetId":4838402}],"dockerImageVersionId":30674,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# CAFA 5 protein function Prediction with TensorFlow\n\nIn this notebook we trained a simple DNN model that predicts the function of the protein i.e., **GO term ID** using it's amino acid sequences.","metadata":{}},{"cell_type":"code","source":"!pip install obonet\n!pip install pyvis\n!pip install biopython","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:24:18.89674Z","iopub.execute_input":"2024-04-20T11:24:18.897642Z","iopub.status.idle":"2024-04-20T11:24:56.770015Z","shell.execute_reply.started":"2024-04-20T11:24:18.897589Z","shell.execute_reply":"2024-04-20T11:24:56.768827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## About the Dataset\n\n### Protein Sequence\n\nEach protein is composed of dozens or hundreds of amino acids that are linked sequentially. Each amino acid in the sequence may be represented by a one-letter or three-letter code. Thus the sequence of a protein is often notated as a string of letters.","metadata":{}},{"cell_type":"markdown","source":"# Gene Ontology\n\nThe functional properties of a proteins are defined by Gene Ontology(GO). Gene Ontology (GO) describes our understanding of the biological domain with respect to three aspects:\n1. Molecular Function (MF)\n2. Biological Process (BP)\n3. Cellular Component (CC)\n\n[Source](http://geneontology.org/docs/ontology-documentation).","metadata":{}},{"cell_type":"markdown","source":"# Dataset description\n\n`train_sequences.fasta` :  contains the sequences for proteins with annotations (labelled proteins).\n\n`train_terms.tsv` :  contains the list of annotated terms (ground truth) for the proteins in `train_sequences.fasta`. \n\nIn `train_terms.tsv` the first column indicates the protein's UniProt accession ID (unique protein id), the second is the `GO Term ID`, and the third indicates in which ontology the term appears.","metadata":{}},{"cell_type":"markdown","source":"# Labels of the dataset\n\nThe objective of our model is to predict the terms (functions) of a protein sequence. One protein sequence can have many functions and can thus be classified into any number of terms. Each term is uniquely identified by a `GO Term ID`. Thus our model has to predict all the `GO Term ID`s for a protein sequence. Hence this is a multi-label classification problem.","metadata":{}},{"cell_type":"markdown","source":"# Protein embeddings for train and test data\n\nTo train a machine learning model we cannot use the alphabetical protein sequences in`train_sequences.fasta` directly. They have to be converted into a vector format. Hence we've converted the alphabetical protein sequences into numerical vector embeddings first and then train the model.\n\nProtein sequences can be converted to vector embeddings using T5Tokenizer and T5Encoder from transformers available on hugging face. This results in a vector embedding of length 1024","metadata":{}},{"cell_type":"markdown","source":"# Import the Required Libraries","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport progressbar","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:24:56.772081Z","iopub.execute_input":"2024-04-20T11:24:56.772404Z","iopub.status.idle":"2024-04-20T11:24:56.778233Z","shell.execute_reply.started":"2024-04-20T11:24:56.772372Z","shell.execute_reply":"2024-04-20T11:24:56.77722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read the data","metadata":{}},{"cell_type":"code","source":"train_terms=pd.read_csv('/kaggle/input/cafa-5-protein-function-prediction/Train/train_terms.tsv', sep=\"\\t\")\ntrain_terms.head()\ntrain_terms.shape","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:24:56.779653Z","iopub.execute_input":"2024-04-20T11:24:56.780179Z","iopub.status.idle":"2024-04-20T11:24:59.427761Z","shell.execute_reply.started":"2024-04-20T11:24:56.780147Z","shell.execute_reply":"2024-04-20T11:24:59.426785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Calculating protein embeddings","metadata":{}},{"cell_type":"markdown","source":"The folllowing code extracts the protein sequence from fasta file and extract embeddings of it using T5Tokenizer and T5Encoder imported from transformers. Each embedding is a vector of length 1024.","metadata":{}},{"cell_type":"code","source":"import os\nimport json\nfrom typing import Dict\nfrom collections import Counter\n\nimport random\nimport obonet\nimport pandas as pd\nimport numpy as np\nfrom Bio import SeqIO\nimport re\nfrom transformers import T5Tokenizer, T5EncoderModel\nimport torch\ndevice = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n\n# Load the tokenizer\ntokenizer = T5Tokenizer.from_pretrained('Rostlab/prot_t5_xl_half_uniref50-enc', do_lower_case=False) #.to(device)\n\n# Load the model\nmodel = T5EncoderModel.from_pretrained(\"Rostlab/prot_t5_xl_half_uniref50-enc\").to(device)\n\ndef get_embeddings(seq):\n    sequence_examples = [\" \".join(list(re.sub(r\"[UZOB]\", \"X\", seq)))]\n\n    ids = tokenizer.batch_encode_plus(sequence_examples, add_special_tokens=True, padding=\"longest\")\n\n    input_ids = torch.tensor(ids['input_ids']).to(device)\n    attention_mask = torch.tensor(ids['attention_mask']).to(device)\n\n    # generate embeddings\n    with torch.no_grad():\n        embedding_repr = model(input_ids=input_ids,\n                               attention_mask=attention_mask)\n\n    # extract residue embeddings for the first ([0,:]) sequence in the batch and remove padded & special tokens ([0,:7]) \n    emb_0 = embedding_repr.last_hidden_state[0]\n    emb_0_per_protein = emb_0.mean(dim=0)\n    \n    return emb_0_per_protein\n\nfile = '/kaggle/input/cafa-5-protein-function-prediction/Train/train_sequences.fasta'\nsequences = SeqIO.parse(file, \"fasta\")\nprint(next(iter(sequences)).seq)","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:24:59.430667Z","iopub.execute_input":"2024-04-20T11:24:59.431093Z","iopub.status.idle":"2024-04-20T11:25:03.380505Z","shell.execute_reply.started":"2024-04-20T11:24:59.431065Z","shell.execute_reply":"2024-04-20T11:25:03.377815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sequences = SeqIO.parse(file, \"fasta\")\nget_embeddings(str(next(iter(sequences)).seq))","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.381172Z","iopub.status.idle":"2024-04-20T11:25:03.381511Z","shell.execute_reply.started":"2024-04-20T11:25:03.381332Z","shell.execute_reply":"2024-04-20T11:25:03.381358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading the protein embeddings\n\n\nWe can calculate the protein embeddings on our own, but the kaggle resources are not enough to collect embeddings for all the sequences in train dataset. So we are using the pre-calculated protein embeddings done by [Sergei Fironov](https://www.kaggle.com/sergeifironov) using the Rost Lab's T5 protein language model.\n\nIf the `t5embeds` is not yet added to the input data of the notebook, it can be added by clicking on `Add Data` and search for `t5embeds` and then click on the `+` beside it.\n\nThe protein embeddings to be used for training are recorded in `train_embeds.npy` and the corresponding protein ids are available in `train_ids.npy`.","metadata":{}},{"cell_type":"markdown","source":"We will load the protein ids of the protein embeddings in the train dataset contained in `train_ids.npy` into a numpy array.","metadata":{}},{"cell_type":"code","source":"train_protein_ids = np.load('/kaggle/input/t5embeds/train_ids.npy')\ntrain_protein_ids_pd=pd.DataFrame(train_protein_ids)\ntrain_protein_ids_pd.head(100)","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.383078Z","iopub.status.idle":"2024-04-20T11:25:03.383447Z","shell.execute_reply.started":"2024-04-20T11:25:03.383248Z","shell.execute_reply":"2024-04-20T11:25:03.383263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<!-- Now, we will load`train_embeds.py` which contains the pre-calculated embeddings of the proteins in the train dataset. with protein_ids (`id`s we loaded previously from the **train_ids.npy**) into a numpy array. This array now contains the precalculated embeddings for the protein_ids( Ids we loaded above from **train_ids.npy**) needed for training. -->\n\nAfter loading the files as numpy arrays, we will convert them into Pandas dataframe.\n\nEach protein embedding is a vector of length 1024. We create the resulting dataframe such that there are 1024 columns to represent the values in each of the 1024 places in the vector.","metadata":{}},{"cell_type":"code","source":"train_embeddings = np.load('/kaggle/input/t5embeds/train_embeds.npy')\ncolumn_num = train_embeddings.shape[1]\ntrain_df  = pd.DataFrame(train_embeddings, columns = [\"Column_\" + str(i) for i in range(1, column_num+1)])\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.384264Z","iopub.status.idle":"2024-04-20T11:25:03.384626Z","shell.execute_reply.started":"2024-04-20T11:25:03.384457Z","shell.execute_reply":"2024-04-20T11:25:03.384472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare the dataset","metadata":{}},{"cell_type":"markdown","source":"There are more than 40,000 labels in the `train_terms.tsv` file, so to simplify we will choose the most frequent 1500 `GO term IDs` as labels of the dataset.","metadata":{}},{"cell_type":"code","source":"# Extracting Go IDs\nnum_of_labels = 1500\nlabels = train_terms['term'].value_counts().index\nlabels=labels[:num_of_labels]\nlabels","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.385662Z","iopub.status.idle":"2024-04-20T11:25:03.386032Z","shell.execute_reply.started":"2024-04-20T11:25:03.385856Z","shell.execute_reply":"2024-04-20T11:25:03.385871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extracting the dataset which contains the top 1500(labels) GO terms\ntrain_terms_updated = train_terms.loc[train_terms['term'].isin(labels)]\ntrain_terms_updated=train_terms_updated.reset_index(drop=True)\ntrain_terms_updated","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.387131Z","iopub.status.idle":"2024-04-20T11:25:03.387577Z","shell.execute_reply.started":"2024-04-20T11:25:03.387341Z","shell.execute_reply":"2024-04-20T11:25:03.387374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plot the most frequent 100 `GO Term ID`s in `train_terms.tsv`.","metadata":{}},{"cell_type":"code","source":"pie_df = train_terms_updated['aspect'].value_counts()\npalette_color = sns.color_palette('bright')\nplt.pie(pie_df.values, labels=np.array(pie_df.index), colors=palette_color, autopct='%.0f%%')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.389065Z","iopub.status.idle":"2024-04-20T11:25:03.389547Z","shell.execute_reply.started":"2024-04-20T11:25:03.389283Z","shell.execute_reply":"2024-04-20T11:25:03.389302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As evident from the pie chart above, majority of the `GO term Id`s have their aspect as BPO(Biological Process Ontology). In the labels array, absence or presence of each `GO term Id` is denoted by 0 or 1.","metadata":{}},{"cell_type":"markdown","source":"Uncomment the next 2 cells if labels are not already available in the datasets directory","metadata":{}},{"cell_type":"code","source":"# train_size = train_protein_ids.shape[0]\n# train_labels = np.zeros((train_size ,num_of_labels))\n# series_train_protein_ids = pd.Series(train_protein_ids)\n\n# bar = progressbar.ProgressBar(maxval=num_of_labels, widgets=[progressbar.Bar('=', '[', ']'), ' ', progressbar.Percentage()])\n# for i in range(num_of_labels):\n#     n_train_terms = train_terms_updated[train_terms_updated['term'] ==  labels[i]]\n#     label_related_proteins = n_train_terms['EntryID'].unique()\n#     train_labels[:,i] =  series_train_protein_ids.isin(label_related_proteins).astype(float)\n#     bar.update(i+1)\n# bar.finish()\n\n# labels_df = pd.DataFrame(data = train_labels, columns = labels)\n# print(labels_df.shape)","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.391829Z","iopub.status.idle":"2024-04-20T11:25:03.392303Z","shell.execute_reply.started":"2024-04-20T11:25:03.392061Z","shell.execute_reply":"2024-04-20T11:25:03.392082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`label_df` is composed of 1500 columns and 142246 entries.","metadata":{}},{"cell_type":"code","source":"# #!mkdir /kaggle/working/labels\n# labels_df = pd.DataFrame(data = train_labels, columns = labels)\n# labels_df.to_csv('/kaggle/working/labels/kaggledata.csv')","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.393381Z","iopub.status.idle":"2024-04-20T11:25:03.393854Z","shell.execute_reply.started":"2024-04-20T11:25:03.393613Z","shell.execute_reply":"2024-04-20T11:25:03.393634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Run the next cell only if labels are already available in the datasets directory","metadata":{}},{"cell_type":"code","source":"labels_df=pd.read_csv('/kaggle/input/labels/kaggledata.csv')\nlabels_df=labels_df.drop(columns='Unnamed: 0')","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.395435Z","iopub.status.idle":"2024-04-20T11:25:03.395771Z","shell.execute_reply.started":"2024-04-20T11:25:03.395613Z","shell.execute_reply":"2024-04-20T11:25:03.395627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels_df.shape","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.397538Z","iopub.status.idle":"2024-04-20T11:25:03.39802Z","shell.execute_reply.started":"2024-04-20T11:25:03.397758Z","shell.execute_reply":"2024-04-20T11:25:03.397778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.39917Z","iopub.status.idle":"2024-04-20T11:25:03.399619Z","shell.execute_reply.started":"2024-04-20T11:25:03.399391Z","shell.execute_reply":"2024-04-20T11:25:03.399411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training\n\nWe will use Tensorflow to train a Deep Neural Network with the protein embeddings to perform mult-label classification.","metadata":{}},{"cell_type":"code","source":"train_df.shape","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.400887Z","iopub.status.idle":"2024-04-20T11:25:03.401204Z","shell.execute_reply.started":"2024-04-20T11:25:03.401048Z","shell.execute_reply":"2024-04-20T11:25:03.401061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels_df.shape","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.402449Z","iopub.status.idle":"2024-04-20T11:25:03.402779Z","shell.execute_reply.started":"2024-04-20T11:25:03.402619Z","shell.execute_reply":"2024-04-20T11:25:03.402633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nfrom sklearn.model_selection import train_test_split\n\n# Split the data into train and validation sets\ntrain_data, val_data, train_labels, val_labels = train_test_split(train_df, labels_df, test_size=0.2, random_state=42)\n\n# Define model architecture\nINPUT_SHAPE = [train_df.shape[1]]\nnum_of_labels = labels_df.shape[1]\n\nBATCH_SIZE = 5120\n\nmodel = tf.keras.Sequential([\n    tf.keras.layers.BatchNormalization(input_shape=INPUT_SHAPE),\n    tf.keras.layers.Dense(units=512, activation='relu'),\n    tf.keras.layers.Dropout(0.2),\n    tf.keras.layers.Dense(units=512, activation='relu'),\n    tf.keras.layers.Dropout(0.2),\n    tf.keras.layers.Dense(units=512, activation='relu'),\n    tf.keras.layers.Dropout(0.2),\n    tf.keras.layers.Dense(units=512, activation='relu'),\n    tf.keras.layers.Dropout(0.2),\n    tf.keras.layers.Dense(units=512, activation='relu'),\n    tf.keras.layers.Dropout(0.2),\n    tf.keras.layers.Dense(units=num_of_labels, activation='sigmoid')\n])\n\nmodel.compile(\n    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),\n    loss='binary_crossentropy',\n    metrics=['binary_accuracy', tf.keras.metrics.AUC()]\n)\n\n# Train the model with validation split\nhistory = model.fit(\n    train_data,\n    train_labels,\n    batch_size=BATCH_SIZE,\n    epochs=20, # change this as needed recommended 100 epochs\n    validation_data=(val_data, val_labels)\n)\n\n# Get the final training and validation metrics\nfinal_train_metrics = model.evaluate(train_data, train_labels)\nfinal_val_metrics = model.evaluate(val_data, val_labels)\n\nprint(\"Final Training Metrics:\")\nprint(\"Loss:\", final_train_metrics[0])\nprint(\"Binary Accuracy:\", final_train_metrics[2])\nprint(\"AUC:\", final_train_metrics[1])\n\nprint(\"\\nFinal Validation Metrics:\")\nprint(\"Loss:\", final_val_metrics[0])\nprint(\"Binary Accuracy:\", final_val_metrics[2])\nprint(\"AUC:\", final_val_metrics[1])\n","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.404361Z","iopub.status.idle":"2024-04-20T11:25:03.404724Z","shell.execute_reply.started":"2024-04-20T11:25:03.404555Z","shell.execute_reply":"2024-04-20T11:25:03.40457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.save(\"/kaggle/working/model1.h5\")","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.405915Z","iopub.status.idle":"2024-04-20T11:25:03.406253Z","shell.execute_reply.started":"2024-04-20T11:25:03.406093Z","shell.execute_reply":"2024-04-20T11:25:03.406107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.save_weights(\"/kaggle/working/model.weights.h5\")","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.407407Z","iopub.status.idle":"2024-04-20T11:25:03.407859Z","shell.execute_reply.started":"2024-04-20T11:25:03.407612Z","shell.execute_reply":"2024-04-20T11:25:03.407631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.load_weights(\"/kaggle/working/model.weights.h5\")","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.409308Z","iopub.status.idle":"2024-04-20T11:25:03.409815Z","shell.execute_reply.started":"2024-04-20T11:25:03.40949Z","shell.execute_reply":"2024-04-20T11:25:03.409504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_train_metrics = model.evaluate(train_data, train_labels)\nfinal_val_metrics = model.evaluate(val_data, val_labels)\n\nprint(\"Final Training Metrics:\")\nprint(\"Loss:\", final_train_metrics[0])\nprint(\"Binary Accuracy:\", final_train_metrics[1])\nprint(\"AUC:\", final_train_metrics[2])\n\nprint(\"\\nFinal Validation Metrics:\")\nprint(\"Loss:\", final_val_metrics[0])\nprint(\"Binary Accuracy:\", final_val_metrics[1])\nprint(\"AUC:\", final_val_metrics[2])\n","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.411154Z","iopub.status.idle":"2024-04-20T11:25:03.411508Z","shell.execute_reply.started":"2024-04-20T11:25:03.411313Z","shell.execute_reply":"2024-04-20T11:25:03.411326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"custom_input_tensor = np.array([[1 for i in range(1024)]])\nprint(custom_input_tensor)\nprint(len(custom_input_tensor[0]))\n# Get predictions for custom input tensor\npredictions = model.predict(custom_input_tensor)\n\n# 'predictions' will contain the model's output for the custom input tensor\nprint(predictions)\nfor i in predictions[0]:\n    x=0 if i<0.5 else 1\n    print(x)\n# print(len(predictions))\n","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.412871Z","iopub.status.idle":"2024-04-20T11:25:03.413226Z","shell.execute_reply.started":"2024-04-20T11:25:03.413058Z","shell.execute_reply":"2024-04-20T11:25:03.413073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot the model's loss and accuracy for each epoch","metadata":{}},{"cell_type":"code","source":"history_df = pd.DataFrame(history.history)\nhistory_df.loc[:, ['loss']].plot(title=\"Cross-entropy\")\nhistory_df.loc[:, ['binary_accuracy']].plot(title=\"Accuracy\")","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.414388Z","iopub.status.idle":"2024-04-20T11:25:03.414694Z","shell.execute_reply.started":"2024-04-20T11:25:03.414543Z","shell.execute_reply":"2024-04-20T11:25:03.414556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# following is code for running inference","metadata":{}},{"cell_type":"markdown","source":"# we have added a test file too","metadata":{}},{"cell_type":"code","source":"# !pip install gradio","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.415727Z","iopub.status.idle":"2024-04-20T11:25:03.416165Z","shell.execute_reply.started":"2024-04-20T11:25:03.415933Z","shell.execute_reply":"2024-04-20T11:25:03.415951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tqdm\nfrom Bio import SeqIO\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport os\nimport json\nfrom typing import Dict\nfrom collections import Counter\nimport random\nimport obonet\nfrom transformers import T5Tokenizer, T5EncoderModel\nimport torch\nimport re\n# import gradio as gr\n\ndevice = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n\n# Load the tokenizer\ntokenizer = T5Tokenizer.from_pretrained('Rostlab/prot_t5_xl_half_uniref50-enc', do_lower_case=False) #.to(device)\n\n# Load the model\nmodel = T5EncoderModel.from_pretrained(\"Rostlab/prot_t5_xl_half_uniref50-enc\").to(device)\n\ndef get_embeddings(seq):\n    sequence_examples = [\" \".join(list(re.sub(r\"[UZOB]\", \"X\", seq)))]\n\n    ids = tokenizer.batch_encode_plus(sequence_examples, add_special_tokens=True, padding=\"longest\")\n\n    input_ids = torch.tensor(ids['input_ids']).to(device)\n    attention_mask = torch.tensor(ids['attention_mask']).to(device)\n\n    # generate embeddings\n    with torch.no_grad():\n        embedding_repr = model(input_ids=input_ids,\n                               attention_mask=attention_mask)\n\n    # extract residue embeddings for the first ([0,:]) sequence in the batch and remove padded & special tokens ([0,:7])\n    emb_0 = embedding_repr.last_hidden_state[0]\n    emb_0_per_protein = emb_0.mean(dim=0)\n\n    return emb_0_per_protein\n\ndef predict(fasta_file):\n    sequences = SeqIO.parse(fasta_file, \"fasta\")\n\n    ids = []\n    num_sequences=sum(1 for seq in sequences)\n    embeds = np.zeros((num_sequences, 1024))\n    i = 0\n    with open(fasta_file, \"r\") as fastafile:\n      # Iterate over each sequence in the file\n      for sequence in SeqIO.parse(fastafile, \"fasta\"):\n        # Access the sequence ID and sequence data\n        seq_id = sequence.id\n        seq_data = str(sequence.seq)\n        embeds[i] = get_embeddings(seq_data).detach().cpu().numpy()\n#         print(embeds[i])\n        ids.append(seq_id)\n        i += 1\n        \n    INPUT_SHAPE=[1024]\n    num_of_labels=1500\n\n    model = tf.keras.Sequential([\n        tf.keras.layers.BatchNormalization(input_shape=INPUT_SHAPE),\n        tf.keras.layers.Dense(units=512, activation='relu'),\n        tf.keras.layers.Dropout(0.2),\n        tf.keras.layers.Dense(units=512, activation='relu'),\n        tf.keras.layers.Dropout(0.2),\n        tf.keras.layers.Dense(units=512, activation='relu'),\n        tf.keras.layers.Dropout(0.2),\n        tf.keras.layers.Dense(units=512, activation='relu'),\n        tf.keras.layers.Dropout(0.2),\n        tf.keras.layers.Dense(units=512, activation='relu'),\n        tf.keras.layers.Dropout(0.2),\n        tf.keras.layers.Dense(units=num_of_labels, activation='sigmoid')\n    ])\n\n    model.compile(\n        optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),\n        loss='binary_crossentropy',\n        metrics=['binary_accuracy', tf.keras.metrics.AUC()]\n    )\n    \n    model.load_weights('/kaggle/working/model.weights.h5') #load model here\n    labels_df=pd.read_csv('/kaggle/input/labels/kaggledata.csv')\n    labels_df=labels_df.drop(columns='Unnamed: 0')\n\n    predictions = model.predict(embeds)\n    predictions_list1=[]\n    predictions_list2=[]\n\n    # 'predictions' will contain the model's output for the custom input tensor\n    # print(predictions)\n    for prediction in predictions:\n        tmp=[]\n        t2=[]\n        for i in prediction:\n            x=0 if i<0.4 else 1\n            tmp.append(x)\n            t2.append(i)\n        predictions_list1.append(tmp.copy())\n        predictions_list2.append(t2.copy())\n\n    label_columns = labels_df.columns\n\n    # Convert the predictions into a DataFrame\n    predictions_df = pd.DataFrame(predictions_list1, columns=label_columns)\n    p21=pd.DataFrame(predictions_list2, columns=label_columns)\n\n    # Save the DataFrame to a CSV file\n    predictions_df.to_csv(\"predictions.csv\", index=False) #output csv\n    p21.to_csv(\"decimal.csv\",index=False)\n    \n","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.417608Z","iopub.status.idle":"2024-04-20T11:25:03.418055Z","shell.execute_reply.started":"2024-04-20T11:25:03.417816Z","shell.execute_reply":"2024-04-20T11:25:03.417836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predict('/kaggle/input/fastaexample/example.fasta') #after this you will get a predictions.csv and deciaml.csv in the working directory","metadata":{"execution":{"iopub.status.busy":"2024-04-20T11:25:03.419887Z","iopub.status.idle":"2024-04-20T11:25:03.420393Z","shell.execute_reply.started":"2024-04-20T11:25:03.420127Z","shell.execute_reply":"2024-04-20T11:25:03.42015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}