{
  "id": 667106,
  "title": "Running the evaluation metric",
  "url": "/competitions/stanford-rna-3d-folding-2/discussion/667106",
  "author_name": "Rhiju Das",
  "post_date": "2026-01-11T04:30:55.593000",
  "votes": 21,
  "comment_count": 10,
  "views": 0,
  "content": "<p>It's great to see so many entries &amp; shared notebooks already! In the early weeks of this competition, we may share some tips learned from the prior competition, which you're very welcome to use or ignore as you see fit.</p>\n<p>Here's one tip, also mentioned in the <a href=\"https://www.kaggle.com/competitions/stanford-rna-3d-folding-2/discussion/666732\" target=\"_blank\">Getting started</a> post:</p>\n<p>To identify problem cases for your code, you may find it helpful to get scores back on the validation targets individually, and it is possible to do this within your notebooks using the competition evaluation metric. </p>\n<p>To make this a little easier, we've updated the <a href=\"https://www.kaggle.com/code/rhijudas/mmseqs2-3d-rna-template-identification-part-2/\" target=\"_blank\">MMseqs2 example notebook</a> to include example code to get these target-by-target validation scores. For ease of reference, the steps are also written out below.</p>\n<p>More tips coming -- looking forward to more discussions and notebook sharing.</p>\n<p>Good luck!\nRhiju</p>\n<p>Here are the steps to get TM-scores on the validation set in your notebook:</p>\n<ul>\n<li><strong>Step 1.</strong>  Go to 'Add input…', and add the following two inputs to your notebook:</li>\n</ul>\n<pre><code>https://www.kaggle.com/code/rhijudas/tm-score-permutechains/data\nhttps://www.kaggle.com/datasets/metric/usalign\n</code></pre>\n<ul>\n<li><strong>Step 2.</strong>  Add the following three code blocks at the end of your notebook:</li>\n</ul>\n<pre><code># install the metric and defined the score() function\n!ls /kaggle/usr/lib/\nimport runpy\nmodule_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py\")\nscore = module_globals['score']\n</code></pre>\n<pre><code># Read in the validation solution and the output submission.csv from your notebook\nimport pandas as pd\nsol = pd.read_csv('/kaggle/input/stanford-rna-3d-folding-2/validation_labels.csv')\nsub = pd.read_csv('/kaggle/working/submission.csv')\n</code></pre>\n<pre><code># Run the eval code on each target and get the score, if this is a notebook run using the public test set (whose size matches the solution file, `validation_labels.csv`]\nsol['target_id'] = sol['ID'].apply(lambda x: '_'.join(str(x).split('_')[:-1]))\nsub['target_id'] = sub['ID'].apply(lambda x: '_'.join(str(x).split('_')[:-1]))\n\nif len(sol)==len(sub): # This tests if we're looking at public val\n    results = []\n    for target_id, group_native in sol.groupby('target_id'):\n        group_predicted = sub[sub['target_id'] == target_id]\n        result = score(group_native,group_predicted,'ID')\n        print(target_id,result)\n        results.append( result )\n    print( 'Mean score:',  \n          float(sum(results) / len(results)) if len(results)&gt;0 else 0.0, \n          f'(n={len(results)})' )\n</code></pre>\n<ul>\n<li><strong>Step 3</strong> Inspect the outputs\nIn the example MMseqs2 notebook, the output on val is:</li>\n</ul>\n<pre><code>8ZNQ 0.03169\n9CFN 0.04012\n9E74 0.56897\n9E75 0.06648\n9E9Q 0.5597\n9EBP 0.04562\n9G4J 0.92386\n9G4P 0.04276\n9G4Q 0.04655\n9G4R 0.03152\n9HRO 0.02404\n9I9W 0.02199\n9IWF 0.03934\n9J09 0.051\n9JFO 0.05872\n9JFS 0.06211\n9JGM 0.0786\n9KGG 0.09895\n9LEC 0.85576\n9LEL 0.68292\n9LJN 0.78599\n9MME 0.12252\n9OBM 0.0334\n9OD4 0.02793\n9QZJ 0.02456\n9RVP 0.02432\n9WHV 0.04899\n9ZCC 0.09892\nMean score: 0.1963332142857143 (n=28)\n</code></pre>\n<p>Inspection of the output reveals several cases that could use some additional thought:</p>\n<ul>\n<li>For example, <code>9MME</code> is an octamer target where MMseqs2 actually does find a 3D template, but the code outputs only a single chain of the octamer instead of all 8.</li>\n<li><code>9J09</code> is a case where there actually is a template that should be identifiable based on the similarity of function of the target (a 'mini'-CRISPR/Cas13 gene editing complex; see the <code>description</code> field) and distantly related homologs (other Cas13 complexes and the natural ancestral complex, TnpB). </li>\n</ul>\n<p>For more discussion of these particular targets (and a few others in the validation set), check out the <a href=\"https://www.biorxiv.org/content/10.64898/2025.12.30.696949v1\" target=\"_blank\">preprint on Part 1 of this competition</a>.</p>",
  "messages": [
    {
      "id": 3389401,
      "postDate": "2026-01-11T04:30:55.593Z",
      "content": "<p>It's great to see so many entries &amp; shared notebooks already! In the early weeks of this competition, we may share some tips learned from the prior competition, which you're very welcome to use or ignore as you see fit.</p>\n<p>Here's one tip, also mentioned in the <a href=\"https://www.kaggle.com/competitions/stanford-rna-3d-folding-2/discussion/666732\" target=\"_blank\">Getting started</a> post:</p>\n<p>To identify problem cases for your code, you may find it helpful to get scores back on the validation targets individually, and it is possible to do this within your notebooks using the competition evaluation metric. </p>\n<p>To make this a little easier, we've updated the <a href=\"https://www.kaggle.com/code/rhijudas/mmseqs2-3d-rna-template-identification-part-2/\" target=\"_blank\">MMseqs2 example notebook</a> to include example code to get these target-by-target validation scores. For ease of reference, the steps are also written out below.</p>\n<p>More tips coming -- looking forward to more discussions and notebook sharing.</p>\n<p>Good luck!\nRhiju</p>\n<p>Here are the steps to get TM-scores on the validation set in your notebook:</p>\n<ul>\n<li><strong>Step 1.</strong>  Go to 'Add input…', and add the following two inputs to your notebook:</li>\n</ul>\n<pre><code>https://www.kaggle.com/code/rhijudas/tm-score-permutechains/data\nhttps://www.kaggle.com/datasets/metric/usalign\n</code></pre>\n<ul>\n<li><strong>Step 2.</strong>  Add the following three code blocks at the end of your notebook:</li>\n</ul>\n<pre><code># install the metric and defined the score() function\n!ls /kaggle/usr/lib/\nimport runpy\nmodule_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py\")\nscore = module_globals['score']\n</code></pre>\n<pre><code># Read in the validation solution and the output submission.csv from your notebook\nimport pandas as pd\nsol = pd.read_csv('/kaggle/input/stanford-rna-3d-folding-2/validation_labels.csv')\nsub = pd.read_csv('/kaggle/working/submission.csv')\n</code></pre>\n<pre><code># Run the eval code on each target and get the score, if this is a notebook run using the public test set (whose size matches the solution file, `validation_labels.csv`]\nsol['target_id'] = sol['ID'].apply(lambda x: '_'.join(str(x).split('_')[:-1]))\nsub['target_id'] = sub['ID'].apply(lambda x: '_'.join(str(x).split('_')[:-1]))\n\nif len(sol)==len(sub): # This tests if we're looking at public val\n    results = []\n    for target_id, group_native in sol.groupby('target_id'):\n        group_predicted = sub[sub['target_id'] == target_id]\n        result = score(group_native,group_predicted,'ID')\n        print(target_id,result)\n        results.append( result )\n    print( 'Mean score:',  \n          float(sum(results) / len(results)) if len(results)&gt;0 else 0.0, \n          f'(n={len(results)})' )\n</code></pre>\n<ul>\n<li><strong>Step 3</strong> Inspect the outputs\nIn the example MMseqs2 notebook, the output on val is:</li>\n</ul>\n<pre><code>8ZNQ 0.03169\n9CFN 0.04012\n9E74 0.56897\n9E75 0.06648\n9E9Q 0.5597\n9EBP 0.04562\n9G4J 0.92386\n9G4P 0.04276\n9G4Q 0.04655\n9G4R 0.03152\n9HRO 0.02404\n9I9W 0.02199\n9IWF 0.03934\n9J09 0.051\n9JFO 0.05872\n9JFS 0.06211\n9JGM 0.0786\n9KGG 0.09895\n9LEC 0.85576\n9LEL 0.68292\n9LJN 0.78599\n9MME 0.12252\n9OBM 0.0334\n9OD4 0.02793\n9QZJ 0.02456\n9RVP 0.02432\n9WHV 0.04899\n9ZCC 0.09892\nMean score: 0.1963332142857143 (n=28)\n</code></pre>\n<p>Inspection of the output reveals several cases that could use some additional thought:</p>\n<ul>\n<li>For example, <code>9MME</code> is an octamer target where MMseqs2 actually does find a 3D template, but the code outputs only a single chain of the octamer instead of all 8.</li>\n<li><code>9J09</code> is a case where there actually is a template that should be identifiable based on the similarity of function of the target (a 'mini'-CRISPR/Cas13 gene editing complex; see the <code>description</code> field) and distantly related homologs (other Cas13 complexes and the natural ancestral complex, TnpB). </li>\n</ul>\n<p>For more discussion of these particular targets (and a few others in the validation set), check out the <a href=\"https://www.biorxiv.org/content/10.64898/2025.12.30.696949v1\" target=\"_blank\">preprint on Part 1 of this competition</a>.</p>",
      "rawMarkdown": "It's great to see so many entries & shared notebooks already! In the early weeks of this competition, we may share some tips learned from the prior competition, which you're very welcome to use or ignore as you see fit.\n\nHere's one tip, also mentioned in the [Getting started](https://www.kaggle.com/competitions/stanford-rna-3d-folding-2/discussion/666732) post:\n\nTo identify problem cases for your code, you may find it helpful to get scores back on the validation targets individually, and it is possible to do this within your notebooks using the competition evaluation metric. \n\nTo make this a little easier, we've updated the [MMseqs2 example notebook](https://www.kaggle.com/code/rhijudas/mmseqs2-3d-rna-template-identification-part-2/) to include example code to get these target-by-target validation scores. For ease of reference, the steps are also written out below.\n\nMore tips coming -- looking forward to more discussions and notebook sharing.\n\nGood luck!\nRhiju\n\nHere are the steps to get TM-scores on the validation set in your notebook:\n\n- **Step 1.**  Go to 'Add input...', and add the following two inputs to your notebook:\n\n```\nhttps://www.kaggle.com/code/rhijudas/tm-score-permutechains/data\nhttps://www.kaggle.com/datasets/metric/usalign\n```\n\n- **Step 2.**  Add the following three code blocks at the end of your notebook:\n```\n# install the metric and defined the score() function\n!ls /kaggle/usr/lib/\nimport runpy\nmodule_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py\")\nscore = module_globals['score']\n```\n\n```\n# Read in the validation solution and the output submission.csv from your notebook\nimport pandas as pd\nsol = pd.read_csv('/kaggle/input/stanford-rna-3d-folding-2/validation_labels.csv')\nsub = pd.read_csv('/kaggle/working/submission.csv')\n```\n\n```\n# Run the eval code on each target and get the score, if this is a notebook run using the public test set (whose size matches the solution file, `validation_labels.csv`]\nsol['target_id'] = sol['ID'].apply(lambda x: '_'.join(str(x).split('_')[:-1]))\nsub['target_id'] = sub['ID'].apply(lambda x: '_'.join(str(x).split('_')[:-1]))\n\nif len(sol)==len(sub): # This tests if we're looking at public val\n    results = []\n    for target_id, group_native in sol.groupby('target_id'):\n        group_predicted = sub[sub['target_id'] == target_id]\n        result = score(group_native,group_predicted,'ID')\n        print(target_id,result)\n        results.append( result )\n    print( 'Mean score:',  \n          float(sum(results) / len(results)) if len(results)>0 else 0.0, \n          f'(n={len(results)})' )\n```\n\n- **Step 3** Inspect the outputs\nIn the example MMseqs2 notebook, the output on val is:\n\n```\n8ZNQ 0.03169\n9CFN 0.04012\n9E74 0.56897\n9E75 0.06648\n9E9Q 0.5597\n9EBP 0.04562\n9G4J 0.92386\n9G4P 0.04276\n9G4Q 0.04655\n9G4R 0.03152\n9HRO 0.02404\n9I9W 0.02199\n9IWF 0.03934\n9J09 0.051\n9JFO 0.05872\n9JFS 0.06211\n9JGM 0.0786\n9KGG 0.09895\n9LEC 0.85576\n9LEL 0.68292\n9LJN 0.78599\n9MME 0.12252\n9OBM 0.0334\n9OD4 0.02793\n9QZJ 0.02456\n9RVP 0.02432\n9WHV 0.04899\n9ZCC 0.09892\nMean score: 0.1963332142857143 (n=28)\n```\n\nInspection of the output reveals several cases that could use some additional thought:\n- For example, `9MME` is an octamer target where MMseqs2 actually does find a 3D template, but the code outputs only a single chain of the octamer instead of all 8.\n- `9J09` is a case where there actually is a template that should be identifiable based on the similarity of function of the target (a 'mini'-CRISPR/Cas13 gene editing complex; see the `description` field) and distantly related homologs (other Cas13 complexes and the natural ancestral complex, TnpB). \n\nFor more discussion of these particular targets (and a few others in the validation set), check out the [preprint on Part 1 of this competition](https://www.biorxiv.org/content/10.64898/2025.12.30.696949v1).",
      "votes": 21
    },
    {
      "id": 3419882,
      "postDate": "2026-03-11T22:13:46.327Z",
      "content": "<p>Hi host, why is that the score you calculated in your example notebook above doesn't match the LB score that notebook received? I thought \"/kaggle/usr/lib/tm-score-permutechains/metric.py\" you used is the official metric? Thanks in advance!</p>",
      "rawMarkdown": "Hi host, why is that the score you calculated in your example notebook above doesn't match the LB score that notebook received? I thought \"/kaggle/usr/lib/tm-score-permutechains/metric.py\" you used is the official metric? Thanks in advance!",
      "replies": [
        {
          "id": 3419904,
          "postDate": "2026-03-12T01:16:36.823Z",
          "content": "<p>Yes this is one of the consuing things about Kaggle. </p>\n<p>When you run the notebook interactively, it runs on the publicly available <code>test_sequences.csv</code> and compares to the publicly available <code>validation_labels.csv</code>. </p>\n<p>But when you submit the notebook to the competition, it will run on a hidden rerun directory that has replaced <code>test_sequences.csv</code> with a different directory. And the <code>test_submission.csv</code> will be scored by the Kaggle server against the hidden solution for the leaderboard. It's not possible to recover the competition score of 0.177 without access to the hidden data.</p>",
          "rawMarkdown": "Yes this is one of the consuing things about Kaggle. \n\nWhen you run the notebook interactively, it runs on the publicly available `test_sequences.csv` and compares to the publicly available `validation_labels.csv`. \n\nBut when you submit the notebook to the competition, it will run on a hidden rerun directory that has replaced `test_sequences.csv` with a different directory. And the `test_submission.csv` will be scored by the Kaggle server against the hidden solution for the leaderboard. It's not possible to recover the competition score of 0.177 without access to the hidden data.",
          "votes": 1,
          "replies": [
            {
              "id": 3419918,
              "postDate": "2026-03-12T01:57:59.990Z",
              "content": "<p>Thanks! Now I understand! So the public LB is evaluated using a different set of test sequence (not test_sequence.csv)? I was assuming the test_sequence.csv is the public test, but it seems public LB uses a different set.</p>",
              "rawMarkdown": "Thanks! Now I understand! So the public LB is evaluated using a different set of test sequence (not test_sequence.csv)? I was assuming the test_sequence.csv is the public test, but it seems public LB uses a different set."
            },
            {
              "id": 3419919,
              "postDate": "2026-03-12T02:04:54.620Z",
              "content": "<p>Yes, you got it.</p>",
              "rawMarkdown": "Yes, you got it."
            }
          ]
        }
      ]
    },
    {
      "id": 3416581,
      "postDate": "2026-03-03T09:18:00.717Z",
      "content": "<p>Here you import from your local library: <code>module_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py\")</code> which does not work for other users, right?</p>\n<p>Also, I dont have any metric.py but only a notebook <code>tm-score-permutechains.ipynb</code></p>",
      "rawMarkdown": "Here you import from your local library: `module_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py\")` which does not work for other users, right?\n\nAlso, I dont have any metric.py but only a notebook `tm-score-permutechains.ipynb`",
      "replies": [
        {
          "id": 3419903,
          "postDate": "2026-03-12T01:14:08.497Z",
          "content": "<p>It should work -- if you import a 'metric' via Kaggle, Kaggle should install this <code>metric.py</code> into <code>/kaggle/usr/lib/</code>. </p>\n<p>Can  you do 'Add Input…' from the menu and   type in <code>https://www.kaggle.com/datasets/metric/usalign</code>? </p>\n<p>After that, try: <code>module_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py</code></p>\n<p>Let us know if that works for you. </p>",
          "rawMarkdown": "It should work -- if you import a 'metric' via Kaggle, Kaggle should install this `metric.py` into `/kaggle/usr/lib/`. \n\nCan  you do 'Add Input...' from the menu and   type in `https://www.kaggle.com/datasets/metric/usalign`? \n\nAfter that, try: `module_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py`\n\nLet us know if that works for you. ",
          "votes": 1,
          "replies": [
            {
              "id": 3420140,
              "postDate": "2026-03-12T14:38:25.310Z",
              "content": "<p>Hello Mr. Das,</p>\n<p>Yes it should work somewhere else.</p>\n<p>I have yet to try your approach, here is what I did:\nI downloaded and uploaded as a dataset of my own, then:</p>\n<pre><code>import sys\nsys.path.append(\"/kaggle/input/datasets/andreshzapke/metric-score\")\n\nfrom metric import score\n</code></pre>\n<p>Best regards</p>",
              "rawMarkdown": "Hello Mr. Das,\n\nYes it should work somewhere else.\n\nI have yet to try your approach, here is what I did:\nI downloaded and uploaded as a dataset of my own, then:\n```python\nimport sys\nsys.path.append(\"/kaggle/input/datasets/andreshzapke/metric-score\")\n\nfrom metric import score\n```\n\nBest regards"
            }
          ]
        }
      ]
    },
    {
      "id": 3407999,
      "postDate": "2026-02-19T15:39:03.950Z",
      "content": "<p>Right, as a beginner in ML, this is exactly what i am thinking!</p>\n<p>If we group these scores in categories, and find out that all \"Cas13 complexes\" perform badly, for example, then what would be the solution, apart from postprocessing? Increase their loss weight or add more Cas13 samples?</p>\n<p>Is anyone fine-tuning Protenix at all or is this very difficult to do?</p>",
      "rawMarkdown": "Right, as a beginner in ML, this is exactly what i am thinking!\n\nIf we group these scores in categories, and find out that all \"Cas13 complexes\" perform badly, for example, then what would be the solution, apart from postprocessing? Increase their loss weight or add more Cas13 samples?\n\nIs anyone fine-tuning Protenix at all or is this very difficult to do?\n\n"
    },
    {
      "id": 3397593,
      "postDate": "2026-01-27T16:25:46.440Z",
      "content": "<p>Could you please pin this topic?</p>",
      "rawMarkdown": "Could you please pin this topic?"
    },
    {
      "id": 3389418,
      "postDate": "2026-01-11T05:53:12.623Z",
      "content": "<p>Thanks - little tricky to get it to work on local machine but you gave me enough clues :)</p>",
      "rawMarkdown": "Thanks - little tricky to get it to work on local machine but you gave me enough clues :)"
    }
  ],
  "comments": [
    {
      "id": 3419882,
      "author_name": "hongan",
      "author_url": "",
      "post_date": "2026-03-11T22:13:46.327000",
      "content": "<p>Hi host, why is that the score you calculated in your example notebook above doesn't match the LB score that notebook received? I thought \"/kaggle/usr/lib/tm-score-permutechains/metric.py\" you used is the official metric? Thanks in advance!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3419904,
          "author_name": "Rhiju Das",
          "author_url": "",
          "post_date": "2026-03-12T01:16:36.823000",
          "content": "<p>Yes this is one of the consuing things about Kaggle. </p>\n<p>When you run the notebook interactively, it runs on the publicly available <code>test_sequences.csv</code> and compares to the publicly available <code>validation_labels.csv</code>. </p>\n<p>But when you submit the notebook to the competition, it will run on a hidden rerun directory that has replaced <code>test_sequences.csv</code> with a different directory. And the <code>test_submission.csv</code> will be scored by the Kaggle server against the hidden solution for the leaderboard. It's not possible to recover the competition score of 0.177 without access to the hidden data.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 3419918,
              "author_name": "hongan",
              "author_url": "",
              "post_date": "2026-03-12T01:57:59.990000",
              "content": "<p>Thanks! Now I understand! So the public LB is evaluated using a different set of test sequence (not test_sequence.csv)? I was assuming the test_sequence.csv is the public test, but it seems public LB uses a different set.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3419919,
              "author_name": "Rhiju Das",
              "author_url": "",
              "post_date": "2026-03-12T02:04:54.620000",
              "content": "<p>Yes, you got it.</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3416581,
      "author_name": "Andres H. Zapke",
      "author_url": "",
      "post_date": "2026-03-03T09:18:00.717000",
      "content": "<p>Here you import from your local library: <code>module_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py\")</code> which does not work for other users, right?</p>\n<p>Also, I dont have any metric.py but only a notebook <code>tm-score-permutechains.ipynb</code></p>",
      "votes": 0,
      "replies": [
        {
          "id": 3419903,
          "author_name": "Rhiju Das",
          "author_url": "",
          "post_date": "2026-03-12T01:14:08.497000",
          "content": "<p>It should work -- if you import a 'metric' via Kaggle, Kaggle should install this <code>metric.py</code> into <code>/kaggle/usr/lib/</code>. </p>\n<p>Can  you do 'Add Input…' from the menu and   type in <code>https://www.kaggle.com/datasets/metric/usalign</code>? </p>\n<p>After that, try: <code>module_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py</code></p>\n<p>Let us know if that works for you. </p>",
          "votes": 1,
          "replies": [
            {
              "id": 3420140,
              "author_name": "Andres H. Zapke",
              "author_url": "",
              "post_date": "2026-03-12T14:38:25.310000",
              "content": "<p>Hello Mr. Das,</p>\n<p>Yes it should work somewhere else.</p>\n<p>I have yet to try your approach, here is what I did:\nI downloaded and uploaded as a dataset of my own, then:</p>\n<pre><code>import sys\nsys.path.append(\"/kaggle/input/datasets/andreshzapke/metric-score\")\n\nfrom metric import score\n</code></pre>\n<p>Best regards</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3407999,
      "author_name": "Andres H. Zapke",
      "author_url": "",
      "post_date": "2026-02-19T15:39:03.950000",
      "content": "<p>Right, as a beginner in ML, this is exactly what i am thinking!</p>\n<p>If we group these scores in categories, and find out that all \"Cas13 complexes\" perform badly, for example, then what would be the solution, apart from postprocessing? Increase their loss weight or add more Cas13 samples?</p>\n<p>Is anyone fine-tuning Protenix at all or is this very difficult to do?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3397593,
      "author_name": "Bridgeport",
      "author_url": "",
      "post_date": "2026-01-27T16:25:46.440000",
      "content": "<p>Could you please pin this topic?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3389418,
      "author_name": "PC Jimmmy",
      "author_url": "",
      "post_date": "2026-01-11T05:53:12.623000",
      "content": "<p>Thanks - little tricky to get it to work on local machine but you gave me enough clues :)</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3389401": "It's great to see so many entries & shared notebooks already! In the early weeks of this competition, we may share some tips learned from the prior competition, which you're very welcome to use or ignore as you see fit.\n\nHere's one tip, also mentioned in the [Getting started](https://www.kaggle.com/competitions/stanford-rna-3d-folding-2/discussion/666732) post:\n\nTo identify problem cases for your code, you may find it helpful to get scores back on the validation targets individually, and it is possible to do this within your notebooks using the competition evaluation metric. \n\nTo make this a little easier, we've updated the [MMseqs2 example notebook](https://www.kaggle.com/code/rhijudas/mmseqs2-3d-rna-template-identification-part-2/) to include example code to get these target-by-target validation scores. For ease of reference, the steps are also written out below.\n\nMore tips coming -- looking forward to more discussions and notebook sharing.\n\nGood luck!\nRhiju\n\nHere are the steps to get TM-scores on the validation set in your notebook:\n\n- **Step 1.**  Go to 'Add input...', and add the following two inputs to your notebook:\n\n```\nhttps://www.kaggle.com/code/rhijudas/tm-score-permutechains/data\nhttps://www.kaggle.com/datasets/metric/usalign\n```\n\n- **Step 2.**  Add the following three code blocks at the end of your notebook:\n```\n# install the metric and defined the score() function\n!ls /kaggle/usr/lib/\nimport runpy\nmodule_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py\")\nscore = module_globals['score']\n```\n\n```\n# Read in the validation solution and the output submission.csv from your notebook\nimport pandas as pd\nsol = pd.read_csv('/kaggle/input/stanford-rna-3d-folding-2/validation_labels.csv')\nsub = pd.read_csv('/kaggle/working/submission.csv')\n```\n\n```\n# Run the eval code on each target and get the score, if this is a notebook run using the public test set (whose size matches the solution file, `validation_labels.csv`]\nsol['target_id'] = sol['ID'].apply(lambda x: '_'.join(str(x).split('_')[:-1]))\nsub['target_id'] = sub['ID'].apply(lambda x: '_'.join(str(x).split('_')[:-1]))\n\nif len(sol)==len(sub): # This tests if we're looking at public val\n    results = []\n    for target_id, group_native in sol.groupby('target_id'):\n        group_predicted = sub[sub['target_id'] == target_id]\n        result = score(group_native,group_predicted,'ID')\n        print(target_id,result)\n        results.append( result )\n    print( 'Mean score:',  \n          float(sum(results) / len(results)) if len(results)>0 else 0.0, \n          f'(n={len(results)})' )\n```\n\n- **Step 3** Inspect the outputs\nIn the example MMseqs2 notebook, the output on val is:\n\n```\n8ZNQ 0.03169\n9CFN 0.04012\n9E74 0.56897\n9E75 0.06648\n9E9Q 0.5597\n9EBP 0.04562\n9G4J 0.92386\n9G4P 0.04276\n9G4Q 0.04655\n9G4R 0.03152\n9HRO 0.02404\n9I9W 0.02199\n9IWF 0.03934\n9J09 0.051\n9JFO 0.05872\n9JFS 0.06211\n9JGM 0.0786\n9KGG 0.09895\n9LEC 0.85576\n9LEL 0.68292\n9LJN 0.78599\n9MME 0.12252\n9OBM 0.0334\n9OD4 0.02793\n9QZJ 0.02456\n9RVP 0.02432\n9WHV 0.04899\n9ZCC 0.09892\nMean score: 0.1963332142857143 (n=28)\n```\n\nInspection of the output reveals several cases that could use some additional thought:\n- For example, `9MME` is an octamer target where MMseqs2 actually does find a 3D template, but the code outputs only a single chain of the octamer instead of all 8.\n- `9J09` is a case where there actually is a template that should be identifiable based on the similarity of function of the target (a 'mini'-CRISPR/Cas13 gene editing complex; see the `description` field) and distantly related homologs (other Cas13 complexes and the natural ancestral complex, TnpB). \n\nFor more discussion of these particular targets (and a few others in the validation set), check out the [preprint on Part 1 of this competition](https://www.biorxiv.org/content/10.64898/2025.12.30.696949v1).",
    "3419882": "Hi host, why is that the score you calculated in your example notebook above doesn't match the LB score that notebook received? I thought \"/kaggle/usr/lib/tm-score-permutechains/metric.py\" you used is the official metric? Thanks in advance!",
    "3416581": "Here you import from your local library: `module_globals = runpy.run_path(\"/kaggle/usr/lib/tm-score-permutechains/metric.py\")` which does not work for other users, right?\n\nAlso, I dont have any metric.py but only a notebook `tm-score-permutechains.ipynb`",
    "3407999": "Right, as a beginner in ML, this is exactly what i am thinking!\n\nIf we group these scores in categories, and find out that all \"Cas13 complexes\" perform badly, for example, then what would be the solution, apart from postprocessing? Increase their loss weight or add more Cas13 samples?\n\nIs anyone fine-tuning Protenix at all or is this very difficult to do?\n\n",
    "3397593": "Could you please pin this topic?",
    "3389418": "Thanks - little tricky to get it to work on local machine but you gave me enough clues :)"
  }
}