{
  "id": 381508,
  "title": "Submission throwing an exception, related to pydicom and the pylibjpeg-libjpeg plugin",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/381508",
  "author_name": "John Detloff",
  "post_date": "2023-01-27T03:06:25.716000",
  "votes": 0,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Apologies if this is a basic question, I'm new to Kaggle and to notebooks</p>\n<p>I'm trying to submit my notebook, however it's throwing an exception with each submission. I believe I know what the exception is - in the first code cell of my notebook I use pip to install pydicom, pylibjpeg, and pylibjpeg-libjpeg, which are used for reading in the dicom images in the test set.</p>\n<p>However, I've found that immediately after installation, pylibjpeg isn't able to access to libjpeg plugin. It's only after the kernel is restart that it can find it. So, in order for my notebook to complete, I need to run that installation cell, restart the kernel, and then run the entire notebook.</p>\n<p>This approach works when I'm running the notebook manually and can manually restart the kernel, but when I submit I get an exception, which I assume is because the kernel isn't being restarted between installing these wheels and running the rest of my notebook. The exception isn't thrown when I don't call into pydicom, so I'm fairly confident that is the root of my issue.</p>\n<p>Is this an issue other's have run into with these libraries? How are others getting around this?</p>\n<p>Thank you for any info!</p>",
  "messages": [
    {
      "id": 2117139,
      "postDate": "2023-01-27T03:56:13.020Z",
      "content": "<p>This competition doesn't allow internet access, so installing modules in the code with <code>!pip install sfsdf</code> won't work at submission. You should download the <code>*.whl</code> files and upload them as a dataset in order to install them without internet access.</p>\n<p>I am using TensorFlowIO, which doesn't require internet or module installation. However, it is quite slow. It takes around one to two seconds to read each image.</p>",
      "rawMarkdown": "This competition doesn't allow internet access, so installing modules in the code with `!pip install sfsdf` won't work at submission. You should download the `*.whl` files and upload them as a dataset in order to install them without internet access.\n\nI am using TensorFlowIO, which doesn't require internet or module installation. However, it is quite slow. It takes around one to two seconds to read each image.",
      "replies": [
        {
          "id": 2117162,
          "postDate": "2023-01-27T04:31:11.340Z",
          "content": "<p>Oh yes, I should have mentioned, I already downloaded the whl's and uploaded them to the notebook, and install them like:</p>\n<p>!pip install wheel_name --no-index --find-links=file:///location-of-wheel</p>\n<p>So the wheel is present, and it does install, it's just that after installation it seems a kernel restart is required in order for pylibjpeg to have access to its plugins</p>",
          "rawMarkdown": "Oh yes, I should have mentioned, I already downloaded the whl's and uploaded them to the notebook, and install them like:\n\n!pip install wheel_name --no-index --find-links=file:///location-of-wheel\n\nSo the wheel is present, and it does install, it's just that after installation it seems a kernel restart is required in order for pylibjpeg to have access to its plugins",
          "replies": [
            {
              "id": 2117351,
              "postDate": "2023-01-27T07:59:45.220Z",
              "content": "<p>I think you should refer to this kernel.<br>\n<a href=\"https://www.kaggle.com/code/yiheng/monai-baseline-inference/notebook\" target=\"_blank\">https://www.kaggle.com/code/yiheng/monai-baseline-inference/notebook</a></p>\n<p>1.Add the following to your dataset</p>\n<pre><code>https://www.kaggle.com/code/vslaykovsky/rsna-2022-whl\n</code></pre>\n<p>2.Run the following command in the cell.</p>\n<pre><code>!pip install /kaggle/input/rsna-2022-whl/{pydicom-2.3.0-py3-none-any.whl,pylibjpeg-1.4.0-py3-none-any.whl,python_gdcm-3.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl}\n</code></pre>",
              "rawMarkdown": "I think you should refer to this kernel.\nhttps://www.kaggle.com/code/yiheng/monai-baseline-inference/notebook\n\n1.Add the following to your dataset\n```\nhttps://www.kaggle.com/code/vslaykovsky/rsna-2022-whl\n```\n2.Run the following command in the cell.\n```\n!pip install /kaggle/input/rsna-2022-whl/{pydicom-2.3.0-py3-none-any.whl,pylibjpeg-1.4.0-py3-none-any.whl,python_gdcm-3.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl}\n```",
              "votes": 1
            },
            {
              "id": 2118352,
              "postDate": "2023-01-28T00:39:05.960Z",
              "content": "<p>Interesting - so this is more or less what I'm currently doing, which is to upload as a dataset the whl's I'm using and then install them from the input directory. It looks like this notebook uses a different set of libraries to accomplish loading the images into a pixel array, which presumably don't require the kernel restart, which is super useful to know about thanks!</p>\n<p>With that said I'd still like to understand at a high level how I would accomplish this with the original set of libraries I used. Is it the case that some python packages require a shell restart and those packages simply cannot be used within Kaggle submissions? Or (more likely) is there a way to restart the kernel within a submission, or work around the need to do that, such that these libraries can be used?</p>",
              "rawMarkdown": "Interesting - so this is more or less what I'm currently doing, which is to upload as a dataset the whl's I'm using and then install them from the input directory. It looks like this notebook uses a different set of libraries to accomplish loading the images into a pixel array, which presumably don't require the kernel restart, which is super useful to know about thanks!\n\nWith that said I'd still like to understand at a high level how I would accomplish this with the original set of libraries I used. Is it the case that some python packages require a shell restart and those packages simply cannot be used within Kaggle submissions? Or (more likely) is there a way to restart the kernel within a submission, or work around the need to do that, such that these libraries can be used?"
            }
          ]
        }
      ]
    },
    {
      "id": 2117123,
      "postDate": "2023-01-27T03:06:25.717Z",
      "content": "<p>Apologies if this is a basic question, I'm new to Kaggle and to notebooks</p>\n<p>I'm trying to submit my notebook, however it's throwing an exception with each submission. I believe I know what the exception is - in the first code cell of my notebook I use pip to install pydicom, pylibjpeg, and pylibjpeg-libjpeg, which are used for reading in the dicom images in the test set.</p>\n<p>However, I've found that immediately after installation, pylibjpeg isn't able to access to libjpeg plugin. It's only after the kernel is restart that it can find it. So, in order for my notebook to complete, I need to run that installation cell, restart the kernel, and then run the entire notebook.</p>\n<p>This approach works when I'm running the notebook manually and can manually restart the kernel, but when I submit I get an exception, which I assume is because the kernel isn't being restarted between installing these wheels and running the rest of my notebook. The exception isn't thrown when I don't call into pydicom, so I'm fairly confident that is the root of my issue.</p>\n<p>Is this an issue other's have run into with these libraries? How are others getting around this?</p>\n<p>Thank you for any info!</p>",
      "rawMarkdown": "Apologies if this is a basic question, I'm new to Kaggle and to notebooks\n\nI'm trying to submit my notebook, however it's throwing an exception with each submission. I believe I know what the exception is - in the first code cell of my notebook I use pip to install pydicom, pylibjpeg, and pylibjpeg-libjpeg, which are used for reading in the dicom images in the test set.\n\nHowever, I've found that immediately after installation, pylibjpeg isn't able to access to libjpeg plugin. It's only after the kernel is restart that it can find it. So, in order for my notebook to complete, I need to run that installation cell, restart the kernel, and then run the entire notebook.\n\nThis approach works when I'm running the notebook manually and can manually restart the kernel, but when I submit I get an exception, which I assume is because the kernel isn't being restarted between installing these wheels and running the rest of my notebook. The exception isn't thrown when I don't call into pydicom, so I'm fairly confident that is the root of my issue.\n\nIs this an issue other's have run into with these libraries? How are others getting around this?\n\nThank you for any info!"
    }
  ],
  "comments": [
    {
      "id": 2117139,
      "author_name": "Alfredo Maussa",
      "author_url": "",
      "post_date": "2023-01-27T03:56:13.020000",
      "content": "<p>This competition doesn't allow internet access, so installing modules in the code with <code>!pip install sfsdf</code> won't work at submission. You should download the <code>*.whl</code> files and upload them as a dataset in order to install them without internet access.</p>\n<p>I am using TensorFlowIO, which doesn't require internet or module installation. However, it is quite slow. It takes around one to two seconds to read each image.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2117162,
          "author_name": "John Detloff",
          "author_url": "",
          "post_date": "2023-01-27T04:31:11.340000",
          "content": "<p>Oh yes, I should have mentioned, I already downloaded the whl's and uploaded them to the notebook, and install them like:</p>\n<p>!pip install wheel_name --no-index --find-links=file:///location-of-wheel</p>\n<p>So the wheel is present, and it does install, it's just that after installation it seems a kernel restart is required in order for pylibjpeg to have access to its plugins</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2117351,
              "author_name": "yukiZ",
              "author_url": "",
              "post_date": "2023-01-27T07:59:45.220000",
              "content": "<p>I think you should refer to this kernel.<br>\n<a href=\"https://www.kaggle.com/code/yiheng/monai-baseline-inference/notebook\" target=\"_blank\">https://www.kaggle.com/code/yiheng/monai-baseline-inference/notebook</a></p>\n<p>1.Add the following to your dataset</p>\n<pre><code>https://www.kaggle.com/code/vslaykovsky/rsna-2022-whl\n</code></pre>\n<p>2.Run the following command in the cell.</p>\n<pre><code>!pip install /kaggle/input/rsna-2022-whl/{pydicom-2.3.0-py3-none-any.whl,pylibjpeg-1.4.0-py3-none-any.whl,python_gdcm-3.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl}\n</code></pre>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2118352,
              "author_name": "John Detloff",
              "author_url": "",
              "post_date": "2023-01-28T00:39:05.960000",
              "content": "<p>Interesting - so this is more or less what I'm currently doing, which is to upload as a dataset the whl's I'm using and then install them from the input directory. It looks like this notebook uses a different set of libraries to accomplish loading the images into a pixel array, which presumably don't require the kernel restart, which is super useful to know about thanks!</p>\n<p>With that said I'd still like to understand at a high level how I would accomplish this with the original set of libraries I used. Is it the case that some python packages require a shell restart and those packages simply cannot be used within Kaggle submissions? Or (more likely) is there a way to restart the kernel within a submission, or work around the need to do that, such that these libraries can be used?</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2117139": "This competition doesn't allow internet access, so installing modules in the code with `!pip install sfsdf` won't work at submission. You should download the `*.whl` files and upload them as a dataset in order to install them without internet access.\n\nI am using TensorFlowIO, which doesn't require internet or module installation. However, it is quite slow. It takes around one to two seconds to read each image.",
    "2117123": "Apologies if this is a basic question, I'm new to Kaggle and to notebooks\n\nI'm trying to submit my notebook, however it's throwing an exception with each submission. I believe I know what the exception is - in the first code cell of my notebook I use pip to install pydicom, pylibjpeg, and pylibjpeg-libjpeg, which are used for reading in the dicom images in the test set.\n\nHowever, I've found that immediately after installation, pylibjpeg isn't able to access to libjpeg plugin. It's only after the kernel is restart that it can find it. So, in order for my notebook to complete, I need to run that installation cell, restart the kernel, and then run the entire notebook.\n\nThis approach works when I'm running the notebook manually and can manually restart the kernel, but when I submit I get an exception, which I assume is because the kernel isn't being restarted between installing these wheels and running the rest of my notebook. The exception isn't thrown when I don't call into pydicom, so I'm fairly confident that is the root of my issue.\n\nIs this an issue other's have run into with these libraries? How are others getting around this?\n\nThank you for any info!"
  }
}