{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# How to add dependencies as a dataset to an offline competition?\n\nYou're probably familiar with this. You're entering a competition where notebooks can only run offline, and you're wondering how to get your dependencies into your notebook.\n\nCertainly, there are several possible solutions for this. Below I show mine, with which I am quite satisfied.","metadata":{}},{"cell_type":"markdown","source":"## Download Dependencies\n\nI will show this below with the fabulous `timm` [(PyTorch Image Models)](https://rwightman.github.io/pytorch-image-models/) library. In the same way, other or further dependencies could also be defined.\n\nThe very first thing I do is create a temporary directory where all the dependencies will be downloaded.","metadata":{}},{"cell_type":"code","source":"TMP_DIR = \"/kaggle/temp/wheels\"\n\n!mkdir -p {TMP_DIR}","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:14:14.132678Z","iopub.execute_input":"2022-12-23T10:14:14.133098Z","iopub.status.idle":"2022-12-23T10:14:14.418258Z","shell.execute_reply.started":"2022-12-23T10:14:14.133019Z","shell.execute_reply":"2022-12-23T10:14:14.41728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Maybe you have already created the directory. Therefore, you should check if the directory already exists and is possibly not empty.","metadata":{}},{"cell_type":"code","source":"!ls -lrth {TMP_DIR}","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:14:14.419723Z","iopub.execute_input":"2022-12-23T10:14:14.420806Z","iopub.status.idle":"2022-12-23T10:14:14.688895Z","shell.execute_reply.started":"2022-12-23T10:14:14.420775Z","shell.execute_reply":"2022-12-23T10:14:14.688037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If this is the case, choose another folder, move the content or delete it if you don't need it any more.","metadata":{}},{"cell_type":"code","source":"!rm -rf {TMP_DIR}/*\n!ls -lrth {TMP_DIR}","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:14:14.690291Z","iopub.execute_input":"2022-12-23T10:14:14.691104Z","iopub.status.idle":"2022-12-23T10:14:15.221113Z","shell.execute_reply.started":"2022-12-23T10:14:14.691071Z","shell.execute_reply":"2022-12-23T10:14:15.219688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After that we are ready to download all the required dependencies.\n\nIt is important that you do this here in the Kaggle Notebook, because it may be that other variants of the packages are loaded if you do it locally on your computer and you use a different hardware architecture there, for example.","metadata":{}},{"cell_type":"code","source":"!pip wheel timm \\\n    --find-links \"https://download.pytorch.org/whl/torch_stable.html\" \\\n    --wheel-dir={TMP_DIR} \\\n    --no-cache-dir   ","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:14:15.224906Z","iopub.execute_input":"2022-12-23T10:14:15.225475Z","iopub.status.idle":"2022-12-23T10:15:23.930003Z","shell.execute_reply.started":"2022-12-23T10:14:15.225444Z","shell.execute_reply":"2022-12-23T10:15:23.928572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Do you want to see which packages have been downloaded?","metadata":{}},{"cell_type":"code","source":"!ls -lrth {TMP_DIR}","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:15:23.931558Z","iopub.execute_input":"2022-12-23T10:15:23.931924Z","iopub.status.idle":"2022-12-23T10:15:24.198571Z","shell.execute_reply.started":"2022-12-23T10:15:23.931889Z","shell.execute_reply":"2022-12-23T10:15:24.197303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Zip dependencies into an archive\n\nNow we want to put the downloaded packages into an archive. There are several ways to do this. The easiest one might be to use a zip archive.","metadata":{}},{"cell_type":"code","source":"!pushd {TMP_DIR} && zip \"/kaggle/working/timm-0.6.12-py3-none-any.zip\" * && popd","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:15:24.200253Z","iopub.execute_input":"2022-12-23T10:15:24.200652Z","iopub.status.idle":"2022-12-23T10:16:21.799084Z","shell.execute_reply.started":"2022-12-23T10:15:24.200615Z","shell.execute_reply":"2022-12-23T10:16:21.797705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls -lrth","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:16:21.800815Z","iopub.execute_input":"2022-12-23T10:16:21.801224Z","iopub.status.idle":"2022-12-23T10:16:22.069258Z","shell.execute_reply.started":"2022-12-23T10:16:21.801187Z","shell.execute_reply":"2022-12-23T10:16:22.068283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Add archive as a dataset\n\nNow the second last step. Add the zip archive as a record to your notebook. For this it is necessary that you save the notebook once \"Save & Run All (Commit)\". \n\nAfter that you can click on \"Add Data\" in your notebook to which you want to add the zip archive and search for your saved notebook and add it as a data set. That's it.\n\nIf you want to repeat the process, it is best to delete the downloaded dependencies. ","metadata":{}},{"cell_type":"code","source":"!rm -rf /kaggle/temp/wheels/*","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:16:22.071353Z","iopub.execute_input":"2022-12-23T10:16:22.07248Z","iopub.status.idle":"2022-12-23T10:16:22.490008Z","shell.execute_reply.started":"2022-12-23T10:16:22.072441Z","shell.execute_reply":"2022-12-23T10:16:22.489119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Installing the packages from the archive\n\nIf you want to install the packages, unpack the archive first. If you want to do this in your notebook and use the archive from the dataset, change `timm-0.6.12-py3-none-any.zip` to `\"/kaggle/input/<dataset_name>/timm-0.6.12-py3-none-any.zip\"`.","metadata":{}},{"cell_type":"code","source":"!unzip -q \"timm-0.6.12-py3-none-any.zip\" -d \"timm-0.6.12\"\n!ls -lrth \"timm-0.6.12\"","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:16:22.491336Z","iopub.execute_input":"2022-12-23T10:16:22.491618Z","iopub.status.idle":"2022-12-23T10:16:33.389101Z","shell.execute_reply.started":"2022-12-23T10:16:22.491592Z","shell.execute_reply":"2022-12-23T10:16:33.388062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Once the packages are unpacked, you can install them with `pip`. If you want to make sure that the subdependencies also use the versions from your archive, just add the flag `--force-reinstall`.","metadata":{}},{"cell_type":"code","source":"!pip install timm --find-links \"timm-0.6.12\" --no-index","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:16:33.393299Z","iopub.execute_input":"2022-12-23T10:16:33.393605Z","iopub.status.idle":"2022-12-23T10:16:43.52697Z","shell.execute_reply.started":"2022-12-23T10:16:33.393579Z","shell.execute_reply":"2022-12-23T10:16:43.525407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Optional: List files in archive without decompressing","metadata":{}},{"cell_type":"code","source":"!zipinfo timm-0.6.12-py3-none-any.zip","metadata":{"execution":{"iopub.status.busy":"2022-12-23T10:16:43.52865Z","iopub.execute_input":"2022-12-23T10:16:43.529085Z","iopub.status.idle":"2022-12-23T10:16:43.798823Z","shell.execute_reply.started":"2022-12-23T10:16:43.529049Z","shell.execute_reply":"2022-12-23T10:16:43.797834Z"},"trusted":true},"execution_count":null,"outputs":[]}]}