{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Deep Learning basics: 5 popular PyTorch methods \n\nHi!\n\nWorking with neural nets, have you ever wondered which function to use between `torch.view` or `reshape`? and what is the difference with `permute`?\n\nIn this short guide on PyTorch, I decided to explore 5 main methods that are a must in most of Deep Learning models nowadays: \n\n- `torch.view()`\n- `torch.reshape()`\n- `torch.permute()`\n- `torch.flatten()`\n- `torch.cat()`\n\nParticularly, I chose them not only because they are famous, but also because some of them can generate some confusion as they behave quite similarly.","metadata":{}},{"cell_type":"code","source":"import torch","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-28T15:34:28.724418Z","iopub.execute_input":"2022-12-28T15:34:28.724831Z","iopub.status.idle":"2022-12-28T15:34:28.732067Z","shell.execute_reply.started":"2022-12-28T15:34:28.724799Z","shell.execute_reply":"2022-12-28T15:34:28.730317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## View\n\n[torch.view()](https://pytorch.org/docs/stable/generated/torch.Tensor.view.html) returns a new tensor with the same data as the original tensor but of a different shape.","metadata":{}},{"cell_type":"code","source":"tensor1 = torch.tensor([[1, 2], [3, 4.]])\ntensor1","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.734266Z","iopub.execute_input":"2022-12-28T15:34:28.734805Z","iopub.status.idle":"2022-12-28T15:34:28.750237Z","shell.execute_reply.started":"2022-12-28T15:34:28.734757Z","shell.execute_reply":"2022-12-28T15:34:28.74861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Imagine we want to reshape our tensor to a (1, 4) shape","metadata":{}},{"cell_type":"code","source":"tensor_view = tensor1.view(1, 4)\ntensor_view","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.752505Z","iopub.execute_input":"2022-12-28T15:34:28.753944Z","iopub.status.idle":"2022-12-28T15:34:28.764877Z","shell.execute_reply.started":"2022-12-28T15:34:28.753881Z","shell.execute_reply":"2022-12-28T15:34:28.763488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Instead of specifying all the output shapes, we can pass `-1` to view and the size is inferred from other dimensions automatically","metadata":{}},{"cell_type":"code","source":"tensor_view_2 = tensor1.view(1, -1)\ntensor_view_2","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.767136Z","iopub.execute_input":"2022-12-28T15:34:28.767771Z","iopub.status.idle":"2022-12-28T15:34:28.777885Z","shell.execute_reply.started":"2022-12-28T15:34:28.767733Z","shell.execute_reply":"2022-12-28T15:34:28.776612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Beware when assigning a value to a `torch.view` variable: also the original tensow will be modified!","metadata":{}},{"cell_type":"code","source":"a = torch.tensor([1, 2, 3, 4])\nb = a.view(2, -1)\n\nprint(a)\nprint(b)\n\nb[0][0] = 22 #Assign 22 in b leads to a change in tensor a\n\nprint(a)\nprint(b)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.780371Z","iopub.execute_input":"2022-12-28T15:34:28.781161Z","iopub.status.idle":"2022-12-28T15:34:28.792137Z","shell.execute_reply.started":"2022-12-28T15:34:28.781076Z","shell.execute_reply":"2022-12-28T15:34:28.7908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`torch.view()` will fail if the new dimensions are not valid given the original ones.\nFor example calling view with (1, 5) would fail as the input size is 4","metadata":{}},{"cell_type":"code","source":"try:\n    tensor1.view(1, 5)\nexcept RuntimeError as e:\n    print(e)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.793914Z","iopub.execute_input":"2022-12-28T15:34:28.795006Z","iopub.status.idle":"2022-12-28T15:34:28.805095Z","shell.execute_reply.started":"2022-12-28T15:34:28.794959Z","shell.execute_reply":"2022-12-28T15:34:28.804004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Same with view (3, -1), since there's no integer N such as N * 3 = 4","metadata":{}},{"cell_type":"code","source":"try:\n    tensor1.view(3, -1)\nexcept RuntimeError as e:\n    print(e)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.806496Z","iopub.execute_input":"2022-12-28T15:34:28.80774Z","iopub.status.idle":"2022-12-28T15:34:28.819063Z","shell.execute_reply.started":"2022-12-28T15:34:28.807662Z","shell.execute_reply":"2022-12-28T15:34:28.817921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Torch would raise an error also if two `-1` parameters are present, since the inferring method cannot figure out all the shapes you have in mind!","metadata":{}},{"cell_type":"code","source":"try:\n    tensor1.view(1, -1, -1)\nexcept RuntimeError as e:\n    print(e)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.820468Z","iopub.execute_input":"2022-12-28T15:34:28.822007Z","iopub.status.idle":"2022-12-28T15:34:28.833492Z","shell.execute_reply.started":"2022-12-28T15:34:28.821948Z","shell.execute_reply":"2022-12-28T15:34:28.831743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`torch.view` comes in very handy in Deep Learning since reshaping tensors is a common procedures while handling data.","metadata":{}},{"cell_type":"markdown","source":"## Reshape\n\n[torch.reshape()](https://pytorch.org/docs/stable/generated/torch.reshape.html) works in a similar way as `torch.view()`: it returns a new tensor with the same data as the original tensor but of a different shape.","metadata":{}},{"cell_type":"code","source":"tensor2 = torch.tensor([[1, 2], [3, 4.]])\ntensor2","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.835235Z","iopub.execute_input":"2022-12-28T15:34:28.836233Z","iopub.status.idle":"2022-12-28T15:34:28.850879Z","shell.execute_reply.started":"2022-12-28T15:34:28.836171Z","shell.execute_reply":"2022-12-28T15:34:28.849224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tensor_reshaped = tensor2.reshape(1, 4)\ntensor_reshaped","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.852712Z","iopub.execute_input":"2022-12-28T15:34:28.853101Z","iopub.status.idle":"2022-12-28T15:34:28.867562Z","shell.execute_reply.started":"2022-12-28T15:34:28.85306Z","shell.execute_reply":"2022-12-28T15:34:28.865769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As for `torch.view()`, a single dimension may be `-1` letting torch inferring the remaining dimensions and the number of elements in it.","metadata":{}},{"cell_type":"code","source":"tensor_reshaped_2 = tensor2.reshape(1, -1)\ntensor_reshaped_2","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.874052Z","iopub.execute_input":"2022-12-28T15:34:28.874498Z","iopub.status.idle":"2022-12-28T15:34:28.8834Z","shell.execute_reply.started":"2022-12-28T15:34:28.874464Z","shell.execute_reply":"2022-12-28T15:34:28.882021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And again, be careful with the dimensions you specify: they should be valid given the orginal input","metadata":{}},{"cell_type":"code","source":"try:\n    tensor2.reshape(1, 5)\nexcept RuntimeError as e:\n    print(e)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.885307Z","iopub.execute_input":"2022-12-28T15:34:28.886226Z","iopub.status.idle":"2022-12-28T15:34:28.89547Z","shell.execute_reply.started":"2022-12-28T15:34:28.886145Z","shell.execute_reply":"2022-12-28T15:34:28.894257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So one question would be: which one should I use? `view()` or `reshape()`?\n\nAs we saw, `torch.view` merely creates a view of the original tensor; the new tensor will always share its data with the original tensor.","metadata":{}},{"cell_type":"code","source":"a = torch.tensor([1, 2, 3, 4])\nb = a.view(2, -1)\n\nprint(a)\nprint(b)\n\nb[0][0] = 22 #Assign 22 in b leads to a change in tensor a\nprint()\n\nprint(a)\nprint(b)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.896894Z","iopub.execute_input":"2022-12-28T15:34:28.897259Z","iopub.status.idle":"2022-12-28T15:34:28.912438Z","shell.execute_reply.started":"2022-12-28T15:34:28.897228Z","shell.execute_reply":"2022-12-28T15:34:28.910799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Another subtle aspect of `view()` is that it works only with contiguous tensor.\nFor instance:","metadata":{}},{"cell_type":"code","source":"a = torch.tensor([[1, 2], [3, 4]])\nprint(a)\n\n# Let's transpose it\nb = a.t()\nprint(b)\nprint()\n\nprint(\"ERROR WILL BE RAISED HERE:\")\ntry:\n    b.view(1, -1)\nexcept RuntimeError as e:\n    print(e)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.91402Z","iopub.execute_input":"2022-12-28T15:34:28.914586Z","iopub.status.idle":"2022-12-28T15:34:28.941006Z","shell.execute_reply.started":"2022-12-28T15:34:28.914539Z","shell.execute_reply":"2022-12-28T15:34:28.938971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This rather unexpected error is better understood if we take into account that after the transpose method `t()`, b is no longer contiguous, that is contiguity is lost because adjacent row entries are no longer in adjacent memory addresses (more at this [link](https://stackoverflow.com/questions/26998223/what-is-the-difference-between-contiguous-and-non-contiguous-arrays/26999092#26999092)).","metadata":{}},{"cell_type":"markdown","source":"`reshape()`, on the other hand, according to the documentation:\n\n> When possible, the returned tensor will be a view of input. Otherwise, it will be a copy. Contiguous inputs and inputs with compatible strides can be reshaped without copying, but you should not depend on the copying vs. viewing behavior.\n\nThat means that `reshape()` may return a copy or a view of the original tensor. You can not count on that to return a view or a copy (use `clone()` if you need a complete copy of the tensor).\n\nMoreover, `reshape()` doesn't impose any contiguity constraints.\n\nLet's see the exact same example as before, using reshape this time","metadata":{}},{"cell_type":"code","source":"a = torch.tensor([[1, 2], [3, 4]])\nprint(a)\n\n# Let's transpose it\nb = a.t()\nprint(b)\n\nb.reshape(1, -1)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.943076Z","iopub.execute_input":"2022-12-28T15:34:28.943934Z","iopub.status.idle":"2022-12-28T15:34:28.954316Z","shell.execute_reply.started":"2022-12-28T15:34:28.943894Z","shell.execute_reply":"2022-12-28T15:34:28.953086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Permute\n\n[torch.permute()](https://pytorch.org/docs/stable/generated/torch.permute.html) rearranges the original tensor according to the desired ordering and returns a view of the original tensor with its dimensions permuted.","metadata":{}},{"cell_type":"code","source":"tensor3 = torch.tensor([[1, 2, 3], [3, 4., 5]])\nprint(tensor3)\nprint(tensor3.size())","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.956742Z","iopub.execute_input":"2022-12-28T15:34:28.957123Z","iopub.status.idle":"2022-12-28T15:34:28.967079Z","shell.execute_reply.started":"2022-12-28T15:34:28.957091Z","shell.execute_reply":"2022-12-28T15:34:28.965791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's rearrange the tensor such as the final output is size (3, 2)","metadata":{}},{"cell_type":"code","source":"tensor_permuted = tensor3.permute(1, 0) #Take the second dimension and place it first\nprint(tensor_permuted)\nprint(tensor_permuted.size())","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.96921Z","iopub.execute_input":"2022-12-28T15:34:28.96966Z","iopub.status.idle":"2022-12-28T15:34:28.987341Z","shell.execute_reply.started":"2022-12-28T15:34:28.969613Z","shell.execute_reply":"2022-12-28T15:34:28.985495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`permute()` accepts as parameters the sequence of indices in desired ordering of dimensions, starting from 0. Also `-1` is a valid parameter, telling torch to figure out the remaining index.\n\nThis can be better undestood with a 3D tensor","metadata":{}},{"cell_type":"code","source":"tensor3 = torch.tensor([[[1, 2, 3]],\n                      [ [5, 6, 7]]])\nprint(tensor3)\nprint(tensor3.size())","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:28.989627Z","iopub.execute_input":"2022-12-28T15:34:28.990318Z","iopub.status.idle":"2022-12-28T15:34:28.99829Z","shell.execute_reply.started":"2022-12-28T15:34:28.990279Z","shell.execute_reply":"2022-12-28T15:34:28.996826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(tensor3.permute(-1, 0, 1)) # 2 is the last index remaining, torch figures it out thanks to -1\nprint(tensor3.permute(-1, 0, 1).size())","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.000895Z","iopub.execute_input":"2022-12-28T15:34:29.003521Z","iopub.status.idle":"2022-12-28T15:34:29.010338Z","shell.execute_reply.started":"2022-12-28T15:34:29.003472Z","shell.execute_reply":"2022-12-28T15:34:29.009089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As always, be careful not to have repeated dims otherwise an error is raised","metadata":{}},{"cell_type":"code","source":"try:\n    print(tensor3.permute(1, 0, 0))\nexcept RuntimeError as e:\n    print(e)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.011402Z","iopub.execute_input":"2022-12-28T15:34:29.011777Z","iopub.status.idle":"2022-12-28T15:34:29.035771Z","shell.execute_reply.started":"2022-12-28T15:34:29.011745Z","shell.execute_reply":"2022-12-28T15:34:29.03479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`permute()` comes in handy particularly when working data batches, as sometimes sometimes rearranging them is required due to model configurations.\n\nOne possibile confusion could raise since `permute()` is rather similar to view/reshape.\n\nLet's break it down","metadata":{}},{"cell_type":"code","source":"tensor_new = torch.tensor([[1, 2, 1], [3, 4, 5]])\nprint(tensor_new)\nprint(tensor_new.size())\nprint(tensor_new.view(3, 2)) #Let's mimick the permute by switching the first with the second dimension (2, 3) to (3, 2)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.036798Z","iopub.execute_input":"2022-12-28T15:34:29.037132Z","iopub.status.idle":"2022-12-28T15:34:29.04597Z","shell.execute_reply.started":"2022-12-28T15:34:29.037102Z","shell.execute_reply":"2022-12-28T15:34:29.044786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tensor_new.permute(1, 0)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.047511Z","iopub.execute_input":"2022-12-28T15:34:29.047904Z","iopub.status.idle":"2022-12-28T15:34:29.064316Z","shell.execute_reply.started":"2022-12-28T15:34:29.047873Z","shell.execute_reply":"2022-12-28T15:34:29.06237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, results are completely different. `permute()` is used to swap axes whereas `view()` changes how the tensor is represented.\n\nThe last example of `permute()` is similar to operating a transpose.","metadata":{}},{"cell_type":"code","source":"tensor_new.t()","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.06543Z","iopub.execute_input":"2022-12-28T15:34:29.065773Z","iopub.status.idle":"2022-12-28T15:34:29.077542Z","shell.execute_reply.started":"2022-12-28T15:34:29.065744Z","shell.execute_reply":"2022-12-28T15:34:29.076564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Flatten\n\n[torch.flatten()](https://pytorch.org/docs/stable/generated/torch.flatten.html), believe it or not, is another view/reshape kind of action on a tensor. According to the documentation, it flattens a contiguous range of dims in a tensor.","metadata":{}},{"cell_type":"code","source":"tensor4 = torch.tensor([[1, 2], [3, 4.]])\ntensor4","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.078662Z","iopub.execute_input":"2022-12-28T15:34:29.079911Z","iopub.status.idle":"2022-12-28T15:34:29.093319Z","shell.execute_reply.started":"2022-12-28T15:34:29.079865Z","shell.execute_reply":"2022-12-28T15:34:29.091796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tensor4.flatten()","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.098889Z","iopub.execute_input":"2022-12-28T15:34:29.09929Z","iopub.status.idle":"2022-12-28T15:34:29.109902Z","shell.execute_reply.started":"2022-12-28T15:34:29.099253Z","shell.execute_reply":"2022-12-28T15:34:29.107598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What it basically does is returning a tensor with the same data but reshaped to a single dimension.\n\nA more complex usage can be achieved exploiting flatten parameters, `start_dim` and `end_dim`.","metadata":{}},{"cell_type":"code","source":"tensor5 = torch.tensor([\n    [\n    [[2., 3], [4, 5]],\n    [[20, 30], [40, 50]],\n    [[6, 7], [8, 9]],\n    [[60, 70], [80, 90]]\n    ],\n    [\n    [[4, 6], [8, 10]],\n    [[40, 60], [80, 100]],\n    [[12, 14], [16, 18]],\n    [[120, 140], [160, 180]],\n    ]\n])\nprint(tensor5)\nprint(tensor5.size())","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.111766Z","iopub.execute_input":"2022-12-28T15:34:29.112772Z","iopub.status.idle":"2022-12-28T15:34:29.126297Z","shell.execute_reply.started":"2022-12-28T15:34:29.112715Z","shell.execute_reply":"2022-12-28T15:34:29.124746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(tensor5.flatten(start_dim=1, end_dim=2))\nprint(tensor5.flatten(start_dim=1, end_dim=2).size())","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.129492Z","iopub.execute_input":"2022-12-28T15:34:29.130413Z","iopub.status.idle":"2022-12-28T15:34:29.139488Z","shell.execute_reply.started":"2022-12-28T15:34:29.130366Z","shell.execute_reply":"2022-12-28T15:34:29.138593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What happend is that axis 1 and 2 (shape 4 and 2 respectively) are flatten (or reshaped) to 8.\n\nThe same could have been obtained with `view()` or `reshape()`","metadata":{}},{"cell_type":"code","source":"tensor5.view(2, -1, 2)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.142146Z","iopub.execute_input":"2022-12-28T15:34:29.142618Z","iopub.status.idle":"2022-12-28T15:34:29.154533Z","shell.execute_reply.started":"2022-12-28T15:34:29.142574Z","shell.execute_reply":"2022-12-28T15:34:29.152769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tensor5.reshape(2, -1, 2)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.155831Z","iopub.execute_input":"2022-12-28T15:34:29.156452Z","iopub.status.idle":"2022-12-28T15:34:29.166168Z","shell.execute_reply.started":"2022-12-28T15:34:29.15642Z","shell.execute_reply":"2022-12-28T15:34:29.164775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Be careful of passing the correct dimension to flatten!","metadata":{}},{"cell_type":"code","source":"try:\n    tensor5.flatten(start_dim=0, end_dim=10)\nexcept IndexError as e:\n    print(e)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.168394Z","iopub.execute_input":"2022-12-28T15:34:29.168785Z","iopub.status.idle":"2022-12-28T15:34:29.183801Z","shell.execute_reply.started":"2022-12-28T15:34:29.168745Z","shell.execute_reply":"2022-12-28T15:34:29.182391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As always, depending on the situation `flatten()` can be useful instead of using view/reshape.\nAll of them are useful functions when dealing with tensors in a Deep Learning context.","metadata":{}},{"cell_type":"markdown","source":"## Cat\n\nThe last function is [torch.cat()](https://pytorch.org/docs/stable/generated/torch.cat.html). It concatenates the given sequence of tensors in the given dimension (default is along 0, thus along rows)","metadata":{}},{"cell_type":"code","source":"tensor6 = torch.tensor([[1, 2], [3, 4.]])\ntensor7 = torch.tensor([[2, 4], [6, 8.]])\n\nprint(tensor6)\nprint(tensor7)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.185533Z","iopub.execute_input":"2022-12-28T15:34:29.186018Z","iopub.status.idle":"2022-12-28T15:34:29.195016Z","shell.execute_reply.started":"2022-12-28T15:34:29.185974Z","shell.execute_reply":"2022-12-28T15:34:29.19379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.cat((tensor6, tensor7))","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.197015Z","iopub.execute_input":"2022-12-28T15:34:29.197643Z","iopub.status.idle":"2022-12-28T15:34:29.209574Z","shell.execute_reply.started":"2022-12-28T15:34:29.197426Z","shell.execute_reply":"2022-12-28T15:34:29.208653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By chaning the `dim` parameters, the concatenation can be achieved along columns","metadata":{}},{"cell_type":"code","source":"torch.cat((tensor6, tensor7), dim=1)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.211002Z","iopub.execute_input":"2022-12-28T15:34:29.211378Z","iopub.status.idle":"2022-12-28T15:34:29.222611Z","shell.execute_reply.started":"2022-12-28T15:34:29.211344Z","shell.execute_reply":"2022-12-28T15:34:29.221454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As stated in the documentation, all tensors must either have the same shape (except in the concatenating dimension) or be empty, otherwise an error is raised","metadata":{}},{"cell_type":"code","source":"tensor8 = torch.tensor([[2, 4, 6], [6, 8., 10]])\nprint(tensor8)\n\nprint(\"\\nERROR WILL BE RAISED HERE:\")\ntry:\n    torch.cat((tensor6, tensor7, tensor8))\nexcept RuntimeError as e:\n    print(e)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.224129Z","iopub.execute_input":"2022-12-28T15:34:29.224762Z","iopub.status.idle":"2022-12-28T15:34:29.249232Z","shell.execute_reply.started":"2022-12-28T15:34:29.224726Z","shell.execute_reply":"2022-12-28T15:34:29.247549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The error is given by the fact that the shape of the last tensor is (2, 3), thus by concatenating along dimension 0 (vertically), the number of columns won't match.\n\nIf we were concatenating with `dim=1`, everything would be fine since tensors sizes match except the concatenating dimension.","metadata":{}},{"cell_type":"code","source":"torch.cat((tensor6, tensor7, tensor8), dim=1)","metadata":{"execution":{"iopub.status.busy":"2022-12-28T15:34:29.251116Z","iopub.execute_input":"2022-12-28T15:34:29.25157Z","iopub.status.idle":"2022-12-28T15:34:29.261972Z","shell.execute_reply.started":"2022-12-28T15:34:29.251528Z","shell.execute_reply":"2022-12-28T15:34:29.260623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`cat()` is essential when dealing with CNN filters, especially in the final layers of the model were all the different outputs need to be concatenated before being fed to the fully connected part.","metadata":{}},{"cell_type":"markdown","source":"## Conclusion\n\nI hope this short introduction avoids some head-scratching on popular yet sometimes misunderstood torch functions!\n\nHappy PyTorch!","metadata":{}},{"cell_type":"markdown","source":"## References\n* [Official documentation](https://pytorch.org/docs/stable/torch.html) for tensor operations\n* A well-written [blog](https://jdhao.github.io/2019/07/10/pytorch_view_reshape_transpose_permute/) with most of the info covered here","metadata":{}}]}