{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction \n`V1.0.2`\n### Who am I\nJust a fellow Kaggle learner. I was creating this Notebook as practice and thought it could be useful to some others \n### Who is this for\nThis Notebook is for people that learn from examples. Forget the boring lectures and follow along for a fun and instructive time :)\n### What can I learn here\nYou learn all the basics needed to create a transfer learning model. You will learn things ranging from Transfer Learning, Data Augmentation, Data Preprocessing and ect. I go over a multitude of steps with explanations. Hopefully this notebook can inspire you and help you to work on your own notebook.\n\n### Things to remember\n+ Please Upvote/Like the Notebook so other people can learn from it\n+ Feel free to give any recommendations/changes. \n+ I will be continuously updating the notebook. Look forward to many more upcoming changes in the future.\n\n### You can also refer to these notebooks that have helped me as well:\n+ https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-baseline-flash-effnet-aug?scriptVersionId=101586230\n+ https://www.kaggle.com/code/jirkaborovec/bloodclots-eda-load-wsi-prune-background\n+ https://www.kaggle.com/code/queyrusi/mayo-clots-vgg11-gpu-train-submission","metadata":{}},{"cell_type":"markdown","source":"# Installs\nHere we are installing specific versions of Tensorflow and Pillow libraries. This is to allow us to use a more recent version of EfficientNet Model and also to make use of the updated methods of Pillow. Please uncomment these cells and install the proper versions. After the install is completed, please restart & clear cell outputs.\n\n![image.png](attachment:7f68272a-2776-462b-a92b-2466d15aca08.png)","metadata":{},"attachments":{"7f68272a-2776-462b-a92b-2466d15aca08.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAggAAAEmCAYAAAAHsmi4AAAgAElEQVR4nOzdeVhUZfvA8e+wyiAIAoo4gLghgsruroC4pOKSS5mKryblnu1m9aotavZmppmVqZn65lqvS0ri9kPFXUBZckFFcEnAhX0R5vcHOoGDMCCK1v25Li+YOec8z33OjMw9z3YUjZyaqhFCCCGEKEGvpgMQQgghnh7ynfk+SRCEEEIIDUVNB/DUkARBCCGEEFokQRBCCCGEFkkQhBBCCKFFEgQhhBBCaJEEQQghhBBaJEEQQgghhBZJEIQQQgihRRIEIYQQQmiRBEEIIYQQWiRBEEIIIYQWSRCEEEIIoUUSBCGEEEJokQRBCCGEEFokQRBCCCGEFkkQhBBCCKHFQOXoWNMxCCGEEOIpY5CcmFjTMQghhBDiKSNdDEIIIYTQYlDTAQghhBDi4RQKPUyUJhjXqoWhkREG+gYo9BQAqIvU3C28S0F+Pnm5ueRk56BWF1VLvZIgCCGEEE8hQ0MjapubYVrbFFCUuY9CX4GRvhFGRkaY1q4NqMnKzCIzPYOCgvxHql8SBCGEEOIpU8fSEjNz8yocqcC0dm1Ma9cmIz2dO7duVTkGSRCEEE+Mh4c7PXp0B2DnzjAiI6NqOKKKScziSTI0NMLS2gojI6NHLsvM3BzjWrW4lZpWpdYEGaQohHhievTojqlSialSqfkAq4xHn5bdEv+BfrhW4oiaj7nyHjVmUTOMjI2xqV+/WpIDTZlGRsVlGhtX+lhJEIQQlTY37ARrxtx7MC+U6DUv63ScqVJZ5u9PxJjFhJ/4gemvTmPFib0sGVPxIVCzMYesOcEfEaGEhe4l+shSQnQ8rkav89+CHa/O/ozhqidXo6GhEdY29dDTr/6PZT19Paxt6mFoWLnEQxIEIcQ/wtyh7Uj5sQPde/XCd84hzF1H1HRIOkk60IvuvfzZke6Jn45JzdPvfdYeOcLhI0dY+2FNx/IgO4Yv/onR3fyYvHzxE0sSLK2tHktycJ+evh6W1laVOkbGIAghqsAcr4kRHH0VMDaH2JqOpyIvY2+bwrlF9x5unM7wjTUakM7sO4USFmZOPfOTfL28pqP5uytODiZ7mwGQn3aVi8mPv9Y6lpbV2q3wMEZGRtSxtNR54KK0IAghqiCdE4s74Nu2A747n8Bf0Ee2jNhkG5q9fu/r4Jj5bPri2WlBmBGZTt75wyyt6WD+1uwYvugHTXIAYNS4H/Mj9hO2dR1fvzcY58dQq6GhUaVnK3iO/oyVP69n4/r1rJzzMp6VONbM3FznrgZJEP7xlCh6ulPr/X6YvOKMnqdhTQckxGMx+8stGI/YTHjoXqKntOTc9tU1HZLOIt7ZwLkWI1jSraYjqbwOL49jcKWa6e0Y/MpoOjyugB7KDCO9Mj449Y0wq9cI7wFvs3LXOmY9b1ettdY2N6t4p1JGMqyHE6b6xY9Mm/RkWPDjqVO6GP6xlBi+/wr1Ap0x1r//3HMwAtSZMVz/4ntydhfUZIDiKTatu9dfD97pRZuaC0V3u2fR12tWTUdRKUuHe91rNVjGcK9lNRxN5dmNWMysV7wxe6EVxmMmsqbCxqa/mvgz2uQzauIarj7uIFXDmfPv1sR89C4rJgaTX7KL4dLvfPe/q7j17EdHFyuMzBrR860fsFKOZdLqR49ModC7twjSk2Va25TbN29VuOKitCD8IxliuODfqHqWTA7+oqjtRoMZM6j93JOPrBTH9vTq1rKGg3iSVHTo3YMO1TgrzrVbEP6tqq88IXQ3nOnB3pgBmFjj6KTLMU44WhdPxzPzHs30YY8xPAC8mf75OPzb+DH5+/kM5yprJgaz6HgGAEaWVhhFfct7/+rNi5/t42o+oG+F96tfMt330Ws3UZrwsBUSH24VP++8SFZh8aOshN/5+afK1qy4V3f5JEH4J5o0ETvPOhW8La2oN/VVdB02M2tzDH/E3PsXFUHY+sW8GVT14b8dZofyx66fWPD1fOY6Upws9G6P/cMOaDWCBZsjiI6J4Y+YE4Qtn0b/xzL9/HU2xZ/j6JKg6i96zGyWTOuP8f0brLaawJKwE/wRE0N0xHpmVeF6prceyYL5s2uguVb809m92RPvOgD5nNn0OrP363LUQWa/uYEz+QBmePd+m+pt0H/Ay+Po1bj4r1zGxRPsBbiXJPxn9Rr+M2YiK+KLd736y7s8/9lB0gCMGtFr7OhHrt64Vq0qHXdyxbuMGjaUwUOHMuq9ZZx8THVLgvCPY0/tQGfKaDjQZuKOxQzd5lAbK43JO7Oadz98j3dnrSYiswnB8zez5pWqJQn9fJpw4/dgWrj0Yloi0G08n80eT68y9x7C8h9m4H/3d754bRLjP1zGeacRfLZ42sMTiirbwE/ff8PX32yt5nJVzHrek6Tdc+79kRrC8h9ep0P6FmaEhPB1tA0vzFvM9EomPUlfbuCEeT/enFzN4QpRLjv+5XtvSF/aMTYtqERzfPJCNh1PK/69uTf/eozTDCf7tSr+EpRzmv+W6s64ysZFC9n4YJfItjfYGF28IqGRmz+P+t/K8AnMXHiUuiVB+Mdxxriu7nubqDx03zk3hc2/bmXzr98wY0wv+q5MxmvKfCYBDJzNpiWvE/zFesJCFxc/hx+TvviJbaGhpVocJi0JJaAB1POcQdj62fQfOJtNI10xN3VlZGgoS7T+V3pgb5PAjkGz+Gn3Pvb++g3jP/mG/0Wm/JUgtBrB3DWbCQsNZdOS10u1LriOms2azaGEha5nyWQ/zfP2Qa+zZH0oYaGbWfPxiHtlJWPfvCftNbup6P/6YjaFhhK2+SfmjvqrS2TSkvXMHT6CBeu1y9biOIEOLVM4/+O9v0hjgvAyjeSnQbNYd+gQS8dPZ29KS/wnqrTL3ryU6UEqXEfNL45j/XyCNd0KGzh0Po+m7XRbyEhXc8POEb03lLCwCKLDlxLy5BcLrLSqLjpU4yavJ/pCBMsH1nQgleGP3b0p9xkXT7ClkkdvOXGR4kZ+K+zK+W/zaIbj3KD4t4zYvazQ8agVe08Vx6Zvh/MjdoEY6FdtGOCjzGKoTN2SIPzjVG6WgkKv6uNYkz7ZRyxNaD8KsFTR1O9lJrneITb2PCm0Z27o10xqX4uYzYvZnNSEkHvfkFMuxJGUBenX4og9k0BKSgIxF1PIy0vhfGwc5689WFMy6dkqAtbM4IX7H4y7v2Hah8uIAHB8mTU/TiPAMJbNP//OnRYv89l395rdBy9lxTQ/OLmarzen4DrhazZNBhynsWTeCJombeHrn09Sq/cM1nzVHgBV0yY0vZd5+M/7gc9GNeHG7tVsjq1FwLRVmlYTVVMPBrz9MubHVrP5pDEdXv/ir9UHH/R8U+xvJXHyXveCvas95tfO84Vmh0OcTEqnXsOef5U9ZQTGx7YQkelKyJzNrPiXGRGbfyfJpgez5szQJEdLE1Mwb9qO/lV8HcuWTsyqXnTv3oHZ510ZOvEJLjn3CJ7FRYfe9FNx/lAKTYOG1HQoldAI63sD5dOur6n84auvFjflY4a1TmMXqqKKMf5cIramjxbB/Vs2V86jz2LQtW6ZxfCPk0pRDlDx+BQA8jOuP0Jd6eTeBU1/Rt5JlvYK0czlXvd2CFvSDxGRCFCP9t0m4NoNZn/+Bh49gjBPeIOp91ZZi2jelQGecOjNN8qYC/4Ng14zZ/m0Icz6ZQSzslI4f2gDS+d+yeZE6PDGCLyubaD70FkkAewzZttv/Rg7ECI87bG5FcmnP65mc+JqTh5oCaeBUS2xL0xm3Y/fsPk0bD65AdfTcQ/UO4FJvVXEfufG+EUAqzlpupc1w96gw/dvAHBj90jGfJ4MrMa89TkGtA+C5WV0TzQwx/jmee6PNepla076jYRSuyy9ksJ0jyaaxzcOjGX858mAig4X2nP+wxC+OADkeRA9pQm9oPhanb9Bem9j6lX0clXRutR0pjfsCTz9o+zrNJnPguUt8TePfUYWHZqGf6NkQp+PpdeqHrzABtbVdEjV4lNebPtpTQchKiAtCP84x8g6n63jvnfIijjz2CKxb92DiQv3Fg8sjH+dDlW5s+l9u+cyprsXLZr1Z/x3+8hr81crgau1OcbOQwi7P4jytxG4Ko0xtgS+28Deu11ZsCuG6PDNzB16r7Fu5XL+l6Ai5JcYoiNC2TTBr4wPWBXmyuS/VucDIqKTSLe01twMKK/wr07MlDvpGCt1+5iOvZWOuVWTUs8F29qQl56ieVyybMjjzgGdiq5mKkIa2nDjyu81UXml5V7by96t8xk/NPiZWHTIfkZ77O+a0/+79pibejL0mRlLconU4j4CrGyHV/7wEXYU91BkkHqxGsMqpYoxDisR2/lHi0BdpK7CUdUxi0G3uiVB+AfKX7GDOzrc+TPvzA7S1z9CRWO64maaTGyZ39SmseSDftQ5uYypw9xo4fIlEelVrKeVH/01Mxzi2LtkOoOG7eRGc0/63dsl/eiXtHBz0/xzauzF8OVA4jLGdHHD6flJfLE1mXoD73cl7GNGfy+cAkN4d9VJ8HydJesnlFG5MbVK9r8bAIWQV5XzMKil6RaIuJ4Olja8oNmookldc9JvVuEv0mNpJzTHbWQo4UdCCam9hdnvPAurKUJe9lY2/7rzXqvV007FWB8bYr7rRfdevejyazJNO5f1HnwareHMva5AMycvzf9DXfXzciqeHkkaV/dVb2R/KRGjqz+6zkkY7d+6OLbCq5z5+dEiuFt4t0rHVccsBl3qlgThn+j4LtIW7SK9nCQh7+IWrs/8P6qS34KKDq/MJ2xKe/L2rWZ2mfuYY2yczvkfV7P3NNgPb0ez8loQCgGDOtiUNRjOeTjvfz2b6ZppgCr6T/HAPiWJSGDp4QSM2/Rj+v3xCd1msy38J6Z7Q6+PNxO+5nXsT+/jp8/nsDchjzqWTWD8Uo6Gzsc/8RChS6bz9ckUjM0f7GffSmyiCv/ZL9/7YPdj7hBPjBMOU+mEPvYG6ZbWf83S+OR3Tpq2Z+y84hFa9q/MZoBzMhErKz97wt7ZHvNbqYRW+siHm9a9GW38e9GlrRu+/Wfdm3nxdFs63Ivu79R0FJWRzIy+HYoTWYBP+tNm6Dc1GlFlbIq61/po5cOgqZWYrKiawiDveyMczx7nx8eYey7ad5p8AJNWDP7Pg2mMHYMnTym1CqTdiMW81KZ49H9+zF4W8WgK8nX4pvaY6FK3JAj/VJs3kTpqFsnhUWTeyaYgJ5uCnDtkJ0ZxY+UsrozaQeGVyhVp3n4aFy+c4+KFvayZ0I683bMYFLLhIXt/w/oDxvTfVdzsHzbFntzyWhBWbmVvShNCdsWw6YMHtm2cxdfb8/Cfv7e4CyF+Lwt65hG6cFZxf+2iN/jikDnBv9zrYvi2H+Zn/o9VxyF030nSXScQHh/DH/GhBNue5KfvV8P2w8Qoe7A8vviYJZ3T2fzD9AcqPsTUT1aT5Drt3vFLGWB8iC/ersIf8ZWHOW/QlPaakerfMHXhIYyDlvJHTAzhb7uStHoWU6vQjTDSWUVS7Ibi8RdCPCFXv9hIRPGiATgP+pLpnXU5qiPTvxiCsxFABse3f/54V1Jc9i2hF4o/KK06v80vswffW3fBmdGLf+KtEcN56/u/7ujof79lI/8SoT/oOu/h4fJyc6t0XHXMYtClbkUjp6ZV+5IoRHVo5Ud/+zxObj9UDR9gLfEf2ATz9AQ2735wQOH97fW48es+rZsPunYLwj47ktBDpb+u2Lfvgacy+SHlVXx8ZYSsOUFI3lx8x5RMqlR06N2UO9u1Y9aJ4wy2hfpxPsS/SslFdfto1oxSj/89o3JLH6scHUlOfLL9A//EmKtN3/n89mHH4j77/Kvs/XIi7/3ykI98l8HM+eQ1/FX3Fi46vvCJLbX89fdTuN9okZ92hoPbfiemQT9e7dGoeJ2EtOMsemUia5LtGD77I9zi/s171bTUckMHFZVbTXEkn/8chFOJxWwubhvK25VqtlRz5XJyhUstyywGUbNO72Pz6eoqLI69v5b3Qf7w7bG7t5b5AZx0aKdOicvDjq+MpdM30OvnIUz32MDsyPvPJhOxvepJxwtv+2F+aPVTkRyIf6BtbxBisZiVk70xM7LD/91fCRtykN9Dt7Bv9z6OJ9vh3a0nPZ7vhb9HI8zufehlxKxhxpNIDgCS1zDplTxmfT6Ono3NMLJyxn+UM/4l97HyZvLyxRi9NpEV08dWW9VqdRFZmVmY1q5dbWXqIiszq8LkACRBEOLpkTiXQdW8JvK6Sf6PZVrc3LBzPGeQwI275tQzjuXrkSEsfcoH/s0NO4H9z/cGp84LJbrhBtoMf4qnZo75iegP7NkbfK/1Z/BSjs6zZ0/jXkyr4NAf/W1LPzGj7P2ehKurJzLqxtvMeWcwzmZg1rgjgyd0ZHBZ4y0LM7i081vemLnxySQH9yVvZMawjfx36PtMHtgRZ5UVZg8uNHgX8jOqv+rM9IxKJgir+HmnG6/dWwuhKrMYMtN1OxFJEIQQVVC8UNLw5fDC8gjGTlSx9BmZyfBMybbBc8wQOLCBSUNdMc+u6lSfmnV15+eM2rmFfhPGENTDC2crMzQr/Rbmk5+ZypnI39n647dsia+5OM+s/5RJ92ZudfhwHXP73utiuHO/i6H66ywoyCcjPR0zc93neZ9c8S6jqjgEIiM9nYIC3QZHSoIghHgkz9JCSc+ca3GktBjCJMeW9LKNJeJa9d9d5Mk5w5Zv3mXLMzERww4nW6u/kgOdblVddXdu3cK4Vi2MHvO9GfLz87lz65bO+8ssBiHEI3h2FkpKz4NadYp/76A0Ji/7Rs0GpJPzrP/DhqE/+GEeuZxnIeK/h9J3dHycycF9t1LTKCqseFxAVRUVFnErNa1Sx0iCIISogqotlJSVnV3m70/C7F8OYzMigrDQvSxpn86OtbqtKVGTMQOsWx4JqnT2zj+k0/56dWpjczNH87jk76IyHnJHx8ekoCCf1JQbjyVJKCosIjXlhs5dC/fJNEchxBPj4eFOjx7dAdi5M4zIyKgajqhiz2LMrlOGkzO4EwAmmw4S+9XqGo5I6MrQ0AhLa6tq627Iz8/nVmpapZMDkARBCCGEeOrUsbSs1MDFsmSkp1dqzMGDZJCiEEII8ZS5c+sW2ZlZ1DY3w7S2KbovpqQmKzOLzPSMKrUalCQtCEIIIcRTTKHQw0RpgnGtWhgaGWGgb4BCrzhhUBepuVt4l4L8fPJyc8nJztFpESRdSAuCEEII8RRTq4vIzsoiOyvridYrsxiEEEIIocVA5VjW/XOFEEII8U9m8KTvMiaEEEKIp590MQghhBBCiyQIQgghhNAiCYIQQgghtEiCIIQQQggtkiAIIYQQQosslCSeCl1zsvkkNQXV3YKaDkUIIf7Rkg0M+crSUpZaFk+Ho5cvYlNYWNNhCCGEAPIUCuliEE8HSQ6EEOLpYaxWSxeDePo4OTXVeV+1Wv2Qx+p7j6srKiGE+PtRaG4SWfxLYuIFzTZJEMQzq3RyoEatBj09PUxqmWBgaIBCoevtUYUQ4p9LrVZzt+AuObk5pZ6XBEH8DfyVHJiZmUliIIQQlaBQKDA0MsTAsHRKIGMQxDOpZOuBWl18O9RaxrUkORBCiCp68O+nJAjiGaemuAVBjaGRYU0HI4QQfxuSIIhnWnHrQXGaIK0HQghRfSRBEM+cB2cuFD8nyYEQQlQnSRDEM0zmMAohxOMiCYL4G5BEQQghqpskCOJvQpIEIYSoTpIgiGefDE4UQohqJwmCeGbJMspCCPH4SIIghBBCCC2SIAghhBBCiyQIQgghhNAiCYIQQgghtEiCIIQQQggtkiAIIYQQQoskCEIIIYTQIgmCEEIIIbRIgiCEEEIILZIgCCGEEEKLJAhCCCGE0GJQ0wEI8SxwcHSkf/9+uLt7Ur9+PYyMjAAoKMgnMzObuLhYdoWFcfDgwRqOVAghqockCEKUQ6k0IeSVV/H3D0CpNNHabmhohKWlER07dqRdu3ZEn4pm5YofiY+Pr4FohRCi+kiCIEQJgYGBTJo8BaXShLCwMBzsHWju3ByFQkF+fgHxcXEcOHiAgwcPkpqSQqfOnenYsQM+Pj6Ym9fB08OTxk6NWbZsGb+Hhtb06QghRJVJgiDEQ3Tt2hUjIyPUajWXExP59ttvOX78eKl9Duzfz4H9+7G2seGVkBA6duqEhYUFo0aNIicnh/D/+78ail4IIR6NJAhCPMT95CAmJoavvvqKy4mJD903NSWF2bNnMzZkLAMGPI+1tTXDXhrGhYQEkpOTqyUepdIETy9vahkba55Lz8jg6JEj1VJ+TbK2scHNzQ2AmJgYUlNSdDpOqTTBt207XF1duXb1KsdPnCj1Orl7uFO7thknTxwnOzvnscRenubOznh6emBpUZcTJ09U+rXybdsWgKNHjuDg6Ejjxo25cOFCue/Fiq6JELqSBEGIcvz553V+WLpU5z+wPyz9Adv6tnTu0gVHB0f69OnDd999Vy2xNHd2ZsKECVhbW5d6Pj0jnfVr17N+/bpqqWfo0BeoXbs2y5cveyJlBQYGEvLKq5iaKgHIyspm6fffsWvXrnLL7tmrF6NHj8HS0oK8vDwMDQ0ZU1hEWNhOvlqwAIBhw17C3t6eefM+Iyoy6pHPR1dKpQnvvTcdbx8fiooKKSpS039Af65cucq33y7h2NGjOpXzSkgIUJwg+Pv5M/D55/n1l19YufLHMvcv+5oUsmP7dhYvXqxz/C4uLnQLDOTk8RNEHIrQ+biKqFQqgoL6kZBwnp07d1ZbueLxkGmOQpTD1rYBXy1cyM6wMN544w2djtm0aRMpKTcwMDDAt207VCpVtcYUGRlJj+7d6dG9O69NmUJSUhIDBg7A3cO9Wsr38vaiQ4f2T6yszp07o1DAnNmzefuttygoyKNz587lHuPj68uo4GCyMjN5f/p0+gUFMXLkSI4dO0q3bt0YOTK4WuKvqomTJtPG3Z3t27czZPBggvr2ZcGCBSiVJox5eUy1vyegvGtyTJM46Kphw4b4+XWlWfPm1RqjtY01nbt01rQWiaebtCAIUQkuLi68On4czs2d0dPTIy0tjZ//+1+2bt2q2Sc+Pp74uHjq1auPlVVdWrZsWW3dDA+Kj4/nxPHjDBo0GDc3N6Iio1AqTRgZPIp27dujNFESHx/HurVrNTMrrG1sGDliJF7enhgaGhMXH8eK5cu5nJjIjJkzadzYCUNDIxYv/obQ0B1s3boVFxcXXnjxRZo3awYUdwOsXrOGy4mJ+LZtywtDhxIXH4eXlzdGRkYsWLCAQYMGlVlWWfLz87h58yZW1tYYGRmTm5tb7nn37t0bfQNDli9frBkXkpqSwufzPuM/X8ynfYf2bNq0Qeu4iq7N/e2+Pr6Y1zEjKSmZ//36K+Hh4QDMmDmTrKwszM3MaNa8OVu2bObn//5cqo6OHTvStq0vR48cYdHChZrnQ3fswMKiDkOHDKVT506s/XlthfFURnnX5NPZc+jUuTNhYTuxa9iQ4cOHc+zYMVavWgVAUFAQfn5+rFu/nvr16tE3KAgTExP8A/xp5NSIWTNn8trUqVhYWHD1ylW6+nVBT6FHZFQUK1as0HQJzZg5E4BZ9342d3ZmzMtjiDkdw9mzZwkeORJzc3O8vLyZO+8zli9bTnLSZV588SU6dupEnTpmXLp0iZ9++olT0acqfQ1E9ZIEQYgSdu3a9dCmbaXShE9nz6GlS0vNc9bW1owMDubmzZul1kBIvJxIQUE+hoaG2DVs+Fhj1tczQK1Wk59XAMD7H3yIq6srUVFRFOTn08bdnbfeeptZH83icmIiUyZPpnWbNhw5fJjbt+8Q0M2fCRMn8NHMmSRdTsLB3gGlqZILFy+QlpqGi4sL701/D6VSyeHDR9DT06N9+/aYmZvz3rRpmJuZ4eTkhJOTExcvXuT0qdPcvnWrzLLKEnHoEG5urRg3fjz169uSn5/Hrt27H3q+SqUJDvYOXE5M1Fp3Ijs7hwnjxz/02IquzbRp7+Hl7UVUZBQnI0/i7ePDq6+OIz0jnajIKOxVKqysrbmZdpOTJ0+SeFG766lZ8+Yo0OPYAwNaAdb+vJa1P6/VOR5dVXRNoqOjCeobhKubG4V37+Lo2IjES5c0+9Sta4VjI0fMzcxIS00jOTkZW1tb/vzzT5IuJwFgZ2eHs7MzzZs1Jy4uFkOj4um9VtZWfDRzJtnZOdg/0DJyP64b1//kxPHjJF6+jJ2dHalpqSReTCQ7K4sxL4+lV69eREVGcvDARfwD/JgwYSKzZ8+WsRM1TBIEIXTk7e2Dg72D1vPm5uZ4eHqW+sN8/dp1CgoKUSpNqGtp+VjicXB0xNvLi65+XcnMzOD06VMEBATQrFkzdmzfrhn7EBAQwLjx4+nePZBlPyzDzs6Oc2fPMWfOHABiY2MwMDAgOzuH5cuX4dzCGSt1Xb74z3+A4haHzZu3kHLjhuab9Pvvv4+rqyturVoBoKdvwIkTx/n4o4808ZVVVln2h/8ffn5+eHl5kZBwnoVfLSz3G3RzZ2dMlCZci71aqeuly7XZt28f0adOsWnjRgD69+9PcPAoTesMwJ07d8r9AK9raUlefh5//nn9kePRVUXX5Nq1a6gVamzr23LlSvmtWRGHIlCaKmnTpjVxsXGlxjvo6emxYeN6/vfr/wB49dVX6dmzJ926BT60dei++Ph4jGsZ4+HhQeKlSyxZ8g0A9vb2/Hn9T2bP/pTs7BxOnYrGytqa1JQbOp+/eDwkQRCihJLrIITu2MH8+fNrOiQtHh4e7AwL0zy+du0aK39cSXx8PJ07d8HAwID6tra88+67ABgZGmKgr49tfVsArl69irePD4u/WcLxY8c4GHGQs2fOPLS+1JQUdmz/jY4dO/HGG2+gsrfHzq4hRUWFGBjoa/bLysys9LkolSb8e+ZMXFu6cuXKFerXq0+TJk0BKhwkp29QuT9fTZs2q/Da7NmzB3cPd8aPn4CdXeBjzLEAACAASURBVANUDg4Y1zJGX++vuu4WFFT4zVZPTw99RflDvHSJp7Iedk1qGRujh6JKZZaUmZHB+fMJmsfRp07hH9ANx0aNqlxmUlISrVu3ZuGir4mKiuLAgf2yhshTQhIEIUq4dfMmubk5KJUm2NSrV2rb8ePHuJx0GVdX11LPp6enE3nyZKnnbBvYYmioT0FBATdv3arWGCMjI3n3nXc0Tf8ZGRkcuvchaqI0wdDQEHuVisKiIs0xN1JSyMzKAmDu3DkMGfICvr6+9B/QnyFDh7B3714+nzevzPp8fH157bUp1KljwZ/Xr3Pt2jUuXrqIo4N2a0pl9evfnxbOLmzbupUdoaFMnz6dl14axu49u/Hz60rS5ctax0RFRpGRkYFdQ7syyyw5tbGkiq6NUmnCh/+egbu7Ozdu3CAtLY0zf/yBubdPpc7pRsoNDA0MUdnbc+KB98X96ZxXr17V6bXSVUXXpH59W4pQk5R0GT29qo9Nzyso4HaJ93Nubg5FRYUYGRpWucyvFy0iOTmJrn5++Pv507dvXyIjI/n4o1k1MjVV/EUSBCFKiP8jnoz0dOrWrYttgwaoVCrNAMPs7By+/+67MgcpPtjv6+jgiKGhEdnZ2Vy9cuXxxBofz8EDB+nTty9DhrzAypU/kpaWSn5ePtu2bWPz5s0PPfa37b9pmo4//Pe/ade2HV26dNF0IZTUtm1bapmY8OX8+ezZswco7mKoDvXr21JUeJf4P/7gcmIiy5Yt4/WpUxk48Hlu377F2bNnyzzu/PnztG/XjqCgoFJN2w6Ojrz22lRu3rzJgf37Sx1T0bXp0qULzZo1ZceO7Sz8qnhwYf/+/fH1qVyCEBMTQ+/efejYsRNhYTtLfcgNev55evbsyYoVK3R+rXRV3jXx8fHmxp83OHz4EK3buJOfn1fqWIWeAhQVtzCYKpWo7O01/yccHRwxNjLmxr3ugIysTMxMa2v211foVZiQKJUmHDhwUNNtMWHCRHr07Enfvv2qbequqBqZ5ihECdnZOVy4cAG1Wo2NtQ2dOncqtT0+Pp6pU17juV696NmjBy8NG6bV9+ri4oJLSxcA0tJuEhcX99ji3REaSlpaGn7+fjg4OhIevp/0jHR69+mDi0txDGNDxvLLr78SHDwKdw93fli2nHfefgdrGxusbWyoU8ecvPw80jPSNeUaGhlpjs9Iz0ChVmBlVbz+Qrdu3TRjD3RRsqwHXUhIQKGnR0BAANY2Nly8eJGEhAQMDAy4fv06iYmXyjxu3bp13EhJYdS/RjF8+HCUShM6duzIe+9No465OXv37NU6pqJrk56RTn5+AXXqWBQPrnN0pEvXrhjXMtYqqzxRkVGE7QyjhUsLZs78iNZtWmNtY8OkyZN5rndvzp4/x+7duyqMp7LKuiYBAQG89940rKys2P7bdrKzc0hOSiIjI5NWrVvj7e1NUFAQPXr0wEC/9PdFPYUeZuZmpZ6rU6cOzw98HmsbG1xcXOjdpw9Z2ZmcOH4CgBvX/8TaxoYRI0fQuk1rho8ciYWFhVasZubmKJUmWNvY8NVXC/n0009xcXFBqTTBwqIORUWF3L5dvS1vovKkBUGIB4SHh+Ph6YmFhQU9e/YiIuJQpUZTDxo0CBubety9e5ejRw4/timOAJcTE9m3dx9Dhg6hX79+fL1oEatXrWJkcDBfLlhAUVERarWaE8ePs3HjerKzc9i+fTuDBg1i5cof0dPTJzc3l02bNmkG4e0P38/o0aP5csECtm3bxpYtW/D08uTlsS/zr9GjKCpScz7hPPXr1a8wvgfL+nrRolLbt27dSkOVip49e7Jq1SoUCgXZWVlERkXh2rIlEydNLrPr43JiIt98s5hXXnmVkcHBjPrXv1Cr1dy6dZs1//0v27ZpD5i7nJhY4bU5cuQI3bv3YMPGTejp6ZGQkEBOBVMuy7JixXKMaxnTo3t3/vOfLwAoKCjg1KlTLFmyhOzsnArjqSxdr0lycjJ79+5h0KBBzJ4zh/z8fCJPntQktQAREQfpFhhIUFAQbdv68vHHnwBw+/YdrKysNK/V7dt3WLFiuWZQ6a7du2narLlmHYqYmBhuleiSiIqMIjIykoCAAFatWsNXXy1g46ZNjB49hvlffklRUSFqtYKwsJ2ykNJTQNHIqam6poMQ4uLF85rfnZyalruvWq0u/RM1hYVFWFtZVVs8b775Jt179EChUBAdFc28z+fptPzv/aWWjYwMSbiQwKcff/JYE4TyuHu4Y2FhWebSxfeXbc7Pz9d5+d/mzs7Y2dlVainkyvBt25b8/LxKr3hYsl+/vMGWJZV3bapSXkV1VbTcc3nxVEV1nwPAZ/PmaValBDAyMq7WZb5927bFyMioxpbFFsWio/8aNyMJgngqPG0JgkqlYurrU2nVqjUASZcvl3mzpvtK3qzJ0NCQ1NRUvv32W7lZk/jbKJkgPMllq8WTVTJBkC4GIcqQnJzMwoWLePutt2nu3BwHR0dmzvqowts9A9y+fZuVK1dKciD+VvJyc8nJyeXu3cKaDkU8IdKCIJ4KT1sLwn1KpQmjR48hsHt3TE1Ny923sLCQ6FPRrFzxY5WWyhVCiJomLQhC6Cg7O4fFixezdds2+vfvh7u7J/Xq2WBkZIRCoaCgIJ/MzGzi4mLZFRamNd1RCCGeVZIgCKGDy4mJLFq4qOIdhRDib0LWQRBCCCGEFkkQhBBCCKFFEgQhhBBCaJEEQQghhBBaJEEQQgghhBZJEIQQQgihRRIEIYQQQmiRBEEIIYQQWiRBEEIIIYQWSRCEEEIIoUUSBCGEEEJokQRBCCGEEFokQRBCCCGEFkkQhBBCCKFFEgQhhBBCaJEEQQghhBBaJEEQQgghhBaDmg5AiAddvHi+agcmJ1ZvIEII8Q8mLQjiqZCnUNR0CEIIIUqQBEE8Fb6yqFvTIQghhLjnkIcXikZOTdU1HYgQlaFWq0v/RE1hYRHWVlY1GZYQQvwtdOjqj6mpqbQgCCGEEEKbJAhCCCGE0CIJghBCCCG0SIIghBBCCC2SIAghhBBCiyQIQgghhNAiKykKoQMHR0f69++Hu7sn9erZYGRkhEKhoKAgn8zMbOLiYtkVFsbBgwdrOlQhhKgWkiAIUQ6l0oTRo8cQ2L07pqamWtsNDY2wtDSiY8eOtGvXjuhT0axc8SPx8fE1EK0QQlQfSRCEeAgHR0fefuttmjs3R6FQkJ9fQHxcHAcOHuDgwYOkpqTQqXNnOnbsgI+PD+bmdfD08KSxU2OWLVvG76GhNX0KQghRZZIgCFEGlUrFlCmTae7cHIDLiYl8++23HD9+vNR+B/bv58D+/Vjb2PBKSAgdO3XCwsKCUaNGkZOTQ/j//V9NhC+EEI9M38Ky7syaDkKIR6VWq1EqldVWXkhICO3bd0ChUBAdFc2sjz7i/PmH32UyOzub/fv3o1Sa0KyZM2ZmZjRUNSQ6Kor09PRqicm3bVsaqlRcuXJFa5uDoyMenp4UqdX4+fvTLaAbsbExFBTcrZa6K6JUmuDn509Qv37Y2tqSlpZGdnb2E6l7xMiReHh6Eh0V9VjKb+7sTJs2bSgsKuLOnTta25VKE9q2a49tgwZlvjaPwsHRkVH/GkXt2rW5ePFitZYtxMPYN3LCyMhIEgTx91CdCULHjh0ZNHgwJiYmXL16lXnz5nHt6lXN9sDAQOZ/uYAxL4/BxsaGQ4cOabadPHmSxo2dcHB0xNzcnKKiIk6cOFEtcQUHBzNo0CCSkpNJTk4ute2NN97gued6c+pUNM8PHEirNq25dPESSUlJ1VJ3eXr26sXMmR/RtWsXHBwa4evrTe/nnkOthtjY2Mdad3NnZ0aNGoWTkxNx8fGkpaVVex2jRo3iX6NHY1u/Pnv27NHa3qvXc0yePBmXFi3YsmWLTmW6uLjw4rBhGOgbkJT88Neoe4/uDBz4PIaGhvzfvn1VPgchKuN+giBdDEI8oEuXLtSpU4f8/AJ+/z2Uy4mJlTp+06ZNtHBpQb169fFt247ffvtN6wO9KmJiYvDx8aF1q9YcLpGUuLi40KRpEy5dusixo0c5dvToI9elK5VKxdChL3Dnzm1mzZxBfHw8Do6OvP766/TrH8Tp06ce64DNs2fO8K9Rox5b+SU1a+6Mj6+v1vVt27YtJiYmlSqrYcOG+Pl1JSM9g4hDEQ/db+OGjWzcsLFK8QrxqCRBEKIEpdKExo0bo1AoSElN4cD+A1r77Nq1i127dj20jPj4eOLj4qlXrz5WVnVp2bJltSQIu3fv4rnevWnj7l7q+TbubVCaKImMPAnAa1On0sCuAcuXLefsmTMolSaMDB5Fu/btUZooiY+PY93atcTHxxMQEEDvPr0J3RGqOacZM2diZGjIkiVLSE5OJjAwkN59+rArLIzt27eXqtuuYUNq167NkcOHNInA5cRENqxbj6e3l2a/8mIAsLaxYeSIkXh5e2JoaExcfBwrli/XJGeBgYH0CeqLg8qBW7dusiM0lE0bN2riBZh176e3tzcvDnsRR0dH7ty+Q1jYLtatWwsUd9O8MHQox44fw8fHF0dHB5KSk/l10y+Eh4c/9NpnZ2djbGxM+/btSiUIPr6+NG3WXKs7pbzzDQoKom9QECYmJvgH+NPIqRGzZs7ktalTsbC0JCszEx8fX/bvD+fo0aMMHz6cY8eOsXrVKgB6Pfccffv0wbaBLdevXWfbb78RumOHTtdRiMqQhZKEKMGlhQtm5uYAXL92rcof7ImXEykoyMfQ0BC7hg2rJbbs7Bz++COe+rb1CAgI0Dzv5upGVnYWJ09GAmBnZ4eDvQNKZfG32vc/+JDnnnuOxEuXOBUdRcuWLXnrrbdxcHTk5q2b2Nk1pE2bNgC0a98eV1dXXN3caNmyJQBt2rRBpVJx89YtrZiuXrlCZmYm7h7u+Pn7a56POBTB14sWaRKA8mIAmDJ5Ml39uhIbE8u+vXtxc23JhIkTisc2+PszbsJ4aitN2bJlC3fS0xn20jD69g0CwF6lwl6lAoq7O6a9N4065nU4duw4KampjAweyZtvvgmAuZkZTk5ODBjwPPn5+USejMRepWJsyFhcXFzKufbZJCcn0aaNB6p7dQG0b98OA0N9riSXHntQ3vmmpaaRnJzM3buF/PnnnyRdTtK8bt5eXrRq1YpT0VGcPXMGczMzHB0bUc/GBoARI0cwYcIEDI2MOHr0GCgUjBs3jv79+1d4HYWoLGlBEKIEy7p1qVWr+I9pyo0bZe4TGBjIpMlTUCpNCN2xg/nz52vtc/3adQoKClEqTahraVlt8UVFRtGlSxfcWrmxZ88e3D3cadykCadPneLsmTNa+wcEBNCsWTN2bN/Od999p3lu3PjxdO8eyLIflnHt2jVatGiBUmmCc3Nn7t4tHtjY3NmZnTt30rhJU65cvVKqW+O+5ORk/rtmNSODg5k2bRqTJ03mXMI5du0M07RI6BKDnZ0d586eY86cOQDExsZgYGBAdnYOjg7FScTGTZv4PTQUlUqFl5cXp06fKvN8c3NzmT9/viY5+eCDD/DxbYuPry8AevoGxEZF8fFHHwEwatS/6D+gP61atS63OyQqKpLez/WmU+dOrP15LSqVijZtPEi8lMjdu3cxNq6r8/kqTZW0adOauNg4Vq78UVNHbm4OS5cu5cD+/UDxe+0+pdKEDh06kpyczFtvvkF2dg4qlYrx48dj7+AAUO51FKKyJEEQ4hkSHh5O/wEDaOXWCqXSBC8vL2oZGxMTE1Pm/k2bNsPAwID6tra88+67ABgZGmKgr49tfVsAzvxxhsDugXh7+9C4sRN/3viTosIimjRtQrv27bGysuLAgf0PjWn37t3s3r0bH19fAgL8ad2qDa+/8QZdu3ZlzpzZOsVw9epVvH18WPzNEo4fO8bBiIOahCfxciIKtYKQkLH4eHtz5MgRwsJ2an3oqVQq6tWvT1JScqkP+rNnz+Hh4YlTIydu3iwexJiVmanZfuVKMkVFhRWOI4iJiaV9+474+Piy9ue1dOjQEUuLOoSF/Y6Hh2elrvnDFBTcJTMzo8xtrdu4U7du8Wtx/9yTk5N5//33NfuUdx2FqCzpYhCihFs3b5KbW/zH16ZevSqXY9vAFkNDfQoKCspsmn8UcXGxWFrVpVOnzrRs6UpaWhq7d5c9JsJEaYKhoSH2KhWNGzemcePGqOztuZGSQmZWFgB//FH8Ydq6TRscGzUi4XwCZ/44g0UdC1q5tSquU4fZCMeOHuWzuZ8xfPhLhIeH49aqFd26BeoUw9y5c1i3dh2o1fQf0J8FCxbw9jvvALBv717mzJnN6dMxtGzZkjfefJPvly7VtAjcZ2FpibGhITdv3iz1/J30O6gV6koPJHxQbm4O0dGRODZyJCAggNatW3Hr9h2tcSq6nG9VKE1MMDDUJz8v/6H7lHcdhagsaUEQooT4P+LJSE+nbt262DZogEqlqtI4BEcHRwwNjcjOzuZqNc+NjzgYgZ+fH507d8bOzo5DhyIe2oSclpZKfl4+27ZtY/PmzWXuEx4ezgsvDqN169bUqlWLs2fOkJubS9euXWnTpjUZ6Xc4fFi7ewEgKCiIgG7d+G3btlIDNy9euEjbtm2pY1FHpxgAftv+m6a5/cN//5t2bdvRpUsXwsPDuX79umYQYqfOnXnttSn07dOn1IDBmNOnuXX7Do0aOZUq934XRVLSZfT0Hu070aFDh+nQoRP+/v40adqUQ4citN4fup5vZSVfuUJOdjYODvalnvdt25b8/DyiIovXgSjvOgpRGdKCIEQJ2dk5XLhwAbVajY21DZ06d6p0GS4uLri0LB7wlpZ2k7i4uGqNMT4+nsRLiXh6eWJoaKD5YChLePh+0jPS6d2nj2YQ3tiQsfzy668EB/81PfBCwnkcHBzIyckhLi6O48ePkZ2dRdNmzbhw4cJDE5Drf/6JnV1Dhg17CW9vb6B4HYmePXuSm5tLTExMhTG4e7jzw7LlvPP2O1jb2GBtY0OdOubk5eeRnpHOW2+/zcKFCxkwcAAA9Wxs0Nc3KHMBqlPRUTRs2JCxIWMB6NC+A35+Xbl+7fpDk5zKOHb0KOfPncXH1xeFQo9Dhw5r7aPrNddT6GFmbqZz3WfPnOHMH2do2bKl5lq88OKLTJs2jS5dulR4HYWoLGlBEOIB4eHheHh6YmFhQc+evYiIOFSpaWKDBg3CxqYed+/e5eiRw9UyxfFBp06dxt3Dg8TEy+V+M7ycmMjqVasYGRzMlwsWUFRUhFqt5sTx42zcuF6z3+nTp+ncpQuXExM18V66dIn6trblJjjHjh5l/fp1DBkylE9nz0ahUKBWq7l58yb/XbNGk7yUF0N2dg7bt29n0KBBrFz5I3p6+uTm5rJp0yaiIqO4efMW9vb2jBs3nrFjQzAwMODMmTNs2Ki9PsCqVT9hYWnJgAHPM2jQYBQKBYmJiXz//XfVNlDvyJEjtGrdmvPnzpa55oQu1zwi4iDdAgMJCgqibVtfPv74E53q/mnVKiwsLBg3bjyvvjoOhUJBzOnT/O9/m7mcmFjudRSishSNnJqqazoIISpDrVaX/omawsIirK2sqq2ON998k+49emiWWp73+TxSU1IqPG5syFgGDHgeIyNDEi4k8OnHnzyWBKEq3D3csbCwJCYmRqdzqUr51lbWXE5KeujAuPJiUCpN8PTyJj8/n6NHjmgd29zZGVXDhpxPSKgwYbO2scHNzY2rV6/W6CC9x3XNHRwdady4MRcuXNC6FhVdRyEq0qGrP6amppIgiGfPk0gQVCoVU1+fSqtWrQFIuny5zJs13VfyZk2Ghoakpqby7bffys2ahBDPnPsJgtyLQfwtVPfNmtLT0/njzBlaOLfAytoKCwsLOnfpioeHB0pTpeZmRJ06d2bYS8N49dVXcG7RAn19fW7fvs2KFSvYs3t3tcUjhBBPyv17MUgLgnjmPIkWhPuUShNGjx5DYPfumJqalrtvYWEh0aeiWbnix8d6/wEhhHicpItBPLOeZIJwn4OjI/3798Pd3ZN69WyKs2uFgoKCfDIzs4mLi2VXWBgHDx58bDEIIcSTIAmCeGbVRIIghBD/FPcTBFkHQQghhBBaJEEQQgghhBZJEIQQQgihRRIEIYQQQmiRBEEIIYQQWiRBEEIIIYQWSRCEEEIIoUUSBCGEEEJokQRBCCGEEFokQRBCCCGEFkkQhBBCCKFFEgQhhBBCaJEEQQghhBBaJEEQQgghhBZJEIQQQgihRRIEIYQQQmiRBEEIIYQQWiRBEEIIIYQWSRCEEEIIoUUSBCGEEEJokQRBCCGEEFokQRBCCCGEFkkQhBBCCKFFEgQhhBBCaJEEQQghhBBaJEEQQgghhBZJEIQQQgihRRIEIYQQQmj8ee0qdwvvomjk1FRd08EIURlqtbr0T9QUFhZhbWVVbXW8NnUq3bv3wMjI8KH7hO7YwalTp5g0eQpKpclD95k/f361xSWEEI9baloa+vp60oIgxIO6dOlCl85dyk0OhBDi786gpgMQ4mljZGSEvkHxfw1dWgB27doFQLdu3RgbEoJVNbZkCCFETZEEQYhqUDI5KCoqQqFQoFAoqrUOpdIETy9vMjMziIqMKrXN2saGli4uXEpM5HJiYrXWWx2aOztTr149Tp44TnZ2Tk2Hg4OjI95eXjSwsyM2NpajRw7XSFy+bdsCcPTIkUcqp6Lzqa56KoqhkaMjcfHxpKaklLuvu4c7tWubab0fynuPV7T9/rYbN25w9syZ6jmpMjyp9/KTeM0qIgmCEI+oZHKQn1/A4cOH8Pb2eei4hKpq7uzMhAkTSEpK0vrj6N6mDePGj2Prlm2sXPljtdZbHfr26YO3jw/z5n1W5h/+iqhUKoKC+pGQcJ6dO3c+Uiz3x5fo6+tRUFBAv379uHXrNitWLOf30NBHKruyXgkJAR7tQ0CX86mOeiri7+dPUL++fLvkW02rWlmUShMmTphIA7uGrF79E2t/XqvZVt57vOR2AwMDvlqwgIMHD2ptO37sGPN1TBB69OhBkyZN2bp1C8nJyTod86jvZV09idesIjIGQYhH8GByEBq6g8OHDtV0WH871jbWdO7SGTc3t0cq59VXX6V79x4cO3aUkSNH0i8oiFkzZpKTm8Oo4GB8fH2rKeInY/ToMXTr1q3U+bw/fTpZmZkEjwzG3cO9pkPU0q1bIFZWVhQW3sXDw7NKZVhYWDDspZceOQl3c3Ojc5fOWNtYP1I5f1fSgiBEFZWVHHy9aBGBgYE1HRqvTZ2KhYUFV69cpatfF/QUekRGRbFixYpSzb8vvPAi3bsHUseiDvFx8Vy6lEjzFs1Zvmw5zs2b4+fnR2xsHB07dSInO4tJkyahVJowMngUvj6+mNcxIykpmf/9+ivh4eGacgcPGUy3wECsrayIj4snIzOzVHz3y2jXvj1KEyXx8XGsW7uW+Ph4rXPxbduW4JEjMTc3x8vLm7nzPmP5suWcPXMGb29vXhz2Io6Ojty5fYewsF2sW7dWqwwo/obZqXMnzp07y+fzPtM0D0ccisDK2oqXx4bQpXNnjh09ClBh2bqcQ6/nnqNvnz7YNrAlMfEyx44dxcfbh3Xr15f5zdDaxobRo0fj4V78wV7Wa1byfLr6dSUhIaHU+Rw/fpw1a1YzYdJEunTpUua33IrqmTFzJgCz7v1s7uzMmJfHEHM6htWrVgEQGBhIn6C+2KtUJCZe5ty5c2Ve9we5ubmRlZ3F+fMJuLi0xMfXV3PNdZWUlIxKZc/48RP44osvHrpfea/ha1On4uXljbm5ORMnTOT/wsM151bydbt+7TrbfvuN0B07SpXdprU7kye/Ru3atSt87csqozKvNcCgwYPp2qUrR44eZs3qNXh7ezNo8GCaNWtKRmYWEQcOsnTp95W6jhWRBEGIKnhYcgDFgxbLa2J9Euzs7HB2dqZ5s+bExcViaGREx44dsbK24qOZM8nOzmH06DEMfH4gFy5e4Oy5czRp0oQWLi7cvXsXpdKEunWtaNa8OQ6ODpw9c5aEhAsATJv2Hl7eXkRFRnEy8iTePj68+uo40jPSiYqMKlXusWPHi8u1tOTu3bua+N7/4ENcXV2JioqiID+fNu7uvPXW28z6aJbWGIqM9HQSL1/Gzs6O1LRUEi8mkp2VRc9evQgJGcutm7c4duw4lpaWjAweiUrVsMwPDScnJ5QmpkRH79HqO966dStbt27VPNal7JLnkJubi6trSz744EM+/888oiKj6Ns3iDFjRpOWlsbRo8ewa2jHoEGD0NfTx9zMTCs+lUrF9Pffx8bGmqjIKAyNjGjfvj22trZ8+MH7WjE7OTlhVtuMffv2aW3bs2cPe/bsKfO98WA9AG3b+tK4SWM+/fgTkpOTsVepSh2jVJrgYO/Ajet/ApR5bt26BaCnKL9RurmzMy1dW5J4KZFDERG0at2K1q1bVTpBSE1N4eKFBDp27MiZM2fZtm2r1j4VvYbXrl4jNS0VE5NaJCYmcuPP4nMbMXIEQ4YM5eKlixw9eowmTZowbtw4jI2M2Lx5MwC1a9emd58+xMfHUZCfj4enB7ZTpzJ79mwuJyZqlWHX0I4JEyZgY23DqlU/6fQalDRg4ACGvTSMY0eP8esvv+Di4sKU16agUCjY+XsYDRra0advH+7evcuKFcsrdS3LIwmCEJVUXnLwNNHT02PDxvX879f/AcXN6z179qRbt0AiIyPp1LkzCQkJvD/9PbKzc1CpVMz66GNMTGppylAoFOzds49vvlmseW7fvn1EnzrFpo0bAejfvz/BwaNwc3MjNSVVq1wHR0dmzZqFsbExAAEBATRr1owd27fz3XffaZ4bN3483bsHsuyHZaXOIz4+HuNaxnh4eJB46RJLlnwDwOQpU8jNzWX+/Pmab24ffPABPr5ty/xWalvfFjVFJCclVXjtAgICyi3b0sICFxcX9uzZzcKvFgLQyehxEwAAF/pJREFUsWNHprw2led6PUdUZBT+Af7cSU/n408+0SQ9X8yfT5MmTcuss1PnTtSrZ8Pq1as1r9mLw15k8ODBdOsWWCqBqez5lNStWyANGjRg3bq1mv7/AQMHEDwymJ69empd/7I8eG5KpQmfzp6Dk1Pjco/z9PTAVGnKqVOnOXBgPwMGDsTT05tlVFzng35atQpHR0f6BvXl1OlTWtsreg3Xr1+HStUQGxsbtv22jajIKJRKEzp06MiFCxc071+l0oT/fDGfLl27ahIEAwMD9uzepXn/3r9+3bsH8vN//0vXLl25cuUK06dN05Tx6ew5+Pn7s3fvnkq9BgMGDmDEyJFER0WzaOFXZGfn0LBhQ5Smpmzbuo0fV6zQ7BcXp90C9ygkQRCiEnRJDgIDAzWLJ9XkQkmZGRmcP5+geRx96hT+Ad1wbNSIlNRUTE1NiYw8qfn2mZyczIWE87iW6OcvLCwiKyurVLl79uzB3cOd8eMnYGfXAJWDA8a1jNHXM0Blb69V7uXERM6fO6cpt2nTZhgYGFDf1pZ33n0XACNDQ/6/vXsPq6pM9Dj+3Vw27e0NQjRhIwpoB/MCgjmKpRJjklaolZa36Rktm5ku05znqDhzfJ6ZylMz2W3OmVPZ5ETjUSdzKptSqeaklE4pKBwURn1EQVREQWFz3/v8sWUFLMCNeQn7ff5R1u1919oL9m+9613v8vP15Ya+N3i1bw6Hgz59+3L0aFGLZt2Cgn8SFzeSgQMGtn1VarEYj7ACPPnkk0xOSTF+zsrK4pWXX77gtgMDA3G5XeTm5BrzMzMzeWD2HBzh/XE4HFwfHMyRw4dbtIjk789n4IABbe5TVGQUADH/EsPgxTcCYLPb8fe30q9faNsHotX+eMPhCKPa6SQ39/+MaXl5+6hyVtE/vL8X65v3zemsZl/ePiL6d7x+XNxIKs6d44svMnE6q9m/fx+3jh9PUlJSuy0e7TlSWMh7773Pgw8+yP3338/mzd90ML3Y82P4iFiuvz6Y4uIifvboY8b0AGsANrsdx/mWlWpndYvtNj9+w0fE0iswkMzMTON3wOms5uDBg0yYMJ7IyEivP4MePXsxb958Dh06yPO/+62xveLiYpxVVaSmTiMqMpKvd+1i+/bMCz490lkKCCJe+i63HHTr1s3UvFtbX0/5mTPGzzU11bhcjVj9/bHbbPj5+1JXW9dinbr6+g7Lsdtt/OrflxMbG8vJkycpKysjf/9+eiaM8sz3Yrs2uw1/f3/CHQ4aXS5j+snSUipbhZH2BAYFEeDvz+nTp1tMrzhbgdvixmYzd14rKTmGxW0hon+EMW3lypWsXLkSu93GK6/8p9fbttlt1NfVU1Fe3mIZZ1UVQUGBxjbq6lofhzpctD14bfcePfDz8yeiVYAoKSnBWe00LX/06BGAFvvTpKPHAbv36EF1dQ2HDh4wphUdPUJdbT3XtXHcWruYfQMYdfPNDBgwkKCgQP54/qoXPCOixsXFdTogAGza9AExQ2IYO3YsxcXFpjp25vwAz/lr9fcjODiYbt26GdPrG+opLyvFfn5afUM9Vc361TQ/fnabDYsFKioqWmy7srISLBasVqvXn0H37t0oKztF5MBIkpKSjVsp+/btY8UzK7jzrruIGTKEkfEjmTd3Hm+++Wabt1sulgKCiBduHT+eHy9YcFXDQXZWNufOnaN37xAcDkeL+5QRAyLAYqGk5JgxrZvdjiM83Fguon8EAdYATpaepKi4mGqnk/79w43l7XYbAwcO7LAOCQmjGDQomo8++pvRtH733Xdz8yhPQDhZWkpdTS1hYS2veEOb/VxWdoq62jo2bdpkNNl2Vm5ODmfKKxgwoGV9m74sm748m8vLy6O8ooJRoxL4ePPHLa7sJ91+O8G9g8nKzvJq29XV1Vj9rURFD2LX7t2A56q1d58QjpeUGNsIcziw223GlV90VBR+vm2P0Hn8+HGioqJYvXq1V0/CFBQUUHaqrN39mTd3Hh/+7UNTQCguLmbQoGgSEkYZHUuHj4ilR88eRlP9uapKenTrbqzja/HBx8cTQHNzcjhXWUloWMt+CmGhoe3uG8Dw4cPw9/NlzZo1LW6LpE6bxpCbhhpX5531P2vWEBkVydSpU/Hz8zXq2NnzAzznb01NDf8sKODpp59uu8ApU7Cf/91q+uybH7+i4mJqampMLUVhoaHU1dRy/MQJrz4DgJJjx/j1b35DWloaDzxwPydOnjBaPkpPneL3r7xs3Mb79a9/Tcodd1zSgKDHHEW8kJeXR9HRIurq6q5qy8HuXbvo27cvP/npT+kfEYHdbmPO3LlMnJjEsWPH2L59m7Fsr169mD5tOr1DQoiJieGOKVOoclay6+tdFOTnk52VzfARI1i8ZAnJycksXryEsLCwDss/e+4sdXX19OoV6Om4FhHBrePHE3Cdp39Bbk4OBw4cYNiwYaROSwVgwcIFRDe77/7559s4e+4sd0yZQkxMjLHMuxs3Mm/e/A7L79Gzp/Fo29492YSFhbFg4QIAxo4Zy4QJ4zlecpwdO8xfsEVFRWzZspkb+vVj6dIlJCUlYbfbmDlzFrNmPUBVZZVx/C607R07d3D23FluS76NmJgY7HYbs+fMISgwiOxszxfyji+/xOFwsGzZL7l98mR+/vMnuWnoUFyuxjb37euvvsLtxvjM7HYbi5cs4Z13NzBp0qQ29+eD9z8gODiYpUuXkJiY6KnH7NnMmTOHk6WlbN1q7iybtXs3jY1upk69k94hIfQOCWH6tOn4+PqwZ88eAE4eP0HvkBDmzJ3D8BHDmT13LoGBgcY2cnJyCA93GMcndVoqI0fGQwctCCNHJnCmvIKMrVuNjrwZGRlkZe0mKLAXcXFx7a7bkaKiItatXYe/nz89e/Yypnt7flit/gQGBgGe8zcvL4/4+Hjj/B07Zixr1qxhxbPPNlvHSsoddxATE2M6fgX5+eTvz2/xO5A6LZX4+HgOHDhAbk6OV59BkyOFhWzcuJGAgOuYcc8M7HYb8+f/iFWrVvHA7NkAOELDuO6666isPHdRx7A9akEQ8cKp0lKe++1zRDS7argaXn31VSw+Ptx+++2sWrUKAJfLRX5+Pv/9hz+06M1eXl5BcHAw6enpWCwWyssrePPNPxr3Tv/wh/+iobGRW24Zx5gxYykqOsrOHTsYOmxYu+VnZ2Wzc+dOfvjDSfzlnQ34+Phw8OBBqmtqjGVWvfEGv/jFL1i06BEWLnyImuoa9u7Zw8BITwe2I4WFvJ2eztx583jhxRdxuVy43W52ff0177yzvt1ys7KySEpKIj39z7z00oukp79FYFAQqanTmTHjHiwWC4WFhbz22qvtjnC3bu1aGhoauPfe+1i8ZAkWiwWXy8XBg4d4849vGFfbF9p2630AqK2tZcvmzaxbu9bYBsCUqVO4aegwysrK+Px/Pydx3Ng265aZmckN/foxc+ZM3n77bdxuNw0NDWRkZLQ7ONSmTR8QcF0A9957H/++fLmxPwcOHOC1115tc1TN1uUAVFZVsn7tev7+2WcAZHzyCdGDBjN37jwAcnNzOdPsdtUf31hF3z59mDHjHqZNm47L5SI7K4uYITFt1jMpKYkb+vbliy8yTT309+7NYdKkyYwePZqjRZ3rcNnk7599xo2DB3N3aqoxzZvzY9v27YyMH0laWhoTJ0xg+fLlvJWeTveePXj44UUsXPgQvr6+FBcV8+6GDca2z5w5Q1VVFStfeAGAmpoaNmzYYBy/5tt4+OFFWCwWcnNyWPXGG15/Bs1t/vhjQvuFMm36NB597HFWvf4a0dFRTJ8+g9TUafj5+VFcfIx3znccvlT0Nkfpci732xy/bSfDK9VJ8ebRo7Hb7eTm5po6Jz373HOEh4fz3HOeqx6rNcD03L3dbsPerXuLdZctW8ZNN93EMytWkJuT027ZvUNCGDp0KMeOHWt3WNvBN95IaGhom/VrEhsXS2BgUIfLXIg3dbnY+nmz7faGDQZMt4IeeuhhJk2exH/9/j87vOd+8+jRWK3WTg3n683+tFUOXPxoff0jIoiMjOTQoUPfySG+4eLPj/4REURHRVFUXNzhOd7RsMveHJ9v8xl4U8eL0fQ2R7UgiHRR3v5BaW842LS0ZURFR7Pmz3/mk08yuGPKFOJGxlFYeKTDcACeFpW2rnSaK8jPv+AfrUsxVK03dWmLN/XzZtvt7cPjjz/OrbeOZ+NfN/LuhndITBzHuFsSqSg/S0FBQYfbvJgvC2/251KU09yR7+i7P5q72PPDm3270DH3Zhvf5jO43MdfAUGklbq6OhrPD+ozOSWlxWNwneFyuS74VMDlUltTQ3V1DQ0Nbd/vBlj/l/X8+McLWPTIIzz62GPU1dWTn7+fVa+/fgVreu3a+Ne/EhISwsyZM5k3bx6NjY0cLjxM+uq3vB73X+Rq0i0G6XIu9y0G+OYFOFZr+72yL1TH/fv38/zzz3/nr7BERJprusWggCBdzpUICCIi31dNAUGPOYqIiIiJAoKIiIiYKCCIiIiIiQKCiIiImCggiIiIiIkCgoiIiJgoIIiIiIiJAoKIiIiYKCCIiIiIiQKCiIiImCggiIiIiIkCgoiIiJgoIIiIiIiJAoKIiIiYKCCIiIiIiQKCiIiImCggiIiIiIkCgoiIiJgoIIiIiIiJAoKIiIiYKCCIiIiIiQKCiIiImCggiIiIiIkCgoiIiJgoIIiIiIiJAoKIiIiYKCCIiIiIiQKCiIiImCggiIiIiInf1a6ASFeRnJzMzx59DLvdxscffcTevXtb/Lxy5cqrXUURkUtGLQgiXVDvkBAmTJzI4BtvvOJl94+IYMLEifSPiLjiZYvIlWMZMDDafbUrIdIZbre75b+4aWx00Ts4+JKX9fgTT5CcnExAQIBXy7tcLvbn72fl8ys5Ulh4yevTZP78H3HfzPsoLCzkX3/xJE5n9WUrq62yp02fzsZ33+VPf1p92cqZNGkSUVHRfPDB+xQVFV22ckSkpVNlZfj6+qgFQaQ9PxgzhsTERK/DAYCPjw83Dr6RybdPvow1g+EjhtPQ0ECfvn0YN+6Wy1rW1TJ06FBuufUWeof0vtpVEfleUh8EkXZ079YNq9UTDrzpY9C8j0L37t0uW71G3XwzoaGh7PjyS0bExhEfH8+WLVtaLJOQkMCs+2cRERFBRXkFW7dmsG7d2ks2v7XJKSlMnTKFG/rdwPGS42z68EM+/ugjwNMKExgURFVlJaNG3cy2bZ/z+1deISEhgZkzZ9E/IgJntZPs3bt5/fXXcDqrefyJJ4iPT6Bnz5789Cc/5X8//5y309Ox223MnTefH4wZg91mZ9++PNatXcu+ffs6LOtC5s//EQkJCfz2d7+9rC0/Il2JWhBEupjRo0fjb/Vn586dFBYeJnrQYBwOhzH/9smTWbJ0CSEhfdi1azdnzpQze85sHnro4U7N79WzF1999TXnqipbzG9tztw5PPLIIzS4GvnHP77C32pl0aJF3H333QCEhoaSEB/PsGHD2Lsnm4L8fCZMnMjSpWnc0K8vX375BYcPH+a25GQeeeQnAJQcK+FU2SnqamspLCzk5IkTACz75a9ISUmh8PBhsrJ2ExUVyS9/+Sti42LbLctbkVGRpKWlqW+FyHlqQRBppXlLgDfTAeOphiaTU1KYnJJyyZ9usNttDBkyhBPHT/Lpp5/Sp28f7rv3PsaOTWT9+nUA3DJuHDU1NfzHihXGlfXSpUvp27eP1/PPnavkN089ZVxNP/XUU4z+wQ/4298+NNVn7NhEDh06xLK0pTid1djtNn73/EpuHT+e9957D4Cammpef/11tm/bBng6Ov7lnfXk5eWxd4/nuL3w0otGp8v169fhcIQREhLCpg83kZ2VzaRJk4iJieHTTz/h5ZdeBiAxMZHHHn+ClMkpZGdlt1lWZ0RGekLCM888o5YE+d5TQBDpQsaNu4W+ffqyefNmAPZk72Hq1KkMHz7s/Jeqg7DwcA4eOGh8+QOsWLECwOv5NTXVzJo1y5jfo2dPevTojiM8vEV9ho+I5frrgykuLuJnjz5mTA+wBmCz242Wjfr6BiorzxnzjxQW4nQ6SUxMZPLkFBxhDhyOME6fPt3uvg8YMBCX20VuTq4xLTMzkwdmz8ER3t+Y1rqszlJIEPFQQBBpJSMjg4yMjA5bDC7kco2LEB8fT/ce3bnn3nu45957jOk+Pr7ExsXS0NBIgL8/dXV1ba4fGBTk1Xyrnz+RkZEt5pWUlOByuVpMs9tsWP39CA4Oplu3b/pd1DfUU15Wir1b230xpk69kwcffBAfXx9OnDhBcVERvn5+BFj92913m91GfV09FeXlLaY7q6oICgpsd72LERkZybJly3jxhRdaBCmR7xMFBJEuwuFwED1oMPn783n//feM6eHh/bnr7ruIj4/njVVvcKa8gjCHA7vdZjz+GBsXi9UawD927rzg/HOVlTirnfz88SfarEfMv8QY/z9ZWkpNTQ3/LCjg6aef9npfRo1KoLrGydNPPW18Ab/w0osdBoSSYyVY/a1ERQ9i1+7dxjHp3SeE4yUlXpftLbfbTZXTecm3K9JVqJOiSCvJycn89b33+bfFiy+q9QA8fRC2bN3Kk08+ecnqNXZsIsHB17Nnb7bRypGRkcHWrVs4ffoMQ4bcBMDePdmEhobxwOzZANx2220sXbqMu+6806v5u3ftYuCASBYsXABATEwMr772Gv/96qv0DglpUafcnBzy8vKIj48ndVqqp55jxrJmzRpWPPtsu/ty9uxZrNYAQvp4+j3cd99MBkQMMC1ntfoTGBgEwI6dOzh77iy3Jd9GTEwMdruN2XPmEBQYRHZ29sUd1HYcOnRItxjke08tCCJdxPDhw3A6nezatavF9KKiIg78s4CEhFEkJSWRnv4WIX36MG3adGbMuAeLxUJhYSHr/7IewKv5gUFBpKZOZ/r0Gfj4+HDmzBlWr17NqdJSU73eSk+ne88ePPzwIhYufAhfX1+Ki4p5d8OGdvfl823bGHLTUNLS0li8+N+oqa7lxIkT+Pn6Gsts276dkfEjSUtLY+KECSxfvpy309OZO28eL7z4IgC1tbVs2byZdWvbfwSzsxQORDw0kqJ0OVdqJMXW717ozDgI34V3M/QOCWHo0KEcO3aszcf9vJ1fXn7GeEKgI/0jIoiOiqKouNjrxwtj42Lp3r0Hu3d93anRIC92vfY0jUx59OhRhQP53msaSVEtCCLXqFOlpfz9s88u2/zWjhQWdvqL1ZvgcSnX68ihg4c0UJJIMwoIIu2orKqirq4Wu91mjGvgjcbGRiorqy5z7eRS+tOfVl/W90qIdEXqpCjSjh1ffklmZia1tbVer+NyucgvyOfjzR9fxpqJiFx+6oMgXc6VfJtjc637GOzdu/c71edARORS0NscRUREpF1qQZAu52q1IIiIfB+oBUFERETapYAgIiIiJgoIIiIiYqKAICIiIiYKCCIiImKigCAiIiImCggiIiJiooAgIiIiJgoIIiIiYqKAICIiIiYKCCIiImKigCAiIiImCggiIiJiooAgIiIiJgoIIiIiYqKAICIiIiYKCNJlWSxXuwYiItcuBQTp+tzuq10DEZFri9utgCDXCgtuBQURkW/F83fU0zyrgCDXAAsW3NTV1V/tioiIdGl1dfVY8IQEBQTpwpp1QrBYqHI61YogInKR3G43VU6n0cFLAUG6HEur3okWi2eay9XI6fJyamtrFRRERLzkdrupra3ldHk5LlcjFosFiwX8rnbFRL4NiwXcbs8tBh8fC65GFxVnz+J2u3GjkCAiciEWLFgsFnwsPvj4Ws63zVoUEKSrswBuT4uY2+I5ud16sEFEpDM8LbE+WPjmEXIFBOmSLJZvnlrw/B8sFje4weLjuXOmkCAicmHGXVt30/89LQoKCHJN+CYkNE1xt17gSldJROS7q9kVVNMthaZ+3039vBQQpMtq3orQ9DO4z5/3FmUCEZH2tPEHsqn1oIkCgnRprUNCy2DQshVBtxxERNrKBm1fTSkgSJfX1BxmfrTR/DikiIh0TLcY5JrTfHwEjYMgIuK91uPLgAKCXKPaOtlFRMR7GklRRERETBQQRERExEQBQUREREwUEERERMREAUFERERMFBBERETERAFBRERETBQQRERExEQBQUREREwUEERERMREAUFERERM/h/TGqWkGqOGhQAAAABJRU5ErkJggg=="}}},{"cell_type":"code","source":"# !pip install tensorflow==2.9.1","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:31:42.03066Z","iopub.execute_input":"2022-08-03T02:31:42.03176Z","iopub.status.idle":"2022-08-03T02:31:42.040166Z","shell.execute_reply.started":"2022-08-03T02:31:42.03163Z","shell.execute_reply":"2022-08-03T02:31:42.038392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !pip install Pillow==9.2.0","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:31:42.043388Z","iopub.execute_input":"2022-08-03T02:31:42.04446Z","iopub.status.idle":"2022-08-03T02:31:42.059457Z","shell.execute_reply.started":"2022-08-03T02:31:42.044408Z","shell.execute_reply":"2022-08-03T02:31:42.058202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Imports\nIn this section, we are going to import all the relevant libraries.","metadata":{}},{"cell_type":"code","source":"# Python Imports\nimport os\nimport gc\nimport csv\nimport string\nimport shutil\nfrom glob import glob\nfrom pathlib import Path\nfrom tqdm.auto import tqdm\n\n# Computing imports\nimport pandas as pd\nimport numpy as np\nimport PIL\n\n# ML imports\nimport tensorflow as tf\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import Model\nfrom tensorflow.keras.optimizers import RMSprop, Adam \nfrom tensorflow.keras.preprocessing.image import img_to_array, load_img\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator, array_to_img\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint\n\n# Plotting imports\nimport matplotlib.pyplot as plt\n\n# I like to disable my Notebook Warnings.\nimport warnings\nfrom warnings import simplefilter\nwarnings.filterwarnings(\"ignore\")\n\n# Allows us to see more information regarding the DataFrame\npd.set_option(\"display.max_rows\", None)\npd.set_option(\"display.max_columns\", None) ","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:31:42.062436Z","iopub.execute_input":"2022-08-03T02:31:42.063193Z","iopub.status.idle":"2022-08-03T02:31:46.826289Z","shell.execute_reply.started":"2022-08-03T02:31:42.063157Z","shell.execute_reply":"2022-08-03T02:31:46.825212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check if the notebook has been updated with the current versions of the libraries.","metadata":{}},{"cell_type":"code","source":"tf.__version__, PIL.__version__","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:31:46.827512Z","iopub.execute_input":"2022-08-03T02:31:46.829153Z","iopub.status.idle":"2022-08-03T02:31:46.841058Z","shell.execute_reply.started":"2022-08-03T02:31:46.829118Z","shell.execute_reply":"2022-08-03T02:31:46.839399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration\nIn this section, we set all the constants and variables that we will re-use throughout the notebook. It makes our lives easier to have all the configurable variables at one place. Note that the image size that we will feed to the model is (300, 300). This is not chosen randomnly. You can find the recommended image size for the model you want to use for transfer learning. In our case, Efficientnetv2B3 recommends a 300x300 input image.","metadata":{}},{"cell_type":"code","source":"IMG_SIZE = (300, 300) # https://tfhub.dev/google/collections/efficientnet_v2/1#:~:text=classification%0Afeature_vector-,Efficientnetv2%20B3,-300\nIMG_CHANNELS = 3      # RGB Channels\nBATCH_SIZE = 64       # You can play with this hyperparameter. I kept it at 64 batch size so that the notebook runs quickly\nEPOCHS = 5            # You can play with this hyperparameter. I kept it at 5 Epochs so that the notebook runs quickly\nLR = 0.0001           # We want learnnig rate to be small since we are only fine tuning the prediction head of the Neural Network","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:31:46.847172Z","iopub.execute_input":"2022-08-03T02:31:46.84785Z","iopub.status.idle":"2022-08-03T02:31:46.861937Z","shell.execute_reply.started":"2022-08-03T02:31:46.847785Z","shell.execute_reply":"2022-08-03T02:31:46.860428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Helper Functions\nThis functions will help us to preprocess our images. They have been taken from this amazing notebook: https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-baseline-flash-effnet-aug?scriptVersionId=101586230","metadata":{}},{"cell_type":"code","source":"from PIL import Image\nImage.MAX_IMAGE_PIXELS = 25_000_000_000  # you have to set this value to allow displaying high-resolution images\n\ndef prune_image_rows_cols(im, mask, thr=0.990):\n    \"\"\"\n    This function removes empty rows and columns. These rows and columns\n    do not provide valuable information to the model and increases the \n    input tensor considerably.\n    \"\"\"\n    # delete empty columns\n    for l in reversed(range(im.shape[1])):\n        if (np.sum(mask[:, l]) / float(mask.shape[0])) > thr:\n            im = np.delete(im, l, 1)\n    # delete empty rows\n    for l in reversed(range(im.shape[0])):\n        if (np.sum(mask[l, :]) / float(mask.shape[1])) > thr:\n            im = np.delete(im, l, 0)\n    return im\n\n\ndef mask_median(im, val=255):\n    \"\"\"\n    Helps to remove background colors\n    \"\"\"\n    masks = [None] * 3\n    for c in range(3):\n        masks[c] = im[..., c] >= np.median(im[:, :, c]) - 5\n    mask = np.logical_and(*masks)\n    im[mask, :] = val\n    return im, mask\n\n\ndef image_load_scale_norm(img_path, prune_thr=0.990, bg_val=255):\n    \"\"\"\n    Function that normalized our images\n    \"\"\"\n    img = Image.open(img_path)\n    if (img.width * img.height) > 1_500_000_000:  # todo: for train images it was fine 4_000_000_000\n        print(img.width, img.height)\n        return None\n    scale = min(img.height / 2e3, img.width / 2e3)\n    tmp_size = int(img.width / scale), int(img.height / scale)\n    img.thumbnail(tmp_size, resample=Image.Resampling.BILINEAR, reducing_gap=scale)\n    im, mask = mask_median(np.array(img), val=bg_val)\n    im = prune_image_rows_cols(im, mask, thr=prune_thr)\n    img = Image.fromarray(im)\n    scale = min(img.height / 1e3, img.width / 1e3)\n    if scale > 1:\n        img = img.resize((int(img.width / scale), int(img.height / scale)), Image.LANCZOS)\n    return img","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:31:46.863259Z","iopub.execute_input":"2022-08-03T02:31:46.864326Z","iopub.status.idle":"2022-08-03T02:31:46.879421Z","shell.execute_reply.started":"2022-08-03T02:31:46.86429Z","shell.execute_reply":"2022-08-03T02:31:46.878144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Importing Data\n1. Since data is in form of csv file we have to use pandas read_csv to load the data\n2. Let's first define the paths that we will need to import our data.\n3. After loading it is important to check the complete information of data. It is important to get a general feel of the data that we are going to be using.","metadata":{}},{"cell_type":"code","source":"MAYO_CLINIC_FOLDER = Path(\"../input/mayo-clinic-strip-ai/\")\nSTROKE_BLOOD_FOLDER = Path(\"../input/stroke-blood-clot-origin-1k-scale-bg-crop\")\n\nORIGINAL_TRAIN_FOLDER = MAYO_CLINIC_FOLDER / \"train\"\nTEST_FOLDER = MAYO_CLINIC_FOLDER / \"test\"\nOTHER_FOLDER = MAYO_CLINIC_FOLDER / \"other\"\n\nNORMALIZED_TRAIN_FOLDER = STROKE_BLOOD_FOLDER / \"train_images\" # taken from https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-eda-load-crop-images/data\nNORMALIZED_TEST_FOLDER =  Path(\"./norm-test-images/normalized_test_images\") \n\nTRAIN_DATA_PATH = MAYO_CLINIC_FOLDER / \"train.csv\"\nTEST_DATA_PATH = MAYO_CLINIC_FOLDER / \"test.csv\"\nOTHER_DATA_PATH = MAYO_CLINIC_FOLDER / \"other.csv\"\nSAMPLE_SUBMISSION_PATH = MAYO_CLINIC_FOLDER / \"sample_submission.csv\"\n\ntrain_df = pd.read_csv(TRAIN_DATA_PATH)\ntest_df = pd.read_csv(TEST_DATA_PATH)\nsample_submission_df = pd.read_csv(SAMPLE_SUBMISSION_PATH)\n\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:31:46.881164Z","iopub.execute_input":"2022-08-03T02:31:46.881951Z","iopub.status.idle":"2022-08-03T02:31:46.948318Z","shell.execute_reply.started":"2022-08-03T02:31:46.881911Z","shell.execute_reply":"2022-08-03T02:31:46.947229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After working on this competition, I realized that these two images cause issues for the training of the model. I chose to remove them from the training data.","metadata":{}},{"cell_type":"code","source":"print(f\"Before drop: {train_df.shape}\")\ntrain_df = train_df[train_df[\"image_id\"] != \"b894f4_0\"] \ntrain_df = train_df[train_df[\"image_id\"] != \"6baf51_0\"]\nprint(f\"After drop: {train_df.shape}\")","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:31:46.949964Z","iopub.execute_input":"2022-08-03T02:31:46.950621Z","iopub.status.idle":"2022-08-03T02:31:46.965708Z","shell.execute_reply.started":"2022-08-03T02:31:46.950586Z","shell.execute_reply":"2022-08-03T02:31:46.964336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the next cell, we create a new folder that will hold our test images in PNG format. These images will be used to create the submission file for the competition.","metadata":{}},{"cell_type":"code","source":"ls_imgs_tif = glob(os.path.join(TEST_FOLDER, \"*.tif\"))\nnames = [os.path.splitext(os.path.basename(p))[0] for p in ls_imgs_tif]\npatient_ids = set([n.split(\"_\")[0] for n in names])\nprint(patient_ids)\n\nif NORMALIZED_TEST_FOLDER.exists():\n    shutil.rmtree(NORMALIZED_TEST_FOLDER)\n    \nNORMALIZED_TEST_FOLDER.mkdir(parents=True, exist_ok=True)\n\nfor img_path in tqdm(ls_imgs_tif):\n    name, _ = os.path.splitext(os.path.basename(img_path))\n    img = image_load_scale_norm(img_path)\n    if not img:\n        print(f\"missing: {name}\")\n        continue\n    img.save(NORMALIZED_TEST_FOLDER / f\"{name}.png\")\n    del img\n    gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:31:46.968608Z","iopub.execute_input":"2022-08-03T02:31:46.970405Z","iopub.status.idle":"2022-08-03T02:34:33.366435Z","shell.execute_reply.started":"2022-08-03T02:31:46.97026Z","shell.execute_reply":"2022-08-03T02:34:33.365065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA/Visualizations\nThe goal is to try and gain insights from the data prior to modeling.","metadata":{}},{"cell_type":"markdown","source":"## Explorating the Dataframe\nIt is useful to use .info() method to quickly have a glance on the general information about the DataFrame. It displays info such as the type of the columnd and also the # of non-null count.","metadata":{}},{"cell_type":"code","source":"train_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:34:33.367718Z","iopub.execute_input":"2022-08-03T02:34:33.368097Z","iopub.status.idle":"2022-08-03T02:34:33.39464Z","shell.execute_reply.started":"2022-08-03T02:34:33.368063Z","shell.execute_reply":"2022-08-03T02:34:33.392824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"let's also see how many images we are working with for training and testing.","metadata":{}},{"cell_type":"code","source":"train_images = glob(str(NORMALIZED_TRAIN_FOLDER / \"*\"))\ntest_images = glob(str(NORMALIZED_TEST_FOLDER / \"*\"))\n\nprint(f\"Number of images in a training set: {len(train_images)}\")\nprint(f\"Number of images in a training set: {len(test_images)}\")","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:34:33.402861Z","iopub.execute_input":"2022-08-03T02:34:33.40403Z","iopub.status.idle":"2022-08-03T02:34:33.645052Z","shell.execute_reply.started":"2022-08-03T02:34:33.403975Z","shell.execute_reply":"2022-08-03T02:34:33.643792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can also visualize one of the training pictures and see how it looks.","metadata":{}},{"cell_type":"code","source":"picture_path = next(NORMALIZED_TRAIN_FOLDER.iterdir())\nplt.style.use(\"seaborn-whitegrid\")\n\nimg = Image.open(picture_path)   \nimg.thumbnail((300,300), Image.Resampling.LANCZOS)\nplt.imshow(img)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:34:33.646717Z","iopub.execute_input":"2022-08-03T02:34:33.649364Z","iopub.status.idle":"2022-08-03T02:34:34.018586Z","shell.execute_reply.started":"2022-08-03T02:34:33.649326Z","shell.execute_reply":"2022-08-03T02:34:34.017172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering / Prepare the data","metadata":{}},{"cell_type":"markdown","source":"Here we are creating a new column in the dataframe that will hold the path to the respective image.","metadata":{}},{"cell_type":"code","source":"train_df[\"path\"] = train_df[\"image_id\"].apply(lambda x: NORMALIZED_TRAIN_FOLDER / f\"{x}.png\")\ntrain_df[\"path\"] = train_df[\"path\"].astype(str)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:34:34.020253Z","iopub.execute_input":"2022-08-03T02:34:34.020591Z","iopub.status.idle":"2022-08-03T02:34:34.037372Z","shell.execute_reply.started":"2022-08-03T02:34:34.020557Z","shell.execute_reply":"2022-08-03T02:34:34.036295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We select only the two columns that we need. In this case, it would be the path column and label column.","metadata":{}},{"cell_type":"code","source":"selected_train_df = train_df[[\"path\", \"label\"]]\nselected_train_df['label'] = selected_train_df['label'].astype('category')\nselected_train_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:34:34.03881Z","iopub.execute_input":"2022-08-03T02:34:34.039641Z","iopub.status.idle":"2022-08-03T02:34:34.057974Z","shell.execute_reply.started":"2022-08-03T02:34:34.039605Z","shell.execute_reply":"2022-08-03T02:34:34.056899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Models\nIn this section, we will be using transfer learning to build our prediction model. We will be using the EfficientNetV2B3 as the backbone feature extractor. You can find more information here: https://www.tensorflow.org/api_docs/python/tf/keras/applications/efficientnet_v2/EfficientNetV2B3","metadata":{}},{"cell_type":"code","source":"# Image size of (300, 300, 3)\nIMG_SHAPE = IMG_SIZE + (IMG_CHANNELS,)\n# preprocess_input = tf.keras.applications.mobilenet_v2.preprocess_input\npre_trained_model = tf.keras.applications.efficientnet_v2.EfficientNetV2B3(input_shape=IMG_SHAPE,\n                                               include_top=False,\n                                               weights='imagenet')\n\n# Set trainable to false for the pretrained model. We only want to train the dense layers we will be addind for prediction\npre_trained_model.trainable = False\n","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:34:34.059452Z","iopub.execute_input":"2022-08-03T02:34:34.059852Z","iopub.status.idle":"2022-08-03T02:34:39.940487Z","shell.execute_reply.started":"2022-08-03T02:34:34.059817Z","shell.execute_reply":"2022-08-03T02:34:39.938985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preparing for Training\nHere we will build our data generators that will feed our model with data for training and testing. Note that we are using data augmentation for the train and validation generators. We do not augment the data for testing.\nWe are also not using any rescaling since the EfficientNetV2B3 model already does rescaling as part of the model.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ndef train_val_generators(train_df, test_df):\n    \n    # No rescaling needed since it is included in the efficientnet model\n    train_datagen = ImageDataGenerator(\n      rotation_range=40,\n      width_shift_range=0.2,\n      height_shift_range=0.2,\n      shear_range=0.2,\n      zoom_range=0.2,\n      horizontal_flip=True,\n      fill_mode='nearest',\n      validation_split=0.3\n    )\n    \n    # Train and validation will be used during training. Validation score is what we look at to tune the hyperparameters and to assess if our model is overfitting or not\n    train_generator = train_datagen.flow_from_dataframe(dataframe=train_df, x_col='path', y_col='label',class_mode='binary', subset='training', color_mode='rgb', target_size=IMG_SIZE, batch_size=BATCH_SIZE)\n    validation_generator = train_datagen.flow_from_dataframe(dataframe=train_df, x_col='path', y_col='label',class_mode='binary', subset='validation', color_mode='rgb', target_size=IMG_SIZE, batch_size=BATCH_SIZE)\n    \n    # The test generator will be used to create the final submission\n    test_datagen = ImageDataGenerator()\n    test_generator = test_datagen.flow_from_directory(\"./norm-test-images\",\n    target_size=IMG_SIZE,\n    batch_size=1,\n    class_mode=\"binary\",\n    shuffle=False,\n)\n\n\n    return train_generator, validation_generator, test_generator","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:34:39.943856Z","iopub.execute_input":"2022-08-03T02:34:39.944822Z","iopub.status.idle":"2022-08-03T02:34:40.119301Z","shell.execute_reply.started":"2022-08-03T02:34:39.944756Z","shell.execute_reply":"2022-08-03T02:34:40.118076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_generator, validation_generator, test_generator = train_val_generators(selected_train_df, test_df)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:34:40.120612Z","iopub.execute_input":"2022-08-03T02:34:40.122306Z","iopub.status.idle":"2022-08-03T02:34:40.653001Z","shell.execute_reply.started":"2022-08-03T02:34:40.122235Z","shell.execute_reply":"2022-08-03T02:34:40.651489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can analyze the shape of one image batch and also the shape of output tensor after it goes through the backbone model.","metadata":{}},{"cell_type":"code","source":"image_batch, label_batch = next(train_generator)\nimage_batch.shape","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:34:40.655133Z","iopub.execute_input":"2022-08-03T02:34:40.656141Z","iopub.status.idle":"2022-08-03T02:34:46.804014Z","shell.execute_reply.started":"2022-08-03T02:34:40.656088Z","shell.execute_reply":"2022-08-03T02:34:46.802987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_batch = pre_trained_model(image_batch)\nprint(feature_batch.shape)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:34:46.805688Z","iopub.execute_input":"2022-08-03T02:34:46.807162Z","iopub.status.idle":"2022-08-03T02:35:04.806453Z","shell.execute_reply.started":"2022-08-03T02:34:46.807113Z","shell.execute_reply":"2022-08-03T02:35:04.805021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can also visualize the training data from the created generator.","metadata":{}},{"cell_type":"code","source":"fit, ax= plt.subplots(nrows=2, ncols=3, figsize=(15,8))\nax=ax.flatten()\nj=0\nfor _ in range(6):\n    img, label = train_generator.next()\n    ax[j].imshow(img[0],)\n    ax[j].set_title(label[0])\n    j=j+1","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:35:04.808414Z","iopub.execute_input":"2022-08-03T02:35:04.809346Z","iopub.status.idle":"2022-08-03T02:35:53.842907Z","shell.execute_reply.started":"2022-08-03T02:35:04.80928Z","shell.execute_reply":"2022-08-03T02:35:53.841445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is important to understand how the classes our mapped in the generator. In our case, CA is 0 and LAA is 1.","metadata":{}},{"cell_type":"code","source":"train_generator.class_indices","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:35:53.845113Z","iopub.execute_input":"2022-08-03T02:35:53.845672Z","iopub.status.idle":"2022-08-03T02:35:53.854685Z","shell.execute_reply.started":"2022-08-03T02:35:53.84562Z","shell.execute_reply":"2022-08-03T02:35:53.853249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pre_trained_model.summary()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-08-03T02:35:53.856287Z","iopub.execute_input":"2022-08-03T02:35:53.857617Z","iopub.status.idle":"2022-08-03T02:35:55.306539Z","shell.execute_reply.started":"2022-08-03T02:35:53.857549Z","shell.execute_reply":"2022-08-03T02:35:55.30508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before creating our full model, let's understand what type of model is our pre trained model. We are trying to understand if it built using the TensorFlow sequential or functional API.","metadata":{}},{"cell_type":"code","source":"# Print the type of the pre-trained model\nprint(f\"The pretrained model has type: {type(pre_trained_model)}\")","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:35:55.308237Z","iopub.execute_input":"2022-08-03T02:35:55.308561Z","iopub.status.idle":"2022-08-03T02:35:55.315699Z","shell.execute_reply.started":"2022-08-03T02:35:55.308528Z","shell.execute_reply":"2022-08-03T02:35:55.314278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We notice that it is built using the functional API. We can now proceed on adding our last few dense layers to complete our model.","metadata":{}},{"cell_type":"code","source":"def create_final_model(pre_trained_model):\n    \n    # Flatten the output layer to 1 dimension\n    x = layers.Flatten()(pre_trained_model.output)\n\n    ### START CODE HERE\n\n    # Add a fully connected layer with 512 hidden units and ReLU activation\n    x = layers.Dense(512, activation='relu')(x)\n    # Add a dropout rate of 0.2 \n    x = layers.Dropout(0.2)(x)  \n    # Add a fully connected layer with 128 hidden units and ReLU activation\n    x = layers.Dense(128, activation='relu')(x)\n    # Add a dropout rate of 0.2\n    x = layers.Dropout(0.2)(x)     \n    # Add a fully connected layer with 64 hidden units and ReLU activation\n    x = layers.Dense(64, activation='relu')(x)\n    # Add a final sigmoid layer for classification\n    x = layers.Dense(1, activation='sigmoid')(x)       \n\n    # Create the complete model by using the Model class\n    model = Model(inputs=pre_trained_model.input, outputs=x)\n\n    # Compile the model\n    model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate=LR), \n                loss = tf.keras.losses.BinaryCrossentropy(from_logits=True),\n                metrics = ['accuracy'])\n\n    return model","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:35:55.317919Z","iopub.execute_input":"2022-08-03T02:35:55.318256Z","iopub.status.idle":"2022-08-03T02:35:55.329184Z","shell.execute_reply.started":"2022-08-03T02:35:55.318224Z","shell.execute_reply":"2022-08-03T02:35:55.327059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save your model in a variable\nmodel = create_final_model(pre_trained_model)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:35:55.331619Z","iopub.execute_input":"2022-08-03T02:35:55.332208Z","iopub.status.idle":"2022-08-03T02:35:56.234677Z","shell.execute_reply.started":"2022-08-03T02:35:55.332157Z","shell.execute_reply":"2022-08-03T02:35:56.233178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Inspect parameters\ntotal_params = model.count_params()\nnum_trainable_params = sum([w.shape.num_elements() for w in model.trainable_weights])\n\nprint(f\"There are {total_params:,} total parameters in this model.\")\nprint(f\"There are {num_trainable_params:,} trainable parameters in this model.\")","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:35:56.236541Z","iopub.execute_input":"2022-08-03T02:35:56.236927Z","iopub.status.idle":"2022-08-03T02:35:56.254715Z","shell.execute_reply.started":"2022-08-03T02:35:56.236892Z","shell.execute_reply":"2022-08-03T02:35:56.253177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\">\n<b>Remark:</b> You can notice that even though we only added 4 dense layers, there are many trainable parameters that have been added that are not included in the backbone model (EfficientNetV2B3). This is due to the fact that we are flattening that last output of the backbone and propagating it through all those 4 dense layers. The amount of trainable parameters, in this case, increases very rapidly.\n</div>","metadata":{}},{"cell_type":"markdown","source":"## Training\nWe've prepared the food (data), time to... FEED THE MACHINE. We use three different callbacks that give additional functionality during training. Pleaes research them to get a deeper understanding on what each of the callback is doing: https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping\n\nearly_stop = EarlyStopping(monitor='val_loss',patience=4)\n\nfilepath = \"./model.h5\"\ncheckpoint = ModelCheckpoint(filepath, monitor=\"val_acc\", verbose=1, save_best_only=True, mode=\"max\")\n\nreduce_lr = ReduceLROnPlateau(monitor=\"val_acc\", factor=0.5, patience=2, verbose=1, mode=\"max\", min_lr=0.00001)\n\ncallbacks_list = [early_stop, checkpoint, reduce_lr]\n\nhistory = model.fit(train_generator,\n                    validation_data = validation_generator,\n                    epochs = EPOCHS,\n                    verbose = 2,\n                    callbacks=callbacks_list)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:35:56.256321Z","iopub.execute_input":"2022-08-03T02:35:56.256898Z","iopub.status.idle":"2022-08-03T02:52:31.800585Z","shell.execute_reply.started":"2022-08-03T02:35:56.256859Z","shell.execute_reply":"2022-08-03T02:52:31.79931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plot training results\nIt is important to plot our training results and inspect if our model is performing well. It is important to look at the validation accuracy and loss to understand if our model is overfitting our training data. In our case, we are only training for 1 Epoch, therefore we are not expecting great results.","metadata":{}},{"cell_type":"code","source":"plt.style.use(\"seaborn-whitegrid\")\n\nacc = history.history['accuracy']\nval_acc = history.history['val_accuracy']\n\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n\nplt.figure(figsize=(8, 8))\nplt.subplot(2, 1, 1)\nplt.plot(acc, label='Training Accuracy')\nplt.plot(val_acc, label='Validation Accuracy')\nplt.legend(loc='lower right')\nplt.ylabel('Accuracy')\nplt.ylim([min(plt.ylim()),1])\nplt.title('Training and Validation Accuracy')\n\nplt.subplot(2, 1, 2)\nplt.plot(loss, label='Training Loss')\nplt.plot(val_loss, label='Validation Loss')\nplt.legend(loc='upper right')\nplt.ylabel('Binary Crossentropy')\nplt.ylim([0,1.0])\nplt.title('Training and Validation Loss')\nplt.xlabel('epoch')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:52:31.802696Z","iopub.execute_input":"2022-08-03T02:52:31.803082Z","iopub.status.idle":"2022-08-03T02:52:32.620062Z","shell.execute_reply.started":"2022-08-03T02:52:31.803046Z","shell.execute_reply":"2022-08-03T02:52:32.618456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Save model\nAfter iterating and playing with your hyperparameters to reach better scores, you can save and load your model with the following code blocks. It would be annoying to always need to train your model to continue working on your notebook... better to load it.","metadata":{}},{"cell_type":"markdown","source":"### Saving only the weights","metadata":{}},{"cell_type":"code","source":"# # Save the weights\n# model.save_weights('./checkpoints/my_checkpoint')","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:52:32.626315Z","iopub.execute_input":"2022-08-03T02:52:32.626686Z","iopub.status.idle":"2022-08-03T02:52:32.632245Z","shell.execute_reply.started":"2022-08-03T02:52:32.626652Z","shell.execute_reply":"2022-08-03T02:52:32.630786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Create a new model instance\n# model = create_final_model(pre_trained_model)\n\n# # Restore the weights\n# model.load_weights('../input/trained-models/checkpoints/my_checkpoint')","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:52:32.634435Z","iopub.execute_input":"2022-08-03T02:52:32.634976Z","iopub.status.idle":"2022-08-03T02:52:32.642985Z","shell.execute_reply.started":"2022-08-03T02:52:32.634942Z","shell.execute_reply":"2022-08-03T02:52:32.641922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Saving the whole model","metadata":{}},{"cell_type":"code","source":"# Save the entire model as a SavedModel.\n# https://www.tensorflow.org/tutorials/keras/save_and_load#savedmodel_format\n\n# !mkdir -p saved_model\n# model.save('saved_model/my_model')\n# !zip -r 'saved_model.zip' './saved_model'","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:52:32.644426Z","iopub.execute_input":"2022-08-03T02:52:32.644868Z","iopub.status.idle":"2022-08-03T02:52:32.654587Z","shell.execute_reply.started":"2022-08-03T02:52:32.644835Z","shell.execute_reply":"2022-08-03T02:52:32.653549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Predicting & Submitting\nHere I am loading a model I trained and tuned the hyperarameters. I will be using this model for predicting the test images.","metadata":{}},{"cell_type":"code","source":"model = tf.keras.models.load_model('../input/mayo-clinic-saved-model/my_model')","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:52:32.65626Z","iopub.execute_input":"2022-08-03T02:52:32.65669Z","iopub.status.idle":"2022-08-03T02:53:04.336692Z","shell.execute_reply.started":"2022-08-03T02:52:32.656656Z","shell.execute_reply":"2022-08-03T02:53:04.3352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will now use the test generator to predict our submission results.","metadata":{}},{"cell_type":"code","source":"ls_imgs_png = glob(os.path.join(\"./norm-test-images/normalized_test_images\", \"*.png\"))\namount_of_test_images = len(ls_imgs_png)\namount_of_test_images","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:53:04.340849Z","iopub.execute_input":"2022-08-03T02:53:04.341242Z","iopub.status.idle":"2022-08-03T02:53:04.351411Z","shell.execute_reply.started":"2022-08-03T02:53:04.341206Z","shell.execute_reply":"2022-08-03T02:53:04.350083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"laa_predictions = model.predict(test_generator, steps=amount_of_test_images, verbose=1)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:53:04.353085Z","iopub.execute_input":"2022-08-03T02:53:04.354569Z","iopub.status.idle":"2022-08-03T02:53:08.206339Z","shell.execute_reply.started":"2022-08-03T02:53:04.354516Z","shell.execute_reply":"2022-08-03T02:53:08.20521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<div class=\"alert alert-block alert-danger\">  \nNote that since LAA is class 1, the model is predicting the probability for LAA. \n</div>","metadata":{}},{"cell_type":"code","source":"ca_preds = 1 - laa_predictions\npreds = np.concatenate((laa_predictions, ca_preds), axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:53:08.207948Z","iopub.execute_input":"2022-08-03T02:53:08.208887Z","iopub.status.idle":"2022-08-03T02:53:08.215101Z","shell.execute_reply.started":"2022-08-03T02:53:08.208849Z","shell.execute_reply":"2022-08-03T02:53:08.214115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Put the predictions into a dataframe\ndf_preds = pd.DataFrame(preds, columns=[\"LAA\", \"CE\"])\ndf_preds.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:53:08.216501Z","iopub.execute_input":"2022-08-03T02:53:08.217149Z","iopub.status.idle":"2022-08-03T02:53:08.236406Z","shell.execute_reply.started":"2022-08-03T02:53:08.217112Z","shell.execute_reply":"2022-08-03T02:53:08.234796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"names = [os.path.splitext(os.path.basename(p))[0] for p in ls_imgs_png]\ndf_preds[\"patient_id\"] = [n.split(\"_\")[0] for n in names]\ndf_preds.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:53:08.238317Z","iopub.execute_input":"2022-08-03T02:53:08.240175Z","iopub.status.idle":"2022-08-03T02:53:08.258804Z","shell.execute_reply.started":"2022-08-03T02:53:08.240122Z","shell.execute_reply":"2022-08-03T02:53:08.257574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we will be adding back the images that were discarded because they were too big. We will be filling the CE and LAA prediction probabilities with hardcoded mean values. In the future, it would be interesting to use the mean of CE and LAA probabilities of all images as the filling value instead of hardcoded values.","metadata":{}},{"cell_type":"code","source":"# fill missing if skipped for too large image size\nmissed = [{\"CE\": 0.6, \"LAA\": 0.4, \"patient_id\": pid} for pid in patient_ids if pid not in df_preds[\"patient_id\"].values]\ndf_preds = df_preds.append(pd.DataFrame(missed), ignore_index=True)\ndf_preds.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:53:08.261868Z","iopub.execute_input":"2022-08-03T02:53:08.262225Z","iopub.status.idle":"2022-08-03T02:53:08.283243Z","shell.execute_reply.started":"2022-08-03T02:53:08.262192Z","shell.execute_reply":"2022-08-03T02:53:08.282041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# in case there are many samples for one patient --> use mean value\ndf_preds = df_preds.groupby(\"patient_id\").mean()\ndf_preds[[\"CE\", \"LAA\"]].round(6)\ndf_preds.reset_index(inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:53:08.285917Z","iopub.execute_input":"2022-08-03T02:53:08.287269Z","iopub.status.idle":"2022-08-03T02:53:08.304161Z","shell.execute_reply.started":"2022-08-03T02:53:08.287216Z","shell.execute_reply":"2022-08-03T02:53:08.302888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_preds.to_csv(\"submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T02:53:08.305819Z","iopub.execute_input":"2022-08-03T02:53:08.306169Z","iopub.status.idle":"2022-08-03T02:53:08.320251Z","shell.execute_reply.started":"2022-08-03T02:53:08.306136Z","shell.execute_reply":"2022-08-03T02:53:08.318773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Final Remarks\nThank you for going through this notebook. Please feel free to show support and comment on the notebooks with advice or improvements. If you found it useful, please let me know as well :)","metadata":{}}]}