{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Proyecto Data Science 1","metadata":{}},{"cell_type":"markdown","source":"### NOTA: \n\n#### 1.\nEl conjunto de datos sobre el que se trabaja a continuación corresponde a una competencia en Kaggle organizada por \"Mayo-Clinic\".\nDado que la base de datos proporcionada es extremadamente pesada (356 GB en total) y compuesta de imagenes de hasta 1.2 GB, se concluyó que la mejor manera de trabajr este conjunto de datos es directamente en un notebook de Kaggle ya que el RAM y CPU al que se tiene acceso aqui es muy superior al de cualquiera de las computadoras PC a las que los integrantes del grupo tienen acceso.\n\n#### 2.\nA continuación se incluyen el código de todos los avances que se realicen a medida que se desarrolla el proyecto. \n","metadata":{}},{"cell_type":"markdown","source":"### Fase 1: EDA\n### Fecha de entrega:  19/9/22","metadata":{}},{"cell_type":"code","source":"import os\nimport shutil\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom matplotlib.image import imread\nfrom PIL import Image\nimport PIL.Image\nPIL.Image.MAX_IMAGE_PIXELS = 6000000000  #esto es para que Kaggle no bloquee la ejecucion por el peso excesivo de las imagenes","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:09:41.710317Z","iopub.execute_input":"2022-11-17T14:09:41.710766Z","iopub.status.idle":"2022-11-17T14:09:42.915167Z","shell.execute_reply.started":"2022-11-17T14:09:41.710673Z","shell.execute_reply":"2022-11-17T14:09:42.914208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Empezamos por importar los datos y los archivos pertinentes a este dataset","metadata":{}},{"cell_type":"code","source":"#Importamos los datos\n\n#estos son los caminos hacia la ubicacion de las fotos en Kaggle\npath_todos = '/kaggle/input/mayo-clinic-strip-ai/'\npath_fotos_train = path_todos + 'train/'\npath_fotos_test = path_todos + 'test/'\npath_fotos_other = path_todos + 'other/'\n\n\n#Creamos 3 dataframes de Pandas uno por csv. \n#El archivo \"sample solo es una muestra pertinente solo para los que esten compitiendo en este Kaggle\n\nDf_train = pd.read_csv(path_todos+'train.csv')\nDf_test = pd.read_csv(path_todos+'test.csv')\nDf_other = pd.read_csv(path_todos+'other.csv')\n\n","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:44.15283Z","iopub.execute_input":"2022-11-16T23:59:44.153256Z","iopub.status.idle":"2022-11-16T23:59:44.193255Z","shell.execute_reply.started":"2022-11-16T23:59:44.153224Z","shell.execute_reply":"2022-11-16T23:59:44.19216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### A continuación le damos un primer vistazo a los dataframes que vienen en formato .csv y revisamos las dimensiones de los archivos, su formato, las distintas variables que comporta cada uno, su propósito en la BD, revisamos si contienen NAs, y el estado general de cada uno.","metadata":{}},{"cell_type":"code","source":"#Primero le damos un vistazo a los dataframes\n\nDf_train.head() #Primeras filas del dataset de entrenamiento","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:45.397749Z","iopub.execute_input":"2022-11-16T23:59:45.398236Z","iopub.status.idle":"2022-11-16T23:59:45.424743Z","shell.execute_reply.started":"2022-11-16T23:59:45.398196Z","shell.execute_reply":"2022-11-16T23:59:45.423775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Ultimas filas del dataset de entrenamiento\n\nDf_train.tail()","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:45.786507Z","iopub.execute_input":"2022-11-16T23:59:45.787138Z","iopub.status.idle":"2022-11-16T23:59:45.800538Z","shell.execute_reply.started":"2022-11-16T23:59:45.787102Z","shell.execute_reply":"2022-11-16T23:59:45.798881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Primeras filas del dataset de prueba (test)\n\nDf_test.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:46.200094Z","iopub.execute_input":"2022-11-16T23:59:46.200569Z","iopub.status.idle":"2022-11-16T23:59:46.213445Z","shell.execute_reply.started":"2022-11-16T23:59:46.200531Z","shell.execute_reply":"2022-11-16T23:59:46.212059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#ultimas filas del dataset de prueba (test)\n\nDf_test.tail()  #!!! Este dataset solo contiene 4 filas. No es para \"probar el modelo\" \n                #sino que es parte de la competencia\n","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:46.624733Z","iopub.execute_input":"2022-11-16T23:59:46.625861Z","iopub.status.idle":"2022-11-16T23:59:46.639153Z","shell.execute_reply.started":"2022-11-16T23:59:46.625816Z","shell.execute_reply":"2022-11-16T23:59:46.637565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Primeras filas del dataset 'other'\n\nDf_other.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:47.010717Z","iopub.execute_input":"2022-11-16T23:59:47.011194Z","iopub.status.idle":"2022-11-16T23:59:47.026391Z","shell.execute_reply.started":"2022-11-16T23:59:47.011155Z","shell.execute_reply":"2022-11-16T23:59:47.023901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Ultimas filas del dataset 'other'\n\nDf_other.tail()","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:47.381717Z","iopub.execute_input":"2022-11-16T23:59:47.38212Z","iopub.status.idle":"2022-11-16T23:59:47.395154Z","shell.execute_reply.started":"2022-11-16T23:59:47.382088Z","shell.execute_reply":"2022-11-16T23:59:47.393963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Dims Df_train: ', Df_train.shape)\nprint('Dims Df_test: ', Df_test.shape)\nprint('Dims Df_other: ', Df_other.shape)","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:47.768833Z","iopub.execute_input":"2022-11-16T23:59:47.769284Z","iopub.status.idle":"2022-11-16T23:59:47.776959Z","shell.execute_reply.started":"2022-11-16T23:59:47.769246Z","shell.execute_reply":"2022-11-16T23:59:47.775601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Df_train.isna().sum()  #Revisando si tenemos NAs ","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:48.191536Z","iopub.execute_input":"2022-11-16T23:59:48.192877Z","iopub.status.idle":"2022-11-16T23:59:48.204796Z","shell.execute_reply.started":"2022-11-16T23:59:48.192817Z","shell.execute_reply":"2022-11-16T23:59:48.203458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Df_test.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:48.948727Z","iopub.execute_input":"2022-11-16T23:59:48.949845Z","iopub.status.idle":"2022-11-16T23:59:48.959988Z","shell.execute_reply.started":"2022-11-16T23:59:48.949801Z","shell.execute_reply":"2022-11-16T23:59:48.959161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Df_other.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:49.610785Z","iopub.execute_input":"2022-11-16T23:59:49.611692Z","iopub.status.idle":"2022-11-16T23:59:49.623154Z","shell.execute_reply.started":"2022-11-16T23:59:49.611648Z","shell.execute_reply":"2022-11-16T23:59:49.621693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Conclusiones preliminares sobre los Datos:\n\n1. Los datasets train.csv, test.csv y other.csv esencialmente contienen información respecto a las imagenes que se encuentran en las carpetas train, test y other respectivamente.\n\n2. El dataset \"train.csv\" contiene 754 lineas y 5 variables en total. Estas variables son:\n\n    i. image_id : Contiene el codigo que identifica una imagen del set de entrenamiento\n    \n    ii. center_id: El identificador del centro de diagnostico donde se capturó la imagen\n    \n    iii. patient_id : El código que identifica el paciente\n    \n    iv. image_num : El numero de la fotografía\n    \n    v. label : Identifica si la foto corresponde a un coagulo tipo CE (Cardioembolic) o LAA (Large Artery Atherosclerosis)\n               \n3. No hay datos faltantes en cuanto a la identificacion de imagenes o sus \"labels\" respectivos, por lo que en principio no debería de requerirse muchas operaciones de limpieza de datos en el futuro.\n","metadata":{}},{"cell_type":"markdown","source":"#### Revisamos las imagenes","metadata":{}},{"cell_type":"markdown","source":"#### Averiguamos cuantas imagenes contiene cada carpeta","metadata":{}},{"cell_type":"code","source":"print('No. de imgs en carpeta \"train\": ', len(os.listdir(path_fotos_train)))\nprint('No. de imgs en carpeta \"test\": ', len(os.listdir(path_fotos_test)))\nprint('No. de imgs en carpeta \"other\": ', len(os.listdir(path_fotos_other)))","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:51.791273Z","iopub.execute_input":"2022-11-16T23:59:51.791749Z","iopub.status.idle":"2022-11-16T23:59:52.214419Z","shell.execute_reply.started":"2022-11-16T23:59:51.791712Z","shell.execute_reply":"2022-11-16T23:59:52.213053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Todo bien! El No de imgs en cada carpeta es consistente con el No. de entradas en los datasets que contienen la informacion pertinente!\n\n#### Ahora revisamos más de cerca las propiedades de algunas de las imagenes.","metadata":{}},{"cell_type":"code","source":"#extraemos una foto  del dataset de entrenamiento\n\nfoto_train = path_fotos_train + os.listdir(path_fotos_train)[136] \n\nfoto_train = imread(foto_train)","metadata":{"execution":{"iopub.status.busy":"2022-11-16T23:59:52.548462Z","iopub.execute_input":"2022-11-16T23:59:52.549685Z","iopub.status.idle":"2022-11-17T00:00:00.627565Z","shell.execute_reply.started":"2022-11-16T23:59:52.549627Z","shell.execute_reply":"2022-11-17T00:00:00.62603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(foto_train)  #Visualizamos la imagen","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:00:00.630141Z","iopub.execute_input":"2022-11-17T00:00:00.631398Z","iopub.status.idle":"2022-11-17T00:00:34.773569Z","shell.execute_reply.started":"2022-11-17T00:00:00.631325Z","shell.execute_reply":"2022-11-17T00:00:34.772267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.listdir(path_fotos_train).index('09644e_4.tif')","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:00:34.775518Z","iopub.execute_input":"2022-11-17T00:00:34.776461Z","iopub.status.idle":"2022-11-17T00:00:34.78572Z","shell.execute_reply.started":"2022-11-17T00:00:34.776423Z","shell.execute_reply":"2022-11-17T00:00:34.784636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Averiguamos el tamaño promedio de una imagen\n\ndim1 = []\ndim2 = []\nfor i in os.listdir(path_fotos_train):\n    \n    img = Image.open(path_fotos_train + i)\n    d1 = img.width\n    d2 = img.height\n    dim1.append(d1)\n    dim2.append(d2)\n    ","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:00:34.789359Z","iopub.execute_input":"2022-11-17T00:00:34.79015Z","iopub.status.idle":"2022-11-17T00:01:00.92871Z","shell.execute_reply.started":"2022-11-17T00:00:34.790115Z","shell.execute_reply":"2022-11-17T00:01:00.927242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.jointplot(x = dim1, y = dim2) #Hacemos un plot de las dimensiones de las imagenes","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:00.930881Z","iopub.execute_input":"2022-11-17T00:01:00.93141Z","iopub.status.idle":"2022-11-17T00:01:01.654094Z","shell.execute_reply.started":"2022-11-17T00:01:00.931342Z","shell.execute_reply":"2022-11-17T00:01:01.652871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Finalmente, averiguamos las dimensiones promedio de las imagenes:\n\nprint('promedio ancho: ', np.mean(dim1))\nprint('promedio altura: ', np.mean(dim2))","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:01.655832Z","iopub.execute_input":"2022-11-17T00:01:01.65619Z","iopub.status.idle":"2022-11-17T00:01:01.663919Z","shell.execute_reply.started":"2022-11-17T00:01:01.65616Z","shell.execute_reply":"2022-11-17T00:01:01.662744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Para terminar con el EDA averiguamos el peso promedio en Bytes de las imagenes\n\npeso_imgs = []\n\nfor i in os.listdir(path_fotos_train):\n    \n    camino = path_fotos_train + i\n    peso = os.stat(camino).st_size\n    peso_imgs.append(peso)\n\n\n    ","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:01.665415Z","iopub.execute_input":"2022-11-17T00:01:01.665841Z","iopub.status.idle":"2022-11-17T00:01:02.086037Z","shell.execute_reply.started":"2022-11-17T00:01:01.665807Z","shell.execute_reply":"2022-11-17T00:01:02.084799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.histplot(data=peso_imgs).set(xlabel = 'Peso en GB')","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:02.087859Z","iopub.execute_input":"2022-11-17T00:01:02.088357Z","iopub.status.idle":"2022-11-17T00:01:02.419677Z","shell.execute_reply.started":"2022-11-17T00:01:02.088307Z","shell.execute_reply":"2022-11-17T00:01:02.418434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Peso promedio de las fotos: ', np.mean(peso_imgs)/1000000000, ' GB')","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:02.421437Z","iopub.execute_input":"2022-11-17T00:01:02.421799Z","iopub.status.idle":"2022-11-17T00:01:02.427878Z","shell.execute_reply.started":"2022-11-17T00:01:02.421766Z","shell.execute_reply":"2022-11-17T00:01:02.426771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Distribución de tipos de célula gráfico de pie \nplot = Df_train['label'].value_counts().plot(kind='pie', autopct='%.2f', \n                                            figsize=(6, 6),\n                                            title='Distribución de tipos de célula')","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:02.430853Z","iopub.execute_input":"2022-11-17T00:01:02.431226Z","iopub.status.idle":"2022-11-17T00:01:02.587217Z","shell.execute_reply.started":"2022-11-17T00:01:02.431193Z","shell.execute_reply":"2022-11-17T00:01:02.584609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"La proporción es 72.55% CE y 27.45% LAA","metadata":{}},{"cell_type":"code","source":"plot = Df_train['center_id'].value_counts().plot(kind='bar',\n                                            title='Centros')","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:08.316121Z","iopub.execute_input":"2022-11-17T00:01:08.31733Z","iopub.status.idle":"2022-11-17T00:01:08.600774Z","shell.execute_reply.started":"2022-11-17T00:01:08.317268Z","shell.execute_reply":"2022-11-17T00:01:08.599059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Observamos que la mayoría de fotos provienen del centro 11, seguidos del centro 4 y 7, respectivamente.","metadata":{}},{"cell_type":"code","source":"plot = pd.crosstab(index=Df_train['center_id'],\n            columns=Df_train['label']\n                  ).apply(lambda r: r/r.sum() *100,\n                          axis=0).plot(kind='bar', stacked=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:09.582471Z","iopub.execute_input":"2022-11-17T00:01:09.582904Z","iopub.status.idle":"2022-11-17T00:01:09.975133Z","shell.execute_reply.started":"2022-11-17T00:01:09.582867Z","shell.execute_reply":"2022-11-17T00:01:09.973657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"En las fotos de las muestras tomadas, los centros 1, 2, 8, 9 y 10 registraron más células de tipo CE. Mientras que el centro 3 predomina la célula tipo LAA. Los otros centros tienen proporciones más similares. ","metadata":{}},{"cell_type":"code","source":"df = pd.crosstab(index=Df_train['patient_id'],\n            columns=Df_train['label'], margins=True)\ndf","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:15.490523Z","iopub.execute_input":"2022-11-17T00:01:15.490956Z","iopub.status.idle":"2022-11-17T00:01:15.560698Z","shell.execute_reply.started":"2022-11-17T00:01:15.490922Z","shell.execute_reply":"2022-11-17T00:01:15.559283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.drop(df[df['CE']==0].index)\ndf = df.drop(df[df['LAA']==0].index)\nwith pd.option_context(\"display.max_rows\", 1000):\n    display(df)","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:15.753164Z","iopub.execute_input":"2022-11-17T00:01:15.753615Z","iopub.status.idle":"2022-11-17T00:01:15.770597Z","shell.execute_reply.started":"2022-11-17T00:01:15.753581Z","shell.execute_reply":"2022-11-17T00:01:15.769293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"La anterior comprueba que si se tomaron varias fotos de una misma prueba, estas fueron identificadas con una única label CE ó LAA, esto quiere decir que todo el conjunto de fotos con un mismo pacient_id le corresponde un único label CE ó LAA","metadata":{}},{"cell_type":"markdown","source":"### Fase 2:\n\n### Fecha de entrega: 31/10/22","metadata":{}},{"cell_type":"markdown","source":"Como vimos en el EDA, el tamaño de los archivos que se encuentran en la BD es enorme: cada imagen puede llegar a medir desde 300MB hasta 3 GB de manera que en total, la BD pesa 390 GB. Esto implica que aun con los recursos mejorados de computo que ofrece Kaggle, estamos limitado respecto a las operaciones que podemos ejecutar en el Data Wrangling. \nPor lo tanto, y considerando que el objetivo final es obtener una base de datos que pueda trabajarse de manera similar al laboratorio trabajado en clase sobre imagenes correspondientes a imagenes de celulas infectadas con malaria , empezamos definiendo claramente los objetivos que queremos cumplir en la fase de Data Wrangling:\n\n1. Dado que el objetivo final es tener un conjunto de imagenes sobre el cuál se pueda entrenar una CNN, es necesario que las imagenes estén ordenadas en el formato requerido por Keras para las CNN. Es decir:\n\n Train:\n \n     Clase 1:\n         img_1.png\n         img_2.png\n         etc...\n     Clase 2:\n         img_1.png\n         img_2.png\n         etc...\n         \n         \nTest:\n\n    Clase 1:\n         img_1.png\n         img_2.png\n         etc...\n     Clase 2:\n         img_1.png\n         img_2.png\n         etc...\n         \n         \n         \n2. Además, como contamos con una cantidad limitada tanto de espacio en la carpeta output, como de RAM en Kaggle, sería bueno comprimir las imagenes para poder entrenar el modelo.\n \n3. Siempre pensando en el RAm limitado, pensamos que un buen inicio también podría ser convertir las imagenes a blanco y negro para entrenar el primer modelo. En caso se dese hacer un intento por mejorar el \"accuracy\" del modelo, entonces crearemos una nueva BD esta vez con imagenes a color.\n \n4. Cuando tengamos la Base de Datos comprimida y clasificada en la carpeta output de Kaggle, la transferiremos del output hacia un nuevo dataset dentro de Kaggle. De esta manera nunca tendremos que descargar y volver a subir por internet la base de datos a Kaggle o algún servidor ya que esto tomaría mucho tiempo. Así, cuando necesitemos entrenar la red convolucional, podremos acceder a las imagenes inmediatamente via la nueva base de datos.\n\n","metadata":{}},{"cell_type":"markdown","source":"#### Pasos 1, 2 y 3:\n\nA continuación realizamos los pasos 1, 2 y 3 mencionados anteriormente.\nEmpezamos creando als carpetas con la estructura desada.","metadata":{}},{"cell_type":"code","source":"if not os.path.exists('/kaggle/working/Datos_T1'):\n    os.makedirs('/kaggle/working/Datos_T1')\nelse:\n    shutil.rmtree('/kaggle/working/Datos_T1')\n    os.makedirs('/kaggle/working/Datos_T1')\n    print('El directorio Datos_T1 ya existía asi que se reeplazo por uno vacío con el mismo nombre')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### NOTA IMPORTANTE: La instruccion que se encuentra a continuación tardó al rededor de 3:30 horas en ejecutarse en Kaggle.\n\nLa intrucción que se programó a continuación efectua las siguientes instrucciones sobre el conjunto de todas las imagenes de entrenaiento:\n\n\n1. Des-saturar la imagen para reducir el peso del archivo\n2. Comprimir la imagen a un tamaño más manejable (se escogió comrimirlas en un factor de 30)\n3. clasificar la imagen en la carpeta correspondiente (Train/LAA o bien Train/CE) dependiendo del 'label' al que corresponda el código de la imagen en el DF 'train.csv'\n\n\nAún con los recursos computacionales  avanzados (30 GB de RAM) que ofrece Kaggle, está es una operación que consume mucho tiempo y recursos. ","metadata":{}},{"cell_type":"code","source":"for i in range(0, 754):\n    \n    url_img = path_fotos_train + os.listdir(path_fotos_train)[i]\n    img = Image.open(url_img)\n    img_decolored = img.convert('L')\n    img_decolored_resized = img_decolored.resize((int(np.rint(img_decolored.width /30)), int(np.rint(img_decolored.height /30))))\n    \n    codigo_img = url_img.split('/')[-1].split('.tif')[0]\n    histologia = Df_train.loc[Df_train['image_id'] == codigo_img, 'label'] \n    \n    if ((histologia == 'LAA').bool()):\n        img_decolored_resized.save('/kaggle/working/Datos_T1/Train/LAA/'+codigo_img + '.png', 'PNG')\n    else:\n        img_decolored_resized.save('/kaggle/working/Datos_T1/Train/CE/'+codigo_img + '.png', 'PNG')\n        \n    print('img_No: ', i)\n    \n    gc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### NOTA: \n\nTras correr la instrucción anterior, se generó un directorio de al rededor de 100 MB que contiene el conjunto completo de entrenamiento, comprimido, des-saturado y clasificado en el formato deseado. \n\nComo sigue siendo una base de datos grande, se determinó que lo más eficiente era transformar el 'output' del notebook en un dataset de Kaggle (kaggle ofrece esta opción después de guardar el dataset, en la rubrica 'data' de la página del notebook)\n\nEl nuevo directorio, ahora llamado \"Datos_T1\" puede encontrarse en el link: ______ y además puede ser llamado directamente desde un notebook de Kaggle como cualquier otro conjunto de datos.","metadata":{}},{"cell_type":"markdown","source":"### Reordenar los datos:\n\nAhora que el conjunto de entrenamiento está listo, es necesario contar también con un conjunto de prueba/validación (en este caso son el mismo porque solo hay 754 imagenes en total).\n\nPor lo tanto volvemos a efectuar una nueva operación sobre el dataset 'Datos_T1' para contar con 2 carpetas: Train y Test, donde cada una contiene 2 subcarpetas (1 por clase): CE y LAA, las cuales a su vez contienen las imagenes correspondientes a la categoría pertinente.\n\nEsta operación generó un nuevo conjunto de datos \"Datos_T2\" el cuál contituye una nueva versión del anterior.\n\n","metadata":{}},{"cell_type":"code","source":"#Reimportar librerias\nimport matplotlib.cm\nimport shutil\nimport gc\nimport sys\nimport random","metadata":{"execution":{"iopub.status.busy":"2022-11-17T00:01:38.654417Z","iopub.execute_input":"2022-11-17T00:01:38.65487Z","iopub.status.idle":"2022-11-17T00:01:38.660589Z","shell.execute_reply.started":"2022-11-17T00:01:38.654836Z","shell.execute_reply":"2022-11-17T00:01:38.659342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Empezamos por crear un directorio llamado \"Datos_T2\" que contendrá las carpetas correspondientes Train y Test y estas a su vez contendrán las carpetas correspondientes a cada clase requerida (CE y LAA).","metadata":{}},{"cell_type":"code","source":"if not os.path.exists('/kaggle/working/Datos_T2'):  #Si aun no existe, crear el directorio\n    os.makedirs('/kaggle/working/Datos_T2')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Luego, creamos las carpetas TRAIN y TEST con las subcarpetas (LAA y CE) cada una.","metadata":{}},{"cell_type":"code","source":"os.makedirs('/kaggle/working/Datos_T2/Train/LAA')  \nos.makedirs('/kaggle/working/Datos_T2/Train/CE')\nos.makedirs('/kaggle/working/Datos_T2/Test/LAA')\nos.makedirs('/kaggle/working/Datos_T2/Test/CE')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_LAA = '/kaggle/input/datos1/Datos_T1/Train/LAA/'  #Registramos las direcciones de las carpetas\npath_CE = '/kaggle/input/datos1/Datos_T1/Train/CE/'","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(os.listdir(path_LAA))*0.2  #Vamos a extraer 20% de las imagenes  tipo LAA","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(os.listdir(path_CE))*0.2  #20% de las imagenes tipo CE","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Los cálcuos anteriores indican que debemos tomar 41 imagenes de LAA y 109 de CE para el conjutno de datos de prueba.mantener las proporciones. En total, esto representa un 20% de todos los datos.","metadata":{}},{"cell_type":"code","source":"#Empezamos listando los datos LAA para los conjuntos test y train respectivamente :\n\nlista_LAA = os.listdir(path_LAA)\n\nlista_test_LAA = random.sample(lista_LAA, 41 )\n\nlista_train_LAA = [id for id in lista_LAA if id not in lista_test_LAA]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(lista_test_LAA))\nprint(len(lista_train_LAA))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Ahora repartimos los datos dentro del nuevo directorio (output/kaggle/working/Datos_T2)\n\n#Primero con los datos de la lista LAA\n\nfor nombre_foto in lista_LAA:\n    url_foto = path_LAA + nombre_foto\n    img = Image.open(url_foto)\n    \n    if (nombre_foto in lista_train_LAA):\n        img.save('/kaggle/working/Datos_T2/Train/LAA/'+nombre_foto , 'PNG')\n        \n    elif(nombre_foto in lista_test_LAA):\n        img.save('/kaggle/working/Datos_T2/Test/LAA/'+nombre_foto , 'PNG')\n\n\n        \n#Ahora con los datos de la lista CE:\n\nfor nombre_foto in lista_CE:\n    url_foto = path_CE + nombre_foto\n    img = Image.open(url_foto)\n    \n    if (nombre_foto in lista_train_CE):\n        img.save('/kaggle/working/Datos_T2/Train/CE/'+nombre_foto , 'PNG')\n        \n    elif(nombre_foto in lista_test_CE):\n        img.save('/kaggle/working/Datos_T2/Test/CE/'+nombre_foto , 'PNG')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Listo! Solo falta revisar que la separación en el directorio Datos_T2 se haya hecho correctamente:","metadata":{}},{"cell_type":"code","source":"path_Datos_T2 = '/kaggle/working/Datos_T2/'\nos.listdir(path_Datos_T2)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(os.listdir(path_Datos_T2+'Test/LAA')))\nprint(len(os.listdir(path_Datos_T2+'Train/LAA')))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Todo bien con este conjunto, ahora averiguamos si las fotos de tipo CE estan bien ordenadas","metadata":{}},{"cell_type":"code","source":"print(len(os.listdir(path_Datos_T2+'Test/CE')))\nprint(len(os.listdir(path_Datos_T2+'Train/CE')))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Todo bien! Al parecer la separación de los datos se hizo de manera adecuada. Ahora actualizamos el dataset nuevo agregandolo como una nueva versión de 'datos1'.","metadata":{}},{"cell_type":"markdown","source":"### Entrenamiento del modelo:\n\nA continuación, entrenamos una CNN replicando el procedimiento visto en clase para el caso de imagenes de celulas infectadas con malaria.","metadata":{}},{"cell_type":"code","source":"#Importa y reimportar librerias\n\nimport os\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom matplotlib.image import imread\n%matplotlib inline\nimport shutil\nfrom PIL import Image\nimport PIL.Image\nPIL.Image.MAX_IMAGE_PIXELS = 6000000000\nimport matplotlib.cm\nimport gc\nimport sys","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:09:42.917841Z","iopub.execute_input":"2022-11-17T14:09:42.918315Z","iopub.status.idle":"2022-11-17T14:09:42.928711Z","shell.execute_reply.started":"2022-11-17T14:09:42.918243Z","shell.execute_reply":"2022-11-17T14:09:42.927342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Empezamos por agregar el data set actualizado \"datos1\" que contiene el nuev directorio \"Datos_T2\" ordenado en el formato exigido por Keras y registrando los 'paths' que llevan hacia las carpetas Train y Test.","metadata":{}},{"cell_type":"code","source":"path_datos = '/kaggle/input/datos1/Datos_T2/'\npath_train = path_datos+'Train/'\npath_test = path_datos + 'Test/'","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:03:40.030505Z","iopub.execute_input":"2022-10-31T13:03:40.031106Z","iopub.status.idle":"2022-10-31T13:03:40.035201Z","shell.execute_reply.started":"2022-10-31T13:03:40.031069Z","shell.execute_reply":"2022-10-31T13:03:40.034568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_img1_train_CE = path_train + 'CE/' + os.listdir(path_train + 'CE/')[0]  #Obtener la trayectoria que lleva a la primera imagen\npath_img1_test_CE = path_test + 'CE/' + os.listdir(path_test + 'CE/')[0]\n\nprint(path_img1_train_CE)  #Revisar la trayectoria\nprint(path_img1_test_CE)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:03:41.284403Z","iopub.execute_input":"2022-10-31T13:03:41.285629Z","iopub.status.idle":"2022-10-31T13:03:41.529038Z","shell.execute_reply.started":"2022-10-31T13:03:41.285577Z","shell.execute_reply":"2022-10-31T13:03:41.528181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_train_CE = imread(path_img1_train_CE)  #Leer el archivo correspondiente a la primera img en la carpeta Train/CE\nimg_test_CE = imread(path_img1_test_CE)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:03:43.118876Z","iopub.execute_input":"2022-10-31T13:03:43.119258Z","iopub.status.idle":"2022-10-31T13:03:43.166703Z","shell.execute_reply.started":"2022-10-31T13:03:43.119227Z","shell.execute_reply":"2022-10-31T13:03:43.165172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Revisamos una ultima vez que la clasificacion de las imagenes en el directorio Datos_T2 sea la esperada:","metadata":{}},{"cell_type":"code","source":"print(img_train_CE.shape)#Averiguar dimensiones\nprint(img_test_CE.shape)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:04:04.465388Z","iopub.execute_input":"2022-10-31T13:04:04.465733Z","iopub.status.idle":"2022-10-31T13:04:04.471539Z","shell.execute_reply.started":"2022-10-31T13:04:04.465704Z","shell.execute_reply":"2022-10-31T13:04:04.470479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize =(18, 15))\nplt.imshow(img_train_CE, cmap='gray')","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:04:10.775023Z","iopub.execute_input":"2022-10-31T13:04:10.775387Z","iopub.status.idle":"2022-10-31T13:04:11.205644Z","shell.execute_reply.started":"2022-10-31T13:04:10.775352Z","shell.execute_reply":"2022-10-31T13:04:11.204526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize =(18, 15))\nplt.imshow(img_test_CE, cmap='gray')","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:04:13.344238Z","iopub.execute_input":"2022-10-31T13:04:13.344564Z","iopub.status.idle":"2022-10-31T13:04:13.67521Z","shell.execute_reply.started":"2022-10-31T13:04:13.344539Z","shell.execute_reply":"2022-10-31T13:04:13.674298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Super! Nuestra clasificación funcionó y las imagenes parecen venir en el nuevo formato comrpimido sin demasiada perdida de informacion! Ahora estamos listos para entrenar el modelo y probarlo!","metadata":{}},{"cell_type":"code","source":"print('No. de imgs. tipo CE en carpeta Train: ', len(os.listdir(path_train + 'CE/')))\nprint('No. de imgs. tipo LAA en carpeta Train: ', len(os.listdir(path_train + 'LAA/')))\nprint('No. de imgs. tipo CE en carpeta Test: ', len(os.listdir(path_test + 'CE/')))\nprint('No. de imgs. tipo LAA en carpeta Test: ', len(os.listdir(path_test + 'LAA/')))","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:04:16.91662Z","iopub.execute_input":"2022-10-31T13:04:16.916975Z","iopub.status.idle":"2022-10-31T13:04:17.012288Z","shell.execute_reply.started":"2022-10-31T13:04:16.916947Z","shell.execute_reply":"2022-10-31T13:04:17.011557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Todo parece bien, procedemos con el entrenamiento del modelo","metadata":{}},{"cell_type":"markdown","source":"Empezamos por revisar que las dimensiones de todas las fotos en el conjunto de datos 'Datos_T2' ahora sea más manejable.","metadata":{}},{"cell_type":"code","source":"dim1 = []  #ancho\ndim2 = []  #largo\nfor nombre_imagen in os.listdir(path_train + 'CE/'):    #obtener el ancho y largo de cada imagen\n    \n    img = imread(path_train + 'CE/' + nombre_imagen)\n    d1,d2 = img.shape\n    dim1.append(d1)\n    dim2.append(d2)\n    \nsns.jointplot(x = dim1, y = dim2)  #scatter plot de las dims","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:05:39.709587Z","iopub.execute_input":"2022-10-31T13:05:39.709937Z","iopub.status.idle":"2022-10-31T13:05:43.851084Z","shell.execute_reply.started":"2022-10-31T13:05:39.709908Z","shell.execute_reply":"2022-10-31T13:05:43.849908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Como vemos en el scatterplot anterior, las dimensiones se redujeron considerablemente! Antes estabamos cerca de los 30,000 pixeles o hasta más en cada dimension, y ahora estamos en imagenes clásicas de 1300X800!","metadata":{}},{"cell_type":"code","source":"print('Avg width', np.mean(dim1)) #Calcular las dimensiones promedio\nprint('Avg height', np.mean(dim2))","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:12:07.027324Z","iopub.execute_input":"2022-10-31T13:12:07.027658Z","iopub.status.idle":"2022-10-31T13:12:07.033861Z","shell.execute_reply.started":"2022-10-31T13:12:07.027632Z","shell.execute_reply":"2022-10-31T13:12:07.032482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"forma_imagen = (1230, 730, 1)  #Definimos las dimensiones estandar de las imagenes para la red.\n                               # Como es una imagen en blanco y negro, el ultimo elemento de la tripla es 1","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:12:50.268722Z","iopub.execute_input":"2022-10-31T13:12:50.269131Z","iopub.status.idle":"2022-10-31T13:12:50.273959Z","shell.execute_reply.started":"2022-10-31T13:12:50.269078Z","shell.execute_reply":"2022-10-31T13:12:50.273024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Preparacion de los parametros de entrenamiento de la CNN","metadata":{}},{"cell_type":"code","source":"#Mas metodos, funciones y librerias\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Activation, Dropout, Flatten, Dense, Conv2D, MaxPooling2D\nfrom tensorflow.keras.callbacks import EarlyStopping\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:04:36.451219Z","iopub.execute_input":"2022-11-17T15:04:36.451676Z","iopub.status.idle":"2022-11-17T15:04:36.46074Z","shell.execute_reply.started":"2022-11-17T15:04:36.451638Z","shell.execute_reply":"2022-11-17T15:04:36.459593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Empezamos creando el objeto ImageDataGenerator, el cual permite seleccionar, manipular y preparar las imagenes antes de ingresarlas a la red. Como solo estamos trabajando con imagenes en blanco y negro por el momento, la unica operacion que se les efectuara a las imagenes a medida que pasan por la red va a ser un 'flip' (rotación horizontal) ","metadata":{}},{"cell_type":"code","source":"gen_imagen = ImageDataGenerator(horizontal_flip = True)  #Creando el generador","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:15:55.755236Z","iopub.execute_input":"2022-10-31T13:15:55.756013Z","iopub.status.idle":"2022-10-31T13:15:55.761634Z","shell.execute_reply.started":"2022-10-31T13:15:55.755978Z","shell.execute_reply":"2022-10-31T13:15:55.760521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_imagen.flow_from_directory(path_train)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:18:30.884394Z","iopub.execute_input":"2022-10-31T13:18:30.884752Z","iopub.status.idle":"2022-10-31T13:18:31.002456Z","shell.execute_reply.started":"2022-10-31T13:18:30.884717Z","shell.execute_reply":"2022-10-31T13:18:31.001293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_imagen.flow_from_directory(path_test)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:18:32.543584Z","iopub.execute_input":"2022-10-31T13:18:32.543964Z","iopub.status.idle":"2022-10-31T13:18:32.654959Z","shell.execute_reply.started":"2022-10-31T13:18:32.543933Z","shell.execute_reply":"2022-10-31T13:18:32.654062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Como vemos en las 2 celdas anteriores, el directorio fue ordenado de manera apropiada ya que tanto en el caso de Train como de Test encontró 2 clases de imagenes (CE y LAA) correspondientes a las 2 tipo de coagulos que contiene la BD.","metadata":{}},{"cell_type":"markdown","source":"#### Agregando las distintas capas de la red.\n\nDado que este es el primer modelo que se entrena empezamos con una seleccion de hiperparametros clásica, y eventualmente los ajustaremos si consideramos que se pueden mejorar los resultados de rendimiento de la red.","metadata":{}},{"cell_type":"code","source":"modelo = Sequential()\n\nmodelo.add(Conv2D(filters = 32, kernel_size = (3, 3), input_shape = forma_imagen, activation = 'relu',))\nmodelo.add(MaxPooling2D(pool_size = (2, 2)))\n\nmodelo.add(Conv2D(filters = 64, kernel_size = (3, 3), input_shape = forma_imagen, activation = 'relu',))\nmodelo.add(MaxPooling2D(pool_size = (2, 2)))\n\nmodelo.add(Conv2D(filters = 64, kernel_size = (3, 3), input_shape = forma_imagen, activation = 'relu',))\nmodelo.add(MaxPooling2D(pool_size = (2, 2)))\n\n\nmodelo.add(Flatten())\n\n\nmodelo.add(Dense(128))\nmodelo.add(Activation('relu'))\n\nmodelo.add(Dropout(0.5))\n\n\nmodelo.add(Dense(1))\nmodelo.add(Activation('sigmoid'))\n\nmodelo.compile(loss = 'binary_crossentropy',\n              optimizer = 'adam',\n              metrics = ['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:27:56.265689Z","iopub.execute_input":"2022-10-31T13:27:56.266003Z","iopub.status.idle":"2022-10-31T13:27:56.90329Z","shell.execute_reply.started":"2022-10-31T13:27:56.265978Z","shell.execute_reply":"2022-10-31T13:27:56.902606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"modelo.summary()","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:28:01.01539Z","iopub.execute_input":"2022-10-31T13:28:01.015732Z","iopub.status.idle":"2022-10-31T13:28:01.022921Z","shell.execute_reply.started":"2022-10-31T13:28:01.015704Z","shell.execute_reply":"2022-10-31T13:28:01.021801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Como las imagenes aun son relativamente grandes escojemos un tamaño de tando de 32 imagenes. Esto implica que cada imagen es subdividida en 32 instancias durante el entrenamiento de la red.","metadata":{}},{"cell_type":"code","source":"tamanio_tanda = 32","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:31:13.981825Z","iopub.execute_input":"2022-10-31T13:31:13.982159Z","iopub.status.idle":"2022-10-31T13:31:13.986665Z","shell.execute_reply.started":"2022-10-31T13:31:13.982133Z","shell.execute_reply":"2022-10-31T13:31:13.985524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_gen_imagen = gen_imagen.flow_from_directory(path_train,\n                                               target_size = forma_imagen[:2],\n                                               color_mode = 'grayscale',\n                                               batch_size = tamanio_tanda,\n                                               class_mode = 'binary')\n\n","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:31:14.407837Z","iopub.execute_input":"2022-10-31T13:31:14.408218Z","iopub.status.idle":"2022-10-31T13:31:14.517369Z","shell.execute_reply.started":"2022-10-31T13:31:14.408188Z","shell.execute_reply":"2022-10-31T13:31:14.516691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_gen_imagen = gen_imagen.flow_from_directory(path_test,\n                                               target_size = forma_imagen[:2],\n                                               color_mode = 'grayscale',\n                                               batch_size = tamanio_tanda,\n                                               class_mode = 'binary',\n                                               shuffle = False)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:31:18.650955Z","iopub.execute_input":"2022-10-31T13:31:18.651366Z","iopub.status.idle":"2022-10-31T13:31:18.760714Z","shell.execute_reply.started":"2022-10-31T13:31:18.651336Z","shell.execute_reply":"2022-10-31T13:31:18.759791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_gen_imagen.class_indices","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:31:26.197541Z","iopub.execute_input":"2022-10-31T13:31:26.198134Z","iopub.status.idle":"2022-10-31T13:31:26.203596Z","shell.execute_reply.started":"2022-10-31T13:31:26.198084Z","shell.execute_reply":"2022-10-31T13:31:26.202525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Como vemos en la celda anterior, el generador reconoce 2 clases: CE y LAA. Justo como lo requiere la red!","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.models import load_model  #esta funcion permite cargar y guardar modelos en formato .h5","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:04:43.800098Z","iopub.execute_input":"2022-11-17T15:04:43.800578Z","iopub.status.idle":"2022-11-17T15:04:43.805977Z","shell.execute_reply.started":"2022-11-17T15:04:43.800541Z","shell.execute_reply":"2022-11-17T15:04:43.804761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Como el modelo se obtuvo de una corrida anterior del notebook, vamos a cargarlo\n#esto con el objetivo de no tener que realizar el entrenamiento de la red a cada\n#vez que se corra el notebook\n#modelo =load_model('/kaggle/input/clot-detector-2/clot_detector_2.h5')","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:08:10.86248Z","iopub.execute_input":"2022-10-31T14:08:10.86281Z","iopub.status.idle":"2022-10-31T14:08:20.760765Z","shell.execute_reply.started":"2022-10-31T14:08:10.862783Z","shell.execute_reply":"2022-10-31T14:08:20.759468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### NOTA! La siguiente celda ejecuta la instrucción de entrenar la red. Esta operación toma al rededor de 30 - 40 minutos con el procesador proporcionado por Kaggle.\n","metadata":{}},{"cell_type":"code","source":"resultados = modelo.fit_generator(train_gen_imagen,epochs = 10,\n                                  validation_data = test_gen_imagen,\n                                  callbacks = [detencion_temprana])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"modelo.save('clot_detector_2.h5') #esta instruccion sirve para guardar el modelo,\n                                   ","metadata":{"execution":{"iopub.status.busy":"2022-10-31T13:57:32.03453Z","iopub.execute_input":"2022-10-31T13:57:32.034879Z","iopub.status.idle":"2022-10-31T13:57:32.039334Z","shell.execute_reply.started":"2022-10-31T13:57:32.034852Z","shell.execute_reply":"2022-10-31T13:57:32.038542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"La instruccion que se encuentra a continuación permite obtener el historial de perdidas del modelo durante su entrenamiento. Como este procedimiento ya se realizó con anterioridad, ingresamos los datos obtenidos en dicha oportunidad.","metadata":{}},{"cell_type":"code","source":"#perdidas = { 'loss': [991.039612, 0.752938, 0.665514, 0.638687],\n#             'accuracy': [0.596026, 0.614238, 0.741722, 0.753311],\n#             'val_loss': [0.734882, 0.682291, 0.706149, 0.737379], \n#             'val_accuracy' : [0.553333, 0.720000, 0.713333, 0.706667]}\n\n#perdidas = pd.DataFrame(perdidas)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:14:37.246418Z","iopub.execute_input":"2022-10-31T14:14:37.247271Z","iopub.status.idle":"2022-10-31T14:14:37.252274Z","shell.execute_reply.started":"2022-10-31T14:14:37.247225Z","shell.execute_reply":"2022-10-31T14:14:37.251383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"perdidas = pd.DataFrame(modelo.history.history)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"perdidas.head()","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:14:40.161079Z","iopub.execute_input":"2022-10-31T14:14:40.161447Z","iopub.status.idle":"2022-10-31T14:14:40.1785Z","shell.execute_reply.started":"2022-10-31T14:14:40.161422Z","shell.execute_reply":"2022-10-31T14:14:40.177384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.style.use('seaborn-whitegrid')\n\nfig = plt.figure()\nfig = plt.figure(figsize =(7, 7))\n\nax = plt.axes()\nplt.ylim([0, 1])\nplt.xlim([0, 3.5])\n\nplt.plot(range(len(perdidas.loss)), perdidas.loss, '-r', label = 'loss')\nplt.plot(range(len(perdidas.accuracy)), perdidas.accuracy, '-b', label = 'accuracy')\n\nleg = ax.legend()\n\nplt.xlabel('iteracion No.')","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:14:49.708163Z","iopub.execute_input":"2022-10-31T14:14:49.708568Z","iopub.status.idle":"2022-10-31T14:14:49.898119Z","shell.execute_reply.started":"2022-10-31T14:14:49.708537Z","shell.execute_reply":"2022-10-31T14:14:49.897172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.style.use('seaborn-whitegrid')\n\nfig = plt.figure()\nfig = plt.figure(figsize =(7, 7))\n\nax = plt.axes()\nplt.ylim([0, 1])\nplt.xlim([0, 3.5])\n\nplt.plot(range(len(perdidas.val_loss)), perdidas.val_loss, '-r', label = 'val_loss')\nplt.plot(range(len(perdidas.val_accuracy)), perdidas.val_accuracy, '-b', label = 'val_accuracy')\n\nleg = ax.legend()\n\nplt.xlabel('iteracion No.')","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:14:53.94531Z","iopub.execute_input":"2022-10-31T14:14:53.945979Z","iopub.status.idle":"2022-10-31T14:14:54.325305Z","shell.execute_reply.started":"2022-10-31T14:14:53.945946Z","shell.execute_reply":"2022-10-31T14:14:54.324173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Como podemos ver, las gráficas anteriores indican una reduccion de la perdida a medida que se incrementa la precision del modelo. Además, cuando la perdida volvió a  aumentar, el sistema de detención temprana se activó en la 4ta época para evitar un sobreajuste a los datos.","metadata":{}},{"cell_type":"markdown","source":"### Evaluación del modelo\n\nAhora procedemos a evaluar el modelo mediante las metricas usuales de rendimiento, es decir la matriz de confusión y las metricas de f1, precision, etc...","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.preprocessing import image","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:31:11.816517Z","iopub.execute_input":"2022-10-31T14:31:11.816964Z","iopub.status.idle":"2022-10-31T14:31:11.82205Z","shell.execute_reply.started":"2022-10-31T14:31:11.816926Z","shell.execute_reply":"2022-10-31T14:31:11.821157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"probabilidades_pred = modelo.predict_generator(test_gen_imagen)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:31:13.811672Z","iopub.execute_input":"2022-10-31T14:31:13.812096Z","iopub.status.idle":"2022-10-31T14:31:40.471493Z","shell.execute_reply.started":"2022-10-31T14:31:13.812058Z","shell.execute_reply":"2022-10-31T14:31:40.469825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"probabilidades_pred  #estas son las predicciones realizadas por el modelo sobre el conjunto de prueba ","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:32:07.537547Z","iopub.execute_input":"2022-10-31T14:32:07.537968Z","iopub.status.idle":"2022-10-31T14:32:07.550057Z","shell.execute_reply.started":"2022-10-31T14:32:07.537939Z","shell.execute_reply":"2022-10-31T14:32:07.548761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_gen_imagen.classes  #estas son los 'labels' reales de cada imagen","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:32:33.797928Z","iopub.execute_input":"2022-10-31T14:32:33.798321Z","iopub.status.idle":"2022-10-31T14:32:33.807951Z","shell.execute_reply.started":"2022-10-31T14:32:33.79829Z","shell.execute_reply":"2022-10-31T14:32:33.806536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"La variable 'predicciones' que se define a continuacion devuelve True si la probabilidad es mayor a 0.5 (LAA) y False si es menor a 0.5 (CE).","metadata":{}},{"cell_type":"code","source":"predicciones = probabilidades_pred > 0.5  \npredicciones","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:40:28.518696Z","iopub.execute_input":"2022-10-31T14:40:28.51913Z","iopub.status.idle":"2022-10-31T14:40:28.52669Z","shell.execute_reply.started":"2022-10-31T14:40:28.519084Z","shell.execute_reply":"2022-10-31T14:40:28.52567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import classification_report, confusion_matrix","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:40:36.943363Z","iopub.execute_input":"2022-10-31T14:40:36.943863Z","iopub.status.idle":"2022-10-31T14:40:37.1169Z","shell.execute_reply.started":"2022-10-31T14:40:36.943813Z","shell.execute_reply":"2022-10-31T14:40:37.115556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Mostramos el reporte de clasificacion.","metadata":{}},{"cell_type":"code","source":"print(classification_report(test_gen_imagen.classes, predicciones))","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:40:59.683665Z","iopub.execute_input":"2022-10-31T14:40:59.684036Z","iopub.status.idle":"2022-10-31T14:40:59.697353Z","shell.execute_reply.started":"2022-10-31T14:40:59.684009Z","shell.execute_reply":"2022-10-31T14:40:59.6962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Como lo muestra el reporte, el modelo parece ser muy bueno para reconocer coágulos de tipo CE (0) pero tiene dificultades para erconocer coagulos tipo LAA (1).\nEsto probablemente se debe al hecho de que la mayoría de las imagenes dentro del dataset original eran efectivamente de tipo (CE)","metadata":{}},{"cell_type":"code","source":"sns.set(rc={'figure.figsize':(9,7)})\nsns.heatmap(confusion_matrix(test_gen_imagen.classes, predicciones), annot=True)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:44:20.949467Z","iopub.execute_input":"2022-10-31T14:44:20.949857Z","iopub.status.idle":"2022-10-31T14:44:21.179018Z","shell.execute_reply.started":"2022-10-31T14:44:20.949829Z","shell.execute_reply":"2022-10-31T14:44:21.177335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"La matriz de confusión confirma la sospecha anterior: tenemos 100 predicciones correectas de un total de 104 muestras tipo CE contra ninguna prediccion correcta para las muestras LAA","metadata":{}},{"cell_type":"markdown","source":"### Prueba de prediccion con una imagen.\n\nAhora hacemos una prueba de predecir una imagen (la primera imagen de prueba que se empleo)","metadata":{}},{"cell_type":"code","source":"img_de_prueba = image.load_img(path_img1_test_CE,grayscale = True, target_size = forma_imagen)\nimg_de_prueba  #mostrar la imagen ahora en formato (1230, 730, 1)","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:50:45.530432Z","iopub.execute_input":"2022-10-31T14:50:45.530956Z","iopub.status.idle":"2022-10-31T14:50:45.582056Z","shell.execute_reply.started":"2022-10-31T14:50:45.530921Z","shell.execute_reply":"2022-10-31T14:50:45.580854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_de_prueba = image.img_to_array(img_de_prueba)  #convertimos de tipo Image a un array de numpy","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:51:28.416998Z","iopub.execute_input":"2022-10-31T14:51:28.41745Z","iopub.status.idle":"2022-10-31T14:51:28.427049Z","shell.execute_reply.started":"2022-10-31T14:51:28.417408Z","shell.execute_reply":"2022-10-31T14:51:28.425928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_de_prueba.shape  #mostrar las dimensiones del array: todo bien!","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:51:40.266588Z","iopub.execute_input":"2022-10-31T14:51:40.267646Z","iopub.status.idle":"2022-10-31T14:51:40.272718Z","shell.execute_reply.started":"2022-10-31T14:51:40.26761Z","shell.execute_reply":"2022-10-31T14:51:40.272092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Expandimos las dimensiones para que sean consistentes con las requeridas por el modelo\nimg_de_prueba = np.expand_dims(img_de_prueba, axis = 0)  ","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:53:07.191962Z","iopub.execute_input":"2022-10-31T14:53:07.192711Z","iopub.status.idle":"2022-10-31T14:53:07.197883Z","shell.execute_reply.started":"2022-10-31T14:53:07.19266Z","shell.execute_reply":"2022-10-31T14:53:07.19702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_de_prueba.shape #listo!","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:53:17.006024Z","iopub.execute_input":"2022-10-31T14:53:17.006395Z","iopub.status.idle":"2022-10-31T14:53:17.012915Z","shell.execute_reply.started":"2022-10-31T14:53:17.006366Z","shell.execute_reply":"2022-10-31T14:53:17.011986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"modelo.predict(img_de_prueba)  #Ahra efectuamos la prediccion","metadata":{"execution":{"iopub.status.busy":"2022-10-31T14:53:33.480329Z","iopub.execute_input":"2022-10-31T14:53:33.480709Z","iopub.status.idle":"2022-10-31T14:53:33.834236Z","shell.execute_reply.started":"2022-10-31T14:53:33.480677Z","shell.execute_reply":"2022-10-31T14:53:33.833398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"El modelo nos indica que la imagen seleccionada es de tipo CE ya que el valor de la prediccion de 0.45 es más cercano a 0 que a 1. \n","metadata":{}},{"cell_type":"markdown","source":"## Entrega final (17/11/22)\n\nEn vista de la entrega final, proponemos entrenar un 2do modelo con el objetivo de comparar su desempeño con el modelo anterior. Para esto, vamos a emplear una version similar a la red convolucional Le-Net 5 implementada para el Laboratorio 10, ya que esta tuvo excelentes resultados con conjuntos de imagenes de baja resolución. Esto nos permitirá determinar hasta qué punto es posible comprimir imagenes médicas manteniendo resultados adecuados. ","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"# preparar las trayectorias hacia los datos\n\npath_datos = '/kaggle/input/datos1/Datos_T2/'\npath_train = path_datos+'Train/'\npath_test = path_datos + 'Test/'\n","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:14:42.330921Z","iopub.execute_input":"2022-11-17T14:14:42.331381Z","iopub.status.idle":"2022-11-17T14:14:42.336699Z","shell.execute_reply.started":"2022-11-17T14:14:42.331343Z","shell.execute_reply":"2022-11-17T14:14:42.33547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### 1era tarea: Preparar los datos en array de numpy.\n\nPara la red que se tiene en mente, es necesario que los datos vengan en un formato un poco distinto. Es decir que en vez de .png's necesitamos un array donde cada foto es representada por su array correspondiente. Por lo tanto empezamos por allí.\n","metadata":{}},{"cell_type":"code","source":"# datos de entrenamiento\nX_train = []\ny_train = []\n\nfor png in os.listdir(path_train + 'CE/'):  #obtener imagenes tipo CE de 'train'\n    \n    path = path_train + 'CE/' + png  #obtener el path\n    img = Image.open(path)  #instanciar como objeto de la clase Image de pillow\n    img = img.resize((32, 32))  #estandarizar el tamaño de las imagenes\n    img = np.array(img)     #convertir a array de numpy\n    X_train.append(img)     #agregar el arraya la lista de datos de entrenamiento\n    y_train.append(0)       #agregar la etiqueta correcta\n    \nfor png in os.listdir(path_train + 'LAA/'):  #obtener imagenes tipo LAA de 'train'\n    \n    path = path_train + 'LAA/' + png  #obtener el path\n    img = Image.open(path)  #instanciar como objeto de la clase Image de pillow\n    img = img.resize((32, 32))  #estandarizar el tamaño de las imagenes\n    img = np.array(img)     #convertir a array de numpy\n    X_train.append(img)     #agregar el arraya la lista de datos de entrenamiento\n    y_train.append(1)       #agregar la etiqueta correcta\n    \n\nX_train = np.array(X_train)\ny_train = np.array(y_train)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:14:43.497723Z","iopub.execute_input":"2022-11-17T14:14:43.498814Z","iopub.status.idle":"2022-11-17T14:14:54.213479Z","shell.execute_reply.started":"2022-11-17T14:14:43.498774Z","shell.execute_reply":"2022-11-17T14:14:54.212262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# datos de prueba\nX_test = []\ny_test = []\n\nfor png in os.listdir(path_test + 'CE/'):  #obtener imagenes tipo CE de 'train'\n    \n    path = path_test + 'CE/' + png  #obtener el path\n    img = Image.open(path)  #instanciar como objeto de la clase Image de pillow\n    img = img.resize((32,32))  #estandarizar el tamaño de las imagenes\n    img = np.array(img)     #convertir a array de numpy\n    X_test.append(img)     #agregar el arraya la lista de datos de entrenamiento\n    y_test.append(0)       #agregar la etiqueta correcta\n    \nfor png in os.listdir(path_test + 'LAA/'):  #obtener imagenes tipo LAA de 'train'\n    \n    path = path_test + 'LAA/' + png  #obtener el path\n    img = Image.open(path)  #instanciar como objeto de la clase Image de pillow\n    img = img.resize((32, 32))  #estandarizar el tamaño de las imagenes\n    img = np.array(img)     #convertir a array de numpy\n    X_test.append(img)     #agregar el arraya la lista de datos de entrenamiento\n    y_test.append(1)       #agregar la etiqueta correcta\n    \n\nX_test = np.array(X_test)\ny_test = np.array(y_test)","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:14:54.215166Z","iopub.execute_input":"2022-11-17T14:14:54.215547Z","iopub.status.idle":"2022-11-17T14:14:56.670222Z","shell.execute_reply.started":"2022-11-17T14:14:54.215515Z","shell.execute_reply":"2022-11-17T14:14:56.669148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"¡Listo!\nAhora debemos \"desordenar\" los arrays, ya que siguen con el orden anterior","metadata":{"execution":{"iopub.status.busy":"2022-11-17T03:45:06.212082Z","iopub.execute_input":"2022-11-17T03:45:06.212611Z","iopub.status.idle":"2022-11-17T03:45:06.223299Z","shell.execute_reply.started":"2022-11-17T03:45:06.212569Z","shell.execute_reply":"2022-11-17T03:45:06.221759Z"}}},{"cell_type":"code","source":"# primero establecemos una permutacion aleatoria de los elementos\nnp.random.seed(314159)\n\npermutacion = np.random.permutation(len(X_train))\n\n# basta seleccionar los datos de acuerdo al orden dictado por la permutacion!\nX_train2 = X_train[permutacion]\ny_train2 = y_train[permutacion]\n\n#Idem para los datos de validacion\n\npermutacion2 = np.random.permutation(len(X_test))\n\nX_test2 = X_test[permutacion2]\ny_test2 = y_test[permutacion2]","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:19:16.860309Z","iopub.execute_input":"2022-11-17T14:19:16.861013Z","iopub.status.idle":"2022-11-17T14:19:16.867878Z","shell.execute_reply.started":"2022-11-17T14:19:16.860976Z","shell.execute_reply":"2022-11-17T14:19:16.866941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train2[-25:]","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:19:19.451329Z","iopub.execute_input":"2022-11-17T14:19:19.452186Z","iopub.status.idle":"2022-11-17T14:19:19.462367Z","shell.execute_reply.started":"2022-11-17T14:19:19.452152Z","shell.execute_reply":"2022-11-17T14:19:19.461319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ahora, solo falta normalizar los datos y estaremos listos para entrenar la red.","metadata":{}},{"cell_type":"code","source":"#importamos el metodo de keras correspondiente\nfrom sklearn import preprocessing","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:19:26.398324Z","iopub.execute_input":"2022-11-17T14:19:26.399402Z","iopub.status.idle":"2022-11-17T14:19:26.463315Z","shell.execute_reply.started":"2022-11-17T14:19:26.399346Z","shell.execute_reply":"2022-11-17T14:19:26.462375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#definimos una funcion para normalizar listas o arrays\n\ndef normalizar(array_imgs):\n\n  nueva_lista = []\n\n  for i in array_imgs:\n\n    normalizado = preprocessing.normalize(i)\n\n    nueva_lista.append(normalizado)\n\n  return np.array(nueva_lista, dtype='float32')","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:19:27.692631Z","iopub.execute_input":"2022-11-17T14:19:27.693032Z","iopub.status.idle":"2022-11-17T14:19:27.699535Z","shell.execute_reply.started":"2022-11-17T14:19:27.693001Z","shell.execute_reply":"2022-11-17T14:19:27.698298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#normalizar \n\nX_train2 = normalizar(X_train2)\nX_test2 = normalizar(X_test2)","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:19:33.257272Z","iopub.execute_input":"2022-11-17T14:19:33.257977Z","iopub.status.idle":"2022-11-17T14:19:33.327195Z","shell.execute_reply.started":"2022-11-17T14:19:33.257929Z","shell.execute_reply":"2022-11-17T14:19:33.326032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#cambiamos los vectores de etiquetas a float 32\ny_train2 = y_train2.astype('float32')\ny_test2 = y_test2.astype('float32')","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:20:07.612663Z","iopub.execute_input":"2022-11-17T14:20:07.613104Z","iopub.status.idle":"2022-11-17T14:20:07.618476Z","shell.execute_reply.started":"2022-11-17T14:20:07.613065Z","shell.execute_reply":"2022-11-17T14:20:07.617317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train2 = X_train2.reshape(604, 32, 32, 1)  #reajustamos el tamño de entrada de los datos\nX_test2 = X_test2.reshape(150,32,32,1)","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:20:31.650604Z","iopub.execute_input":"2022-11-17T14:20:31.651008Z","iopub.status.idle":"2022-11-17T14:20:31.655792Z","shell.execute_reply.started":"2022-11-17T14:20:31.650972Z","shell.execute_reply":"2022-11-17T14:20:31.654638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test2.shape #funciona!","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:20:34.764626Z","iopub.execute_input":"2022-11-17T14:20:34.765037Z","iopub.status.idle":"2022-11-17T14:20:34.772455Z","shell.execute_reply.started":"2022-11-17T14:20:34.765002Z","shell.execute_reply":"2022-11-17T14:20:34.771199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Entrenamiento 2da red","metadata":{}},{"cell_type":"code","source":"#reimportamos las librerias necesarias\nimport tensorflow as tf\nfrom tensorflow import keras\nimport numpy as np","metadata":{"execution":{"iopub.status.busy":"2022-11-17T14:21:05.346003Z","iopub.execute_input":"2022-11-17T14:21:05.346429Z","iopub.status.idle":"2022-11-17T14:21:10.310211Z","shell.execute_reply.started":"2022-11-17T14:21:05.346394Z","shell.execute_reply":"2022-11-17T14:21:10.309106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Crear el modelo\n\nmodelo_leNet = keras.models.Sequential()\n\n#PASO 1:\nmodelo_leNet.add(keras.layers.Conv2D(filters = 6, kernel_size=10, input_shape =(32,32,1),  activation = 'relu'))\nmodelo_leNet.add(keras.layers.AveragePooling2D())\n\n#PASO 2:\nmodelo_leNet.add(keras.layers.Conv2D(filters = 16, kernel_size=10, activation = 'relu'))\nmodelo_leNet.add(keras.layers.AveragePooling2D())\n\n#PASO 3:\nmodelo_leNet.add(keras.layers.Flatten())\n\n#PASO 4:\nmodelo_leNet.add(keras.layers.Dense(300, activation='relu'))\n\n#PASO 5:\nmodelo_leNet.add(keras.layers.Dense(100, activation='relu'))\n\n#PASO 6:  \nmodelo_leNet.add(keras.layers.Dense(1, activation = 'sigmoid')) \n","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:29:08.139697Z","iopub.execute_input":"2022-11-17T15:29:08.140941Z","iopub.status.idle":"2022-11-17T15:29:08.209548Z","shell.execute_reply.started":"2022-11-17T15:29:08.140886Z","shell.execute_reply":"2022-11-17T15:29:08.208304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"modelo_leNet.compile(loss = 'binary_crossentropy',\n              optimizer = 'adam',\n              metrics = ['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:29:10.146173Z","iopub.execute_input":"2022-11-17T15:29:10.146646Z","iopub.status.idle":"2022-11-17T15:29:10.157416Z","shell.execute_reply.started":"2022-11-17T15:29:10.146606Z","shell.execute_reply":"2022-11-17T15:29:10.156413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"modelo_leNet.summary()   #Revisamo las capas del modelo. Todo se ve bien!","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:29:12.238958Z","iopub.execute_input":"2022-11-17T15:29:12.239367Z","iopub.status.idle":"2022-11-17T15:29:12.245748Z","shell.execute_reply.started":"2022-11-17T15:29:12.239332Z","shell.execute_reply":"2022-11-17T15:29:12.244884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# establecemos una detencion temprana para evitar overfitting\nearly_stopping = EarlyStopping(monitor = 'val_loss', patience = 5, verbose = 1)","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:29:19.486167Z","iopub.execute_input":"2022-11-17T15:29:19.486824Z","iopub.status.idle":"2022-11-17T15:29:19.490917Z","shell.execute_reply.started":"2022-11-17T15:29:19.486787Z","shell.execute_reply":"2022-11-17T15:29:19.490006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"modelo_leNet.fit(X_train2, y_train2,epochs=100, \n                 validation_data=(X_test2, y_test2), \n                 verbose = 1, \n                 callbacks = [early_stopping]\n                )  ","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:29:31.897647Z","iopub.execute_input":"2022-11-17T15:29:31.898073Z","iopub.status.idle":"2022-11-17T15:29:35.641078Z","shell.execute_reply.started":"2022-11-17T15:29:31.898036Z","shell.execute_reply":"2022-11-17T15:29:35.639856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### ¡Vaya! El resultado parece ser casi equivalente al del modelo anterior!\n\nProcedemos a evaluar las métricas del modelo.","metadata":{"execution":{"iopub.status.busy":"2022-11-17T05:50:09.466479Z","iopub.execute_input":"2022-11-17T05:50:09.467029Z","iopub.status.idle":"2022-11-17T05:50:09.477998Z","shell.execute_reply.started":"2022-11-17T05:50:09.466988Z","shell.execute_reply":"2022-11-17T05:50:09.476386Z"}}},{"cell_type":"code","source":"historial = pd.DataFrame(modelo_leNet.history.history)","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:31:12.591331Z","iopub.execute_input":"2022-11-17T15:31:12.591799Z","iopub.status.idle":"2022-11-17T15:31:12.598686Z","shell.execute_reply.started":"2022-11-17T15:31:12.591758Z","shell.execute_reply":"2022-11-17T15:31:12.59745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generar gráfica de Exactitud y de Entrenamiento y Validación\nhistorial[['accuracy', 'val_accuracy']].plot()","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:31:13.119123Z","iopub.execute_input":"2022-11-17T15:31:13.119785Z","iopub.status.idle":"2022-11-17T15:31:54.782806Z","shell.execute_reply.started":"2022-11-17T15:31:13.119746Z","shell.execute_reply":"2022-11-17T15:31:54.781271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generar gráfica de Pérdida de Entrenamiento y Validación\nhistorial[['loss', 'val_loss']].plot()","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:31:54.784972Z","iopub.execute_input":"2022-11-17T15:31:54.785771Z","iopub.status.idle":"2022-11-17T15:31:55.028119Z","shell.execute_reply.started":"2022-11-17T15:31:54.785711Z","shell.execute_reply":"2022-11-17T15:31:55.027206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vemos que las gráficas anteriores indican una reduccion en la perdida tanto respecto a conjunto de validación como el conjunto de entrenamiento. \n\nAl mismo tiempo hay un incremento en la precisión, a medida que la precisión respecto al conjunto de validación no decae. \n\nEsto indica que no estamos frente a un caso de overfitting, por lo tanto guardamos el modelo.\n","metadata":{}},{"cell_type":"code","source":"#guardar el modelo\nmodelo_leNet.save('clot_detector_leNet.h5') ","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:36:58.11728Z","iopub.execute_input":"2022-11-17T15:36:58.117734Z","iopub.status.idle":"2022-11-17T15:36:58.158171Z","shell.execute_reply.started":"2022-11-17T15:36:58.117701Z","shell.execute_reply":"2022-11-17T15:36:58.157048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ahora, continuamos evaluando las métricas de rendimiento del modelo.","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:50:18.137736Z","iopub.execute_input":"2022-11-17T15:50:18.138995Z","iopub.status.idle":"2022-11-17T15:50:18.179973Z","shell.execute_reply.started":"2022-11-17T15:50:18.138953Z","shell.execute_reply":"2022-11-17T15:50:18.178849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#revisar las clases predichas\nclases = np.argmax(modelo_leNet.predict(X_test2),axis=1)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:51:23.545876Z","iopub.execute_input":"2022-11-17T15:51:23.546322Z","iopub.status.idle":"2022-11-17T15:51:23.618419Z","shell.execute_reply.started":"2022-11-17T15:51:23.546281Z","shell.execute_reply":"2022-11-17T15:51:23.617335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#matriz de confusión\nm_conf = confusion_matrix(y_test2, clases)\n\nplt.figure(figsize=(15,15))\nsns.heatmap(m_conf, annot=True, fmt='d')\nplt.xlabel('Predicción')\nplt.ylabel('Etiqueta')","metadata":{"execution":{"iopub.status.busy":"2022-11-17T15:52:07.915319Z","iopub.execute_input":"2022-11-17T15:52:07.91571Z","iopub.status.idle":"2022-11-17T15:52:08.210492Z","shell.execute_reply.started":"2022-11-17T15:52:07.915677Z","shell.execute_reply":"2022-11-17T15:52:08.209292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Una vez más tenemos un modelo cesgado, es decir muy adecuado para predecir coágulos de tipo CE pero bastante malo para predecir los de tipo LAA.","metadata":{}},{"cell_type":"markdown","source":"### Conclusiónes\n\n1. El primer modelo presentaba un nivel de precisión levemente superior al 2do.\n\n2. Sin embargo, el 2do modelo tiene la ventaja de haber sido entrenado con imagenes mucho más pequeñas.\n\n3. El 2do modelo tiene un peso de 600 KB en vez de 1.5 GB,\n\n4. La fuerte reducción en el peso del 2do modelo a comparación del 1ero permite justificar el sacrificio de 0.02 puntos de precisión. \n\n5. La puesta en producción del 2do modelo debería de resultar much más sencilla debido a que peso mucho \n   menos.","metadata":{}}]}