{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":99552,"databundleVersionId":13851420,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Chunk 0 — Imports\n\nLoads the libraries we need.","metadata":{}},{"cell_type":"code","source":"from glob import glob\nimport os, json, random\nimport numpy as np\nimport pandas as pd\nimport pydicom\nimport matplotlib.pyplot as plt\n\nfrom ipywidgets import interact, widgets, fixed\nfrom pathlib import Path\nimport os, random, numpy as np, pandas as pd, pydicom, cv2, gc\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T02:55:15.748692Z","iopub.execute_input":"2025-10-05T02:55:15.749247Z","iopub.status.idle":"2025-10-05T02:55:17.321264Z","shell.execute_reply.started":"2025-10-05T02:55:15.749221Z","shell.execute_reply":"2025-10-05T02:55:17.320376Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Chunk 1 - Dataset locations & reading the training table\n\nPoints to the RSNA competition folders on Kaggle and loads the main label table (`train.csv`). Each row in train.csv corresponds to one imaging series (i.e., one folder of DICOM slices) and includes the target (Aneurysm Present) plus some metadata.\n\n`train.csv` — the main table you’ll train with. One row per series (via `SeriesInstanceUID`). Includes:\n\n- Demographics: `PatientAge`, `PatientSex`.\n\n- `Modality (CTA/MRA/MRI...)`.\n\n- 14 labels:\n\n    - `Aneurysm Present` (binary, “is there any aneurysm in this series?”).\n\n    - 13 location columns (binary each), e.g. “Left Middle Cerebral Artery”, “Anterior Communicating Artery”, etc.\n\n**This is the primary file for training/evaluating a classifier.**\n\n- For a simple baseline, we start with the single target Aneurysm Present.\n\n- Later, we can switch to multi-label training to predict the 13 locations as well.\n\n\n`train_localizers.csv`: Slice-level hints for where an aneurysm is. These are points, not segmentations. Each row has:\n\n- `SeriesInstanceUID`: which 3D series (matches `train.csv`).\n\n- `SOPInstanceUID`: which slice inside that series.\n\n- `coordinates`: `{\"x\": ..., \"y\": ...}` pixel location (near aneurysm center).\n\n- `location`: A text description of the aneurysm's location.\n\n\n- Find series folder via `SeriesInstanceUID`.\n\n- Find the exact slice via `SOPInstanceUID`.\n\n- Plot (x, y) on that slice.\n\n\nFor a simple 3D CNN demo: use train.csv (series-level labels).\n\nUse train_localizers.csv as an optional add-on for visuals or later detection work.","metadata":{}},{"cell_type":"code","source":"# Kaggle dataset root (mounted read-only in Kaggle notebooks)\nBASE = Path(\"/kaggle/input/rsna-intracranial-aneurysm-detection\")\n\n# Where each DICOM series lives: series/<SeriesInstanceUID>/*.dcm\nSERIES_DIR = BASE / \"series\" \n\n# CSV files provided by the competition\nTRAIN_CSV  = BASE / \"train.csv\"             # main labels per SeriesInstanceUID\n# LOCAL_CSV  = BASE / \"train_localizers.csv\"  # per-slice point annotations\n\n# Load the main training table\ntrain_df = pd.read_csv(TRAIN_CSV)\n\n# Quick peek (optional in class)\nprint(\"Rows in train.csv:\", len(train_df))\ntrain_df.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T02:55:20.427795Z","iopub.execute_input":"2025-10-05T02:55:20.428593Z","iopub.status.idle":"2025-10-05T02:55:20.497826Z","shell.execute_reply.started":"2025-10-05T02:55:20.428552Z","shell.execute_reply":"2025-10-05T02:55:20.497246Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Each row = one imaging series, identified by a unique SeriesInstanceUID.\n\n- A series is a 3D stack of slices from one scan session and one modality (CTA, MRA, MRI T1post, MRI T2, etc.).","metadata":{}},{"cell_type":"markdown","source":"## Chunk 2a — Quick EDA \n\nShows total sample count, how many “aneurysm present,” modality mix, and a few simple plots (age histogram, modality pie, yes/no bar). This gives us a feel for the data distribution.","metadata":{}},{"cell_type":"code","source":"# Basic counts\nprint(\"Total samples:\", len(train_df))\nprint(\"Aneurysm Present positive rate:\", round(train_df['Aneurysm Present'].mean()*100, 2), \"%\")\nprint(\"\\nModality counts:\")\nprint(train_df['Modality'].value_counts())\n\n# Age distribution\nages = pd.to_numeric(train_df['PatientAge'], errors='coerce').dropna()\nplt.figure(figsize=(6,4))\nplt.hist(ages, bins=20)\nplt.xlabel('Age'); plt.ylabel('Count'); plt.title('Patient Age Distribution')\nplt.show()\n\n# Aneurysm presence (0/1)\npos = train_df['Aneurysm Present'].value_counts().sort_index()\nplt.figure(figsize=(5,4))\nplt.bar(['No (0)', 'Yes (1)'], pos.values)\nplt.title('Aneurysm Present Count'); plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T02:55:23.505424Z","iopub.execute_input":"2025-10-05T02:55:23.505719Z","iopub.status.idle":"2025-10-05T02:55:23.971206Z","shell.execute_reply.started":"2025-10-05T02:55:23.505697Z","shell.execute_reply":"2025-10-05T02:55:23.970514Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Chunk 2b — Minimal interactive viewer (pick a series, slide through slices)","metadata":{}},{"cell_type":"code","source":"# --- Minimal slider viewer: one dropdown + one slider ---\nfrom ipywidgets import Dropdown, IntSlider, VBox\nfrom IPython.display import display, clear_output\nimport functools, pydicom, matplotlib.pyplot as plt\n\ndef list_series_uids(limit=50):\n    uids = [uid for uid in train_df['SeriesInstanceUID'] if (SERIES_DIR/str(uid)).exists()]\n    uids = sorted(uids)[:limit]    # remove shuffle, keep deterministic order\n    return uids\n\n# Read a series and cache the result to avoid repeated disk reads\n@functools.lru_cache(maxsize=256)\ndef read_series_stack(series_uid: str):\n    series_path = SERIES_DIR / series_uid\n    files = list(series_path.glob(\"*.dcm\"))\n\n    # Sort by InstanceNumber (good enough for classroom viewing)\n    def inst_no(p):\n        try:\n            d = pydicom.dcmread(str(p), stop_before_pixels=True, force=True)\n            return int(getattr(d, 'InstanceNumber', 0))\n        except:\n            return 0\n    files = sorted(files, key=inst_no)\n\n    # Load pixel arrays into a list of 2D slices\n    imgs = []\n    for p in files:\n        try:\n            d = pydicom.dcmread(str(p), force=True)\n            imgs.append(d.pixel_array)\n        except:\n            pass\n    return imgs  # list of 2D numpy arrays\n\n# UI controls: dropdown for series, slider for slice index\nuids = list_series_uids(60)\nuid_dd = Dropdown(options=uids, description='Series')\nsl     = IntSlider(description='Slice', min=1, max=1, step=1, value=1)\n\nout = widgets.Output()\n\ndef refresh_plot(*_):\n    \"\"\"Redraw the selected slice for the selected series.\"\"\"\n    with out:\n        out.clear_output(wait=True)\n        imgs = read_series_stack(str(uid_dd.value))\n        if not imgs:\n            print(\"No readable slices for this series.\")\n            return\n\n        # Keep slider range in sync with number of slices\n        sl.max = max(1, len(imgs))\n        idx = min(sl.value - 1, len(imgs) - 1)  # convert to 0-based index\n\n        # Display the current slice\n        plt.figure(figsize=(5,5))\n        plt.imshow(imgs[idx], cmap='gray')\n        plt.title(f'{uid_dd.value}\\nSlice {idx+1}/{len(imgs)}')\n        plt.axis('off')\n        plt.show()\n\n# Redraw when series or slice changes\nuid_dd.observe(lambda ch: refresh_plot(), names='value')\nsl.observe(lambda ch: refresh_plot(), names='value')\n\n# Initial draw\nrefresh_plot()\ndisplay(VBox([uid_dd, sl, out]))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T02:55:31.041279Z","iopub.execute_input":"2025-10-05T02:55:31.041572Z","iopub.status.idle":"2025-10-05T02:55:42.57241Z","shell.execute_reply.started":"2025-10-05T02:55:31.041549Z","shell.execute_reply":"2025-10-05T02:55:42.571842Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Chunk 3 — Load data: Build a tiny training subset (keep it fast and balanced)\n\nFrom the big CSV, pick a small sample (N=60), try to balance positives/negatives, then split into train/val. This keeps our demo model runs quick.","metadata":{}},{"cell_type":"code","source":"# Small fixed voxel size for memory & speed (D, H, W)\nDEPTH, HEIGHT, WIDTH = 48, 96, 96\nBATCH_SIZE = 4\nEPOCHS = 2\nN_SERIES = 60\nSEED = 1337\nrandom.seed(SEED); np.random.seed(SEED)\n\ndf = pd.read_csv(TRAIN_CSV)[[\"SeriesInstanceUID\",\"Aneurysm Present\"]].dropna()\ndf[\"Aneurysm Present\"] = df[\"Aneurysm Present\"].astype(int)\ndf = df[df[\"SeriesInstanceUID\"].apply(lambda s: (SERIES_DIR/s).exists())].reset_index(drop=True)\n\n# Try to balance classes within the small sample\npos = df[df[\"Aneurysm Present\"]==1].sample(min(N_SERIES//2, df[\"Aneurysm Present\"].sum()), \n                                           random_state=SEED)\nneg = df[df[\"Aneurysm Present\"]==0].sample(N_SERIES-len(pos), random_state=SEED)\ndf_small = pd.concat([pos,neg]).sample(frac=1, random_state=SEED).reset_index(drop=True)\n\n# Simple split \nval_n = max(8, int(0.2*len(df_small)))\ndf_val = df_small.iloc[:val_n].reset_index(drop=True)\ndf_tr  = df_small.iloc[val_n:].reset_index(drop=True)\n\nlen(df_tr), len(df_val)  # helps sanity-check: e.g., 48 train / 12 val","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-10-05T02:55:51.088544Z","iopub.execute_input":"2025-10-05T02:55:51.088783Z","iopub.status.idle":"2025-10-05T02:55:51.168494Z","shell.execute_reply.started":"2025-10-05T02:55:51.088764Z","shell.execute_reply":"2025-10-05T02:55:51.167777Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Chunk 4 — DICOM to 3D volume (normalize + resize + depth sampling)\n\nGiven a series folder, read all slices, sort them, convert to a 3D array, clip extreme values, resize each slice to common Height×Witdth, and sample to a fixed depth. Returns a (D,H,W,1) block in [0,1].","metadata":{}},{"cell_type":"markdown","source":"## Chunk 5 — TensorFlow setup + tf.data input pipeline\n\n- Makes sure TensorFlow sees the GPU\n- Builds a tf.data pipeline that calls our Python loader, batches, and prefetches.","metadata":{}},{"cell_type":"code","source":"def _sort_key(d):\n    \"\"\"Sort slices. Prefer z-position; fall back to InstanceNumber.\"\"\"\n    z = None\n    if hasattr(d, \"ImagePositionPatient\") and len(d.ImagePositionPatient)>=3:\n        try:\n            z = float(d.ImagePositionPatient[2])\n        except:\n            pass\n    if z is not None:\n        return (0, z)\n    return (1, float(getattr(d, \"InstanceNumber\", 0)))\n\ndef load_series(series_id, target_shape=(DEPTH,HEIGHT,WIDTH), clip=(1,99)):\n    \"\"\"Read a series and turn it into a fixed-size, normalized 3D volume (D,H,W,1).\"\"\"\n    sdir = SERIES_DIR/series_id\n    files = [p for p in sdir.glob(\"*.dcm\")]\n    if not files:\n        return np.zeros((*target_shape,1), np.float32)\n\n    # Read headers, ensure pixels exist\n    headers = []\n    for fp in files:\n        try:\n            d = pydicom.dcmread(str(fp), force=True, stop_before_pixels=False)\n            if hasattr(d, \"PixelData\"):\n                headers.append(d)\n        except:\n            pass\n    if not headers:\n        return np.zeros((*target_shape,1), np.float32)\n\n    # Sort slices (z if available, otherwise instance number)\n    headers.sort(key=_sort_key)\n\n    # Extract pixels; handle multi-frame DICOMs too\n    slices = []\n    for d in headers:\n        try:\n            arr = d.pixel_array.astype(np.float32)\n            slope = float(getattr(d,\"RescaleSlope\",1.0))\n            inter = float(getattr(d,\"RescaleIntercept\",0.0))\n            arr = arr*slope + inter\n            if arr.ndim==2:\n                slices.append(arr)\n            elif arr.ndim==3:\n                # handle (F,H,W) or (H,W,F)\n                if arr.shape[0] < 8 and arr.shape[-1] > arr.shape[0]:\n                    arr = np.moveaxis(arr, -1, 0)\n                for k in range(arr.shape[0]):\n                    slices.append(arr[k])\n        except:\n            continue\n    if not slices:\n        return np.zeros((*target_shape,1), np.float32)\n\n    vol = np.stack(slices,0)  # (D0,H0,W0)\n\n    # Robust intensity clamp (1–99th percentile), then normalize to [0,1]\n    lo, hi = np.percentile(vol, clip)\n    vol = np.clip(vol, lo, hi)\n    D0,H0,W0 = vol.shape\n\n    # Resize each slice to target H×W\n    resized = np.zeros((D0, target_shape[1], target_shape[2]), np.float32)\n    for i in range(D0):\n        resized[i] = cv2.resize(vol[i], (target_shape[2], target_shape[1]), interpolation=cv2.INTER_LINEAR)\n\n    # Depth-uniform sampling to target D\n    idx = np.linspace(0, D0-1, target_shape[0]).astype(int)\n    vol = resized[idx]\n\n    # Normalize to [0,1]\n    mn, mx = vol.min(), vol.max()\n    if mx>mn:\n        vol = (vol-mn)/(mx-mn)\n\n    return vol[...,None].astype(np.float32)  # (D,H,W,1)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T02:55:56.729821Z","iopub.execute_input":"2025-10-05T02:55:56.730104Z","iopub.status.idle":"2025-10-05T02:55:56.740848Z","shell.execute_reply.started":"2025-10-05T02:55:56.730082Z","shell.execute_reply":"2025-10-05T02:55:56.740035Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\n# Make GPU memory growth \"on demand\" (avoid grabbing all VRAM at once)\ngpus = tf.config.list_physical_devices('GPU')\nfor g in gpus:\n    try:\n        tf.config.experimental.set_memory_growth(g, True)\n    except:\n        pass\n# print(\"GPUs visible to TF:\", gpus)\n\n# Optional mixed precision for speed (safe because output Dense is float32)\nfrom tensorflow.keras import mixed_precision\nmixed_precision.set_global_policy(\"mixed_float16\")\n\ndef make_ds(df_part, shuffle):\n    \"\"\"Build a tf.data dataset that yields (volume, label).\"\"\"\n    sids = df_part[\"SeriesInstanceUID\"].astype(str).values\n    ys   = df_part[\"Aneurysm Present\"].astype(np.float32).values\n\n    def py_load(sid_bytes):\n        sid = sid_bytes.decode()\n        x = load_series(sid)                 # (D,H,W,1) float32 in [0,1]\n        return x.astype(np.float32)\n\n    def wrapper(sid, y):\n        # Bridge Python/Numpy to TF tensors (shape is known)\n        vol = tf.numpy_function(py_load, [sid], tf.float32)\n        vol.set_shape((DEPTH,HEIGHT,WIDTH,1))\n        return vol, y\n\n    ds = tf.data.Dataset.from_tensor_slices((sids, ys))\n    if shuffle:\n        ds = ds.shuffle(min(len(df_part),256), seed=SEED, reshuffle_each_iteration=True)\n\n    # Map (parallel), batch, prefetch; deterministic=False allows better throughput\n    ds = ds.map(wrapper, num_parallel_calls=tf.data.AUTOTUNE, deterministic=False)\\\n           .batch(BATCH_SIZE)\\\n           .prefetch(tf.data.AUTOTUNE)\n    return ds\n\n# Final train/val datasets\ntrain_ds = make_ds(df_tr, shuffle=True)\nval_ds   = make_ds(df_val, shuffle=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T02:56:02.772338Z","iopub.execute_input":"2025-10-05T02:56:02.772995Z","iopub.status.idle":"2025-10-05T02:56:25.692936Z","shell.execute_reply.started":"2025-10-05T02:56:02.772972Z","shell.execute_reply":"2025-10-05T02:56:25.692105Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import math\n\nn_train = len(df_tr)\nn_val   = len(df_val)\nsteps_train = math.ceil(n_train / BATCH_SIZE)\nsteps_val   = math.ceil(n_val / BATCH_SIZE)\n\nprint(f\"Train series: {n_train}  |  steps/epoch ≈ {steps_train}\")\nprint(f\"Val   series: {n_val}    |  val_steps   ≈ {steps_val}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T02:56:58.309706Z","iopub.execute_input":"2025-10-05T02:56:58.310008Z","iopub.status.idle":"2025-10-05T02:56:58.314962Z","shell.execute_reply.started":"2025-10-05T02:56:58.309989Z","shell.execute_reply":"2025-10-05T02:56:58.314329Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Grab one training batch\ntrain_batch = next(iter(train_ds))\nx_tr, y_tr = train_batch  # x: (B, D, H, W, 1), y: (B,)\nprint(\"Train batch X shape:\", x_tr.shape, \"dtype:\", x_tr.dtype)\nprint(\"Train batch y shape:\", y_tr.shape, \"dtype:\", y_tr.dtype)\nprint(\"First few train labels:\", y_tr.numpy()[:8])\n\n# Grab one validation batch\nval_batch = next(iter(val_ds))\nx_va, y_va = val_batch\nprint(\"Val batch X shape:\", x_va.shape, \"dtype:\", x_va.dtype)\nprint(\"Val batch y shape:\", y_va.shape, \"dtype:\", y_va.dtype)\nprint(\"First few val labels:\", y_va.numpy()[:8])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T02:57:00.605577Z","iopub.execute_input":"2025-10-05T02:57:00.606162Z","iopub.status.idle":"2025-10-05T02:58:20.211146Z","shell.execute_reply.started":"2025-10-05T02:57:00.606139Z","shell.execute_reply":"2025-10-05T02:58:20.210375Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\n# pick the first example of the training batch\nvol = x_tr[0].numpy()  # (D,H,W,1) in [0,1]  ## try to change to 1, 2, ... observe some thing?\nvol = np.squeeze(vol, axis=-1)  # (D,H,W)\nD,H,W = vol.shape\nmid = D // 2\n\n# 1) show the middle slice\nplt.figure(figsize=(4,4))\nplt.imshow(vol[mid], cmap='gray')\nplt.title(f\"One training example\\nmiddle slice {mid+1}/{D}\")\nplt.axis('off')\nplt.show()\n\n# 2) show a small montage (e.g., 12 evenly spaced slices)\nk = 12\nidxs = np.linspace(0, D-1, k).astype(int)\ncols = 6\nrows = int(np.ceil(k/cols))\nplt.figure(figsize=(10, 3.5))\nfor i, z in enumerate(idxs):\n    ax = plt.subplot(rows, cols, i+1)\n    ax.imshow(vol[z], cmap='gray')\n    ax.set_title(f\"z={z+1}\", fontsize=8)\n    ax.axis('off')\nplt.suptitle(\"Evenly spaced slices (overview)\", y=1.02)\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:08:11.818434Z","iopub.execute_input":"2025-10-05T03:08:11.819163Z","iopub.status.idle":"2025-10-05T03:08:12.527151Z","shell.execute_reply.started":"2025-10-05T03:08:11.81914Z","shell.execute_reply":"2025-10-05T03:08:12.526343Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def build_tiny_3dcnn():\n    inp = keras.Input((DEPTH,HEIGHT,WIDTH,1))\n    x = layers.Conv3D(16,3,padding=\"same\",activation=\"relu\")(inp)  # 16 filters, 3 by 3 by 3\n    x = layers.MaxPool3D()(x)  # by default, 2 by 2 by 2.\n    x = layers.Conv3D(32,3,padding=\"same\",activation=\"relu\")(x) # 32 filters, 3 by 3 by 3\n    x = layers.GlobalAveragePooling3D()(x)\n    \n    x = layers.Dropout(0.2)(x) # set the rate to 0.2\n    \n    out = layers.Dense(1, activation=\"sigmoid\", dtype=\"float32\")(x) \n\n    model = keras.Model(inp, out)\n    model.compile(\n        optimizer=keras.optimizers.Adam(1e-3),\n        loss=\"binary_crossentropy\",\n        metrics=[\"accuracy\"],          \n        steps_per_execution=32       \n    )\n    return model\n\nmodel = build_tiny_3dcnn()\n\nmodel.summary()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:08:33.668329Z","iopub.execute_input":"2025-10-05T03:08:33.668933Z","iopub.status.idle":"2025-10-05T03:08:34.592591Z","shell.execute_reply.started":"2025-10-05T03:08:33.668911Z","shell.execute_reply":"2025-10-05T03:08:34.59198Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Don't run this cell\n# callbacks = [\n#     keras.callbacks.ReduceLROnPlateau(monitor=\"val_loss\", patience=1, factor=0.5, verbose=1),\n#     keras.callbacks.EarlyStopping(monitor=\"val_accuracy\", mode=\"max\", patience=2, restore_best_weights=True),\n# ]\n \n# history = model.fit(\n#     train_ds,\n#     validation_data=val_ds,\n#     epochs=1,\n#     callbacks=callbacks,\n#     verbose=1,\n# )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:08:45.069844Z","iopub.execute_input":"2025-10-05T03:08:45.07013Z","iopub.status.idle":"2025-10-05T03:08:45.073735Z","shell.execute_reply.started":"2025-10-05T03:08:45.070111Z","shell.execute_reply":"2025-10-05T03:08:45.073052Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Don't run this cell\n# val_metrics = model.evaluate(val_ds, return_dict=True, verbose=0)\n# print(val_metrics)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:08:49.283606Z","iopub.execute_input":"2025-10-05T03:08:49.284246Z","iopub.status.idle":"2025-10-05T03:08:49.287443Z","shell.execute_reply.started":"2025-10-05T03:08:49.284222Z","shell.execute_reply":"2025-10-05T03:08:49.286854Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Make the process more efficient","metadata":{}},{"cell_type":"code","source":"def make_ds_cache(df_part, shuffle):\n    \"\"\"With memory caching: first epoch is slow, later epochs much faster\"\"\"\n    sids = df_part[\"SeriesInstanceUID\"].astype(str).values\n    ys   = df_part[\"Aneurysm Present\"].astype(np.float32).values\n\n    def py_load(sid_bytes):\n        sid = sid_bytes.decode()\n        x = load_series(sid)  # (D,H,W,1) float32 in [0,1]\n        return x.astype(np.float32)\n\n    def wrapper(sid, y):\n        vol = tf.numpy_function(py_load, [sid], tf.float32)\n        vol.set_shape((DEPTH,HEIGHT,WIDTH,1))\n        return vol, y\n\n    ds = tf.data.Dataset.from_tensor_slices((sids, ys))\n    if shuffle:\n        ds = ds.shuffle(min(len(df_part),256), seed=SEED, reshuffle_each_iteration=True)\n\n    # key: map -> cache -> batch -> prefetch\n    ds = ds.map(wrapper, num_parallel_calls=tf.data.AUTOTUNE, deterministic=False)\n    ds = ds.cache()   # store mapped volumes in memory; reuse instead of reloading\n    ds = ds.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n    return ds\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:08:52.0457Z","iopub.execute_input":"2025-10-05T03:08:52.04598Z","iopub.status.idle":"2025-10-05T03:08:52.051848Z","shell.execute_reply.started":"2025-10-05T03:08:52.04596Z","shell.execute_reply":"2025-10-05T03:08:52.051178Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_ds_nocache = make_ds(df_tr, shuffle=True)\nval_ds_nocache   = make_ds(df_val, shuffle=False)\n\ntrain_ds_cache = make_ds_cache(df_tr, shuffle=True)\nval_ds_cache   = make_ds_cache(df_val, shuffle=False)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:09:00.189119Z","iopub.execute_input":"2025-10-05T03:09:00.189361Z","iopub.status.idle":"2025-10-05T03:09:00.287503Z","shell.execute_reply.started":"2025-10-05T03:09:00.189344Z","shell.execute_reply":"2025-10-05T03:09:00.286974Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import time\n\n# First, run the version without caching\nmodel = build_tiny_3dcnn()\nt0 = time.perf_counter()\nhistory = model.fit(\n    train_ds_nocache,\n    validation_data=val_ds_nocache,\n    epochs=2,\n    verbose=1,\n)\nt1 = time.perf_counter()\nprint(f\"[NoCache] Total time: {t1 - t0:.1f}s\")\n\n# Now, run the cached version (use a new model for a fair comparison)\nmodel2 = build_tiny_3dcnn()\n\n# Optional: touch the datasets once to fill the cache\nfor _ in train_ds_cache.take(1): pass\nfor _ in val_ds_cache.take(1):   pass\n\nt2 = time.perf_counter()\nhistory2 = model2.fit(\n    train_ds_cache,\n    validation_data=val_ds_cache,\n    epochs=4,\n    verbose=1,\n)\nt3 = time.perf_counter()\nprint(f\"[Cache]   Total time: {t3 - t2:.1f}s\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:09:05.032008Z","iopub.execute_input":"2025-10-05T03:09:05.032288Z","iopub.status.idle":"2025-10-05T03:19:22.618995Z","shell.execute_reply.started":"2025-10-05T03:09:05.032268Z","shell.execute_reply":"2025-10-05T03:19:22.618334Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Homework\n\nLet’s make our model a little more complex and see how it changes training process and accuracy.\n\n**Tasks**\n\n- Remove the GlobalAveragePooling3D layer from the previous tiny model.\n\n- Add another `MaxPool3D()` layer after the second convolution.\n\n- Add a `Flatten()` layer to turn the 3D feature maps into a 1D vector.\n\n- Add a `Dense()` layer after flattening.\n\n- Increase the Dropout rate from 0.2 to 0.4.\n\n\n**Train the model and compare:**\n\n- How long does training take compared to the tiny model?\n\n- Does the accuracy improve?\n\n- Does validation accuracy behave differently? Why? (consider the validation data size)","metadata":{}},{"cell_type":"code","source":"# def bigger_3dcnn():\n#     inp = keras.Input((DEPTH, HEIGHT, WIDTH, 1))\n#     x = layers.Conv3D(16, 3, padding=\"same\", activation=\"relu\")(inp)\n#     x = layers.MaxPool3D()(x)\n#     x = layers.Conv3D(32, 3, padding=\"same\", activation=\"relu\")(x)\n#     ... # Another max pooling layer\n\n#     ... # Add flatten layer\n#     ... # Add a dense layer with 128 neurons.\n#     x = layers.Dropout(...)(x) # increase the dropout rate to 0.4\n\n#     out = layers.Dense(1, activation=\"sigmoid\", dtype=\"float32\")(x)\n\n#     model = keras.Model(inp, out)\n#     model.compile(\n#         optimizer=keras.optimizers.Adam(1e-3),\n#         loss=\"binary_crossentropy\",\n#         metrics=[\"accuracy\"]\n#     )\n#     return model","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Answer","metadata":{}},{"cell_type":"code","source":"# def bigger_3dcnn():\n#     inp = keras.Input((DEPTH, HEIGHT, WIDTH, 1))\n#     x = layers.Conv3D(16, 3, padding=\"same\", activation=\"relu\")(inp)\n#     x = layers.MaxPool3D()(x)\n#     x = layers.Conv3D(32, 3, padding=\"same\", activation=\"relu\")(x)\n#     x = layers.MaxPool3D()(x)\n\n#     x = layers.Flatten()(x)\n#     x = layers.Dense(128, activation=\"relu\")(x)\n#     x = layers.Dropout(0.4)(x)\n\n#     out = layers.Dense(1, activation=\"sigmoid\", dtype=\"float32\")(x)\n\n#     model = keras.Model(inp, out)\n#     model.compile(\n#         optimizer=keras.optimizers.Adam(1e-3),\n#         loss=\"binary_crossentropy\",\n#         metrics=[\"accuracy\"]\n#     )\n#     return model","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:41:28.463975Z","iopub.execute_input":"2025-10-05T03:41:28.464581Z","iopub.status.idle":"2025-10-05T03:41:28.470131Z","shell.execute_reply.started":"2025-10-05T03:41:28.464553Z","shell.execute_reply":"2025-10-05T03:41:28.469494Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# train_ds_cache = make_ds_cache(df_tr, shuffle=True)\n# val_ds_cache   = make_ds_cache(df_val, shuffle=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:41:30.592307Z","iopub.execute_input":"2025-10-05T03:41:30.592879Z","iopub.status.idle":"2025-10-05T03:41:30.784544Z","shell.execute_reply.started":"2025-10-05T03:41:30.592857Z","shell.execute_reply":"2025-10-05T03:41:30.783959Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# # Now, run the cached version (use a new model for a fair comparison)\n# model3 = bigger_3dcnn()\n\n# # Optional: touch the datasets once to fill the cache\n# for _ in train_ds_cache.take(1): pass\n# for _ in val_ds_cache.take(1):   pass\n\n# t4 = time.perf_counter()\n# history3 = model3.fit(\n#     train_ds_cache,\n#     validation_data=val_ds_cache,\n#     epochs=4,\n#     verbose=1,\n# )\n# t5 = time.perf_counter()\n# print(f\"[Cache]   Total time: {t3 - t2:.1f}s\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:41:36.788094Z","iopub.execute_input":"2025-10-05T03:41:36.788756Z","iopub.status.idle":"2025-10-05T03:45:37.187423Z","shell.execute_reply.started":"2025-10-05T03:41:36.788733Z","shell.execute_reply":"2025-10-05T03:45:37.186843Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# val_metrics3 = model3.evaluate(val_ds_cache , return_dict=True, verbose=0)\n# print(val_metrics3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T03:56:08.578968Z","iopub.execute_input":"2025-10-05T03:56:08.579654Z","iopub.status.idle":"2025-10-05T03:56:08.625643Z","shell.execute_reply.started":"2025-10-05T03:56:08.57963Z","shell.execute_reply":"2025-10-05T03:56:08.624734Z"}},"outputs":[],"execution_count":null}]}