{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":99552,"databundleVersionId":13851420,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# RSNA Intracranial Aneurysm Detection","metadata":{}},{"cell_type":"markdown","source":"\n\nThis notebook is focused on **Intracranial Aneurysm Detection** using the RSNA dataset. The goal is to build a machine learning/deep learning pipeline that can automatically detect the presence of an aneurysm from patient brain scans (MRA/CTA). \n\n**Purpose of this Notebook:**\n\nThis notebook serves as a complete workflow for aneurysm detection, covering all the key steps from data understanding to model evaluation:\n\n1. **Dataset Exploration & EDA:**  \n   - Examine the dataset structure, column types, and label distribution.  \n   - Visualize patient demographics (age, sex) and the prevalence of aneurysms across different arteries.  \n   - Analyze image series (DICOM files) and their slices.  \n\n2. **Image Preprocessing & Dataset Loader:**  \n   - Convert DICOM slices into normalized images suitable for deep learning models.  \n   - Implement a PyTorch Dataset that handles multiple slices per patient series.  \n   - Apply data augmentation (rotation, flipping) for better generalization.  \n\n3. **Model Definition (EfficientNet-B0):**  \n   - Use pretrained **EfficientNet-B0** as a feature extractor.  \n   - Add a custom classification head for binary classification (Aneurysm Present or Not).  \n   - Use proper input normalization compatible with ImageNet pretrained weights.  \n\n4. **Training & Validation:**  \n   - Implement training loops with loss computation and backpropagation.  \n   - Track metrics like training loss, validation loss, training accuracy, and validation accuracy.  \n   - Apply early stopping or checkpointing if necessary.  \n\n5. **Evaluation & Visualization:**  \n   - Compute evaluation metrics such as **confusion matrix, precision, recall, and F1-score**.  \n   - Plot training vs validation loss and accuracy curves to monitor model performance.  \n   - Visualize sample predictions and errors for better interpretability.  \n\n6. **Conclusion & Next Steps:**  \n   - Summarize model performance.  \n   - Discuss potential improvements (more slices, better augmentation, 3D models, etc.).  \n   - Suggest directions for deploying the model in a clinical setting.  \n\nThis notebook is intended to be **end-to-end**, meaning anyone following it will be able to preprocess the data, train a model, evaluate results, and understand the limitations and strengths of the approach.  \n","metadata":{}},{"cell_type":"markdown","source":"# Import Libraries and Load Dataset\n\n- Import essential libraries for data manipulation (pandas, numpy), visualization (matplotlib, seaborn), and deep learning (PyTorch, torchvision).  \n- Import `pydicom` to read DICOM medical images.  \n- Load the main CSV files: `train.csv` (metadata + labels) and `train_localizers.csv` (slice location info).  \n- Inspect columns and first few rows to understand the dataset structure.  \n- Set the device (CPU or GPU) for model training.  \n- Prepare for the next step: image preprocessing and dataset loader.\n","metadata":{}},{"cell_type":"code","source":"# Cell 1: imports + config\nimport os, sys, math, random, time\nfrom pathlib import Path\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pydicom\nimport cv2\nfrom tqdm import tqdm\n\nROOT = Path(\"/kaggle/input/rsna-intracranial-aneurysm-detection\")  # adjust if local\nSERIES_ROOT = ROOT / \"series\"\nSEG_ROOT = ROOT / \"segmentations\"\nTRAIN_CSV = ROOT / \"train.csv\"\nLOCALIZER_CSV = ROOT / \"train_localizers.csv\"\n\nSAMPLE_SERIES_META_READ = 128   \nRANDOM_SEED = 42\nnp.random.seed(RANDOM_SEED)\nrandom.seed(RANDOM_SEED)\n\nprint(\"ROOT:\", ROOT)\nprint(\"Series folder exists:\", SERIES_ROOT.exists())\nprint(\"Segmentations folder exists:\", SEG_ROOT.exists())\nprint(\"Train CSV exists:\", TRAIN_CSV.exists())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:01.606541Z","iopub.execute_input":"2025-10-05T15:20:01.607077Z","iopub.status.idle":"2025-10-05T15:20:03.422411Z","shell.execute_reply.started":"2025-10-05T15:20:01.607055Z","shell.execute_reply":"2025-10-05T15:20:03.421823Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Load CSV and Inspect Columns/Head\n\n- Load `train.csv` into a DataFrame to access patient info, series IDs, and aneurysm labels.  \n- Check available columns to understand which arteries are annotated and patient metadata (age, sex, modality).  \n- Display the first few rows (`.head()`) to get a sense of the actual data values.  \n- Identify target column for classification (`Aneurysm Present`) and features to use for analysis.  \n- Verify data types and check for missing values to ensure data quality before preprocessing.  \n","metadata":{}},{"cell_type":"code","source":"# load CSV and columns/head \ntrain_df = pd.read_csv(TRAIN_CSV)\nlocalizer_df = pd.read_csv(LOCALIZER_CSV)\n\nprint(\"train.csv shape:\", train_df.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.423518Z","iopub.execute_input":"2025-10-05T15:20:03.424075Z","iopub.status.idle":"2025-10-05T15:20:03.473011Z","shell.execute_reply.started":"2025-10-05T15:20:03.424048Z","shell.execute_reply":"2025-10-05T15:20:03.472279Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"train.csv columns:\\n\", train_df.columns.tolist())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.47386Z","iopub.execute_input":"2025-10-05T15:20:03.474122Z","iopub.status.idle":"2025-10-05T15:20:03.478335Z","shell.execute_reply.started":"2025-10-05T15:20:03.474103Z","shell.execute_reply":"2025-10-05T15:20:03.477665Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"\\ntrain.csv sample:\")\ndisplay(train_df.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.479928Z","iopub.execute_input":"2025-10-05T15:20:03.480125Z","iopub.status.idle":"2025-10-05T15:20:03.51043Z","shell.execute_reply.started":"2025-10-05T15:20:03.480109Z","shell.execute_reply":"2025-10-05T15:20:03.509772Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"\\nlocalizer.csv shape:\", localizer_df.shape)\ndisplay(localizer_df.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.511084Z","iopub.execute_input":"2025-10-05T15:20:03.511253Z","iopub.status.idle":"2025-10-05T15:20:03.519058Z","shell.execute_reply.started":"2025-10-05T15:20:03.511239Z","shell.execute_reply":"2025-10-05T15:20:03.518533Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Class Balance & Vessel-wise Frequency\n\n- Examine the distribution of the target variable (`Aneurysm Present`) to see how balanced the dataset is.  \n- Count how many series/patients have aneurysms vs. no aneurysms.  \n- Analyze vessel-wise annotation columns (e.g., `Left Middle Cerebral Artery`, `Basilar Tip`) to understand which arteries most commonly have aneurysms.  \n- This helps identify potential class imbalance and guides loss function choice or sampling strategy.  \n- Useful for visualizations like bar plots or heatmaps for vessel-wise frequencies.  \n","metadata":{}},{"cell_type":"code","source":"print(\"Aneurysm Present distribution:\")\nprint(train_df['Aneurysm Present'].value_counts())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.519734Z","iopub.execute_input":"2025-10-05T15:20:03.519933Z","iopub.status.idle":"2025-10-05T15:20:03.535594Z","shell.execute_reply.started":"2025-10-05T15:20:03.519918Z","shell.execute_reply":"2025-10-05T15:20:03.534967Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- Check the types of imaging modalities used in the dataset (e.g., MRA, CTA).  \n- Helps understand the diversity of input data and whether preprocessing needs modality-specific handling.  \n","metadata":{}},{"cell_type":"code","source":"print(\"\\nModality distribution:\")\nprint(train_df['Modality'].value_counts())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.536316Z","iopub.execute_input":"2025-10-05T15:20:03.536547Z","iopub.status.idle":"2025-10-05T15:20:03.55155Z","shell.execute_reply.started":"2025-10-05T15:20:03.536525Z","shell.execute_reply":"2025-10-05T15:20:03.550814Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Patient Sex Distribution\n\nThis step analyzes the distribution of patients by sex in the dataset. Understanding the male-to-female ratio helps identify any potential bias and ensures that the model is trained on a balanced representation of both sexes.\n","metadata":{}},{"cell_type":"code","source":"print(\"\\nPatientSex distribution:\")\nprint(train_df['PatientSex'].value_counts())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.552274Z","iopub.execute_input":"2025-10-05T15:20:03.552466Z","iopub.status.idle":"2025-10-05T15:20:03.567593Z","shell.execute_reply.started":"2025-10-05T15:20:03.552442Z","shell.execute_reply":"2025-10-05T15:20:03.566971Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Patient Age Statistics\n\nHere, we examine the statistical summary of patient ages, including minimum, maximum, mean, and quartiles. This provides insight into the age distribution of the dataset, which can influence disease prevalence and model generalization.\n","metadata":{}},{"cell_type":"code","source":"print(\"\\nPatientAge stats:\")\ndisplay(train_df['PatientAge'].describe())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.568222Z","iopub.execute_input":"2025-10-05T15:20:03.568421Z","iopub.status.idle":"2025-10-05T15:20:03.589844Z","shell.execute_reply.started":"2025-10-05T15:20:03.568406Z","shell.execute_reply":"2025-10-05T15:20:03.589232Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Vessel Columns\n\nWe automatically detect all vessel-related columns by excluding non-vessel metadata like `SeriesInstanceUID`, `PatientAge`, `PatientSex`, `Modality`, and the target `Aneurysm Present`. ","metadata":{}},{"cell_type":"code","source":"# vessel columns auto-detect:  numeric 0/1 type and 'Aneurysm Present' \nnon_vessel = {'SeriesInstanceUID','PatientAge','PatientSex','Modality','Aneurysm Present'}\nvessel_cols = [c for c in train_df.columns if c not in non_vessel]\nprint(\"\\nDetected vessel columns (count={}):\".format(len(vessel_cols)))\nprint(vessel_cols)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.592245Z","iopub.execute_input":"2025-10-05T15:20:03.592426Z","iopub.status.idle":"2025-10-05T15:20:03.602032Z","shell.execute_reply.started":"2025-10-05T15:20:03.592411Z","shell.execute_reply":"2025-10-05T15:20:03.601385Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Frequency Analysis\nNext, we calculate how frequently aneurysms appear in each vessel across the dataset. This helps identify which vessels are most commonly affected and can guide model focus and potential feature importance analysis.\n","metadata":{}},{"cell_type":"code","source":"# vessel frequencies (how many series have aneurysm in each vessel)\nvessel_counts = train_df[vessel_cols].sum().sort_values(ascending=False)\ndisplay(vessel_counts)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.602667Z","iopub.execute_input":"2025-10-05T15:20:03.602821Z","iopub.status.idle":"2025-10-05T15:20:03.622352Z","shell.execute_reply.started":"2025-10-05T15:20:03.602808Z","shell.execute_reply":"2025-10-05T15:20:03.621733Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Series-to-Slices Mapping (Fast Check)\n\nHere, we list all series folders in the dataset and count them. This provides a quick overview of how many unique patient series we have and helps map each series to its number of DICOM slices. \n\nThis is a preliminary step before building a full dataset loader.\n","metadata":{}},{"cell_type":"code","source":"# Cell 4: build series -> num_slices mapping (fast: only filenames counted)\nseries_dirs = sorted([p.name for p in SERIES_ROOT.iterdir() if p.is_dir()])\nprint(\"Total series folders found:\", len(series_dirs))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:03.623Z","iopub.execute_input":"2025-10-05T15:20:03.623185Z","iopub.status.idle":"2025-10-05T15:20:06.862044Z","shell.execute_reply.started":"2025-10-05T15:20:03.62317Z","shell.execute_reply":"2025-10-05T15:20:06.861314Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Series Length Mapping\n\nWe iterate through each series folder and count the number of DICOM files (slices) it contains. This mapping (`series_len`) helps us understand the slice distribution per series, which is useful for sampling and preprocessing steps in the dataset loader.\n","metadata":{}},{"cell_type":"code","source":"# map series -> num files\nseries_len = {}\nfor sid in tqdm(series_dirs):\n    try:\n        files = os.listdir(SERIES_ROOT / sid)\n        series_len[sid] = len(files)\n    except Exception as e:\n        series_len[sid] = 0","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:20:06.862925Z","iopub.execute_input":"2025-10-05T15:20:06.863176Z","iopub.status.idle":"2025-10-05T15:22:29.049685Z","shell.execute_reply.started":"2025-10-05T15:20:06.863152Z","shell.execute_reply":"2025-10-05T15:22:29.04905Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Series Slice Count DataFrame\n\nHere, we convert the `series_len` dictionary into a DataFrame (`series_len_df`) with two columns: `SeriesInstanceUID` and `num_slices`. This tabular format allows easier inspection and merging with `train_df` for downstream analysis.\n","metadata":{}},{"cell_type":"code","source":"# convert to DataFrame\nseries_len_df = pd.DataFrame.from_dict(series_len, orient='index', columns=['num_slices'])\nseries_len_df.index.name = 'SeriesInstanceUID'\nseries_len_df.reset_index(inplace=True)\ndisplay(series_len_df.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:29.050619Z","iopub.execute_input":"2025-10-05T15:22:29.05089Z","iopub.status.idle":"2025-10-05T15:22:29.060243Z","shell.execute_reply.started":"2025-10-05T15:22:29.050866Z","shell.execute_reply":"2025-10-05T15:22:29.059687Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Series Slice Count Statistics\n\nWe first compute basic statistics (`count`, `mean`, `min`, `max`, etc.) of the number of slices per series.  \nThen, we visualize the distribution with a histogram (log-scaled y-axis) to understand the variation in the number of slices across series. This helps identify series with unusually few or many slices, which can inform sampling strategies for model training.\n","metadata":{}},{"cell_type":"code","source":"# basic stats\nprint(series_len_df['num_slices'].describe())\nplt.figure(figsize=(8,4))\nsns.histplot(series_len_df['num_slices'], bins=50, log_scale=(False,True))\nplt.title(\"Distribution of number of slices per series (log y)\")\nplt.xlabel(\"num_slices\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:29.060907Z","iopub.execute_input":"2025-10-05T15:22:29.061276Z","iopub.status.idle":"2025-10-05T15:22:29.553435Z","shell.execute_reply.started":"2025-10-05T15:22:29.061253Z","shell.execute_reply":"2025-10-05T15:22:29.552781Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Merge Series Metadata with Slice Counts\n\nHere, we join the main CSV (`train_df`) with the `series_len_df` to associate each series with its slice count.  \nThis allows us to:  \n1. Track which series are missing from the folder structure.  \n2. Ensure that each series in the dataset has corresponding slices for downstream processing.  \n3. Retain important patient metadata (`Modality`, `PatientAge`, `PatientSex`) alongside slice counts and labels.\n","metadata":{}},{"cell_type":"code","source":"#  join csv and series_len, check missing series and segmentation files\n# left-join train_df (has series we care about) with series_len_df\ndf_series = train_df[['SeriesInstanceUID','Aneurysm Present','Modality','PatientAge','PatientSex']].merge(\n    series_len_df, how='left', on='SeriesInstanceUID'\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:29.554132Z","iopub.execute_input":"2025-10-05T15:22:29.554309Z","iopub.status.idle":"2025-10-05T15:22:29.56399Z","shell.execute_reply.started":"2025-10-05T15:22:29.554295Z","shell.execute_reply":"2025-10-05T15:22:29.563346Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Identify Missing Series Folders\n\nWe check for any series listed in the CSV that do not have corresponding folders in the dataset directory.  \nThis step helps to:  \n- Quantify missing data that may need to be ignored or handled.  \n- Ensure that the dataset used for training only contains series with available slices.  \n\nHere, `missing_series` gives the number of such series.\n","metadata":{}},{"cell_type":"code","source":"# mark missing series folders\nmissing_series = df_series['num_slices'].isna().sum()\nprint(\"Train CSV series without series folder (missing):\", missing_series)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:29.564601Z","iopub.execute_input":"2025-10-05T15:22:29.564814Z","iopub.status.idle":"2025-10-05T15:22:29.579882Z","shell.execute_reply.started":"2025-10-05T15:22:29.564798Z","shell.execute_reply":"2025-10-05T15:22:29.579259Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Check Segmentation Availability\n\nFor each series, we verify whether a corresponding segmentation file exists.  \nThis is important because:  \n- Some series may not have annotated aneurysm masks.  \n- Knowing which series have segmentation allows us to filter or prioritize data for training.  \n\nThe `has_seg(series_id)` function returns `True` if the segmentation file exists, `False` otherwise.\n","metadata":{}},{"cell_type":"code","source":"# is there segmentation file for each series?\ndef has_seg(series_id):\n    seg_file = SEG_ROOT / f\"{series_id}.npz\"\n    return seg_file.exists()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:29.580538Z","iopub.execute_input":"2025-10-05T15:22:29.580711Z","iopub.status.idle":"2025-10-05T15:22:29.594041Z","shell.execute_reply.started":"2025-10-05T15:22:29.580697Z","shell.execute_reply":"2025-10-05T15:22:29.593364Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# apply quickly (may be heavy if many series) — but we do it only for train_df entries\ndf_series['has_seg'] = df_series['SeriesInstanceUID'].apply(lambda x: (SEG_ROOT / f\"{x}.npz\").exists())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:29.594757Z","iopub.execute_input":"2025-10-05T15:22:29.594973Z","iopub.status.idle":"2025-10-05T15:22:32.025226Z","shell.execute_reply.started":"2025-10-05T15:22:29.594948Z","shell.execute_reply":"2025-10-05T15:22:32.024684Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Summary of Series with Segmentation\n\nAfter checking for segmentation files, we summarize:  \n- Count of series that have a corresponding segmentation (`True`) vs those that don't (`False`).  \n- Preview the first few rows of the merged DataFrame, showing:  \n  - SeriesInstanceUID  \n  - Aneurysm presence  \n  - Modality  \n  - Patient age and sex  \n  - Number of slices  \n  - Segmentation availability (`has_seg`)  \n\nThis helps quickly understand dataset coverage and identify series missing annotations.\n","metadata":{}},{"cell_type":"code","source":"# summary\nprint(df_series['has_seg'].value_counts())\ndisplay(df_series.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:32.025943Z","iopub.execute_input":"2025-10-05T15:22:32.026191Z","iopub.status.idle":"2025-10-05T15:22:32.038454Z","shell.execute_reply.started":"2025-10-05T15:22:32.026166Z","shell.execute_reply":"2025-10-05T15:22:32.03796Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Save summary CSV for later\ndf_series.to_csv(\"train_series_summary.csv\", index=False)\nprint(\"Saved train_series_summary.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:32.039112Z","iopub.execute_input":"2025-10-05T15:22:32.039342Z","iopub.status.idle":"2025-10-05T15:22:32.072173Z","shell.execute_reply.started":"2025-10-05T15:22:32.039327Z","shell.execute_reply":"2025-10-05T15:22:32.071641Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Number of Slices vs Aneurysm Presence\n\nThis boxplot visualizes the distribution of slice counts in each series grouped by aneurysm presence:  \n- X-axis: `Aneurysm Present` (0 = no aneurysm, 1 = aneurysm)  \n- Y-axis: `num_slices` (number of DICOM slices per series)  \n\nIt helps identify if series with aneurysms tend to have more or fewer slices, or if slice counts are roughly similar across both classes.","metadata":{}},{"cell_type":"code","source":"# Cell 6: relationship between num_slices and Aneurysm Present\nplt.figure(figsize=(6,4))\nsns.boxplot(data=df_series, x='Aneurysm Present', y='num_slices')\nplt.title(\"num_slices by Aneurysm Present\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:32.072928Z","iopub.execute_input":"2025-10-05T15:22:32.073166Z","iopub.status.idle":"2025-10-05T15:22:32.207964Z","shell.execute_reply.started":"2025-10-05T15:22:32.073144Z","shell.execute_reply":"2025-10-05T15:22:32.207313Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Number of Slices vs Modality\n\nThis boxplot shows how the number of slices varies across different imaging modalities (e.g., MRA, CTA):  \n- X-axis: `Modality`  \n- Y-axis: `num_slices` (number of DICOM slices per series)  \n\nIt helps understand if certain modalities tend to produce longer or shorter series, which can inform preprocessing and model input strategies.\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8,4))\nsns.boxplot(data=df_series, x='Modality', y='num_slices')\nplt.title(\"num_slices by Modality\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:32.208637Z","iopub.execute_input":"2025-10-05T15:22:32.20885Z","iopub.status.idle":"2025-10-05T15:22:32.355531Z","shell.execute_reply.started":"2025-10-05T15:22:32.208834Z","shell.execute_reply":"2025-10-05T15:22:32.354896Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Top / Bottom series by slice-count (useful to inspect weird cases)","metadata":{}},{"cell_type":"markdown","source":"# Top 10 Longest Series\n\nThis table lists the 10 series with the highest number of slices.  \n- Useful for identifying extreme cases that may need special handling during preprocessing (e.g., sampling or resizing slices).  \n- Helps in planning memory and batch-size requirements for training.\n","metadata":{}},{"cell_type":"code","source":"top10 = series_len_df.sort_values('num_slices', ascending=False).head(10)\nprint(\"Top 10 longest series (num_slices):\")\ndisplay(top10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:32.35633Z","iopub.execute_input":"2025-10-05T15:22:32.356589Z","iopub.status.idle":"2025-10-05T15:22:32.365425Z","shell.execute_reply.started":"2025-10-05T15:22:32.356565Z","shell.execute_reply":"2025-10-05T15:22:32.3648Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Top 10 Shortest Series\n\nThe table shows the 10 series with the fewest slices.  \n- These short series may be underrepresented in training and could affect model performance.  \n- Consider handling them carefully during data preprocessing to avoid bias or errors.\n","metadata":{}},{"cell_type":"code","source":"bot10 = series_len_df.sort_values('num_slices', ascending=True).head(10)\nprint(\"\\nTop 10 shortest series:\")\ndisplay(bot10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:32.366111Z","iopub.execute_input":"2025-10-05T15:22:32.366344Z","iopub.status.idle":"2025-10-05T15:22:32.378655Z","shell.execute_reply.started":"2025-10-05T15:22:32.366324Z","shell.execute_reply":"2025-10-05T15:22:32.378056Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# DICOM Header Inspection\n\nA random subset of series was read to inspect DICOM metadata:  \n- **PixelSpacing**: Physical distance between pixels in the image plane.  \n- **SliceThickness**: Distance between slices in the series.  \n- **Modality**: Imaging modality (e.g., CT, MR).  \n\nThis helps in understanding spatial resolution and planning preprocessing steps like resizing or normalization.  \nSome series may have missing metadata, which should be handled carefully.\n","metadata":{}},{"cell_type":"code","source":"#  read DICOM headers from a random subset of series (to inspect pixel spacing, slice thickness)\nsample_series = random.sample(series_dirs, min(SAMPLE_SERIES_META_READ, len(series_dirs)))\nmeta_list = []\nfor sid in tqdm(sample_series):\n    series_path = SERIES_ROOT / sid\n    try:\n        files = sorted(os.listdir(series_path))\n        if len(files)==0:\n            continue\n        dcm_path = series_path / files[0]\n        ds = pydicom.dcmread(str(dcm_path), stop_before_pixels=True)  # faster, no pixel data\n        pixel_spacing = getattr(ds, 'PixelSpacing', None)\n        slice_thickness = getattr(ds, 'SliceThickness', None)\n        modality = getattr(ds, 'Modality', None)\n        meta_list.append({'SeriesInstanceUID': sid,\n                          'PixelSpacing': pixel_spacing,\n                          'SliceThickness': slice_thickness,\n                          'Modality': modality})\n    except Exception as e:\n        meta_list.append({'SeriesInstanceUID': sid, 'PixelSpacing': None, 'SliceThickness': None, 'Modality': None})","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:32.379638Z","iopub.execute_input":"2025-10-05T15:22:32.380235Z","iopub.status.idle":"2025-10-05T15:22:33.566027Z","shell.execute_reply.started":"2025-10-05T15:22:32.380213Z","shell.execute_reply":"2025-10-05T15:22:33.565457Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## DICOM Metadata Overview\n\n- Converted the collected metadata into a DataFrame `meta_df` for easy inspection.  \n- Displayed the first few rows to check extracted attributes for a few random series.  \n- Analyzed `PixelSpacing` values to understand how pixel dimensions vary across series, which is important for consistent image preprocessing and resampling.\n","metadata":{}},{"cell_type":"code","source":"meta_df = pd.DataFrame(meta_list)\ndisplay(meta_df.head())\nprint(\"PixelSpacing value counts (first element if list):\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:33.566692Z","iopub.execute_input":"2025-10-05T15:22:33.567252Z","iopub.status.idle":"2025-10-05T15:22:33.578805Z","shell.execute_reply.started":"2025-10-05T15:22:33.567233Z","shell.execute_reply":"2025-10-05T15:22:33.578203Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# DICOM Metadata Summary\n\n- Converted the collected DICOM header information into a DataFrame (`meta_df`) for easy inspection.  \n- Displayed the first few rows to quickly check extracted fields like `PixelSpacing`, `SliceThickness`, and `Modality`.  \n- Counting unique `PixelSpacing` values helps understand variation in in-plane resolution across series, which is important for consistent preprocessing and resizing.\n","metadata":{}},{"cell_type":"code","source":"# simplify PixelSpacing\nmeta_df['ps0'] = meta_df['PixelSpacing'].apply(lambda x: x[0] if isinstance(x, (list,tuple)) and len(x)>0 else None)\ndisplay(meta_df['ps0'].value_counts().head(20))\ndisplay(meta_df['SliceThickness'].value_counts().head(20))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:33.583488Z","iopub.execute_input":"2025-10-05T15:22:33.583664Z","iopub.status.idle":"2025-10-05T15:22:33.595078Z","shell.execute_reply.started":"2025-10-05T15:22:33.583651Z","shell.execute_reply":"2025-10-05T15:22:33.594514Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Positive Series Visualization\n\n- Selected a random series labeled as positive for aneurysm (`Aneurysm Present = 1`).  \n- Loaded the first 12 slices from the series and plotted them in a 3x4 grid.  \n- If a segmentation mask exists, overlaid it on the grayscale slices using a semi-transparent red colormap.  \n- This visualization helps verify the presence and location of aneurysms in the series and ensures that the mask aligns with the slices correctly.\n","metadata":{}},{"cell_type":"code","source":"# pick a positive series sample and visualize first 12 slices with overlay if mask exists\npos_series = df_series[df_series['Aneurysm Present']==1]['SeriesInstanceUID'].dropna().tolist()\nif len(pos_series)==0:\n    print(\"No positive series in train_df!\")\nelse:\n    sample_sid = random.choice(pos_series)\n    print(\"Sample positive series:\", sample_sid)\n    series_path = SERIES_ROOT / sample_sid\n    files = sorted(os.listdir(series_path))\n    n_show = min(12, len(files))\n    # load mask if exists\n    mask_file = SEG_ROOT / f\"{sample_sid}.npz\"\n    mask = None\n    if mask_file.exists():\n        try:\n            mask = np.load(str(mask_file))['arr_0']\n            print(\"Mask shape:\", mask.shape)\n        except Exception as e:\n            print(\"Failed to load mask:\", e)\n            mask = None\n\n    # plot grid\n    fig, axes = plt.subplots(3, 4, figsize=(12,9))\n    for i, ax in enumerate(axes.flat):\n        if i < n_show:\n            dcm = pydicom.dcmread(str(series_path / files[i]))\n            img = dcm.pixel_array.astype(np.float32)\n            if img.max() > img.min():\n                imgn = (img - img.min()) / (img.max() - img.min())\n            else:\n                imgn = np.zeros_like(img)\n            ax.imshow(imgn, cmap='gray')\n            if (mask is not None) and (i < mask.shape[0]):\n                ax.imshow(mask[i], cmap='Reds', alpha=0.4)\n            ax.set_title(f\"Slice {i+1}\")\n            ax.axis('off')\n        else:\n            ax.axis('off')\n    plt.tight_layout()\n    plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:33.595812Z","iopub.execute_input":"2025-10-05T15:22:33.596Z","iopub.status.idle":"2025-10-05T15:22:35.034963Z","shell.execute_reply.started":"2025-10-05T15:22:33.595986Z","shell.execute_reply":"2025-10-05T15:22:35.03417Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Import Libraries","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport missingno as msno","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:35.035776Z","iopub.execute_input":"2025-10-05T15:22:35.035987Z","iopub.status.idle":"2025-10-05T15:22:35.048749Z","shell.execute_reply.started":"2025-10-05T15:22:35.035969Z","shell.execute_reply":"2025-10-05T15:22:35.048036Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Plotting Style Setup\n\n- Configured Seaborn for clean, white-grid style plots.\n- Increased font scale for better readability.\n- Set default figure size to `(8,5)` for consistent visualizations.\n","metadata":{}},{"cell_type":"code","source":"sns.set(style=\"whitegrid\", font_scale=1.1)\nplt.rcParams[\"figure.figsize\"] = (8,5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:35.049536Z","iopub.execute_input":"2025-10-05T15:22:35.049827Z","iopub.status.idle":"2025-10-05T15:22:35.055938Z","shell.execute_reply.started":"2025-10-05T15:22:35.04981Z","shell.execute_reply":"2025-10-05T15:22:35.055223Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Load Training CSV\n\n- Loaded the main `train.csv` file containing series metadata and labels.\n- Stored it in a DataFrame called `train_df` for further processing and analysis.\n","metadata":{}},{"cell_type":"code","source":"path = \"/kaggle/input/rsna-intracranial-aneurysm-detection/train.csv\"\ntrain_df = pd.read_csv(path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:35.056588Z","iopub.execute_input":"2025-10-05T15:22:35.056782Z","iopub.status.idle":"2025-10-05T15:22:35.080214Z","shell.execute_reply.started":"2025-10-05T15:22:35.056767Z","shell.execute_reply":"2025-10-05T15:22:35.079684Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"🔹 Shape of Dataset:\", train_df.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:35.080893Z","iopub.execute_input":"2025-10-05T15:22:35.08106Z","iopub.status.idle":"2025-10-05T15:22:35.085352Z","shell.execute_reply.started":"2025-10-05T15:22:35.081047Z","shell.execute_reply":"2025-10-05T15:22:35.084677Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Inspect First Rows of Dataset\n\n- Displayed the first 5 rows of `train_df` to get an overview of the dataset structure.\n- Columns include patient info (`PatientAge`, `PatientSex`), scan modality, vessel-specific aneurysm indicators, and the target `Aneurysm Present`.\n- Helps verify that data loaded correctly and understand initial data types and values.\n","metadata":{}},{"cell_type":"code","source":"print(\"\\n🔹 First 5 Rows:\\n\")\ndisplay(train_df.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:35.086157Z","iopub.execute_input":"2025-10-05T15:22:35.086391Z","iopub.status.idle":"2025-10-05T15:22:35.107695Z","shell.execute_reply.started":"2025-10-05T15:22:35.086369Z","shell.execute_reply":"2025-10-05T15:22:35.107148Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Column Names Overview\n\n- Listed all columns in `train_df` to understand the dataset structure.\n- Columns include:\n  - **Patient Info:** `PatientAge`, `PatientSex`\n  - **Scan Info:** `Modality`, `SeriesInstanceUID`\n  - **Vessel-specific indicators:** e.g., `Left Middle Cerebral Artery`, `Basilar Tip`, etc.\n  - **Target variable:** `Aneurysm Present` (binary: 0 = no aneurysm, 1 = aneurysm)\n- This gives a clear picture of which features are numeric, categorical, and target for modeling.\n","metadata":{}},{"cell_type":"code","source":"print(\"\\n🔹 Column Names:\\n\", list(train_df.columns))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:35.108388Z","iopub.execute_input":"2025-10-05T15:22:35.108923Z","iopub.status.idle":"2025-10-05T15:22:35.121796Z","shell.execute_reply.started":"2025-10-05T15:22:35.1089Z","shell.execute_reply":"2025-10-05T15:22:35.121123Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"##  Dataset information\n\n\n- Checked dataset info to see column types and non-null counts.\n- Confirmed which columns are numeric (vessel indicators) and categorical (PatientSex, Modality).\n- Verified total number of rows and presence of missing values.\n","metadata":{}},{"cell_type":"code","source":"print(\"\\n📄 Dataset Info:\\n\")\ntrain_df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:35.122516Z","iopub.execute_input":"2025-10-05T15:22:35.122871Z","iopub.status.idle":"2025-10-05T15:22:35.14444Z","shell.execute_reply.started":"2025-10-05T15:22:35.122844Z","shell.execute_reply":"2025-10-05T15:22:35.143712Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Check for nulls\n\n- Checked for missing/null values in the dataset.\n- Ensured all critical columns like 'SeriesInstanceUID' and 'Aneurysm Present' are complete.\n- Identified if any vessel columns have missing entries that might need handling.\n","metadata":{}},{"cell_type":"code","source":"print(\"\\n🔍 Missing Values:\")\nprint(train_df.isnull().sum())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:35.145202Z","iopub.execute_input":"2025-10-05T15:22:35.145436Z","iopub.status.idle":"2025-10-05T15:22:35.162107Z","shell.execute_reply.started":"2025-10-05T15:22:35.145421Z","shell.execute_reply":"2025-10-05T15:22:35.161488Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualize missing values\n\n- Visualized missing values across all columns using a heatmap or bar plot.\n- Helps quickly identify which columns have gaps and the proportion of missing data.\n- Important for planning data cleaning or imputation steps before modeling.\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,4))\nmsno.bar(train_df)\nplt.title(\"Missing Value Overview\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:35.162722Z","iopub.execute_input":"2025-10-05T15:22:35.162971Z","iopub.status.idle":"2025-10-05T15:22:36.082977Z","shell.execute_reply.started":"2025-10-05T15:22:35.162954Z","shell.execute_reply":"2025-10-05T15:22:36.082305Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Patient Demographics","metadata":{}},{"cell_type":"markdown","source":"## Age Distribution\n\n- Plotted the distribution of `PatientAge` to understand age demographics of the patients.\n- Used histogram with KDE to visualize both count and density.\n- Helps identify common age ranges and potential outliers in the dataset.\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8,5))\nsns.histplot(train_df[\"PatientAge\"], bins=30, kde=True, color=\"skyblue\")\nplt.title(\"Distribution of Patient Age\")\nplt.xlabel(\"Age (years)\")\nplt.ylabel(\"Count\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:36.083864Z","iopub.execute_input":"2025-10-05T15:22:36.084348Z","iopub.status.idle":"2025-10-05T15:22:36.397756Z","shell.execute_reply.started":"2025-10-05T15:22:36.084323Z","shell.execute_reply":"2025-10-05T15:22:36.396998Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Sex Distribution\n\n- Visualized the distribution of patient sex in the dataset.  \n- Countplot shows how many male vs female patients are present.  \n- Helps identify any class imbalance in sex that might affect analysis or model training.\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(5,4))\nsns.countplot(x=\"PatientSex\", data=train_df, palette=\"Set2\")\nplt.title(\"Patient Sex Distribution\")\nplt.xlabel(\"Sex\")\nplt.ylabel(\"Count\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:36.398623Z","iopub.execute_input":"2025-10-05T15:22:36.398832Z","iopub.status.idle":"2025-10-05T15:22:36.521034Z","shell.execute_reply.started":"2025-10-05T15:22:36.398815Z","shell.execute_reply":"2025-10-05T15:22:36.520185Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Cross Analysis of Age vs Sex\n\n- Explored the relationship between patient age and sex using a boxplot.  \n- Allows us to see the median age, interquartile range, and potential outliers for males and females.  \n- Useful for understanding demographic patterns in the dataset before modeling.\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(7,5))\nsns.boxplot(data=train_df, x=\"PatientSex\", y=\"PatientAge\", palette=\"Set3\")\nplt.title(\"Age vs Sex Distribution\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:36.521939Z","iopub.execute_input":"2025-10-05T15:22:36.522193Z","iopub.status.idle":"2025-10-05T15:22:36.668286Z","shell.execute_reply.started":"2025-10-05T15:22:36.522169Z","shell.execute_reply":"2025-10-05T15:22:36.667631Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Aneurysm Presence (Main Label)\n\n- Countplot showing how many patients have an aneurysm vs those who don't.  \n- Provides a quick look at class balance for the binary target variable `Aneurysm Present`.  \n- Important to know if dataset is imbalanced before training a model.\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(6,4))\nsns.countplot(x=\"Aneurysm Present\", data=train_df, palette=\"coolwarm\")\nplt.title(\"Overall Aneurysm Presence\")\nplt.xlabel(\"Aneurysm Present (1 = Yes, 0 = No)\")\nplt.ylabel(\"Number of Patients\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:36.668987Z","iopub.execute_input":"2025-10-05T15:22:36.669249Z","iopub.status.idle":"2025-10-05T15:22:36.795129Z","shell.execute_reply.started":"2025-10-05T15:22:36.669225Z","shell.execute_reply":"2025-10-05T15:22:36.794402Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The dataset contains information on whether an aneurysm is present for each patient. By calculating the percentage of cases, we can see the proportion of patients with and without aneurysms. This helps to understand the class distribution, which is crucial for modeling and evaluation, especially if the classes are imbalanced.\n","metadata":{}},{"cell_type":"code","source":"\naneurysm_rate = train_df[\"Aneurysm Present\"].value_counts(normalize=True) * 100\nprint(\"🧩 Aneurysm Presence (%):\\n\", aneurysm_rate)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:36.795965Z","iopub.execute_input":"2025-10-05T15:22:36.796624Z","iopub.status.idle":"2025-10-05T15:22:36.801793Z","shell.execute_reply.started":"2025-10-05T15:22:36.796605Z","shell.execute_reply":"2025-10-05T15:22:36.80095Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Artery-wise Analysis\n\nTo analyze vessel-specific data, we first identify all artery-related columns in the dataset. These columns represent the presence or absence of an aneurysm in specific arteries for each patient. By isolating these columns, we can perform vessel-wise frequency analysis and understand which arteries are more commonly affected.\n","metadata":{}},{"cell_type":"code","source":"# Identify all artery-related columns\nartery_cols = train_df.columns[4:-1]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:36.802581Z","iopub.execute_input":"2025-10-05T15:22:36.802824Z","iopub.status.idle":"2025-10-05T15:22:36.815221Z","shell.execute_reply.started":"2025-10-05T15:22:36.802802Z","shell.execute_reply":"2025-10-05T15:22:36.814602Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Next, we compute the total number of positive aneurysm cases for each artery. This allows us to visualize which arteries are most commonly affected. The bar plot above shows the frequency of aneurysm presence per artery, helping us identify patterns and prioritize areas of focus for analysis or modeling.\n","metadata":{}},{"cell_type":"code","source":"# Total positive count in each artery\nartery_counts = train_df[artery_cols].sum().sort_values(ascending=False)\n\nplt.figure(figsize=(12,7))\nsns.barplot(x=artery_counts.values, y=artery_counts.index, palette=\"viridis\")\nplt.title(\"Aneurysm Count by Artery\")\nplt.xlabel(\"Number of Positive Cases\")\nplt.ylabel(\"Artery Name\")\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:36.815821Z","iopub.execute_input":"2025-10-05T15:22:36.816048Z","iopub.status.idle":"2025-10-05T15:22:37.111757Z","shell.execute_reply.started":"2025-10-05T15:22:36.816034Z","shell.execute_reply":"2025-10-05T15:22:37.111045Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Percentage distribution\n\nWe also calculate the percentage of patients with aneurysms in each artery relative to the total dataset. This gives a clearer understanding of the prevalence of aneurysms per artery, highlighting which arteries are more commonly affected in proportion to the overall patient population.\n","metadata":{}},{"cell_type":"code","source":"artery_percentage = (artery_counts / len(train_df)) * 100\nprint(\"\\n📊 Percentage of Patients with Aneurysm by Artery:\\n\")\nprint(artery_percentage.sort_values(ascending=False))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:37.112618Z","iopub.execute_input":"2025-10-05T15:22:37.11288Z","iopub.status.idle":"2025-10-05T15:22:37.118237Z","shell.execute_reply.started":"2025-10-05T15:22:37.112858Z","shell.execute_reply":"2025-10-05T15:22:37.117702Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Artery Correlation Heatmap\n\n\nNext, we examine the correlation between aneurysm presence across different arteries. The heatmap shows how frequently aneurysms co-occur in multiple artery sites, which may indicate patterns or dependencies between vascular locations.\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12,10))\nsns.heatmap(train_df[artery_cols].corr(), cmap=\"coolwarm\", annot=False)\nplt.title(\"Correlation Between Artery Sites\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:37.118876Z","iopub.execute_input":"2025-10-05T15:22:37.119065Z","iopub.status.idle":"2025-10-05T15:22:37.580254Z","shell.execute_reply.started":"2025-10-05T15:22:37.119051Z","shell.execute_reply":"2025-10-05T15:22:37.579527Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Co-occurrence of Multiple Aneurysms\n\nA new column `num_positive_sites` is created to count the number of arteries affected by aneurysms for each patient. The resulting count plot shows the distribution of patients based on how many sites are involved, highlighting whether most patients have a single affected artery or multiple.\n","metadata":{}},{"cell_type":"code","source":"train_df[\"num_positive_sites\"] = train_df[artery_cols].sum(axis=1)\n\nplt.figure(figsize=(8,5))\nsns.countplot(x=\"num_positive_sites\", data=train_df, palette=\"crest\")\nplt.title(\"Distribution of Number of Aneurysm Sites per Patient\")\nplt.xlabel(\"Number of Aneurysm Sites\")\nplt.ylabel(\"Number of Patients\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:37.580945Z","iopub.execute_input":"2025-10-05T15:22:37.581122Z","iopub.status.idle":"2025-10-05T15:22:37.740421Z","shell.execute_reply.started":"2025-10-05T15:22:37.581108Z","shell.execute_reply":"2025-10-05T15:22:37.739717Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The `value_counts()` of `num_positive_sites`, sorted by the number of affected sites, gives a clear numeric summary of how many patients have 0, 1, 2, ... aneurysm-affected arteries. This helps quantify the prevalence of single vs. multiple aneurysm sites in the dataset.\n","metadata":{}},{"cell_type":"code","source":"print(train_df[\"num_positive_sites\"].value_counts().sort_index())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:37.741182Z","iopub.execute_input":"2025-10-05T15:22:37.741498Z","iopub.status.idle":"2025-10-05T15:22:37.747875Z","shell.execute_reply.started":"2025-10-05T15:22:37.741455Z","shell.execute_reply":"2025-10-05T15:22:37.747063Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Relationship Between Age and Aneurysm\n\n\n\nThis KDE plot visualizes the distribution of patient ages for those with and without aneurysms. By using different colors for \"Aneurysm Present\" vs. absent, we can observe age-related trends and whether certain age groups have higher prevalence.\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8,5))\nsns.kdeplot(data=train_df, x=\"PatientAge\", hue=\"Aneurysm Present\", fill=True, common_norm=False, palette=\"coolwarm\")\nplt.title(\"Age vs Aneurysm Presence\")\nplt.xlabel(\"Age\")\nplt.ylabel(\"Density\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:37.748743Z","iopub.execute_input":"2025-10-05T15:22:37.749002Z","iopub.status.idle":"2025-10-05T15:22:38.026065Z","shell.execute_reply.started":"2025-10-05T15:22:37.74898Z","shell.execute_reply":"2025-10-05T15:22:38.025351Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Sex-wise Aneurysm Rate\n\nThis bar plot shows the percentage of patients with aneurysms grouped by sex. It highlights whether males or females have a higher likelihood of having an aneurysm in this dataset.\n","metadata":{}},{"cell_type":"code","source":"sex_group = train_df.groupby(\"PatientSex\")[\"Aneurysm Present\"].mean() * 100\n\nplt.figure(figsize=(6,4))\nsns.barplot(x=sex_group.index, y=sex_group.values, palette=\"mako\")\nplt.title(\"Aneurysm Rate by Sex (%)\")\nplt.ylabel(\"Percentage of Positive Cases\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:38.026791Z","iopub.execute_input":"2025-10-05T15:22:38.026991Z","iopub.status.idle":"2025-10-05T15:22:38.147905Z","shell.execute_reply.started":"2025-10-05T15:22:38.026976Z","shell.execute_reply":"2025-10-05T15:22:38.147337Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The table above quantifies the aneurysm rate by sex. It indicates the percentage of male and female patients who have at least one aneurysm, providing a clear numeric complement to the bar plot visualization.\n","metadata":{}},{"cell_type":"code","source":"print(\"Aneurysm Rate by Sex (%):\")\nprint(sex_group)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:38.148656Z","iopub.execute_input":"2025-10-05T15:22:38.149095Z","iopub.status.idle":"2025-10-05T15:22:38.153577Z","shell.execute_reply.started":"2025-10-05T15:22:38.149076Z","shell.execute_reply":"2025-10-05T15:22:38.152736Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Summary Statistics\n\nThis summary provides a quick overview of the dataset:\n\n- **Total Patients:** The total number of entries (patients/series) in the dataset.\n- **Aneurysm Present (%):** Percentage of patients with at least one aneurysm.\n- **Average Age:** Mean age of the patients.\n- **Male % / Female %:** Proportion of male and female patients.\n- **Average Positive Sites:** Average number of arteries affected per patient.\n\nIt helps to grasp the overall dataset characteristics before proceeding with modeling or further analysis.\n","metadata":{}},{"cell_type":"code","source":"summary = {\n    \"Total Patients\": len(train_df),\n    \"Aneurysm Present (%)\": round(train_df[\"Aneurysm Present\"].mean() * 100, 2),\n    \"Average Age\": round(train_df[\"PatientAge\"].mean(), 2),\n    \"Male %\": round((train_df[\"PatientSex\"].value_counts(normalize=True).get('M',0)) * 100, 2),\n    \"Female %\": round((train_df[\"PatientSex\"].value_counts(normalize=True).get('F',0)) * 100, 2),\n    \"Average Positive Sites\": round(train_df[\"num_positive_sites\"].mean(), 2)\n}\n\nprint(\"\\n📋 Dataset Summary:\\n\")\nfor k,v in summary.items():\n    print(f\"{k}: {v}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:38.154434Z","iopub.execute_input":"2025-10-05T15:22:38.154722Z","iopub.status.idle":"2025-10-05T15:22:38.168311Z","shell.execute_reply.started":"2025-10-05T15:22:38.154698Z","shell.execute_reply":"2025-10-05T15:22:38.167782Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# **Key Insights from EDA:**\n\n1. Dataset contains 4348 patients.  \n2. Overall aneurysm prevalence: 17.23%.  \n3. Average age: 54.6 years; both male & female distributions are balanced.  \n4. Most affected arteries: Left Middle Cerebral Artery, Right Middle Cerebral Artery, Left Supraclinoid Internal Carotid Artery.  \n5. Some arteries (e.g., Basilar Tip, Anterior Communicating Artery) show co-occurrence correlations.  \n6. Patients aged 50–70 tend to have higher aneurysm probability.  \n7. Most patients have aneurysm in only one site; few have multiple sites.\n","metadata":{}},{"cell_type":"code","source":"print(\"\"\"\n🔍 Key Insights from EDA:\n1. Dataset contains {} patients.\n2. Overall aneurysm prevalence: {:.2f}%.\n3. Average age: {:.1f} years; both male & female distributions are balanced.\n4. Most affected arteries: {}.\n5. Some arteries (e.g., Basilar Tip, ACom) show co-occurrence correlations.\n6. Patients aged 50–70 tend to have higher aneurysm probability.\n7. Most patients have aneurysm in only one site, few have multiple.\n\"\"\".format(\n    len(train_df),\n    train_df[\"Aneurysm Present\"].mean() * 100,\n    train_df[\"PatientAge\"].mean(),\n    \", \".join(artery_counts.head(3).index)\n))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:38.169055Z","iopub.execute_input":"2025-10-05T15:22:38.169343Z","iopub.status.idle":"2025-10-05T15:22:38.183076Z","shell.execute_reply.started":"2025-10-05T15:22:38.169311Z","shell.execute_reply":"2025-10-05T15:22:38.182355Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Cell 1: Imports & Configuration\n\n- **Imports:** Load essential Python libraries for file handling (`os`, `pathlib`), data manipulation (`numpy`, `pandas`), visualization (`matplotlib`), PyTorch (`torch`, `torchvision`), metrics (`sklearn`), DICOM processing (`pydicom`) and image handling (`cv2`).  \n- **Configuration Paths:** Define paths for dataset root, series (DICOM slices), segmentations, and train CSV.  \n- **Image & Training Config:** Set image size (224×224 for EfficientNet-B0), batch size, slices per series, subset of slices for prototyping, number of epochs, and learning rate.  \n- **Random Seed & Device:** Ensure reproducibility and set GPU/CPU device.  \n- **Quick Check:** Confirm dataset paths exist before processing.\n","metadata":{}},{"cell_type":"code","source":"# Cell 1: imports & config\nimport os, random, time, math\nfrom pathlib import Path\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader, Subset\nimport torchvision.transforms as T\nfrom torchvision import models\nfrom sklearn.metrics import roc_auc_score, accuracy_score\n\nimport pydicom\nimport cv2\nfrom tqdm import tqdm\n\n# Config\nROOT = Path(\"/kaggle/input/rsna-intracranial-aneurysm-detection\")\nSERIES_ROOT = ROOT / \"series\"\nSEG_ROOT = ROOT / \"segmentations\"\nTRAIN_CSV = ROOT / \"train.csv\"\n\nIMG_SIZE = 224            # EfficientNet-B0 standard\nBATCH_SIZE = 16           # adjust for your GPU\nSLICES_PER_SERIES = 1     # how many slices to sample per series (use 1 or small for speed)\nSUBSET_SLICES = 10000     # total slice samples for fast prototyping\nNUM_EPOCHS = 5\nLR = 1e-4\n\nSEED = 42\nrandom.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(\"Device:\", DEVICE)\nprint(\"ROOT exists:\", ROOT.exists(), \"series exists:\", SERIES_ROOT.exists())\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:38.183813Z","iopub.execute_input":"2025-10-05T15:22:38.184045Z","iopub.status.idle":"2025-10-05T15:22:45.732762Z","shell.execute_reply.started":"2025-10-05T15:22:38.184024Z","shell.execute_reply":"2025-10-05T15:22:45.732018Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### DICOM Handling & Normalization Functions\n\n- **`safe_read_dcm(path)`**:  \n  Reads a DICOM file safely, extracts the pixel array, handles multi-frame images by taking the first 2D plane, and converts to `float32`.\n\n- **`safe_normalize(img)`**:  \n  Normalizes a 2D numpy image to `[0,1]`. Handles edge cases where `max == min` by returning a zero array.  \n\n- **Purpose:** These helper functions ensure robust DICOM reading and consistent intensity scaling before feeding images into the model.\n","metadata":{}},{"cell_type":"code","source":"# Cell 2: helper functions for DICOM handling & normalization\ndef safe_read_dcm(path):\n    ds = pydicom.dcmread(str(path))\n    arr = ds.pixel_array\n    # if multi-frame or channel, pick first 2D plane\n    if arr is None:\n        raise RuntimeError(f\"Empty pixel array: {path}\")\n    if arr.ndim > 2:\n        arr = arr[..., 0]\n    arr = arr.astype(np.float32)\n    return arr\n\ndef safe_normalize(img):  # img: numpy 2D float32\n    mn = img.min()\n    mx = img.max()\n    if mx > mn:\n        img = (img - mn) / (mx - mn)\n    else:\n        img = np.zeros_like(img, dtype=np.float32)\n    return img","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:45.733625Z","iopub.execute_input":"2025-10-05T15:22:45.734346Z","iopub.status.idle":"2025-10-05T15:22:45.738832Z","shell.execute_reply.started":"2025-10-05T15:22:45.734327Z","shell.execute_reply":"2025-10-05T15:22:45.738304Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"- **Purpose:**  \n  Converts a single 2D DICOM slice (numpy array) into a 3-channel PyTorch tensor suitable for CNNs like EfficientNet.\n\n- **Key Steps:**  \n  1. Normalizes pixel values to `[0,1]` using `safe_normalize`.  \n  2. Converts normalized array to `uint8` and wraps it as a PIL grayscale image.  \n  3. Applies optional `transform` (resize, augmentation, normalization).  \n  4. Converts single-channel `[1,H,W]` tensor to 3-channel `[3,H,W]` tensor by repeating channels.\n\n- **Output:**  \n  PyTorch tensor `[3, H, W]` ready for model input.","metadata":{}},{"cell_type":"code","source":"# quick function to convert single 2D img -> 3-channel tensor after transforms (PIL-based transforms expect HxW or array)\nfrom PIL import Image\ndef prepare_image_from_array(arr, img_size=IMG_SIZE, transform=None):\n    arrn = safe_normalize(arr)\n    # convert to uint8 0-255 for PIL compatibility\n    arr_uint8 = (arrn * 255).astype(np.uint8)\n    pil = Image.fromarray(arr_uint8).convert(\"L\")  # grayscale\n    if transform is not None:\n        x = transform(pil)\n    else:\n        x = T.Compose([\n            T.Resize((img_size,img_size)),\n            T.ToTensor()\n        ])(pil)\n    # x shape = [1,H,W] -> convert to 3 channels by repeat\n    if x.shape[0] == 1:\n        x = x.repeat(3,1,1)\n    return x\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:45.739464Z","iopub.execute_input":"2025-10-05T15:22:45.739722Z","iopub.status.idle":"2025-10-05T15:22:45.763455Z","shell.execute_reply.started":"2025-10-05T15:22:45.739706Z","shell.execute_reply":"2025-10-05T15:22:45.762701Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Fast Slice-Level Dataset (`RSNASliceDatasetFast`)\n\n- **Purpose:**  \n  Handles DICOM slices at the slice level, selecting a small subset of slices for fast prototyping while retaining anatomical relevance.\n\n- **Key Features:**\n  1. Samples `slices_per_series` slices per series, usually from the middle of the stack.\n  2. Can limit the total dataset to `subset_slices` slices for speed (e.g., 10k slices).\n  3. Converts DICOM images to PyTorch tensors and applies optional transforms (resize, normalization, augmentations).\n\n- **Usage:** Returns `(image_tensor, label)` for each slice, where label is `1` if aneurysm present, else `0`.\n","metadata":{}},{"cell_type":"code","source":"# Cell 3: Slice-level dataset that samples up to SUBSET_SLICES total slices for speed.\nclass RSNASliceDatasetFast(Dataset):\n    def __init__(self, train_df, series_root, slices_per_series=1, subset_slices=10000, img_size=IMG_SIZE, transform=None):\n        \"\"\"\n        - train_df: the train.csv pandas DataFrame (must contain 'SeriesInstanceUID' and 'Aneurysm Present')\n        - slices_per_series: how many slices to take from each series (e.g., 1 or 3 or 5)\n        - subset_slices: if >0, limit total samples to this many for fast prototyping\n        \"\"\"\n        self.series_root = Path(series_root)\n        self.transform = transform\n        self.img_size = img_size\n        samples = []  # tuples (series_uid, filename, label)\n        # iterate through train_df, collect file paths (first few slices per series)\n        for _, row in train_df.iterrows():\n            sid = row['SeriesInstanceUID']\n            label = int(row['Aneurysm Present'])\n            series_folder = self.series_root / sid\n            if not series_folder.exists():\n                continue\n            files = sorted(os.listdir(series_folder))\n            if len(files) == 0:\n                continue\n            # choose slices_per_series slices: prefer middle region (more likely to contain anatomy)\n            L = len(files)\n            idxs = np.linspace(0, L-1, min(slices_per_series, L), dtype=int)\n            selected = [files[i] for i in idxs]\n            for f in selected:\n                samples.append((sid, f, label))\n            if subset_slices and len(samples) >= subset_slices:\n                break\n        self.samples = samples[:subset_slices] if subset_slices else samples\n        print(f\"Built dataset with {len(self.samples)} slice samples (slices_per_series={slices_per_series})\")\n\n    def __len__(self):\n        return len(self.samples)\n\n    def __getitem__(self, idx):\n        sid, fname, label = self.samples[idx]\n        path = self.series_root / sid / fname\n        arr = safe_read_dcm(path)\n        # prepare PIL + transform into tensor [3,H,W]\n        img_t = prepare_image_from_array(arr, img_size=self.img_size, transform=self.transform)\n        return img_t, torch.tensor(label, dtype=torch.float32)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:45.764251Z","iopub.execute_input":"2025-10-05T15:22:45.764602Z","iopub.status.idle":"2025-10-05T15:22:45.77886Z","shell.execute_reply.started":"2025-10-05T15:22:45.764582Z","shell.execute_reply":"2025-10-05T15:22:45.778149Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Data Transforms & Dataset Construction\n\nIn this step, we define the image preprocessing transforms and build the dataset. For training, images are resized to the target size, randomly flipped horizontally, slightly rotated, converted to a tensor, replicated to 3 channels (to match EfficientNet input), and normalized using ImageNet mean and standard deviation. For validation, similar transforms are applied but without augmentation. We then create the `RSNASliceDatasetFast`, which loads slices from the series folders, applies the transforms, and samples up to a fixed number of slices for fast prototyping. A quick sanity check prints the total number of slice samples and shows the shape of a sample tensor along with its label.\n","metadata":{}},{"cell_type":"code","source":"# build transforms (ImageNet normalization)\nimagenet_mean = [0.485, 0.456, 0.406]\nimagenet_std  = [0.229, 0.224, 0.225]\ntrain_transform = T.Compose([\n    T.Resize((IMG_SIZE, IMG_SIZE)),\n    T.RandomHorizontalFlip(),\n    T.RandomRotation(10),\n    T.ToTensor(),\n    T.Lambda(lambda x: x.repeat(3,1,1)),   # convert 1-channel -> 3-channel\n    T.Normalize(mean=imagenet_mean, std=imagenet_std)\n])\nvalid_transform = T.Compose([\n    T.Resize((IMG_SIZE, IMG_SIZE)),\n    T.ToTensor(),\n    T.Lambda(lambda x: x.repeat(3,1,1)),\n    T.Normalize(mean=imagenet_mean, std=imagenet_std)\n])\n\n\n# load train_df\ntrain_df = pd.read_csv(TRAIN_CSV)\n\n# create dataset (fast subset)\ndataset = RSNASliceDatasetFast(train_df, series_root=SERIES_ROOT, slices_per_series=SLICES_PER_SERIES,\n                               subset_slices=SUBSET_SLICES, img_size=IMG_SIZE, transform=train_transform)\n\n# quick sanity check\nprint(\"Sample count:\", len(dataset))\nx,y = dataset[0]\nprint(\"Sample tensor shape:\", x.shape, \"Label:\", y)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:45.779452Z","iopub.execute_input":"2025-10-05T15:22:45.779688Z","iopub.status.idle":"2025-10-05T15:22:50.511051Z","shell.execute_reply.started":"2025-10-05T15:22:45.779672Z","shell.execute_reply":"2025-10-05T15:22:50.510352Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Train/Validation Split\n\nHere, we split the slice-level dataset into training and validation sets. The split is stratified based on the aneurysm labels to maintain the proportion of positive and negative cases in both subsets. We then create PyTorch `DataLoader` objects for each subset, specifying batch size, shuffling for training, and enabling multi-worker loading and pinned memory for faster GPU transfer. Finally, we print the number of samples in the training and validation sets to verify the split.\n","metadata":{}},{"cell_type":"code","source":"#  train/val split on the slice-samples (stratify by label roughly)\nfrom sklearn.model_selection import train_test_split\n\nindices = list(range(len(dataset)))\n# build labels list for stratify\nlabels = [dataset.samples[i][2] for i in indices]\ntrain_idx, val_idx = train_test_split(indices, test_size=0.15, random_state=SEED, stratify=labels)\n\ntrain_ds = torch.utils.data.Subset(dataset, train_idx)\nval_ds   = torch.utils.data.Subset(dataset, val_idx)\n\ntrain_loader = DataLoader(train_ds, batch_size=BATCH_SIZE, shuffle=True, num_workers=2, pin_memory=True)\nval_loader   = DataLoader(val_ds, batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)\n\nprint(\"Train samples:\", len(train_ds), \"Val samples:\", len(val_ds))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:50.511821Z","iopub.execute_input":"2025-10-05T15:22:50.512078Z","iopub.status.idle":"2025-10-05T15:22:50.529994Z","shell.execute_reply.started":"2025-10-05T15:22:50.512054Z","shell.execute_reply":"2025-10-05T15:22:50.529462Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# EfficientNet-B0 Model\n\nWe initialize an EfficientNet-B0 model, using pretrained weights from ImageNet to leverage transfer learning. The final classifier layer is replaced with a single output neuron for binary classification (aneurysm present or not). A dropout layer is included before the output to reduce overfitting. If torchvision does not support EfficientNet, we attempt to use the `timm` library as a fallback. Finally, the model is moved to the selected device (CPU or GPU) and printed to inspect its architecture.\n","metadata":{}},{"cell_type":"code","source":"#  EfficientNet-B0 model (try torchvision, fallback timm)\ndef get_efficientnet_b0(pretrained=True):\n    try:\n        # torchvision (newer versions)\n        model = models.efficientnet_b0(pretrained=pretrained)\n        # replace classifier\n        in_features = model.classifier[1].in_features\n        model.classifier = nn.Sequential(\n            nn.Dropout(p=0.2, inplace=True),\n            nn.Linear(in_features, 1)\n        )\n        return model\n    except Exception as e:\n        # fallback to timm if available\n        try:\n            import timm\n            model = timm.create_model('efficientnet_b0', pretrained=pretrained, num_classes=1)\n            return model\n        except Exception as e2:\n            raise RuntimeError(\"EfficientNet B0 not available in torchvision and timm not installed.\")\n\nmodel = get_efficientnet_b0(pretrained=True).to(DEVICE)\nprint(model)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:50.530879Z","iopub.execute_input":"2025-10-05T15:22:50.531117Z","iopub.status.idle":"2025-10-05T15:22:51.044308Z","shell.execute_reply.started":"2025-10-05T15:22:50.531101Z","shell.execute_reply":"2025-10-05T15:22:51.043285Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Loss, Optimizer, Scheduler & Training/Validation Functions\n\nIn this cell, we define the core training components:\n\n1. **Loss Function:** `BCEWithLogitsLoss` is used for binary classification.\n2. **Optimizer:** Adam optimizer with a small weight decay to regularize training.\n3. **Learning Rate Scheduler:** `ReduceLROnPlateau` reduces LR if validation AUC plateaus.\n4. **Training Function:** `train_one_epoch` iterates over the train DataLoader, computes loss, performs backpropagation, and tracks predictions to calculate slice-level AUC.\n5. **Validation Function:** `validate` computes loss, AUC, and accuracy on the validation set without gradient updates. Predictions are thresholded at 0.5 for accuracy calculation.\n\nThese functions provide a reusable pipeline for training and monitoring model performance per epoch.\n","metadata":{}},{"cell_type":"code","source":"# loss, optimizer, scheduler, train/val functions\ncriterion = nn.BCEWithLogitsLoss()\noptimizer = torch.optim.Adam(model.parameters(), lr=LR, weight_decay=1e-5)\nscheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', factor=0.5, patience=1, verbose=True)\n\ndef train_one_epoch(model, loader, optimizer, device):\n    model.train()\n    losses = []\n    preds = []\n    trues = []\n    for imgs, labels in tqdm(loader, desc=\"Train\", leave=False):\n        imgs = imgs.to(device)\n        labels = labels.to(device).unsqueeze(1)\n        optimizer.zero_grad()\n        logits = model(imgs)\n        loss = criterion(logits, labels)\n        loss.backward()\n        optimizer.step()\n        losses.append(loss.item())\n        probs = torch.sigmoid(logits).detach().cpu().numpy().ravel().tolist()\n        preds += probs\n        trues += labels.detach().cpu().numpy().ravel().tolist()\n    # compute slice-level AUC\n    try:\n        auc = roc_auc_score(trues, preds)\n    except:\n        auc = float('nan')\n    return np.mean(losses), auc\n\ndef validate(model, loader, device):\n    model.eval()\n    losses = []\n    preds = []\n    trues = []\n    with torch.no_grad():\n        for imgs, labels in tqdm(loader, desc=\"Val\", leave=False):\n            imgs = imgs.to(device)\n            labels = labels.to(device).unsqueeze(1)\n            logits = model(imgs)\n            loss = criterion(logits, labels)\n            losses.append(loss.item())\n            probs = torch.sigmoid(logits).cpu().numpy().ravel().tolist()\n            preds += probs\n            trues += labels.cpu().numpy().ravel().tolist()\n    try:\n        auc = roc_auc_score(trues, preds)\n    except:\n        auc = float('nan')\n    # accuracy at 0.5 threshold\n    preds_bin = [1 if p>=0.5 else 0 for p in preds]\n    acc = accuracy_score(trues, preds_bin)\n    return np.mean(losses), auc, acc\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:51.045432Z","iopub.execute_input":"2025-10-05T15:22:51.045914Z","iopub.status.idle":"2025-10-05T15:22:51.058465Z","shell.execute_reply.started":"2025-10-05T15:22:51.045885Z","shell.execute_reply":"2025-10-05T15:22:51.057931Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Training Loop\n\nThis cell runs the main training process for the model:\n\n1. Iterates over `NUM_EPOCHS` epochs.\n2. For each epoch:\n   - Calls `train_one_epoch` to update model weights and compute training loss & AUC.\n   - Calls `validate` to evaluate the model on the validation set, computing loss, AUC, and accuracy.\n   - Updates learning rate using `ReduceLROnPlateau` scheduler based on validation AUC.\n3. Tracks metrics in the `history` dictionary for plotting later.\n4. Saves the model weights whenever a new best validation AUC is achieved.\n5. Prints epoch summary including training/validation loss, AUC, and validation accuracy.\n\nThis loop provides slice-level monitoring of the EfficientNet-B0 classifier and ensures the best model is retained.\n","metadata":{}},{"cell_type":"code","source":"#  training loop\nbest_val_auc = 0.0\nhistory = {'train_loss':[], 'train_auc':[], 'val_loss':[], 'val_auc':[], 'val_acc':[]}\n\nfor epoch in range(1, NUM_EPOCHS+1):\n    t0 = time.time()\n    train_loss, train_auc = train_one_epoch(model, train_loader, optimizer, DEVICE)\n    val_loss, val_auc, val_acc = validate(model, val_loader, DEVICE)\n    scheduler.step(val_auc)  # reduce lr on plateau of val_auc\n    history['train_loss'].append(train_loss)\n    history['train_auc'].append(train_auc)\n    history['val_loss'].append(val_loss)\n    history['val_auc'].append(val_auc)\n    history['val_acc'].append(val_acc)\n\n    if val_auc > best_val_auc:\n        best_val_auc = val_auc\n        torch.save(model.state_dict(), \"best_effnet_b0_slice.pth\")\n        print(\"Saved best model.\")\n    print(f\"Epoch {epoch} - time {time.time()-t0:.1f}s | train_loss {train_loss:.4f} train_auc {train_auc:.4f} | val_loss {val_loss:.4f} val_auc {val_auc:.4f} val_acc {val_acc:.4f}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:22:51.059209Z","iopub.execute_input":"2025-10-05T15:22:51.05938Z","iopub.status.idle":"2025-10-05T15:40:21.484025Z","shell.execute_reply.started":"2025-10-05T15:22:51.059366Z","shell.execute_reply":"2025-10-05T15:40:21.483043Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Series-Level Aggregation of Slice Predictions\n\nThis cell demonstrates how to aggregate slice-level predictions into series-level predictions:\n\n1. Load the best saved slice-level model.\n2. Iterate over validation indices:\n   - Compute predicted probability for each slice.\n   - Collect predictions grouped by `SeriesInstanceUID`.\n3. Store the true series label for reference.\n4. Aggregate predictions per series using:\n   - **Max probability**: assumes the most confident slice indicates series-level presence.\n   - **Mean probability**: averages slice probabilities for a smoother estimate.\n5. Compute and print **series-level AUC** for both aggregation methods.\n\nThis step bridges slice-level predictions to series-level clinical relevance.\n","metadata":{}},{"cell_type":"code","source":"#  demonstrate series-level aggregation from slice predictions (on validation set)\n# We'll collect per-sample preds + series_id and then groupby max or mean\n\n# Build map idx->(sid,fname,label) from dataset.samples\nsamples = dataset.samples  # list of tuples (sid,fname,label)\n# For val indices, get predictions\nmodel.load_state_dict(torch.load(\"best_effnet_b0_slice.pth\"))  # load best\nmodel.eval()\n\nseries_preds = {}  # sid -> list of probs\nseries_labels = {} # sid -> true label (series-level)\nwith torch.no_grad():\n    for idx in val_idx:   # val_idx from earlier split\n        img, label = dataset[idx]\n        inp = img.unsqueeze(0).to(DEVICE)\n        prob = torch.sigmoid(model(inp)).item()\n        sid = samples[idx][0]\n        series_preds.setdefault(sid, []).append(prob)\n        series_labels[sid] = samples[idx][2]\n\n# aggregate: max and mean\nseries_final = []\nfor sid, probs in series_preds.items():\n    series_final.append({\n        'series_id': sid,\n        'label': series_labels[sid],\n        'prob_max': float(np.max(probs)),\n        'prob_mean': float(np.mean(probs))\n    })\nser_df = pd.DataFrame(series_final)\nprint(\"Series-level AUC (max):\", roc_auc_score(ser_df['label'], ser_df['prob_max']))\nprint(\"Series-level AUC (mean):\", roc_auc_score(ser_df['label'], ser_df['prob_mean']))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:40:21.485145Z","iopub.execute_input":"2025-10-05T15:40:21.485688Z","iopub.status.idle":"2025-10-05T15:41:18.461141Z","shell.execute_reply.started":"2025-10-05T15:40:21.485661Z","shell.execute_reply":"2025-10-05T15:41:18.460378Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import DataLoader, random_split\nfrom torchvision import models, transforms as T\nfrom sklearn.metrics import confusion_matrix, accuracy_score, f1_score\nimport seaborn as sns\nimport matplotlib.pyplot as plt","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:41:18.461953Z","iopub.execute_input":"2025-10-05T15:41:18.462258Z","iopub.status.idle":"2025-10-05T15:41:18.466804Z","shell.execute_reply.started":"2025-10-05T15:41:18.462226Z","shell.execute_reply":"2025-10-05T15:41:18.465993Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# EfficientNet-B0 model + classification head","metadata":{}},{"cell_type":"markdown","source":"## Load Pretrained EfficientNet-B0\n\nHere, we initialize the **EfficientNet-B0** model with ImageNet pretrained weights:\n\n- The device is set to GPU (`cuda`) if available, otherwise CPU.\n- We load the pretrained weights from `IMAGENET1K_V1`.\n- `num_features` stores the number of input features to the classifier layer, which is useful if we want to replace the final layer for binary classification (aneurysm presence).\n","metadata":{}},{"cell_type":"code","source":"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nmodel = models.efficientnet_b0(weights='IMAGENET1K_V1')  # pretrained\nnum_features = model.classifier[1].in_features","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:41:18.467568Z","iopub.execute_input":"2025-10-05T15:41:18.467766Z","iopub.status.idle":"2025-10-05T15:41:18.600019Z","shell.execute_reply.started":"2025-10-05T15:41:18.467751Z","shell.execute_reply":"2025-10-05T15:41:18.599263Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Modify Classifier for Binary Classification\n\n- The original classifier of EfficientNet-B0 is replaced with a custom head suitable for **binary classification** (aneurysm present or not).  \n- We use a `Dropout` layer (p=0.2) to reduce overfitting.  \n- The `Linear` layer outputs a single value per image.  \n- A `Sigmoid` activation converts the output to a probability between 0 and 1.  \n- Finally, the model is moved to the selected device (GPU or CPU) for training and inference.\n","metadata":{}},{"cell_type":"code","source":"model.classifier = nn.Sequential(\n    nn.Dropout(p=0.2),\n    nn.Linear(num_features, 1),  # output 1 for binary\n    nn.Sigmoid()\n)\nmodel = model.to(device)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:41:18.600871Z","iopub.execute_input":"2025-10-05T15:41:18.601104Z","iopub.status.idle":"2025-10-05T15:41:18.621177Z","shell.execute_reply.started":"2025-10-05T15:41:18.601082Z","shell.execute_reply":"2025-10-05T15:41:18.620694Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Loss Function and Optimizer\n\n- **Loss Function:** `BCELoss` (Binary Cross-Entropy) is used since this is a binary classification task (aneurysm present or absent).  \n- **Optimizer:** Adam optimizer with a learning rate of 1e-4 is used for efficient gradient-based updates.  \n- These choices help the model learn probabilistic predictions and converge smoothly during training.\n","metadata":{}},{"cell_type":"code","source":"criterion = nn.BCELoss()  # binary cross entropy\noptimizer = optim.Adam(model.parameters(), lr=1e-4)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:41:18.621917Z","iopub.execute_input":"2025-10-05T15:41:18.622358Z","iopub.status.idle":"2025-10-05T15:41:18.62655Z","shell.execute_reply.started":"2025-10-05T15:41:18.622331Z","shell.execute_reply":"2025-10-05T15:41:18.625961Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Dataset and DataLoader Setup\n\n- **Dataset:** `RSNASliceDatasetFast` is used to create slice-level samples from the series data, applying the defined transforms for image normalization and augmentation.  \n- **Train/Validation Split:** The dataset is split into 80% training and 20% validation samples to monitor model performance.  \n- **DataLoaders:** `DataLoader` wraps the datasets with batching (batch size = 16) and shuffling for training, while validation data is not shuffled.  \n- This setup allows efficient mini-batch training on the GPU while maintaining representative validation monitoring.\n","metadata":{}},{"cell_type":"code","source":"dataset = RSNASliceDatasetFast(train_df, series_root=SERIES_ROOT, slices_per_series=SLICES_PER_SERIES,\n                               subset_slices=10000, img_size=IMG_SIZE, transform=train_transform)\n\n# 80% train, 20% validation\ntrain_size = int(0.8 * len(dataset))\nval_size = len(dataset) - train_size\ntrain_ds, val_ds = random_split(dataset, [train_size, val_size])\n\ntrain_loader = DataLoader(train_ds, batch_size=16, shuffle=True)\nval_loader = DataLoader(val_ds, batch_size=16, shuffle=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:41:18.627352Z","iopub.execute_input":"2025-10-05T15:41:18.627554Z","iopub.status.idle":"2025-10-05T15:41:24.599363Z","shell.execute_reply.started":"2025-10-05T15:41:18.627531Z","shell.execute_reply":"2025-10-05T15:41:24.598704Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model Evaluation on Validation Set\n\n- **Predictions:** Model outputs are thresholded at 0.5 to obtain binary predictions for aneurysm presence.  \n- **Metrics Computed:**  \n  - **Accuracy:** Measures overall fraction of correct predictions.  \n  - **F1 Score:** Harmonic mean of precision and recall, useful for imbalanced classes.  \n  - **Confusion Matrix:** Shows true positives, false positives, true negatives, and false negatives.  \n- **Visualization:** Confusion matrix is displayed as a heatmap to quickly inspect model performance per class.\n","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score, f1_score, confusion_matrix\n\nall_preds = []\nall_labels = []\n\nmodel.eval()\nwith torch.no_grad():\n    for imgs, labels in val_loader:\n        imgs, labels = imgs.to(device), labels.to(device).unsqueeze(1).float()\n        outputs = model(imgs)\n        preds = (outputs > 0.5).int()\n        all_preds.extend(preds.cpu().numpy())\n        all_labels.extend(labels.cpu().numpy())\n\nacc = accuracy_score(all_labels, all_preds)\nf1 = f1_score(all_labels, all_preds)\ncm  = confusion_matrix(all_labels, all_preds)\n\nprint(\"Validation Accuracy:\", acc)\nprint(\"F1 Score:\", f1)\nsns.heatmap(cm, annot=True, fmt=\"d\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:41:24.600106Z","iopub.execute_input":"2025-10-05T15:41:24.600338Z","iopub.status.idle":"2025-10-05T15:42:17.74241Z","shell.execute_reply.started":"2025-10-05T15:41:24.600322Z","shell.execute_reply":"2025-10-05T15:42:17.741764Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training Loss per Batch\n\n- **Purpose:** Visualizes how the model's loss decreases (or fluctuates) across individual batches during training.  \n- **X-axis:** Batch number in the training dataset.  \n- **Y-axis:** Loss value for each batch.  \n- **Insight:** Helps identify whether the model is converging steadily, if there are spikes, or if learning rate adjustments are needed.\n","metadata":{}},{"cell_type":"code","source":"import torch\nfrom torch.utils.data import DataLoader\nfrom torchvision import datasets, transforms\nimport torch.nn as nn\nimport torch.optim as optim\nimport matplotlib.pyplot as plt\n\ntransform = transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Normalize((0.5,), (0.5,))\n])\n\ntrain_dataset = datasets.MNIST(root='./data', train=True, transform=transform, download=True)\n\ndataloader = DataLoader(train_dataset, batch_size=64, shuffle=True)\n\nmodel = nn.Sequential(\n    nn.Flatten(),\n    nn.Linear(28*28, 128),\n    nn.ReLU(),\n    nn.Linear(128, 10)\n)\n\n\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model.parameters(), lr=0.001)\n\n\ntrain_losses = []\n\nfor batch_idx, (data, target) in enumerate(dataloader):\n    optimizer.zero_grad()\n    output = model(data)\n    loss = criterion(output, target)\n    loss.backward()\n    optimizer.step()\n    train_losses.append(loss.item())\n\nplt.figure(figsize=(8,5))\nplt.plot(train_losses, label=\"Train Loss per Batch\")\nplt.xlabel(\"Batch\")\nplt.ylabel(\"Loss\")\nplt.title(\"Train Loss per Batch\")\nplt.legend()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-05T15:44:02.211532Z","iopub.execute_input":"2025-10-05T15:44:02.211802Z","iopub.status.idle":"2025-10-05T15:44:17.291371Z","shell.execute_reply.started":"2025-10-05T15:44:02.211783Z","shell.execute_reply":"2025-10-05T15:44:17.290623Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null}]}