{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.13"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":99552,"databundleVersionId":13190393,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"b1d17e13","cell_type":"markdown","source":"# 🧠 RSNA Intracranial Aneurysm Detection - EDA & 3D Visualization\n\nIntracranial aneurysms affect approximately 3% of the global population, with a devastating rupture rate leading to nearly 500,000 deaths annually—half of them under the age of 50. Early detection is critical, yet many aneurysms go undiagnosed until it's too late.\n\nIn this Kaggle competition, organized by **RSNA** in collaboration with **ASNR**, **SNIS**, and **ESNR**, the goal is to develop machine learning models that can accurately **detect and localize intracranial aneurysms** across diverse medical imaging modalities—CTA, MRA, and MRI-collected from multiple institutions with real-world variability.\n\n---\n\n### 📘 What this Notebook Covers:\n\n- 🔍 Exploratory Data Analysis (EDA) of dicom images, segmentations and train files  \n- 🧩 3D volume rendering of medical scans and aneurysm segmentations using `nibabel` and `widgets.interactive`\n- 📊 Visual summaries and plots to gain insights into the dataset  \n\nThis notebook is part of my effort to understand and visualize the problem space before diving into modeling. I hope it’s helpful for others exploring this important task.\n\n> 👤 Author: [Tuyen Dam](https://www.linkedin.com/in/tuy%C3%AAn-%C4%91%C3%A0m-tr%E1%BB%8Dng-b8007a183/)","metadata":{}},{"id":"0b20a429","cell_type":"markdown","source":"# Import requirements","metadata":{}},{"id":"b740f6b3","cell_type":"code","source":"from glob import glob\nfrom tqdm import tqdm\nimport pandas as pd\nimport pydicom\nfrom pydicom.filereader import dcmread\nimport pyvista as pv\nimport natsort\nimport nibabel as nib\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport os\nimport seaborn as sns\nfrom pathlib import Path\nfrom datetime import datetime\nimport warnings\nfrom collections import Counter, defaultdict\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport plotly.offline as pyo\nfrom joblib import Parallel, delayed\nimport cv2\nwarnings.filterwarnings('ignore')\n\n# Set up plotting style\nplt.style.use('default')\nsns.set_palette(\"husl\")\nplt.rcParams['figure.figsize'] = (12, 8)\npv.set_jupyter_backend('static')","metadata":{},"outputs":[],"execution_count":null},{"id":"81b045a8","cell_type":"markdown","source":"# Basic statistic","metadata":{}},{"id":"07acb8b0","cell_type":"code","source":"DATA_PATH = '../data'","metadata":{},"outputs":[],"execution_count":null},{"id":"888f2b01","cell_type":"code","source":"print(f\"Number of segmentation folders: {len(glob(f'{DATA_PATH}/segmentations/**'))}\"\n      f\"\\nNumber of series folders: {len(glob(f'{DATA_PATH}/series/**'))}\")","metadata":{},"outputs":[],"execution_count":null},{"id":"06bc5ae9","cell_type":"code","source":"print(f\"Number of segmentation files: {len(glob(f'{DATA_PATH}/segmentations/**/*.nii', recursive=True))}\")\nprint(f\"Number of series files: {len(glob(f'{DATA_PATH}/series/**/*.dcm', recursive=True))}\")","metadata":{},"outputs":[],"execution_count":null},{"id":"0938cef4","cell_type":"code","source":"number_of_dicom_per_folder = []\nfor folder in tqdm(glob(f'{DATA_PATH}/series/**')):\n    dicom_files = glob(f\"{folder}/**/*.dcm\", recursive=True)\n    number_of_dicom_per_folder.append(len(dicom_files))\n\nplt.figure(figsize=(10, 5))\nplt.hist(number_of_dicom_per_folder, bins=100, color='blue', alpha=0.7)\nplt.title('Distribution of DICOM Files per Series Folder')\nplt.xlabel('Number of DICOM Files')\nplt.ylabel('Frequency')\nplt.grid(axis='y', alpha=0.75)\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"41a7691c","cell_type":"markdown","source":"# Dicom images","metadata":{}},{"id":"4973ebe0","cell_type":"markdown","source":"## Extract metadata from DICOM files","metadata":{}},{"id":"25e77fd2","cell_type":"code","source":"print(\"Loading DICOM files...\")\ndicom_files = glob(f'{DATA_PATH}/series/**/*.dcm', recursive=True)\n\n\ndef extract_dicom_metadata(file_path):\n    \"\"\"Extract relevant metadata from a DICOM file\"\"\"\n    try:\n        ds = pydicom.dcmread(file_path, force=True)\n        \n        metadata = {\n            'file_path': file_path,\n            'study_date': getattr(ds, 'StudyDate', None),\n            'series_description': getattr(ds, 'SeriesDescription', None),\n            'manufacturer_model': getattr(ds, 'ManufacturerModelName', None),\n            'patient_sex': getattr(ds, 'PatientSex', None),\n            'bits_stored': getattr(ds, 'BitsStored', None),\n            'patient_weight': getattr(ds, 'PatientWeight', None),\n            'slice_thickness': getattr(ds, 'SliceThickness', None),\n            'spacing_between_slices': getattr(ds, 'SpacingBetweenSlices', None),\n            'modality': getattr(ds, 'Modality', None),\n            'manufacturer': getattr(ds, 'Manufacturer', None),\n            'study_description': getattr(ds, 'StudyDescription', None),\n            'patient_id': getattr(ds, 'PatientID', None),\n            'series_number': getattr(ds, 'SeriesNumber', None),\n            'instance_number': getattr(ds, 'InstanceNumber', None),\n        }\n        \n        # Extract private tag data if exists\n        try:\n            # Look for private tags (typically starting with odd group numbers)\n            private_tags = []\n            for tag in ds.keys():\n                if tag.group % 2 == 1:  # Odd group numbers are private\n                    try:\n                        value = str(ds[tag].value)\n                        if len(value) < 100:  # Avoid very long values\n                            private_tags.append(value)\n                    except:\n                        pass\n            metadata['private_tag_data'] = ','.join(private_tags) if private_tags else None\n        except:\n            metadata['private_tag_data'] = None\n            \n        return metadata\n    except Exception as e:\n        print(f\"Error reading {file_path}: {str(e)}\")\n        return None\n\n\nprint(\"Extracting metadata from DICOM files...\")\nsample_size = len(dicom_files)\nsample_files = dicom_files[:sample_size]\n\nn_jobs = -1  # Use all available cores\n\n# Parallel processing\nresults = Parallel(n_jobs=n_jobs)(\n    delayed(extract_dicom_metadata)(file_path)\n    for file_path in tqdm(sample_files, desc=\"Extracting metadata\", unit=\"file\")\n)\n\n# Filter out None results\nmetadata_list = [metadata for metadata in results if metadata is not None]","metadata":{},"outputs":[],"execution_count":null},{"id":"258b0bb1","cell_type":"code","source":"# Create DataFrame\ndf = pd.DataFrame(metadata_list)\n\n\n# Convert study_date to datetime\ndef parse_dicom_date(date_str):\n    \"\"\"Parse DICOM date format YYYYMMDD\"\"\"\n    if pd.isna(date_str) or date_str == '':\n        return None\n    try:\n        return datetime.strptime(str(date_str), '%Y%m%d')\n    except:\n        return None\n\ndf['study_date_parsed'] = df['study_date'].apply(parse_dicom_date)\n\n# Convert numeric columns\nnumeric_cols = ['bits_stored', 'patient_weight', 'slice_thickness', 'spacing_between_slices']\nfor col in numeric_cols:\n    df[col] = pd.to_numeric(df[col], errors='coerce')\n","metadata":{},"outputs":[],"execution_count":null},{"id":"3f30080a","cell_type":"markdown","source":"## EDA plots","metadata":{}},{"id":"bfcad410","cell_type":"code","source":"# Set up the plotting environment\nfig = plt.figure(figsize=(20, 24))\n\n\n# PLOT 1: Study Date Distribution by Month\nprint(\"\\n1. Creating Study Date distribution by month...\")\nplt.subplot(4, 3, 1)\n\nif df['study_date_parsed'].notna().sum() > 0:\n    # Extract month-year for grouping\n    df['month_year'] = df['study_date_parsed'].dt.to_period('M')\n    date_counts = df['month_year'].value_counts().sort_index()\n    \n    # Convert to datetime for proper x-axis formatting\n    dates = [pd.to_datetime(str(period)) for period in date_counts.index]\n    \n    plt.plot(dates, date_counts.values, marker='o', linewidth=2, markersize=6)\n    plt.title('Study Date Distribution by Month', fontsize=14, fontweight='bold')\n    plt.xlabel('Date', fontsize=12)\n    plt.ylabel('Number of Studies', fontsize=12)\n    plt.xticks(rotation=45)\n    plt.grid(True, alpha=0.3)\nelse:\n    plt.text(0.5, 0.5, 'No valid study dates found', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title('Study Date Distribution by Month', fontsize=14, fontweight='bold')\n\n\n# PLOT 2: Series Description Distribution (Pie Chart)\nprint(\"2. Creating Series Description distribution pie chart...\")\nplt.subplot(4, 3, 2)\n\nseries_desc_counts = df['series_description'].value_counts().head(8)  # Top 8 categories\nif len(series_desc_counts) > 0:\n    colors = plt.cm.Set3(np.linspace(0, 1, len(series_desc_counts)))\n    wedges, texts, autotexts = plt.pie(series_desc_counts.values, \n                                      labels=series_desc_counts.index,\n                                      autopct='%1.1f%%',\n                                      colors=colors,\n                                      startangle=90)\n    plt.setp(autotexts, size=9, weight=\"bold\")\n    plt.setp(texts, size=8)\n    plt.title('Series Description Distribution', fontsize=14, fontweight='bold')\nelse:\n    plt.text(0.5, 0.5, 'No series description data', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title('Series Description Distribution', fontsize=14, fontweight='bold')\n\n\n# PLOT 3: Manufacturer's Model Name Distribution (Pie Chart)\nprint(\"3. Creating Manufacturer's Model Name distribution...\")\nplt.subplot(4, 3, 3)\n\nmodel_counts = df['manufacturer_model'].value_counts().head(6)\nif len(model_counts) > 0:\n    colors = plt.cm.Pastel1(np.linspace(0, 1, len(model_counts)))\n    wedges, texts, autotexts = plt.pie(model_counts.values,\n                                      labels=model_counts.index,\n                                      autopct='%1.1f%%',\n                                      colors=colors,\n                                      startangle=45)\n    plt.setp(autotexts, size=9, weight=\"bold\")\n    plt.setp(texts, size=8)\n    plt.title(\"Manufacturer's Model Name Distribution\", fontsize=14, fontweight='bold')\nelse:\n    plt.text(0.5, 0.5, 'No model name data', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title(\"Manufacturer's Model Name Distribution\", fontsize=14, fontweight='bold')\n\n\n# PLOT 4: Patient's Sex Distribution (Pie Chart)\nprint(\"4. Creating Patient's Sex distribution...\")\nplt.subplot(4, 3, 4)\n\nsex_counts = df['patient_sex'].value_counts()\nif len(sex_counts) > 0:\n    colors = ['#ff9999', '#66b3ff', '#99ff99'][:len(sex_counts)]\n    wedges, texts, autotexts = plt.pie(sex_counts.values,\n                                      labels=sex_counts.index,\n                                      autopct='%1.1f%%',\n                                      colors=colors,\n                                      startangle=90)\n    plt.setp(autotexts, size=11, weight=\"bold\")\n    plt.setp(texts, size=10)\n    plt.title(\"Patient's Sex Distribution\", fontsize=14, fontweight='bold')\nelse:\n    plt.text(0.5, 0.5, 'No sex data', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title(\"Patient's Sex Distribution\", fontsize=14, fontweight='bold')\n\n\n# PLOT 5: Bits Stored Distribution (Pie Chart)\nprint(\"5. Creating Bits Stored distribution...\")\nplt.subplot(4, 3, 5)\n\nbits_counts = df['bits_stored'].value_counts()\nif len(bits_counts) > 0:\n    colors = plt.cm.Set2(np.linspace(0, 1, len(bits_counts)))\n    wedges, texts, autotexts = plt.pie(bits_counts.values,\n                                      labels=[f'{int(x)} bits' for x in bits_counts.index],\n                                      autopct='%1.1f%%',\n                                      colors=colors,\n                                      startangle=0)\n    plt.setp(autotexts, size=10, weight=\"bold\")\n    plt.setp(texts, size=9)\n    plt.title('Bits Stored Distribution', fontsize=14, fontweight='bold')\nelse:\n    plt.text(0.5, 0.5, 'No bits stored data', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title('Bits Stored Distribution', fontsize=14, fontweight='bold')\n\n\n# PLOT 6: Patient's Weight Distribution (Histogram)\nprint(\"6. Creating Patient's Weight histogram...\")\nplt.subplot(4, 3, 6)\n\nweight_data = df['patient_weight'].dropna()\nif len(weight_data) > 0:\n    plt.hist(weight_data, bins=40, alpha=0.7, color='skyblue', edgecolor='black')\n    plt.title(\"Patient's Weight Distribution\", fontsize=14, fontweight='bold')\n    plt.xlabel('Weight (kg)', fontsize=12)\n    plt.ylabel('Frequency', fontsize=12)\n    plt.grid(True, alpha=0.3)\n    \n    # Add statistics\n    mean_weight = weight_data.mean()\n    plt.axvline(mean_weight, color='red', linestyle='--', linewidth=2, label=f'Mean: {mean_weight:.1f} kg')\n    plt.legend()\nelse:\n    plt.text(0.5, 0.5, 'No weight data available', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title(\"Patient's Weight Distribution\", fontsize=14, fontweight='bold')\n\n\n# PLOT 7: Slice Thickness Distribution (Histogram)\nprint(\"7. Creating Slice Thickness histogram...\")\nplt.subplot(4, 3, 7)\n\nthickness_data = df['slice_thickness'].dropna()\n# Filter out outliers > 5 mm\nthickness_data = thickness_data[thickness_data <= 5]\n\nif len(thickness_data) > 0:\n    plt.hist(thickness_data, bins=100, alpha=0.7, color='lightgreen', edgecolor='black')\n    plt.title('Slice Thickness Distribution (≤ 5 mm)', fontsize=14, fontweight='bold')\n    plt.xlabel('Slice Thickness (mm)', fontsize=12)\n    plt.ylabel('Frequency', fontsize=12)\n    plt.grid(True, alpha=0.3)\n\n    # Add statistics\n    mean_thickness = thickness_data.mean()\n    plt.axvline(mean_thickness, color='red', linestyle='--', linewidth=2, label=f'Mean: {mean_thickness:.2f} mm')\n    plt.legend()\nelse:\n    plt.text(0.5, 0.5, 'No slice thickness data', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title('Slice Thickness Distribution', fontsize=14, fontweight='bold')\n\n\n# PLOT 8: Spacing Between Slices Distribution (Histogram)\nprint(\"8. Creating Spacing Between Slices histogram...\")\nplt.subplot(4, 3, 8)\n\nspacing_data = df['spacing_between_slices'].dropna()\n# Filter out outliers > 5 mm\nspacing_data = spacing_data[spacing_data <= 5]\n\nif len(spacing_data) > 0:\n    plt.hist(spacing_data, bins=40, alpha=0.7, color='coral', edgecolor='black')\n    plt.title('Spacing Between Slices Distribution (≤ 5 mm)', fontsize=14, fontweight='bold')\n    plt.xlabel('Spacing (mm)', fontsize=12)\n    plt.ylabel('Frequency', fontsize=12)\n    plt.grid(True, alpha=0.3)\n    \n    # Add statistics\n    mean_spacing = spacing_data.mean()\n    plt.axvline(mean_spacing, color='red', linestyle='--', linewidth=2, label=f'Mean: {mean_spacing:.2f} mm')\n    plt.legend()\nelse:\n    plt.text(0.5, 0.5, 'No spacing data available', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title('Spacing Between Slices Distribution', fontsize=14, fontweight='bold')\n\n\n# PLOT 9: Private Tag Data Analysis\nprint(\"9. Creating Private Tag Data analysis...\")\nplt.subplot(4, 3, 9)\n\n# Analyze private tag data frequencies (top 10)\nprivate_data = df['private_tag_data'].dropna()\nif len(private_data) > 0:\n    # Get top 10 most frequent values\n    top_counts = private_data.value_counts().head(10)\n\n    # Shorten labels to first 4 characters + '...'\n    short_labels = [str(val)[:30] + '...' for val in top_counts.index]\n\n    plt.bar(short_labels, top_counts.values, color='mediumpurple', edgecolor='black')\n    plt.title('Top 10 Most Frequent Private Tags', fontsize=14, fontweight='bold')\n    plt.xlabel('Private Tag (truncated)', fontsize=12)\n    plt.ylabel('Frequency', fontsize=12)\n    plt.xticks(rotation=45, ha='right', fontsize=6)\n    plt.grid(axis='y', alpha=0.3)\nelse:\n    plt.text(0.5, 0.5, 'No private tag data', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title('Top 10 Most Frequent Private Tags', fontsize=14, fontweight='bold')\n\n# PLOT 10: Modality Distribution\nprint(\"10. Creating Modality distribution...\")\nplt.subplot(4, 3, 10)\n\nmodality_counts = df['modality'].value_counts()\nif len(modality_counts) > 0:\n    plt.bar(modality_counts.index, modality_counts.values, color='gold', alpha=0.8, edgecolor='black')\n    plt.title('Imaging Modality Distribution', fontsize=14, fontweight='bold')\n    plt.xlabel('Modality', fontsize=12)\n    plt.ylabel('Count', fontsize=12)\n    plt.xticks(rotation=45)\n    plt.grid(True, alpha=0.3)\nelse:\n    plt.text(0.5, 0.5, 'No modality data', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title('Imaging Modality Distribution', fontsize=14, fontweight='bold')\n\n\n# PLOT 11: Manufacturer Distribution\nprint(\"11. Creating Manufacturer distribution...\")\nplt.subplot(4, 3, 11)\n\nmfr_counts = df['manufacturer'].value_counts().head(5)\nif len(mfr_counts) > 0:\n    colors = plt.cm.viridis(np.linspace(0, 1, len(mfr_counts)))\n    bars = plt.bar(range(len(mfr_counts)), mfr_counts.values, color=colors, alpha=0.8, edgecolor='black')\n    plt.title('Manufacturer Distribution', fontsize=14, fontweight='bold')\n    plt.xlabel('Manufacturer', fontsize=12)\n    plt.ylabel('Count', fontsize=12)\n    plt.xticks(range(len(mfr_counts)), [mfr[:15] + '...' if len(mfr) > 15 else mfr for mfr in mfr_counts.index], rotation=45)\n    plt.grid(True, alpha=0.3)\nelse:\n    plt.text(0.5, 0.5, 'No manufacturer data', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title('Manufacturer Distribution', fontsize=14, fontweight='bold')\n\n\n# PLOT 12: Series Number vs Instance Number Scatter\nprint(\"12. Creating Series vs Instance Number scatter plot...\")\nplt.subplot(4, 3, 12)\n\nseries_data = df[['series_number', 'instance_number']].dropna()\nif len(series_data) > 0:\n    plt.scatter(series_data['series_number'], series_data['instance_number'], \n               alpha=0.6, color='darkblue', s=30)\n    plt.title('Series Number vs Instance Number', fontsize=14, fontweight='bold')\n    plt.xlabel('Series Number', fontsize=12)\n    plt.ylabel('Instance Number', fontsize=12)\n    plt.grid(True, alpha=0.3)\nelse:\n    plt.text(0.5, 0.5, 'No series/instance data', ha='center', va='center', transform=plt.gca().transAxes)\n    plt.title('Series Number vs Instance Number', fontsize=14, fontweight='bold')\n\nplt.tight_layout(pad=3.0)\nplt.show()\n\n\n# Summary Statistics\nprint(\"\\n\" + \"=\"*50)\nprint(\"SUMMARY STATISTICS\")\nprint(\"=\"*50)\n\nprint(f\"\\nDataset Overview:\")\nprint(f\"Total DICOM files processed: {len(df)}\")\n\nprint(f\"\\nMissing Data Analysis:\")\nmissing_data = df.isnull().sum()\nmissing_percentage = (missing_data / len(df)) * 100\nmissing_df = pd.DataFrame({\n    'Missing Count': missing_data,\n    'Missing Percentage': missing_percentage\n}).sort_values('Missing Percentage', ascending=False)\nprint(missing_df)\n\nprint(f\"\\nNumerical Statistics:\")\nnumerical_cols = ['patient_weight', 'slice_thickness', 'spacing_between_slices', 'bits_stored']\nfor col in numerical_cols:\n    if df[col].notna().sum() > 0:\n        print(f\"\\n{col.upper()}:\")\n        print(f\"  Mean: {df[col].mean():.3f}\")\n        print(f\"  Median: {df[col].median():.3f}\")\n        print(f\"  Std: {df[col].std():.3f}\")\n        print(f\"  Min: {df[col].min():.3f}\")\n        print(f\"  Max: {df[col].max():.3f}\")\n\nprint(f\"\\nCategorical Data Summary:\")\ncategorical_cols = ['series_description', 'manufacturer_model', 'patient_sex', 'modality', 'manufacturer']\nfor col in categorical_cols:\n    unique_count = df[col].nunique()\n    if unique_count > 0:\n        print(f\"\\n{col.upper()}: {unique_count} unique values\")\n        print(f\"  Top values: {list(df[col].value_counts().head(4).index)}\")\n\nprint(f\"\\nDate Range Analysis:\")\nif df['study_date_parsed'].notna().sum() > 0:\n    date_range = df['study_date_parsed'].dropna()\n    print(f\"  Earliest study: {date_range.min().strftime('%Y-%m-%d')}\")\n    print(f\"  Latest study: {date_range.max().strftime('%Y-%m-%d')}\")\n    print(f\"  Date range: {(date_range.max() - date_range.min()).days} days\")\n","metadata":{},"outputs":[],"execution_count":null},{"id":"3e850b30","cell_type":"code","source":"print(f\"Studies from the future (after Aug 2025): {df['month_year'].gt('2025-08').sum()}\")\nslice_thickness_data = df['slice_thickness'].dropna()\n# outlier slice thickness (>5mm)\nif len(slice_thickness_data[slice_thickness_data > 5]) > 0:\n    print(f\"Outlier slice thickness (>5mm): {len(slice_thickness_data[slice_thickness_data > 5])} instances\")\n# outlier spacing between slices (>5mm)\nspacing_data = df['spacing_between_slices'].dropna()\nif len(spacing_data[spacing_data > 5]) > 0:\n    print(f\"Outlier spacing between slices (>5mm): {len(spacing_data[spacing_data > 5])} instances\")","metadata":{},"outputs":[],"execution_count":null},{"id":"7d9144cd","cell_type":"markdown","source":"# CSV training data","metadata":{}},{"id":"34c86606","cell_type":"code","source":"df = pd.read_csv(f'{DATA_PATH}/train.csv')\ndf.head(5)","metadata":{},"outputs":[],"execution_count":null},{"id":"3ce4919f","cell_type":"code","source":"meta_cols = ['SeriesInstanceUID', 'PatientAge', 'PatientSex', 'Modality']\n\n# Artery-level aneurysm labels (13 columns)\nartery_cols = [\n    'Left Infraclinoid Internal Carotid Artery',\n    'Right Infraclinoid Internal Carotid Artery',\n    'Left Supraclinoid Internal Carotid Artery',\n    'Right Supraclinoid Internal Carotid Artery',\n    'Left Middle Cerebral Artery',\n    'Right Middle Cerebral Artery',\n    'Anterior Communicating Artery',\n    'Left Anterior Cerebral Artery',\n    'Right Anterior Cerebral Artery',\n    'Left Posterior Communicating Artery',\n    'Right Posterior Communicating Artery',\n    'Basilar Tip',\n    'Other Posterior Circulation',\n]\n\n# Target label\ntarget_col = 'Aneurysm Present'","metadata":{},"outputs":[],"execution_count":null},{"id":"c3d30f6a","cell_type":"code","source":"# Preprocess\ndf['PatientAge'] = pd.to_numeric(df['PatientAge'], errors='coerce')\n\n# Define column groups\nmeta_cols = ['SeriesInstanceUID', 'PatientAge', 'PatientSex', 'Modality']\ntarget_col = 'Aneurysm Present'\nartery_cols = df.columns[4:-1].tolist()  # All artery-specific binary labels\n\n# Initialize figure\nplt.figure(figsize=(18, 16))\nplt.suptitle(\"Exploratory Data Analysis - Aneurysm Detection Dataset\", fontsize=18, fontweight='bold')\n\n# 1. Target distribution\nplt.subplot(3, 3, 1)\ndf[target_col].value_counts().plot(kind='bar', color='salmon', edgecolor='black')\nplt.title('Aneurysm Presence')\nplt.xlabel('Label')\nplt.ylabel('Count')\n\n# 2. Patient Age distribution\nplt.subplot(3, 3, 2)\ndf['PatientAge'].hist(bins=30, color='skyblue', edgecolor='black')\nplt.title('Patient Age Distribution')\nplt.xlabel('Age')\nplt.ylabel('Count')\n\n# 3. Patient Sex distribution\nplt.subplot(3, 3, 3)\ndf['PatientSex'].value_counts().plot(kind='bar', color='plum', edgecolor='black')\nplt.title('Patient Sex Distribution')\nplt.xlabel('Sex')\nplt.ylabel('Count')\n\n# 4. Imaging Modality\nplt.subplot(3, 3, 4)\ndf['Modality'].value_counts().plot(kind='bar', color='lightgreen', edgecolor='black')\nplt.title('Modality Distribution')\nplt.xlabel('Modality')\nplt.ylabel('Count')\n\n# 5. Aneurysm count per artery\nplt.subplot(3, 3, 5)\ndf[artery_cols].sum().sort_values(ascending=False).plot(kind='bar', color='steelblue', edgecolor='black')\nplt.title('Aneurysm Count per Artery')\nplt.xticks(rotation=90)\nplt.ylabel('Count')\n\n# 6. Artery involvement when Aneurysm Present = 1\nplt.subplot(3, 3, 6)\ndf[df[target_col] == 1][artery_cols].sum().sort_values(ascending=False).plot(kind='bar', color='tomato', edgecolor='black')\nplt.title('Artery Involvement (when Aneurysm Present = 1)')\nplt.xticks(rotation=90)\nplt.ylabel('Count')\n\n\n\nplt.tight_layout(rect=[0, 0.03, 1, 0.95])\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"c0706c0d","cell_type":"markdown","source":"### How many arteries have aneurysms per scan","metadata":{}},{"id":"f58bcf51","cell_type":"code","source":"# Count how many artery columns are labeled 1 per scan\ndf['n_arteries_positive'] = df[artery_cols].sum(axis=1)\n\n# Plot distribution\nplt.figure(figsize=(6, 4))\ndf['n_arteries_positive'].value_counts().sort_index().plot(kind='bar', color='indigo', edgecolor='black')\nplt.title('Number of Arteries with Aneurysms per Scan', fontsize=14)\nplt.xlabel('Number of Positive Artery Labels')\nplt.ylabel('Count of Scans')\nplt.grid(axis='y', alpha=0.3)\nplt.tight_layout()\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"fab3d0eb","cell_type":"markdown","source":"###  Proportion of aneurysm-positive scans that involve multiple arteries","metadata":{}},{"id":"224b1263","cell_type":"code","source":"multi_artery_cases = df[df['Aneurysm Present'] == 1]['n_arteries_positive']\nmulti_artery_rate = (multi_artery_cases > 1).mean()\n\nprint(f\"👉 {multi_artery_rate:.1%} of aneurysm-positive scans involve more than one artery.\")","metadata":{},"outputs":[],"execution_count":null},{"id":"dabfb902","cell_type":"markdown","source":"### Artery Aneurysm Count (Distribution)","metadata":{}},{"id":"f8b5753a","cell_type":"code","source":"# Compute aneurysm counts and percentage (w.r.t aneurysm-positive cases)\nartery_counts = df[artery_cols].sum().sort_values(ascending=False)\ntotal_positive = df['Aneurysm Present'].sum()\nartery_percent = (artery_counts / total_positive * 100).round(1)\n\n# Create combined DataFrame\nartery_stats = pd.DataFrame({\n    'Aneurysm Count': artery_counts,\n    'Percent of Aneurysm-Positive Scans': artery_percent\n})\n\n# Plot\nplt.figure(figsize=(12, 6))\nbars = plt.bar(artery_stats.index, artery_stats['Aneurysm Count'], color='cornflowerblue', edgecolor='black')\nplt.title('Aneurysm Distribution per Artery', fontsize=14, fontweight='bold')\nplt.ylabel('Aneurysm Count')\nplt.xticks(rotation=45, ha='right')\nplt.grid(axis='y', alpha=0.3)\n\n# Add percentage labels above bars\nfor bar, pct in zip(bars, artery_stats['Percent of Aneurysm-Positive Scans']):\n    height = bar.get_height()\n    plt.text(bar.get_x() + bar.get_width() / 2, height + 1, f\"{pct}%\", ha='center', va='bottom', fontsize=9)\n\nplt.tight_layout()\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"23740504","cell_type":"markdown","source":"## Train location","metadata":{}},{"id":"7cc2c440","cell_type":"code","source":"localizer_df = pd.read_csv(f\"{DATA_PATH}/train_localizers.csv\")","metadata":{},"outputs":[],"execution_count":null},{"id":"ccfbd882","cell_type":"code","source":"import ast\n\n# Parse coordinate strings\nlocalizer_df['coordinates'] = localizer_df['coordinates'].apply(ast.literal_eval)\n\n# Separate x and y\nlocalizer_df['x'] = localizer_df['coordinates'].apply(lambda c: c['x'])\nlocalizer_df['y'] = localizer_df['coordinates'].apply(lambda c: c['y'])\n\n# Quick stats\nprint(localizer_df[['x', 'y']].describe())","metadata":{},"outputs":[],"execution_count":null},{"id":"505046e9","cell_type":"markdown","source":"### Aneurysm Coordinate Distribution","metadata":{}},{"id":"4105fd4a","cell_type":"code","source":"plt.figure(figsize=(6, 6))\nplt.scatter(localizer_df['x'], localizer_df['y'], alpha=0.4, s=10, c='crimson')\nplt.title('Aneurysm Coordinates Distribution (in-plane)', fontsize=14)\nplt.xlabel('X position (pixels)')\nplt.ylabel('Y position (pixels)')\nplt.grid(True, alpha=0.3)\nplt.axis('equal')\nplt.tight_layout()\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"478477a0","cell_type":"markdown","source":"### Modality × Localization Coverage","metadata":{}},{"id":"5edbbf43","cell_type":"code","source":"df_modality = df[['SeriesInstanceUID', 'Modality']]\nmodality_counts = pd.merge(localizer_df[['SeriesInstanceUID']], df_modality, on='SeriesInstanceUID', how='left')\nprint(modality_counts['Modality'].value_counts())","metadata":{},"outputs":[],"execution_count":null},{"id":"76707ba9","cell_type":"markdown","source":"###  Heatmap of Localizer Locations by Artery","metadata":{}},{"id":"2275604a","cell_type":"code","source":"plt.figure(figsize=(8, 4))\nsns.countplot(y='location', data=localizer_df, order=localizer_df['location'].value_counts().index)\nplt.title('Aneurysm Location Frequency', fontsize=14)\nplt.xlabel('Count')\nplt.ylabel('Artery Location')\nplt.tight_layout()\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"e18a73b1","cell_type":"markdown","source":"# Visuzlize 3D series","metadata":{}},{"id":"935d8a87","cell_type":"code","source":"# https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/593948 @umarali1\ndef load_sorted_dicom_series(series_path):\n    '''\n    Since the dicom series path files are not in spatial order by file name, we have\n    to use the metadata contained within the slices to proper sort them\n    '''\n    dcm_files = [os.path.join(series_path, f) for f in os.listdir(series_path) if f.endswith('.dcm')]\n    slices = [pydicom.dcmread(f) for f in dcm_files]\n\n    try:\n        slices.sort(key=lambda s: float(s.ImagePositionPatient[2]))\n    except (AttributeError, IndexError):\n        slices.sort(key=lambda s: int(s.InstanceNumber))\n    return slices","metadata":{},"outputs":[],"execution_count":null},{"id":"c1fd5be9","cell_type":"code","source":"slices = load_sorted_dicom_series(f'{DATA_PATH}/series/1.2.826.0.1.3680043.8.498.10004044428023505108375152878107656647')\n# Ensure consistent spacing (check pixel spacing and slice thickness)\npixel_spacing = slices[0].PixelSpacing  # [row, col]\nslice_thickness = float(slices[0].SliceThickness)\nspacing = (slice_thickness, pixel_spacing[0], pixel_spacing[1])\n\n# Stack into 3D volume\nimage_3d = np.stack([s.pixel_array for s in slices]).astype(np.int16)\n\n# Flip if needed (some modalities store slices reversed)\nimage_3d = np.flip(image_3d, axis=0)\n\nprint(f\"Volume shape: {image_3d.shape}, spacing: {spacing}\")","metadata":{},"outputs":[],"execution_count":null},{"id":"a19fe145","cell_type":"code","source":"Z, Y, X = image_3d.shape\n# The number of slices to display\nnum_slices = image_3d.shape[0] # Number of slices along the Z-axis\n\n# Determine the grid size for the subplots\n# We'll use a square grid to make it look neat\ncols = 8 # Number of columns in the subplot grid\nrows = (num_slices + cols - 1) // cols # Calculate required number of rows\n\n# Create a figure and a grid of subplots\nfig, axes = plt.subplots(rows, cols, figsize=(20, 3 * rows))\nfig.suptitle(f'All {num_slices} Slices of 3D Volume (Z-axis)', fontsize=16)\n\n# Flatten the axes array for easier iteration\naxes = axes.flatten()\n\n# Loop through each slice and plot it\nfor i in range(num_slices):\n    ax = axes[i]\n    # The slice is a 2D array of shape (Y, X)\n    slice_data = image_3d[i, :, :]\n\n    # Display the slice. The `extent` argument can be used to\n    # correctly scale the axes based on your spacing.\n    # The spacing is (Z, Y, X), so spacing[1] and spacing[2] correspond to Y and X axes.\n    extent = [0, X * spacing[2], 0, Y * spacing[1]]\n    ax.imshow(slice_data, cmap='gray', origin='lower', extent=extent)\n\n    ax.set_title(f'Slice {i+1}/{num_slices}')\n    ax.set_xlabel('X-axis')\n    ax.set_ylabel('Y-axis')\n\n    # Turn off the axis ticks for a cleaner look\n    ax.set_xticks([])\n    ax.set_yticks([])\n\n# Hide any unused subplots\nfor i in range(num_slices, len(axes)):\n    axes[i].axis('off')\n\n# Adjust the layout to prevent titles from overlapping\nplt.tight_layout(rect=[0, 0.03, 1, 0.95])\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"b87bd611","cell_type":"code","source":"# You can still use the trame backend for interactive viewing\n# pv.set_jupyter_backend('trame')  # for local Jupyter Notebook\nvolume = pv.wrap(image_3d)\nvolume.spacing = spacing[::-1]\nvolume.origin = (0, 0, 0)\n\np = pv.Plotter(notebook=True)\np.add_volume(volume, cmap='viridis')\np.add_mesh(volume.outline(), color='gray')\np.add_axes()\np.show_grid()\n\np.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"85c7b39d","cell_type":"code","source":"import ipywidgets as widgets\nfrom IPython.display import display\n\ndef plot_slice(slice_index):\n    fig, ax = plt.subplots(figsize=(8, 8))\n    slice_data = image_3d[slice_index, :, :]\n    extent = [0, X * spacing[2], 0, Y * spacing[1]]\n    ax.imshow(slice_data, cmap='gray', origin='lower', extent=extent)\n    ax.set_title(f'Interactive Slice {slice_index+1}/{num_slices}')\n    ax.set_xlabel('X-axis')\n    ax.set_ylabel('Y-axis')\n    plt.show()\n\n# Create a slider widget\nslice_slider = widgets.IntSlider(\n    value=0,\n    min=0,\n    max=num_slices - 1,\n    step=1,\n    description='Slice Index:',\n    continuous_update=False\n)\n\n# Use `ipywidgets.interactive` to connect the slider and the function\ninteractive_plot = widgets.interactive(plot_slice, slice_index=slice_slider)\ndisplay(interactive_plot)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"12c64a3f","cell_type":"markdown","source":"# Visualize aneurysms location data","metadata":{}},{"id":"5bad5e08","cell_type":"code","source":"unique_locs = localizer_df.groupby(\"location\").first().reset_index()\n\nn = len(unique_locs)\ncols = 3\nrows = (n + cols - 1) // cols\nplt.figure(figsize=(5 * cols, 5 * rows))\n\nfor i, row in unique_locs.iterrows():\n    try:\n        series_uid = row[\"SeriesInstanceUID\"]\n        sop_uid = row[\"SOPInstanceUID\"]\n        coords = row[\"coordinates\"]\n\n        # Load DICOM\n        dcm_path = ff\"{DATA_PATH}/series/{series_uid}/{sop_uid}.dcm\"\n        if not os.path.exists(dcm_path):\n            print(f\"Missing DICOM: {dcm_path}\")\n            continue\n\n        dcm = pydicom.dcmread(dcm_path)\n        image = dcm.pixel_array\n\n        # Normalize\n        image = image.astype(np.float32)\n        image = (image - np.min(image)) / (np.max(image) - np.min(image))\n        image = (image * 255).astype(np.uint8)\n\n        # Draw marker\n        x, y = int(coords[\"x\"]), int(coords[\"y\"])\n        img_rgb = cv2.cvtColor(image, cv2.COLOR_GRAY2RGB)\n        cv2.circle(img_rgb, (x, y), radius=10, color=(255, 0, 0), thickness=2)\n\n        # Plot\n        plt.subplot(rows, cols, i + 1)\n        plt.imshow(img_rgb)\n        plt.title(row[\"location\"], fontsize=10)\n        plt.axis(\"off\")\n    except Exception as e:\n        print(f\"Error at index {i}: {e}\")\n\nplt.tight_layout()\nplt.show()","metadata":{},"outputs":[],"execution_count":null},{"id":"9ba0b802","cell_type":"markdown","source":"# Visualize segmentations","metadata":{}},{"id":"be525462","cell_type":"code","source":"seg_dirs = sorted(glob(f\"{DATA_PATH}/segmentations/*/\"))\nprint(f\"Found {len(seg_dirs)} segmentation folders\")\n\ndef show_segmentation_overlay(id_path):\n    # Find both files\n    image_path = glob(os.path.join(id_path, \"*.nii\"))[1]            # vessel image\n    mask_path = glob(os.path.join(id_path, \"*_cowseg.nii\"))[0]       # segmentation mask\n\n    # Load NIfTI files\n    img = nib.load(image_path).get_fdata()\n    mask = nib.load(mask_path).get_fdata()\n\n    # Pick a mid slice (sagittal/axial/coronal)\n    slice_idx = img.shape[2] // 2\n    img_slice = img[:, :, slice_idx]\n    mask_slice = mask[:, :, slice_idx]\n\n    # Normalize image for display\n    img_slice = (img_slice - img_slice.min()) / np.ptp(img_slice)\n\n    # Plot\n    plt.figure(figsize=(12, 5))\n\n    plt.subplot(1, 2, 1)\n    plt.imshow(img_slice, cmap='gray')\n    plt.title('Vessel Image')\n    plt.axis('off')\n\n    plt.subplot(1, 2, 2)\n    plt.imshow(img_slice, cmap='gray')\n    plt.imshow(mask_slice, cmap='Reds', alpha=0.5)\n    plt.title('Overlay: Vessel + Segmentation')\n    plt.axis('off')\n\n    plt.suptitle(f\"Segmentation from: {os.path.basename(id_path.strip('/'))}\")\n    plt.tight_layout()\n    plt.show()\n\nfor path in seg_dirs[:3]:  # show first 4\n    show_segmentation_overlay(path)","metadata":{},"outputs":[],"execution_count":null},{"id":"78ebdf2a","cell_type":"code","source":"","metadata":{},"outputs":[],"execution_count":null}]}