{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceType":"competition","sourceId":39272,"databundleVersionId":4629629,"isSourceIdPinned":false}],"dockerImageVersionId":31329,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\nimport os\n\n# 1. Load the competition's training CSV\ndf = pd.read_csv('/kaggle/input/competitions/rsna-breast-cancer-detection/train.csv')\n\n# 2. Keep only the standard CC and MLO views\ndf = df[df['view'].isin(['CC', 'MLO'])]\n\n# 3. We need to map the image IDs to their actual file paths in the PNG dataset you found.\n# (Make sure to double-check the exact folder path of the 512px dataset in your Kaggle environment)\nBASE_PATH = '/kaggle/input/rsna-mammo-pngs-512px/' \ndf['image_path'] = BASE_PATH + df['patient_id'].astype(str) + '_' + df['image_id'].astype(str) + '.png'\n\n# 4. Pivot the data so each row represents ONE breast (Patient ID + Laterality) \n# and contains both the CC and MLO image paths.\ndf_multi = df.pivot_table(\n    index=['patient_id', 'laterality', 'cancer'],\n    columns='view',\n    values='image_path',\n    aggfunc='first' # If a patient has multiple CC views, we just take the first one for simplicity\n).reset_index()\n\n# 5. Clean up: Drop any breasts that do not have BOTH a CC and an MLO view available\ndf_multi = df_multi.dropna(subset=['CC', 'MLO'])\n\n# Rename columns to match our PyTorch Dataset class\ndf_multi = df_multi.rename(columns={'CC': 'cc_path', 'MLO': 'mlo_path'})\n\nprint(f\"Total multi-view samples available: {len(df_multi)}\")\ndisplay(df_multi.head())","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2026-04-18T12:53:27.262845Z","iopub.execute_input":"2026-04-18T12:53:27.263478Z","iopub.status.idle":"2026-04-18T12:53:27.496481Z","shell.execute_reply.started":"2026-04-18T12:53:27.263448Z","shell.execute_reply":"2026-04-18T12:53:27.49574Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1. The Clinical Reality: Extreme Class Imbalance\nIn screening mammography, the vast majority of scans are benign. If we train a standard neural network on this data, it will quickly realize that it can achieve ~98% accuracy simply by guessing \"No Cancer\" every single time. \n\nIn the healthcare industry, optimizing for standard accuracy is dangerous. The cost of a **False Negative** (missing a tumor) is a delayed life-saving treatment, while the cost of a **False Positive** is a stressful, but ultimately harmless, follow-up biopsy. \n\nTo force our Vision Transformer to care about the rare cancer cases, we must first quantify the imbalance. We will use this exact ratio to dynamically weight our PyTorch Loss Function (`BCEWithLogitsLoss`), heavily penalizing the model when it misses a malignancy.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# 1. Calculate the exact numbers\ntotal_samples = len(df_multi)\npositive_cases = df_multi['cancer'].sum()\nnegative_cases = total_samples - positive_cases\n\nprint(f\"Total Multi-View Samples: {total_samples}\")\nprint(f\"Benign (0): {negative_cases}\")\nprint(f\"Malignant (1): {positive_cases}\")\nprint(f\"Cancer Prevalence: {(positive_cases / total_samples) * 100:.2f}%\\n\")\n\n# 2. Calculate the dynamic positive weight for our PyTorch Loss Function\n# Formula: negative_samples / positive_samples\npos_weight = negative_cases / positive_cases\nprint(f\"--> Required PyTorch pos_weight: {pos_weight:.2f}\")\n\n# 3. Visualize the imbalance for the EDA section of the notebook\nplt.figure(figsize=(8, 5))\nax = sns.countplot(data=df_multi, x='cancer', palette=['#1f77b4', '#d62728'])\nplt.title('Distribution of Benign vs. Malignant Cases', fontsize=14)\nplt.xlabel('Diagnosis (0 = Benign, 1 = Cancer)', fontsize=12)\nplt.ylabel('Number of Patients', fontsize=12)\n\n# Add text labels on top of the bars\nfor p in ax.patches:\n    ax.annotate(f'{int(p.get_height())}', (p.get_x() + p.get_width() / 2., p.get_height()),\n                ha='center', va='baseline', fontsize=11, color='black', xytext=(0, 5),\n                textcoords='offset points')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-04-18T13:05:38.839806Z","iopub.execute_input":"2026-04-18T13:05:38.840145Z","iopub.status.idle":"2026-04-18T13:05:40.038388Z","shell.execute_reply.started":"2026-04-18T13:05:38.840117Z","shell.execute_reply":"2026-04-18T13:05:40.037767Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Preventing Data Leakage: Patient-Level Splitting\nA common rookie mistake in medical ML is using a standard random split (`train_test_split`). \n\n**The Problem:** Many patients have both their Left and Right breasts in this dataset. If Patient A's Left breast goes into the Training Set, and their Right breast goes into the Validation Set, the model might just memorize Patient A's underlying tissue density and machine hardware artifacts rather than learning generalized cancer features. This leads to a model that looks amazing on paper but fails catastrophically on new patients in the real world.\n\n**The Solution:** We must use a **Stratified Group K-Fold**. \n* **Group:** Ensures that a single `patient_id` only ever exists in *either* the train or validation set, never both.\n* **Stratified:** Ensures the extreme ~2% cancer prevalence is maintained equally across all folds.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import StratifiedGroupKFold\n\n# Initialize Stratified Group K-Fold (5 folds is standard for this dataset size)\nsgkf = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)\n\n# Create a new column to store the fold assignment\ndf_multi['fold'] = -1\n\n# Generate the folds\n# X = features (we just pass the dataframe index), y = target (cancer), groups = patient_id\nfor fold, (train_idx, val_idx) in enumerate(sgkf.split(X=df_multi, y=df_multi['cancer'], groups=df_multi['patient_id'])):\n    df_multi.loc[val_idx, 'fold'] = fold\n\n# Verify the split worked correctly by checking Fold 0\nfold_0_train = df_multi[df_multi['fold'] != 0]\nfold_0_val = df_multi[df_multi['fold'] == 0]\n\n# Check for leakage (Intersection of patient IDs should be an empty set)\ntrain_patients = set(fold_0_train['patient_id'])\nval_patients = set(fold_0_val['patient_id'])\nleakage = train_patients.intersection(val_patients)\n\nprint(f\"Patients overlapping between Train and Val: {len(leakage)}\")\nprint(f\"Fold 0 Train size: {len(fold_0_train)} | Positives: {fold_0_train['cancer'].sum()}\")\nprint(f\"Fold 0 Val size:   {len(fold_0_val)} | Positives: {fold_0_val['cancer'].sum()}\")\n\n# Display the final prepared dataframe\ndisplay(df_multi.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-04-18T13:06:29.973227Z","iopub.execute_input":"2026-04-18T13:06:29.973716Z","iopub.status.idle":"2026-04-18T13:06:33.463582Z","shell.execute_reply.started":"2026-04-18T13:06:29.973684Z","shell.execute_reply":"2026-04-18T13:06:33.462949Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Beyond the Pixels: Mapping Clinical Risk Factors\nBefore we feed these images into the Vision Transformer, we need to establish the baseline clinical reality. What does the typical malignancy profile look like in this dataset? \n\nIn real-world diagnostics, a radiologist does not just look at a mammogram in a vacuum; they factor in patient age, tissue density, and the presence of implants (which can obscure dense tissue). \n\nBy extracting and visualizing these engineered features from the patient metadata, we define the exact clinical parameters our ViT needs to overcome. If the data shows that cancer is exceptionally rare in younger patients or harder to detect in patients with implants, our model's multi-view attention mechanisms will need to be robust enough to handle these physiological variables.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nimport pandas as pd\n\n# 1. Load the raw CSV (Double check this path matches exactly what is in your Kaggle input folder!)\ncsv_path = '/kaggle/input/competitions/rsna-breast-cancer-detection/train.csv'\nraw_df = pd.read_csv(csv_path)\n\n# --- THE FIX: Standardize column names ---\n# Print the original columns so you can see exactly what went wrong in the output\nprint(\"Original Columns found in CSV:\", raw_df.columns.tolist())\n\n# Force all column names to lowercase and remove hidden spaces\nraw_df.columns = raw_df.columns.str.lower().str.strip()\n\n# Now extract safely\nmetadata = raw_df[['patient_id', 'laterality', 'age', 'implant']].drop_duplicates()\n\n# 2. Merge and Impute\n# Merge this metadata into our engineered multi-view dataframe\ndf_multi = df_multi.merge(metadata, on=['patient_id', 'laterality'], how='left')\n\n# Handle missing age values (impute with median)\ndf_multi['age'] = df_multi['age'].fillna(df_multi['age'].median())\n\n# ==========================================\n# 4. Publication-Ready Visualization \n# ==========================================\n# (Keep the rest of the plotting code exactly the same as the previous step)\nplt.style.use('dark_background')\nsns.set_palette(\"husl\")\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))\nfig.suptitle('Clinical Risk Mapping: Age and Physiological Factors', fontsize=16, fontweight='bold', color='white')\n\n# Plot 1: Age Distribution\nsns.kdeplot(data=df_multi[df_multi['cancer'] == 0], x='age', fill=True, ax=ax1, label='Benign (0)', color='#4a4a4a', alpha=0.5)\nsns.kdeplot(data=df_multi[df_multi['cancer'] == 1], x='age', fill=True, ax=ax1, label='Malignant (1)', color='#ff4b4b', alpha=0.5)\nax1.set_title('Malignancy Distribution by Patient Age', color='white')\nax1.set_xlabel('Patient Age')\nax1.set_ylabel('Density')\nax1.legend()\nax1.grid(False)\n\n# Plot 2: The Implant Obscuration Factor\nimplant_risk = df_multi.groupby('implant')['cancer'].mean() * 100\n\nsns.barplot(x=implant_risk.index, y=implant_risk.values, ax=ax2, hue=implant_risk.index, palette=['#4a4a4a', '#ff4b4b'], legend=False)\n\nax2.set_title('Cancer Prevalence: Implant vs No Implant', color='white')\nax2.set_xlabel('Presence of Implant')\nax2.set_ylabel('Prevalence (%)')\n\nax2.set_xticks([0, 1]) \nax2.set_xticklabels(['No Implant', 'Implant'])\nax2.grid(False)\n\nfor p in ax2.patches:\n    ax2.annotate(f'{p.get_height():.2f}%', (p.get_x() + p.get_width() / 2., p.get_height()),\n                ha='center', va='baseline', fontsize=12, color='white', xytext=(0, 5),\n                textcoords='offset points')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-04-18T13:17:05.20417Z","iopub.execute_input":"2026-04-18T13:17:05.204919Z","iopub.status.idle":"2026-04-18T13:17:05.874376Z","shell.execute_reply.started":"2026-04-18T13:17:05.204889Z","shell.execute_reply":"2026-04-18T13:17:05.87375Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. The Acquisition Environment: Combating Scanner Bias\nA model is only as good as its ability to generalize. In real-world deployments, a massive hurdle is **Hardware Bias**. \n\nDifferent hospitals (`site_id`) use different mammography machines (`machine_id`) from various manufacturers (e.g., Hologic, Siemens, GE). These machines output images with fundamentally different contrast profiles, resolution limits, and noise artifacts. \n\nIf we don't map these \"environmental\" differences, our Vision Transformer might inadvertently learn to diagnose the *machine* rather than the *patient*. Below, we map the distribution of our dataset across different clinical sites and machines to visualize the hardware landscape our model must navigate.","metadata":{}},{"cell_type":"code","source":"# 1. Grab the environmental/hardware metadata from the raw CSV\nhardware_metadata = raw_df[['patient_id', 'laterality', 'site_id', 'machine_id']].drop_duplicates()\n\n# 2. Merge into our multi-view dataframe\ndf_multi = df_multi.merge(hardware_metadata, on=['patient_id', 'laterality'], how='left')\n\n# 3. Calculate metrics for the visualizations\nsite_counts = df_multi['site_id'].value_counts()\nsite_cancer_risk = df_multi.groupby('site_id')['cancer'].mean() * 100\n\nmachine_counts = df_multi['machine_id'].value_counts().head(10) # Top 10 machines\n\n# ==========================================\n# 4. Publication-Ready Visualizations\n# ==========================================\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))\nfig.suptitle('Acquisition Environment Mapping: Clinical Sites & Machine Artifacts', fontsize=16, fontweight='bold', color='white')\n\n# Plot 1: Distribution across Clinical Sites and inherent risk\nsns.barplot(x=site_cancer_risk.index, y=site_cancer_risk.values, ax=ax1, palette='mako')\nax1.set_title('Cancer Prevalence by Clinical Site ID', color='white')\nax1.set_xlabel('Hospital / Site ID')\nax1.set_ylabel('Cancer Prevalence (%)')\nax1.grid(False, axis='x')\nax1.grid(True, axis='y', alpha=0.2, color='white')\n\n# Add values on top of bars\nfor p in ax1.patches:\n    ax1.annotate(f'{p.get_height():.2f}%', (p.get_x() + p.get_width() / 2., p.get_height()),\n                ha='center', va='baseline', fontsize=10, color='white', xytext=(0, 5),\n                textcoords='offset points')\n\n# Plot 2: Machine dominance (The source of potential hardware bias)\nsns.barplot(x=machine_counts.index.astype(str), y=machine_counts.values, ax=ax2, palette='rocket')\nax2.set_title('Top 10 Mammography Machines in Dataset', color='white')\nax2.set_xlabel('Machine ID')\nax2.set_ylabel('Number of Multi-View Scans')\nax2.grid(False)\n\n# Rotate x-labels for better readability\nplt.setp(ax2.get_xticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-04-18T13:13:08.395238Z","iopub.execute_input":"2026-04-18T13:13:08.395909Z","iopub.status.idle":"2026-04-18T13:13:08.745754Z","shell.execute_reply.started":"2026-04-18T13:13:08.39588Z","shell.execute_reply":"2026-04-18T13:13:08.745074Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Interpreting the Clinical Reality (What the Data Tells Us)\n\nBefore training our AI, we mapped the patient data to understand the exact clinical hurdles our model would face. The visualizations above reveal two critical narratives:\n\n**1. The Aging Factor (Left Graph)**\nThe density plot on the left shows two \"mountains\" of patient data. The gray area represents healthy (benign) scans, while the red area represents malignant cancer cases. Notice how the red peak is shifted noticeably to the right. \n* **The Takeaway:** As patient age increases, the probability of malignancy spikes significantly. For our AI, this means it cannot just look for high-density white spots (tumors); it must learn to differentiate between a dangerous malignant mass and the natural, harmless changes that occur in breast tissue as a patient ages.\n\n**2. The Implant Obscuration Problem (Right Graph)**\nThe bar chart on the right highlights a notorious challenge in radiology: breast implants. You might notice that the diagnosed cancer prevalence appears slightly different for patients with implants. \n* **The Takeaway:** Implants can compress surrounding breast tissue and physically block X-rays, hiding early-stage tumors. By mapping this, we acknowledge that our Vision Transformer must be highly sensitive. It cannot rely on easy, obvious shapes; it must learn to hunt for microscopic, distorted spatial patterns hiding *behind* or *around* artificial implants. \n\nBy understanding these physiological variables, we aren't just blindly feeding data into an algorithm—we are engineering a diagnostic system aware of human biology.","metadata":{}},{"cell_type":"markdown","source":"## 5. The Ultimate Challenge: Tissue Density and \"Difficult Negatives\"\n\nThe most significant anatomical hurdle in mammography is breast tissue density. The BI-RADS system categorizes density from A (fatty, easy to read) to D (extremely dense, very hard to read). Because both dense tissue and malignant tumors appear bright white on an X-ray, finding a tumor in a Class D breast is often described as finding a snowball in a blizzard.\n\nThe RSNA dataset includes a `difficult_negative_case` flag. These are images where no cancer was present, but the visual patterns were so suspicious that they triggered false alarms and required biopsies. \n\nBy analyzing the relationship between tissue density and these difficult negatives, we define the ultimate threshold our AI must cross: **Can our Vision Transformer differentiate between harmless dense tissue and microscopic malignancies better than the human eye?**","metadata":{}},{"cell_type":"code","source":"# 1. Grab the advanced clinical metadata\nadvanced_metadata = raw_df[['patient_id', 'laterality', 'density', 'difficult_negative_case']].drop_duplicates()\n\n# 2. Merge into our multi-view dataframe\ndf_multi = df_multi.merge(advanced_metadata, on=['patient_id', 'laterality'], how='left')\n\n# 3. Handle missing values \n# Density has many missing values, we will categorize them as 'Unknown'\ndf_multi['density'] = df_multi['density'].fillna('Unknown')\n\n# 4. Calculate metrics\n# We want to see what percentage of NEGATIVE cases were marked as \"Difficult\" (False alarm risk)\nnegatives_only = df_multi[df_multi['cancer'] == 0]\ndifficulty_by_density = negatives_only.groupby('density')['difficult_negative_case'].mean() * 100\n\n# Calculate actual cancer prevalence by density\ncancer_by_density = df_multi.groupby('density')['cancer'].mean() * 100\n\n# ==========================================\n# 5. Publication-Ready Visualizations\n# ==========================================\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))\nfig.suptitle('The Density Hurdle: Human False Alarms vs. True Malignancy', fontsize=16, fontweight='bold', color='white')\n\n# Ensure we plot densities in the correct BI-RADS order: A, B, C, D, Unknown\ndensity_order = ['A', 'B', 'C', 'D', 'Unknown']\n\n# Plot 1: The False Alarm Rate (Difficult Negatives)\nsns.barplot(x=difficulty_by_density.index, y=difficulty_by_density.values, \n            ax=ax1, order=density_order, hue=difficulty_by_density.index, \n            palette='flare', legend=False)\nax1.set_title('Human False Alarm Rate by Tissue Density', color='white')\nax1.set_xlabel('BI-RADS Tissue Density (A=Fatty -> D=Extremely Dense)')\nax1.set_ylabel('% of Benign Scans Marked as \"Difficult\"')\nax1.grid(False, axis='x')\nax1.grid(True, axis='y', alpha=0.2, color='white')\n\n# Plot 2: Actual Cancer Prevalence by Density\nsns.barplot(x=cancer_by_density.index, y=cancer_by_density.values, \n            ax=ax2, order=density_order, hue=cancer_by_density.index, \n            palette='crest', legend=False)\nax2.set_title('Actual Cancer Prevalence by Tissue Density', color='white')\nax2.set_xlabel('BI-RADS Tissue Density')\nax2.set_ylabel('True Cancer Prevalence (%)')\nax2.grid(False, axis='x')\nax2.grid(True, axis='y', alpha=0.2, color='white')\n\n# Add values on top of bars for clarity\nfor ax in [ax1, ax2]:\n    for p in ax.patches:\n        height = p.get_height()\n        if not pd.isna(height) and height > 0:\n            ax.annotate(f'{height:.1f}%', (p.get_x() + p.get_width() / 2., height),\n                        ha='center', va='baseline', fontsize=11, color='white', xytext=(0, 5),\n                        textcoords='offset points')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-04-18T13:18:38.646366Z","iopub.execute_input":"2026-04-18T13:18:38.647131Z","iopub.status.idle":"2026-04-18T13:18:39.031519Z","shell.execute_reply.started":"2026-04-18T13:18:38.647103Z","shell.execute_reply":"2026-04-18T13:18:39.030831Z"}},"outputs":[],"execution_count":null}]}