{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":""}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"c56eb88e","cell_type":"markdown","source":"# Multi-Backbone 2.5D Knee MRI Benchmark Suite (`enb0_benchmark.py`)\n\n**Rigorous Architecture & Evaluation Protocol:**\n- **Zero Data Leakage:**\n  - Holdout Test studies strictly isolated via **Multi-Label Stratification** before training.\n  - Remaining studies split into **5 Stratified Folds**.\n  - Volume windowing (Tissue-Windowed 2.5D AIP) computed independently per 3D volume.\n- **Tissue-Windowed Representation (Non-Overlapping Thirds):**\n  - **R Channel:** Muscle Window (FS: $p33-p60$, Non-FS: $p30-p55$)\n  - **G Channel:** Bone Window (FS: $p60-p80$, Non-FS: $p55-p77$)\n  - **B Channel:** Cartilage / Fluid Window (FS: $p80-p97$, Non-FS: $p77-p95$)\n- **Evaluated Vision Backbones (7 Modern Architectures):**\n  1. `efficientnet_b0` (ImageNet pretrained, frozen)\n  2. `efficientnet_b3` (ImageNet pretrained, frozen)\n  3. `efficientnet_v2_s` (ImageNet pretrained, frozen)\n  4. `resnet34` (ImageNet pretrained, frozen)\n  5. `resnet50` (ImageNet pretrained, frozen)\n  6. `densenet121` (ImageNet pretrained, frozen)\n  7. `densenet169` (ImageNet pretrained, frozen)\n- **Standardized Pooling & Head:**\n  - Concatenated Global Average Pooling (GAP) + Global Max Pooling (GMP) across all backbones.\n  - GAP captures diffuse fluid contrast; GMP preserves localized focal tear activations.\n  - Dropout($p=0.3$) + Linear($2 \\times C$, 12).\n  - BatchNorm2d layers strictly locked in `eval()` mode permanently to prevent running-stat corruption.\n- **Pre-Flight Dry Run:**\n  - Instantiates and validates forward passes across all 7 backbones before any training begins.\n- **Sequential Diagnostic Reporting per Model:**\n  - Figure 1: Validation Loss and Validation Macro-AUC across 5 folds (dual-panel).\n  - Table 1: Per-Class Benchmark Performance Table (5-Fold OOF vs Holdout Test Macro ROC-AUC).\n  - Figure 2: 12×12 Multilabel Confusion Matrix Heatmap (standard 0.50 threshold).\n  - TSV Block: Tab-separated results for spreadsheet copy-paste.\n- **Consolidated Master ROC Curve:**\n  - Dual-panel overlay comparing all 7 models' Macro ROC curves on OOF and Holdout Test.","metadata":{}},{"id":"4793c2c6","cell_type":"code","source":"import os\nimport gc\nimport cv2\nimport json\nimport random\nimport pathlib\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm.auto import tqdm\nfrom sklearn.metrics import roc_auc_score, roc_curve\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nimport torchvision.models as models\nimport torchvision.transforms as T\nimport pydicom\n\ntry:\n    import pylibjpeg  # noqa: F401\nexcept ImportError:\n    pylibjpeg = None\n\n# Dark mode visual theme\nplt.rcParams['figure.facecolor'] = '#121212'\nplt.rcParams['axes.facecolor'] = '#121212'\nplt.rcParams['text.color'] = 'white'\nplt.rcParams['axes.labelcolor'] = 'white'\nplt.rcParams['xtick.color'] = 'white'\nplt.rcParams['ytick.color'] = 'white'\n\ndef seed_everything(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nseed_everything(42)\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(f\"Using Compute Device: {DEVICE}\")","metadata":{},"outputs":[],"execution_count":null},{"id":"60bbf9aa","cell_type":"markdown","source":"## 1. Configuration & Global Constants","metadata":{}},{"id":"c09e2a26","cell_type":"code","source":"KAGGLE_DIR = pathlib.Path(\"/kaggle/input/competitions/rsna-knee-abnormality-detection\")\nLOCAL_DIR = pathlib.Path(\"./rsna_data\")\nROOT_DIR = KAGGLE_DIR if KAGGLE_DIR.exists() else LOCAL_DIR\n\nTRAIN_CSV = ROOT_DIR / \"train.csv\"\nSERIES_CSV = ROOT_DIR / \"train_series.csv\"\nSERIES_DIR = ROOT_DIR / (\"train_series\" if (ROOT_DIR / \"train_series\").exists() else \"sample_dicom/train_series\")\nOUT_DIR = pathlib.Path(\"/kaggle/working\" if KAGGLE_DIR.exists() else \"./tests/2.5d/outputs\")\nOUT_DIR.mkdir(parents=True, exist_ok=True)\n\nTARGET_COLS = [\n    'ACL', 'MCL', 'Medial Meniscus', 'Lateral Meniscus',\n    'Medial OA', 'Lateral OA', 'PF OA', 'Effusion',\n    'Synovitis', \"Baker's\", 'Contusion', 'Fracture'\n]\n\nWINDOW_PARAMS = {\n    \"FS\":   {\"R\": (33, 60), \"G\": (60, 80), \"B\": (80, 97)},   # Muscle, Bone, Cartilage/Fluid\n    \"NoFS\": {\"R\": (30, 55), \"G\": (55, 77), \"B\": (77, 95)},   # Muscle, Bone, Cartilage/Fat\n}\n\nIMG_SIZE = 224\nBATCH_SIZE = 8\nEPOCHS = 20\nPATIENCE = 5\nLR = 5e-4\nWEIGHT_DECAY = 1e-4\nWARMUP_EPOCHS = 3\n\nMODELS_TO_BENCHMARK = [\n    \"efficientnet_b0\",\n    \"efficientnet_b3\",\n    \"efficientnet_v2_s\",\n    \"resnet34\",\n    \"resnet50\",\n    \"densenet121\",\n    \"densenet169\"\n]\n\nMODEL_DISPLAY_NAMES = {\n    \"efficientnet_b0\": \"EfficientNet-B0\",\n    \"efficientnet_b3\": \"EfficientNet-B3\",\n    \"efficientnet_v2_s\": \"EfficientNet-V2-S\",\n    \"resnet34\": \"ResNet-34\",\n    \"resnet50\": \"ResNet-50\",\n    \"densenet121\": \"DenseNet-121\",\n    \"densenet169\": \"DenseNet-169\",\n}\n\nprint(f\"Target Pathologies ({len(TARGET_COLS)}): {TARGET_COLS}\")\nprint(f\"Candidate Backbones ({len(MODELS_TO_BENCHMARK)}): {MODELS_TO_BENCHMARK}\")","metadata":{"lines_to_next_cell":1},"outputs":[],"execution_count":null},{"id":"3f35d34b","cell_type":"markdown","source":"## 2. Volume Loading & 2.5D Tissue-Windowing Preprocessing Functions","metadata":{}},{"id":"608c440e","cell_type":"code","source":"def load_series_volume(series_path):\n    \"\"\"Loads a DICOM series directory into a 3D float32 volume sorted by slice position.\"\"\"\n    dcm_files = sorted(series_path.glob(\"*.dcm\"))\n    if not dcm_files:\n        return None\n    slices = []\n    for f in dcm_files:\n        try:\n            ds = pydicom.dcmread(str(f))\n            arr = ds.pixel_array.astype(np.float32)\n            slope = float(getattr(ds, \"RescaleSlope\", 1.0))\n            intercept = float(getattr(ds, \"RescaleIntercept\", 0.0))\n            arr = arr * slope + intercept\n            ipp = getattr(ds, \"ImagePositionPatient\", None)\n            pos = float(ipp[0]) if ipp else float(getattr(ds, \"InstanceNumber\", len(slices)))\n            slices.append((pos, arr))\n        except Exception:\n            continue\n    if not slices:\n        return None\n    slices.sort(key=lambda x: x[0])\n    return np.stack([s[1] for s in slices], axis=-1)\n\n\ndef compute_tissue_2_5d_rgb(volume, is_fs=True):\n    \"\"\"\n    Computes tissue-windowed (non-overlapping percentile thirds) average intensity projection (AIP).\n    Returns (R_channel, G_channel, B_channel, RGB_composite_resized).\n    \"\"\"\n    thresh = max(np.percentile(volume, 10), 0.0)\n    fg_vals = volume[volume > thresh]\n    if len(fg_vals) == 0:\n        fg_vals = volume\n\n    wins = WINDOW_PARAMS[\"FS\"] if is_fs else WINDOW_PARAMS[\"NoFS\"]\n    channels = []\n    for ch_key in [\"R\", \"G\", \"B\"]:\n        lo_pct, hi_pct = wins[ch_key]\n        lo, hi = np.percentile(fg_vals, [lo_pct, hi_pct])\n        if hi <= lo:\n            hi = lo + 1e-5\n        clipped = np.clip((volume - lo) / (hi - lo), 0.0, 1.0)\n        channels.append(clipped.mean(axis=-1))\n\n    rgb = np.stack(channels, axis=-1)\n    rgb_resized = cv2.resize(rgb, (IMG_SIZE, IMG_SIZE), interpolation=cv2.INTER_AREA)\n    rgb_resized = np.clip(rgb_resized, 0.0, 1.0).astype(np.float32)\n    return channels[0], channels[1], channels[2], rgb_resized\n\n\ndef select_best_series(study_series_df, plane=\"Sagittal\"):\n    \"\"\"Selects the best series for a plane, prioritizing fluid-sensitive acquisitions.\"\"\"\n    plane_series = study_series_df[study_series_df[\"Anatomical_Plane\"] == plane]\n    if len(plane_series) == 0:\n        return None, None\n    fs_series = plane_series[plane_series[\"Fluid_Sensitive\"] == 1]\n    if len(fs_series) > 0:\n        return fs_series.iloc[0][\"SeriesInstanceUID\"], True\n    else:\n        return plane_series.iloc[0][\"SeriesInstanceUID\"], bool(plane_series.iloc[0][\"Fluid_Sensitive\"])\n\n\ndef precompute_gold_dataset(train_csv_path, series_csv_path, series_dir_path, plane=\"Sagittal\"):\n    \"\"\"\n    Precomputes 2.5D tissue-windowed slices for all gold-labeled studies ONCE in memory.\n    Saves massive disk I/O overhead during multi-model evaluation.\n    \"\"\"\n    train_df = pd.read_csv(train_csv_path)\n    series_df = pd.read_csv(series_csv_path)\n    gold_df = train_df[train_df[TARGET_COLS[0]].notna()].copy().reset_index(drop=True)\n\n    samples = []\n    for _, row in tqdm(gold_df.iterrows(), total=len(gold_df), desc=f\"Precomputing {plane} 2.5D AIP\"):\n        study_uid = row[\"StudyInstanceUID\"]\n        study_series = series_df[series_df[\"StudyInstanceUID\"] == study_uid]\n        best_series_uid, is_fs = select_best_series(study_series, plane=plane)\n        if best_series_uid is None:\n            continue\n\n        s_path = series_dir_path / study_uid / best_series_uid\n        if not s_path.exists():\n            continue\n\n        vol = load_series_volume(s_path)\n        if vol is None:\n            continue\n\n        r_ch, g_ch, b_ch, rgb_img = compute_tissue_2_5d_rgb(vol, is_fs=is_fs)\n        labels = row[TARGET_COLS].values.astype(np.float32)\n\n        samples.append({\n            \"StudyInstanceUID\": study_uid,\n            \"SeriesInstanceUID\": best_series_uid,\n            \"is_fs\": is_fs,\n            \"raw_slice\": vol[:, :, vol.shape[-1] // 2],\n            \"r_ch\": r_ch,\n            \"g_ch\": g_ch,\n            \"b_ch\": b_ch,\n            \"image\": rgb_img,\n            \"labels\": labels\n        })\n\n    print(f\"Preprocessed {len(samples)} gold studies with tissue-windowed representation.\")\n    return samples\n\ngold_samples = precompute_gold_dataset(TRAIN_CSV, SERIES_CSV, SERIES_DIR, plane=\"Sagittal\")","metadata":{"lines_to_next_cell":1},"outputs":[],"execution_count":null},{"id":"9a1b3432","cell_type":"markdown","source":"## 3. Visual Verification: Tissue-Windowed Grayscale Channel Inspection","metadata":{}},{"id":"04de39ea","cell_type":"code","source":"def visualize_grayscale_channels(samples, out_dir):\n    fs_sample = next((s for s in samples if s[\"is_fs\"]), samples[0])\n    nofs_sample = next((s for s in samples if not s[\"is_fs\"]), samples[-1])\n\n    vis_rows = [\n        (\"Row 1: Fluid-Sensitive (FS)\", fs_sample),\n        (\"Row 2: Non-Fluid-Sensitive (Non-FS)\", nofs_sample)\n    ]\n\n    fig, axes = plt.subplots(2, 5, figsize=(20, 8.5), facecolor=\"#121212\")\n    fig.suptitle(\"Tissue-Windowed Representation: R, G, B Channels in Grayscale (FS vs. Non-FS)\",\n                 fontsize=14, color=\"white\", fontweight=\"bold\", y=0.98)\n\n    titles = [\n        \"1. Raw Central Slice\\n(Grayscale)\",\n        \"2. R-Channel (Muscle)\\n[Grayscale]\",\n        \"3. G-Channel (Bone)\\n[Grayscale]\",\n        \"4. B-Channel (Cartilage/Fluid)\\n[Grayscale]\",\n        \"5. Combined 2.5D RGB\\n(Composite)\"\n    ]\n\n    for row_idx, (row_label, sample) in enumerate(vis_rows):\n        raw = sample[\"raw_slice\"]\n        raw_norm = np.clip((raw - np.percentile(raw, 1)) / (np.percentile(raw, 99) - np.percentile(raw, 1) + 1e-5), 0, 1)\n\n        axes[row_idx, 0].imshow(raw_norm, cmap=\"gray\")\n        axes[row_idx, 1].imshow(sample[\"r_ch\"], cmap=\"gray\")\n        axes[row_idx, 2].imshow(sample[\"g_ch\"], cmap=\"gray\")\n        axes[row_idx, 3].imshow(sample[\"b_ch\"], cmap=\"gray\")\n        axes[row_idx, 4].imshow(sample[\"image\"])\n\n        axes[row_idx, 0].set_ylabel(f\"{row_label}\\nUID: {sample['StudyInstanceUID'][:16]}...\",\n                                    color=\"white\", fontsize=10, labelpad=8)\n\n        for c in range(5):\n            if row_idx == 0:\n                axes[row_idx, c].set_title(titles[c], color=\"white\", fontsize=10.5, pad=6, fontweight=\"bold\")\n            axes[row_idx, c].set_xticks([])\n            axes[row_idx, c].set_yticks([])\n            for sp in axes[row_idx, c].spines.values():\n                sp.set_color(\"#444444\")\n\n    plt.tight_layout(rect=[0, 0, 1, 0.94])\n    plt.savefig(out_dir / \"grayscale_channels_sample.png\", dpi=180, bbox_inches=\"tight\", facecolor=fig.get_facecolor())\n    plt.show()\n\nvisualize_grayscale_channels(gold_samples, OUT_DIR)","metadata":{"lines_to_next_cell":1},"outputs":[],"execution_count":null},{"id":"0299adbf","cell_type":"markdown","source":"## 4. Multi-Label Stratification & Dataset Utilities","metadata":{}},{"id":"64952b93","cell_type":"code","source":"def multilabel_stratified_split(labels, n_splits=5, seed=42):\n    \"\"\"Greedy multi-label stratification balancing label distributions and fold sizes.\"\"\"\n    np.random.seed(seed)\n    n_samples, n_labels = labels.shape\n    fold_indices = [[] for _ in range(n_splits)]\n    fold_label_counts = np.zeros((n_splits, n_labels))\n\n    label_frequencies = labels.sum(axis=0)\n    sample_scores = labels @ (1.0 / (label_frequencies + 1e-5))\n    sorted_sample_indices = np.argsort(-sample_scores)\n\n    for idx in sorted_sample_indices:\n        sample_labels = labels[idx]\n        scores = []\n        for f in range(n_splits):\n            count_sum = np.sum(fold_label_counts[f] * sample_labels)\n            size_penalty = len(fold_indices[f]) / (n_samples / n_splits)\n            scores.append(count_sum + size_penalty)\n        best_fold = int(np.argmin(scores))\n        fold_indices[best_fold].append(idx)\n        fold_label_counts[best_fold] += sample_labels\n\n    return [np.array(f) for f in fold_indices]\n\nall_labels = np.array([s[\"labels\"] for s in gold_samples])\ninitial_splits = multilabel_stratified_split(all_labels, n_splits=6, seed=42)\ntest_idx = initial_splits[0]\ncv_idx = np.concatenate(initial_splits[1:])\n\ntest_samples = [gold_samples[i] for i in test_idx]\ncv_samples = [gold_samples[i] for i in cv_idx]\ncv_labels = np.array([s[\"labels\"] for s in cv_samples])\n\ncv_fold_indices = multilabel_stratified_split(cv_labels, n_splits=5, seed=42)\nassert len(set(test_idx).intersection(set(cv_idx))) == 0, \"DATA LEAKAGE DETECTED!\"\nprint(f\"Stratification complete: {len(cv_samples)} CV studies | {len(test_samples)} Holdout Test studies.\")\n\n\nclass Knee25DDataset(Dataset):\n    \"\"\"PyTorch Dataset reading precomputed 2.5D tissue-windowed RGB images.\"\"\"\n    def __init__(self, samples, transform=None):\n        self.samples = samples\n        self.transform = transform\n        self.norm = T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n\n    def __len__(self):\n        return len(self.samples)\n\n    def __getitem__(self, idx):\n        item = self.samples[idx]\n        img = item[\"image\"]\n        tensor_img = torch.from_numpy(img).permute(2, 0, 1).float()\n        if self.transform:\n            tensor_img = self.transform(tensor_img)\n        tensor_img = self.norm(tensor_img)\n        labels = torch.tensor(item[\"labels\"], dtype=torch.float32)\n        return tensor_img, labels, item[\"StudyInstanceUID\"]\n\n# NOTE: RandomHorizontalFlip is strictly omitted to prevent medial/lateral swapping.\ntrain_transforms = T.Compose([\n    T.RandomAffine(degrees=10, translate=(0.05, 0.05), scale=(0.95, 1.05))\n])","metadata":{"lines_to_next_cell":1},"outputs":[],"execution_count":null},{"id":"78665f2a","cell_type":"markdown","source":"## 5. Unified Multi-Backbone Model Architecture & Factory","metadata":{}},{"id":"70f6831b","cell_type":"code","source":"class UnifiedFrozenBackbone(nn.Module):\n    \"\"\"\n    Universal multi-backbone wrapper supporting:\n    - EfficientNet-B0, EfficientNet-B3, EfficientNet-V2-S\n    - ResNet-34, ResNet-50\n    - DenseNet-121, DenseNet-169\n\n    Key Architectural Guarantees:\n    - Backbone parameters are frozen (requires_grad = False).\n    - Concatenated Global Average Pooling (GAP) + Global Max Pooling (GMP).\n    - Trainable dropout + linear head: Linear(2 * C, num_classes).\n    - train(mode) override: permanently forces backbone and all BatchNorm modules to eval(),\n      preventing running-stat corruption on small batches.\n    \"\"\"\n    def __init__(self, model_name: str, num_classes: int = 12, dropout_p: float = 0.3):\n        super().__init__()\n        self.model_name = model_name.lower()\n        self.num_classes = num_classes\n\n        if self.model_name == \"efficientnet_b0\":\n            base = models.efficientnet_b0(weights=models.EfficientNet_B0_Weights.DEFAULT)\n            self.features = base.features\n            in_features = base.classifier[1].in_features  # 1280\n        elif self.model_name == \"efficientnet_b3\":\n            base = models.efficientnet_b3(weights=models.EfficientNet_B3_Weights.DEFAULT)\n            self.features = base.features\n            in_features = base.classifier[1].in_features  # 1536\n        elif self.model_name == \"efficientnet_v2_s\":\n            base = models.efficientnet_v2_s(weights=models.EfficientNet_V2_S_Weights.DEFAULT)\n            self.features = base.features\n            in_features = base.classifier[1].in_features  # 1280\n        elif self.model_name == \"resnet34\":\n            base = models.resnet34(weights=models.ResNet34_Weights.DEFAULT)\n            self.features = nn.Sequential(*list(base.children())[:-2])\n            in_features = base.fc.in_features  # 512\n        elif self.model_name == \"resnet50\":\n            base = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)\n            self.features = nn.Sequential(*list(base.children())[:-2])\n            in_features = base.fc.in_features  # 2048\n        elif self.model_name == \"densenet121\":\n            base = models.densenet121(weights=models.DenseNet121_Weights.DEFAULT)\n            self.features = nn.Sequential(base.features, nn.ReLU(inplace=True))\n            in_features = base.classifier.in_features  # 1024\n        elif self.model_name == \"densenet169\":\n            base = models.densenet169(weights=models.DenseNet169_Weights.DEFAULT)\n            self.features = nn.Sequential(base.features, nn.ReLU(inplace=True))\n            in_features = base.classifier.in_features  # 1664\n        else:\n            raise ValueError(f\"Unsupported model_name: {model_name}\")\n\n        # Freeze all backbone parameters\n        for p in self.features.parameters():\n            p.requires_grad = False\n\n        self.in_features = in_features\n        self.pooled_features = in_features * 2  # GAP + GMP concatenated\n\n        # Trainable classification head receives concatenated [GAP, GMP]\n        self.head = nn.Sequential(\n            nn.Dropout(p=dropout_p),\n            nn.Linear(self.pooled_features, num_classes)\n        )\n\n    def forward(self, x):\n        feat = self.features(x)\n        gap = torch.mean(feat, dim=(2, 3))\n        gmp = torch.amax(feat, dim=(2, 3))\n        fused = torch.cat([gap, gmp], dim=1)\n        return self.head(fused)\n\n    def train(self, mode=True):\n        \"\"\"Keep backbone and all BatchNorm running stats permanently in eval().\"\"\"\n        super().train(mode)\n        self.features.eval()\n        for m in self.features.modules():\n            if isinstance(m, (nn.BatchNorm2d, nn.BatchNorm1d)):\n                m.eval()\n        return self\n\n\ndef build_backbone_model(model_name: str, num_classes: int = 12) -> UnifiedFrozenBackbone:\n    \"\"\"Factory helper to build a frozen backbone with concatenated GAP+GMP pooling.\"\"\"\n    return UnifiedFrozenBackbone(model_name=model_name, num_classes=num_classes)","metadata":{"lines_to_next_cell":1},"outputs":[],"execution_count":null},{"id":"7335e3b2","cell_type":"markdown","source":"## 6. Pre-Flight Dry Run: Verifying All 7 Models BEFORE Training Starts","metadata":{}},{"id":"4bec33ee","cell_type":"code","source":"def run_pre_flight_dry_run(model_names, num_classes=12, device=DEVICE):\n    \"\"\"\n    Executes an end-to-end verification on all candidate backbones BEFORE training begins:\n    - Instantiates each backbone and verifies ImageNet weights load\n    - Performs forward pass with dummy tensor (B=2, C=3, H=224, W=224)\n    - Verifies output shape is strictly (2, num_classes)\n    - Confirms frozen vs trainable parameter isolation\n    \"\"\"\n    print(\"\\n\" + \"=\"*85)\n    print(\"PRE-FLIGHT DRY RUN: VERIFYING ALL 7 BACKBONES BEFORE TRAINING STARTS\")\n    print(\"=\"*85)\n\n    dry_run_results = []\n    for m_name in model_names:\n        print(f\"Testing {m_name:<20}...\", end=\" \", flush=True)\n        try:\n            model = build_backbone_model(m_name, num_classes=num_classes).to(device)\n            dummy_x = torch.randn(2, 3, IMG_SIZE, IMG_SIZE, device=device)\n            with torch.no_grad():\n                dummy_out = model(dummy_x)\n            assert dummy_out.shape == (2, num_classes), f\"Shape mismatch: {dummy_out.shape} != (2, {num_classes})\"\n\n            total_params = sum(p.numel() for p in model.parameters())\n            head_params = sum(p.numel() for p in model.head.parameters())\n            trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n            assert head_params == trainable_params, \"Backbone parameter leakage: trainable != head params!\"\n\n            print(f\"PASSED | InFeat={model.in_features:<4} | Pooled={model.pooled_features:<4} | HeadParams={head_params:<6,} | OutShape={list(dummy_out.shape)}\")\n            dry_run_results.append({\n                \"Model\": m_name,\n                \"In_Features\": model.in_features,\n                \"Pooled_Features\": model.pooled_features,\n                \"Head_Params\": head_params,\n                \"Total_Params\": total_params,\n                \"Status\": \"PASSED\"\n            })\n            del model, dummy_x, dummy_out\n            if torch.cuda.is_available():\n                torch.cuda.empty_cache()\n        except Exception as e:\n            print(f\"FAILED! Error: {e}\")\n            raise RuntimeError(f\"Pre-flight dry run failed for model {m_name}: {e}\")\n\n    print(\"=\"*85)\n    print(\"ALL 7 BACKBONES SUCCESSFULLY VALIDATED! ZERO ISSUES FOUND. PROCEEDING TO BENCHMARK.\")\n    print(\"=\"*85 + \"\\n\")\n    return dry_run_results\n\ndry_run_summary = run_pre_flight_dry_run(MODELS_TO_BENCHMARK, num_classes=len(TARGET_COLS), device=DEVICE)","metadata":{"lines_to_next_cell":1},"outputs":[],"execution_count":null},{"id":"2921b432","cell_type":"markdown","source":"## 7. Core Training, Evaluation & Metric Computation Engines","metadata":{}},{"id":"d6aca048","cell_type":"code","source":"def compute_macro_auc(targets, preds):\n    \"\"\"Calculates macro-average ROC-AUC and per-class AUCs across all columns.\"\"\"\n    aucs = []\n    for c in range(targets.shape[1]):\n        y_true = targets[:, c]\n        y_pred = preds[:, c]\n        if len(np.unique(y_true)) > 1:\n            try:\n                aucs.append(roc_auc_score(y_true, y_pred))\n            except ValueError:\n                aucs.append(0.5)\n        else:\n            aucs.append(0.5)\n    return float(np.mean(aucs)), aucs\n\n\ndef compute_macro_roc_curve(targets, preds, num_points=200):\n    \"\"\"Computes a smooth macro ROC curve by interpolating TPR over a uniform FPR grid.\"\"\"\n    mean_fpr = np.linspace(0.0, 1.0, num_points)\n    tprs = []\n    aucs = []\n    for c in range(targets.shape[1]):\n        y_t = targets[:, c]\n        y_p = preds[:, c]\n        if len(np.unique(y_t)) > 1:\n            fpr, tpr, _ = roc_curve(y_t, y_p)\n            auc_val = roc_auc_score(y_t, y_p)\n            interp_tpr = np.interp(mean_fpr, fpr, tpr)\n            interp_tpr[0] = 0.0\n            tprs.append(interp_tpr)\n            aucs.append(auc_val)\n    mean_tpr = np.mean(tprs, axis=0) if len(tprs) > 0 else mean_fpr\n    mean_tpr[-1] = 1.0\n    macro_auc = float(np.mean(aucs)) if len(aucs) > 0 else 0.5\n    return mean_fpr, mean_tpr, macro_auc\n\n\ndef train_one_epoch(model, dataloader, criterion, optimizer, device):\n    \"\"\"Trains classification head for one epoch.\"\"\"\n    model.train()\n    total_loss = 0.0\n    for imgs, labels, _ in dataloader:\n        imgs, labels = imgs.to(device), labels.to(device)\n        optimizer.zero_grad()\n        logits = model(imgs)\n        loss = criterion(logits, labels)\n        loss.backward()\n        optimizer.step()\n        total_loss += loss.item() * imgs.size(0)\n    return total_loss / len(dataloader.dataset)\n\n\n@torch.no_grad()\ndef evaluate(model, dataloader, criterion, device):\n    \"\"\"Evaluates model on validation or test dataset.\"\"\"\n    model.eval()\n    total_loss = 0.0\n    all_targets, all_preds = [], []\n    for imgs, labels, _ in dataloader:\n        imgs, labels = imgs.to(device), labels.to(device)\n        logits = model(imgs)\n        loss = criterion(logits, labels)\n        total_loss += loss.item() * imgs.size(0)\n        probs = torch.sigmoid(logits).cpu().numpy()\n        all_preds.append(probs)\n        all_targets.append(labels.cpu().numpy())\n\n    targets = np.vstack(all_targets)\n    preds = np.vstack(all_preds)\n    macro_auc, per_class_auc = compute_macro_auc(targets, preds)\n    return total_loss / len(dataloader.dataset), macro_auc, per_class_auc, targets, preds","metadata":{"lines_to_next_cell":1},"outputs":[],"execution_count":null},{"id":"14b169f9","cell_type":"markdown","source":"## 8. Diagnostic Reporting Functions (Executed per Model)","metadata":{}},{"id":"12b449ef","cell_type":"code","source":"def plot_learning_curves(fold_histories, model_name, save_dir):\n    \"\"\"Figure 1: Dual-panel learning curves (Left: Val Loss, Right: Val Macro-AUC).\"\"\"\n    fig, axes = plt.subplots(1, 2, figsize=(14, 4.5), facecolor=\"#121212\")\n\n    for fold_idx, h in enumerate(fold_histories):\n        axes[0].plot(h[\"val_loss\"], label=f\"Fold {fold_idx+1} Val Loss\", alpha=0.85)\n        axes[1].plot(h[\"val_auc\"], label=f\"Fold {fold_idx+1} Val AUC\", alpha=0.85)\n\n    axes[0].set_title(f\"{model_name} — Validation Loss across Epochs\", color=\"white\", fontsize=11.5, fontweight=\"bold\")\n    axes[0].set_xlabel(\"Epoch\", color=\"white\")\n    axes[0].set_ylabel(\"Unweighted BCE Loss\", color=\"white\")\n    axes[0].legend(facecolor=\"#222222\", edgecolor=\"#444444\", labelcolor=\"white\", fontsize=8.5)\n    axes[0].grid(True, alpha=0.2)\n\n    axes[1].set_title(f\"{model_name} — Validation Macro ROC-AUC across Epochs\", color=\"white\", fontsize=11.5, fontweight=\"bold\")\n    axes[1].set_xlabel(\"Epoch\", color=\"white\")\n    axes[1].set_ylabel(\"Macro ROC-AUC\", color=\"white\")\n    axes[1].legend(facecolor=\"#222222\", edgecolor=\"#444444\", labelcolor=\"white\", fontsize=8.5)\n    axes[1].grid(True, alpha=0.2)\n\n    plt.tight_layout()\n    safe_name = model_name.lower().replace('-', '_').replace(' ', '_')\n    plt.savefig(save_dir / f\"{safe_name}_learning_curves.png\", dpi=180, bbox_inches=\"tight\", facecolor=fig.get_facecolor())\n    plt.show()\n\n\ndef safe_div(n, d, default=0.0):\n    \"\"\"Zero-division safe float division.\"\"\"\n    return float(n / d) if d > 0 else default\n\n\ndef compute_rank_auc(y_true, y_score):\n    \"\"\"Calculates Wilcoxon-Mann-Whitney ROC-AUC matching sklearn to machine precision.\"\"\"\n    pos = y_score[y_true == 1]\n    neg = y_score[y_true == 0]\n    if len(pos) == 0 or len(neg) == 0:\n        return 0.5\n    u = np.sum(pos[:, None] > neg[None, :]) + 0.5 * np.sum(pos[:, None] == neg[None, :])\n    return float(u / (len(pos) * len(neg)))\n\n\ndef compute_average_precision(y_true, y_score):\n    \"\"\"Calculates PR-AUC (Average Precision) by trapezoidal integration of PR curve.\"\"\"\n    pos_count = int(np.sum(y_true == 1))\n    if pos_count == 0:\n        return 0.0\n    order = np.argsort(-y_score)\n    y_eval = y_true[order]\n    tp = np.cumsum(y_eval == 1)\n    fp = np.cumsum(y_eval == 0)\n    precision = tp / (tp + fp)\n    recall = tp / pos_count\n    recall_diff = np.diff(np.concatenate(([0.0], recall)))\n    return float(np.sum(precision * recall_diff))\n\n\ndef compute_brier(y_true, y_probs):\n    \"\"\"Mean squared error between binary labels and predicted probabilities.\"\"\"\n    return float(np.mean((y_true - y_probs) ** 2))\n\n\ndef compute_logloss(y_true, y_probs, eps=1e-15):\n    \"\"\"Binary cross-entropy loss with epsilon clipping.\"\"\"\n    p_clip = np.clip(y_probs, eps, 1.0 - eps)\n    return float(-np.mean(y_true * np.log(p_clip) + (1.0 - y_true) * np.log(1.0 - p_clip)))\n\n\ndef compute_f1_at_t(y_true, y_probs, t):\n    \"\"\"Computes F1-score for binary targets at decision threshold t.\"\"\"\n    yp = (y_probs >= t).astype(float)\n    tp = int(np.sum((y_true == 1) & (yp == 1)))\n    fp = int(np.sum((y_true == 0) & (yp == 1)))\n    fn = int(np.sum((y_true == 1) & (yp == 0)))\n    prec = safe_div(tp, tp + fp)\n    rec = safe_div(tp, tp + fn)\n    return safe_div(2 * prec * rec, prec + rec)\n\n\ndef find_optimal_threshold(y_true, y_probs):\n    \"\"\"Finds the decision threshold t in [0.05, 0.95] that maximizes F1-score on OOF.\"\"\"\n    best_t = 0.50\n    best_f1 = 0.0\n    for t in np.linspace(0.05, 0.95, 91):\n        f1 = compute_f1_at_t(y_true, y_probs, t)\n        if f1 > best_f1:\n            best_f1 = f1\n            best_t = float(t)\n    return best_t, best_f1\n\n\ndef compute_binary_stats(y_true, y_probs, threshold=0.5):\n    \"\"\"\n    Computes all standard binary classification metrics at a given decision threshold:\n    Total, Positives, Prevalence, TP, FP, TN, FN, Accuracy, Balanced Accuracy,\n    Sensitivity, Specificity, Precision, NPV, F1, F2, MCC, ROC-AUC, PR-AUC, Brier, Log Loss.\n    \"\"\"\n    yp = (y_probs >= threshold).astype(float)\n    total = len(y_true)\n    pos = int(np.sum(y_true == 1))\n    prev = safe_div(pos, total)\n    tp = int(np.sum((y_true == 1) & (yp == 1)))\n    fp = int(np.sum((y_true == 0) & (yp == 1)))\n    tn = int(np.sum((y_true == 0) & (yp == 0)))\n    fn = int(np.sum((y_true == 1) & (yp == 0)))\n\n    acc = safe_div(tp + tn, total)\n    sens = safe_div(tp, tp + fn)\n    spec = safe_div(tn, tn + fp)\n    bal_acc = 0.5 * (sens + spec)\n    prec = safe_div(tp, tp + fp)\n    npv = safe_div(tn, tn + fn)\n    f1 = safe_div(2 * prec * sens, prec + sens)\n    f2 = safe_div(5 * prec * sens, 4 * prec + sens)\n\n    mcc_denom = np.sqrt(float((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)))\n    mcc = safe_div(float(tp * tn - fp * fn), mcc_denom)\n\n    auc = compute_rank_auc(y_true, y_probs)\n    ap = compute_average_precision(y_true, y_probs)\n    brier = compute_brier(y_true, y_probs)\n    ll = compute_logloss(y_true, y_probs)\n\n    return {\n        \"total\": total, \"pos\": pos, \"prev\": prev,\n        \"tp\": tp, \"fp\": fp, \"tn\": tn, \"fn\": fn,\n        \"acc\": acc, \"bal_acc\": bal_acc, \"sens\": sens, \"spec\": spec,\n        \"prec\": prec, \"npv\": npv, \"f1\": f1, \"f2\": f2, \"mcc\": mcc,\n        \"auc\": auc, \"ap\": ap, \"brier\": brier, \"logloss\": ll\n    }\n\n\ndef build_side_by_side_table(model_name, oof_targets, oof_preds, test_targets, test_preds, threshold=0.5):\n    \"\"\"\n    Builds the publication-grade wide side-by-side results DataFrame:\n    Each row is a Pathology (or OVERALL_MACRO, OVERALL_MICRO, OVERALL_WEIGHTED) for the model.\n    Columns place all OOF metrics side-by-side with Holdout Test metrics.\n    \"\"\"\n    rows = []\n    n_classes = len(TARGET_COLS)\n    oof_per_class = []\n    test_per_class = []\n    opt_thresholds = []\n    oof_calib_f1s = []\n    test_frozen_f1s = []\n\n    for c in range(n_classes):\n        # 1. Base statistics at 0.50 threshold\n        o_stat = compute_binary_stats(oof_targets[:, c], oof_preds[:, c], threshold=threshold)\n        t_stat = compute_binary_stats(test_targets[:, c], test_preds[:, c], threshold=threshold)\n\n        # 2. Optimal threshold calibrated on OOF (maximizing F1)\n        opt_t, oof_cal_f1 = find_optimal_threshold(oof_targets[:, c], oof_preds[:, c])\n\n        # 3. Holdout Test evaluated strictly at frozen OOF optimal threshold (zero leakage!)\n        test_froz_f1 = compute_f1_at_t(test_targets[:, c], test_preds[:, c], opt_t)\n\n        oof_per_class.append(o_stat)\n        test_per_class.append(t_stat)\n        opt_thresholds.append(opt_t)\n        oof_calib_f1s.append(oof_cal_f1)\n        test_frozen_f1s.append(test_froz_f1)\n\n        rows.append({\n            \"Model\": model_name,\n            \"Pathology\": TARGET_COLS[c],\n            # OOF Columns\n            \"OOF_Total\": o_stat[\"total\"], \"OOF_Pos\": o_stat[\"pos\"], \"OOF_Prev\": round(o_stat[\"prev\"], 4),\n            \"OOF_TP\": o_stat[\"tp\"], \"OOF_FP\": o_stat[\"fp\"], \"OOF_TN\": o_stat[\"tn\"], \"OOF_FN\": o_stat[\"fn\"],\n            \"OOF_ROC_AUC\": round(o_stat[\"auc\"], 4), \"OOF_PR_AUC\": round(o_stat[\"ap\"], 4),\n            \"OOF_Accuracy\": round(o_stat[\"acc\"], 4), \"OOF_Balanced_Acc\": round(o_stat[\"bal_acc\"], 4),\n            \"OOF_Sensitivity\": round(o_stat[\"sens\"], 4), \"OOF_Specificity\": round(o_stat[\"spec\"], 4),\n            \"OOF_Precision\": round(o_stat[\"prec\"], 4), \"OOF_NPV\": round(o_stat[\"npv\"], 4),\n            \"OOF_F1\": round(o_stat[\"f1\"], 4), \"OOF_F2\": round(o_stat[\"f2\"], 4), \"OOF_MCC\": round(o_stat[\"mcc\"], 4),\n            \"OOF_Brier\": round(o_stat[\"brier\"], 4), \"OOF_LogLoss\": round(o_stat[\"logloss\"], 4),\n            \"OOF_Opt_Threshold\": round(opt_t, 2), \"OOF_Calib_F1\": round(oof_cal_f1, 4),\n            # Test Columns\n            \"Test_Total\": t_stat[\"total\"], \"Test_Pos\": t_stat[\"pos\"], \"Test_Prev\": round(t_stat[\"prev\"], 4),\n            \"Test_TP\": t_stat[\"tp\"], \"Test_FP\": t_stat[\"fp\"], \"Test_TN\": t_stat[\"tn\"], \"Test_FN\": t_stat[\"fn\"],\n            \"Test_ROC_AUC\": round(t_stat[\"auc\"], 4), \"Test_PR_AUC\": round(t_stat[\"ap\"], 4),\n            \"Test_Accuracy\": round(t_stat[\"acc\"], 4), \"Test_Balanced_Acc\": round(t_stat[\"bal_acc\"], 4),\n            \"Test_Sensitivity\": round(t_stat[\"sens\"], 4), \"Test_Specificity\": round(t_stat[\"spec\"], 4),\n            \"Test_Precision\": round(t_stat[\"prec\"], 4), \"Test_NPV\": round(t_stat[\"npv\"], 4),\n            \"Test_F1\": round(t_stat[\"f1\"], 4), \"Test_F2\": round(t_stat[\"f2\"], 4), \"Test_MCC\": round(t_stat[\"mcc\"], 4),\n            \"Test_Brier\": round(t_stat[\"brier\"], 4), \"Test_LogLoss\": round(t_stat[\"logloss\"], 4),\n            \"Test_Frozen_Opt_F1\": round(test_froz_f1, 4)\n        })\n\n    # Summary Row 1: OVERALL_MACRO (Unweighted average across pathologies)\n    rows.append({\n        \"Model\": model_name,\n        \"Pathology\": \"OVERALL_MACRO\",\n        \"OOF_Total\": len(oof_targets), \"OOF_Pos\": sum(x[\"pos\"] for x in oof_per_class),\n        \"OOF_Prev\": round(float(np.mean([x[\"prev\"] for x in oof_per_class])), 4),\n        \"OOF_TP\": sum(x[\"tp\"] for x in oof_per_class), \"OOF_FP\": sum(x[\"fp\"] for x in oof_per_class),\n        \"OOF_TN\": sum(x[\"tn\"] for x in oof_per_class), \"OOF_FN\": sum(x[\"fn\"] for x in oof_per_class),\n        \"OOF_ROC_AUC\": round(float(np.mean([x[\"auc\"] for x in oof_per_class])), 4),\n        \"OOF_PR_AUC\": round(float(np.mean([x[\"ap\"] for x in oof_per_class])), 4),\n        \"OOF_Accuracy\": round(float(np.mean([x[\"acc\"] for x in oof_per_class])), 4),\n        \"OOF_Balanced_Acc\": round(float(np.mean([x[\"bal_acc\"] for x in oof_per_class])), 4),\n        \"OOF_Sensitivity\": round(float(np.mean([x[\"sens\"] for x in oof_per_class])), 4),\n        \"OOF_Specificity\": round(float(np.mean([x[\"spec\"] for x in oof_per_class])), 4),\n        \"OOF_Precision\": round(float(np.mean([x[\"prec\"] for x in oof_per_class])), 4),\n        \"OOF_NPV\": round(float(np.mean([x[\"npv\"] for x in oof_per_class])), 4),\n        \"OOF_F1\": round(float(np.mean([x[\"f1\"] for x in oof_per_class])), 4),\n        \"OOF_F2\": round(float(np.mean([x[\"f2\"] for x in oof_per_class])), 4),\n        \"OOF_MCC\": round(float(np.mean([x[\"mcc\"] for x in oof_per_class])), 4),\n        \"OOF_Brier\": round(float(np.mean([x[\"brier\"] for x in oof_per_class])), 4),\n        \"OOF_LogLoss\": round(float(np.mean([x[\"logloss\"] for x in oof_per_class])), 4),\n        \"OOF_Opt_Threshold\": round(float(np.mean(opt_thresholds)), 2),\n        \"OOF_Calib_F1\": round(float(np.mean(oof_calib_f1s)), 4),\n        # Test Macro\n        \"Test_Total\": len(test_targets), \"Test_Pos\": sum(x[\"pos\"] for x in test_per_class),\n        \"Test_Prev\": round(float(np.mean([x[\"prev\"] for x in test_per_class])), 4),\n        \"Test_TP\": sum(x[\"tp\"] for x in test_per_class), \"Test_FP\": sum(x[\"fp\"] for x in test_per_class),\n        \"Test_TN\": sum(x[\"tn\"] for x in test_per_class), \"Test_FN\": sum(x[\"fn\"] for x in test_per_class),\n        \"Test_ROC_AUC\": round(float(np.mean([x[\"auc\"] for x in test_per_class])), 4),\n        \"Test_PR_AUC\": round(float(np.mean([x[\"ap\"] for x in test_per_class])), 4),\n        \"Test_Accuracy\": round(float(np.mean([x[\"acc\"] for x in test_per_class])), 4),\n        \"Test_Balanced_Acc\": round(float(np.mean([x[\"bal_acc\"] for x in test_per_class])), 4),\n        \"Test_Sensitivity\": round(float(np.mean([x[\"sens\"] for x in test_per_class])), 4),\n        \"Test_Specificity\": round(float(np.mean([x[\"spec\"] for x in test_per_class])), 4),\n        \"Test_Precision\": round(float(np.mean([x[\"prec\"] for x in test_per_class])), 4),\n        \"Test_NPV\": round(float(np.mean([x[\"npv\"] for x in test_per_class])), 4),\n        \"Test_F1\": round(float(np.mean([x[\"f1\"] for x in test_per_class])), 4),\n        \"Test_F2\": round(float(np.mean([x[\"f2\"] for x in test_per_class])), 4),\n        \"Test_MCC\": round(float(np.mean([x[\"mcc\"] for x in test_per_class])), 4),\n        \"Test_Brier\": round(float(np.mean([x[\"brier\"] for x in test_per_class])), 4),\n        \"Test_LogLoss\": round(float(np.mean([x[\"logloss\"] for x in test_per_class])), 4),\n        \"Test_Frozen_Opt_F1\": round(float(np.mean(test_frozen_f1s)), 4)\n    })\n\n    # Summary Row 2: OVERALL_MICRO (Global pooled across all samples and pathologies)\n    o_mic = compute_binary_stats(oof_targets.ravel(), oof_preds.ravel(), threshold=threshold)\n    t_mic = compute_binary_stats(test_targets.ravel(), test_preds.ravel(), threshold=threshold)\n    opt_t_mic, oof_cal_f1_mic = find_optimal_threshold(oof_targets.ravel(), oof_preds.ravel())\n    test_froz_f1_mic = compute_f1_at_t(test_targets.ravel(), test_preds.ravel(), opt_t_mic)\n\n    rows.append({\n        \"Model\": model_name,\n        \"Pathology\": \"OVERALL_MICRO\",\n        \"OOF_Total\": o_mic[\"total\"], \"OOF_Pos\": o_mic[\"pos\"], \"OOF_Prev\": round(o_mic[\"prev\"], 4),\n        \"OOF_TP\": o_mic[\"tp\"], \"OOF_FP\": o_mic[\"fp\"], \"OOF_TN\": o_mic[\"tn\"], \"OOF_FN\": o_mic[\"fn\"],\n        \"OOF_ROC_AUC\": round(o_mic[\"auc\"], 4), \"OOF_PR_AUC\": round(o_mic[\"ap\"], 4),\n        \"OOF_Accuracy\": round(o_mic[\"acc\"], 4), \"OOF_Balanced_Acc\": round(o_mic[\"bal_acc\"], 4),\n        \"OOF_Sensitivity\": round(o_mic[\"sens\"], 4), \"OOF_Specificity\": round(o_mic[\"spec\"], 4),\n        \"OOF_Precision\": round(o_mic[\"prec\"], 4), \"OOF_NPV\": round(o_mic[\"npv\"], 4),\n        \"OOF_F1\": round(o_mic[\"f1\"], 4), \"OOF_F2\": round(o_mic[\"f2\"], 4), \"OOF_MCC\": round(o_mic[\"mcc\"], 4),\n        \"OOF_Brier\": round(o_mic[\"brier\"], 4), \"OOF_LogLoss\": round(o_mic[\"logloss\"], 4),\n        \"OOF_Opt_Threshold\": round(opt_t_mic, 2), \"OOF_Calib_F1\": round(oof_cal_f1_mic, 4),\n        # Test Micro\n        \"Test_Total\": t_mic[\"total\"], \"Test_Pos\": t_mic[\"pos\"], \"Test_Prev\": round(t_mic[\"prev\"], 4),\n        \"Test_TP\": t_mic[\"tp\"], \"Test_FP\": t_mic[\"fp\"], \"Test_TN\": t_mic[\"tn\"], \"Test_FN\": t_mic[\"fn\"],\n        \"Test_ROC_AUC\": round(t_mic[\"auc\"], 4), \"Test_PR_AUC\": round(t_mic[\"ap\"], 4),\n        \"Test_Accuracy\": round(t_mic[\"acc\"], 4), \"Test_Balanced_Acc\": round(t_mic[\"bal_acc\"], 4),\n        \"Test_Sensitivity\": round(t_mic[\"sens\"], 4), \"Test_Specificity\": round(t_mic[\"spec\"], 4),\n        \"Test_Precision\": round(t_mic[\"prec\"], 4), \"Test_NPV\": round(t_mic[\"npv\"], 4),\n        \"Test_F1\": round(t_mic[\"f1\"], 4), \"Test_F2\": round(t_mic[\"f2\"], 4), \"Test_MCC\": round(t_mic[\"mcc\"], 4),\n        \"Test_Brier\": round(t_mic[\"brier\"], 4), \"Test_LogLoss\": round(t_mic[\"logloss\"], 4),\n        \"Test_Frozen_Opt_F1\": round(test_froz_f1_mic, 4)\n    })\n\n    # Summary Row 3: OVERALL_WEIGHTED (Pathology-weighted average by positive prevalence support)\n    oof_weights = np.array([x[\"pos\"] for x in oof_per_class], dtype=float)\n    oof_w_norm = oof_weights / np.maximum(np.sum(oof_weights), 1.0)\n    test_weights = np.array([x[\"pos\"] for x in test_per_class], dtype=float)\n    test_w_norm = test_weights / np.maximum(np.sum(test_weights), 1.0)\n\n    rows.append({\n        \"Model\": model_name,\n        \"Pathology\": \"OVERALL_WEIGHTED\",\n        \"OOF_Total\": len(oof_targets), \"OOF_Pos\": sum(x[\"pos\"] for x in oof_per_class),\n        \"OOF_Prev\": round(float(np.sum(oof_w_norm * [x[\"prev\"] for x in oof_per_class])), 4),\n        \"OOF_TP\": sum(x[\"tp\"] for x in oof_per_class), \"OOF_FP\": sum(x[\"fp\"] for x in oof_per_class),\n        \"OOF_TN\": sum(x[\"tn\"] for x in oof_per_class), \"OOF_FN\": sum(x[\"fn\"] for x in oof_per_class),\n        \"OOF_ROC_AUC\": round(float(np.sum(oof_w_norm * [x[\"auc\"] for x in oof_per_class])), 4),\n        \"OOF_PR_AUC\": round(float(np.sum(oof_w_norm * [x[\"ap\"] for x in oof_per_class])), 4),\n        \"OOF_Accuracy\": round(float(np.sum(oof_w_norm * [x[\"acc\"] for x in oof_per_class])), 4),\n        \"OOF_Balanced_Acc\": round(float(np.sum(oof_w_norm * [x[\"bal_acc\"] for x in oof_per_class])), 4),\n        \"OOF_Sensitivity\": round(float(np.sum(oof_w_norm * [x[\"sens\"] for x in oof_per_class])), 4),\n        \"OOF_Specificity\": round(float(np.sum(oof_w_norm * [x[\"spec\"] for x in oof_per_class])), 4),\n        \"OOF_Precision\": round(float(np.sum(oof_w_norm * [x[\"prec\"] for x in oof_per_class])), 4),\n        \"OOF_NPV\": round(float(np.sum(oof_w_norm * [x[\"npv\"] for x in oof_per_class])), 4),\n        \"OOF_F1\": round(float(np.sum(oof_w_norm * [x[\"f1\"] for x in oof_per_class])), 4),\n        \"OOF_F2\": round(float(np.sum(oof_w_norm * [x[\"f2\"] for x in oof_per_class])), 4),\n        \"OOF_MCC\": round(float(np.sum(oof_w_norm * [x[\"mcc\"] for x in oof_per_class])), 4),\n        \"OOF_Brier\": round(float(np.sum(oof_w_norm * [x[\"brier\"] for x in oof_per_class])), 4),\n        \"OOF_LogLoss\": round(float(np.sum(oof_w_norm * [x[\"logloss\"] for x in oof_per_class])), 4),\n        \"OOF_Opt_Threshold\": round(float(np.sum(oof_w_norm * opt_thresholds)), 2),\n        \"OOF_Calib_F1\": round(float(np.sum(oof_w_norm * oof_calib_f1s)), 4),\n        # Test Weighted\n        \"Test_Total\": len(test_targets), \"Test_Pos\": sum(x[\"pos\"] for x in test_per_class),\n        \"Test_Prev\": round(float(np.sum(test_w_norm * [x[\"prev\"] for x in test_per_class])), 4),\n        \"Test_TP\": sum(x[\"tp\"] for x in test_per_class), \"Test_FP\": sum(x[\"fp\"] for x in test_per_class),\n        \"Test_TN\": sum(x[\"tn\"] for x in test_per_class), \"Test_FN\": sum(x[\"fn\"] for x in test_per_class),\n        \"Test_ROC_AUC\": round(float(np.sum(test_w_norm * [x[\"auc\"] for x in test_per_class])), 4),\n        \"Test_PR_AUC\": round(float(np.sum(test_w_norm * [x[\"ap\"] for x in test_per_class])), 4),\n        \"Test_Accuracy\": round(float(np.sum(test_w_norm * [x[\"acc\"] for x in test_per_class])), 4),\n        \"Test_Balanced_Acc\": round(float(np.sum(test_w_norm * [x[\"bal_acc\"] for x in test_per_class])), 4),\n        \"Test_Sensitivity\": round(float(np.sum(test_w_norm * [x[\"sens\"] for x in test_per_class])), 4),\n        \"Test_Specificity\": round(float(np.sum(test_w_norm * [x[\"spec\"] for x in test_per_class])), 4),\n        \"Test_Precision\": round(float(np.sum(test_w_norm * [x[\"prec\"] for x in test_per_class])), 4),\n        \"Test_NPV\": round(float(np.sum(test_w_norm * [x[\"npv\"] for x in test_per_class])), 4),\n        \"Test_F1\": round(float(np.sum(test_w_norm * [x[\"f1\"] for x in test_per_class])), 4),\n        \"Test_F2\": round(float(np.sum(test_w_norm * [x[\"f2\"] for x in test_per_class])), 4),\n        \"Test_MCC\": round(float(np.sum(test_w_norm * [x[\"mcc\"] for x in test_per_class])), 4),\n        \"Test_Brier\": round(float(np.sum(test_w_norm * [x[\"brier\"] for x in test_per_class])), 4),\n        \"Test_LogLoss\": round(float(np.sum(test_w_norm * [x[\"logloss\"] for x in test_per_class])), 4),\n        \"Test_Frozen_Opt_F1\": round(float(np.sum(test_w_norm * test_frozen_f1s)), 4)\n    })\n\n    return pd.DataFrame(rows)\n\n\ndef build_comprehensive_metrics_table(oof_targets, oof_preds, test_targets, test_preds, threshold=0.5):\n    \"\"\"Alias for backwards compatibility.\"\"\"\n    return build_side_by_side_table(\"Model\", oof_targets, oof_preds, test_targets, test_preds, threshold=threshold)\n\n\ndef plot_multilabel_confusion_matrix(oof_targets, oof_preds, model_name, save_dir):\n    \"\"\"Figure 2: 12x12 Multilabel Confusion Matrix Heatmap at standard 0.50 threshold.\"\"\"\n    y_true = oof_targets.astype(float)\n    y_pred_bin = (oof_preds >= 0.5).astype(float)\n\n    cm_12x12 = y_pred_bin.T @ y_true  # Rows: Predicted i, Columns: True j\n    col_totals = y_true.sum(axis=0, keepdims=True)\n    cm_norm = cm_12x12 / np.maximum(col_totals, 1.0)\n\n    annot_labels = np.empty((len(TARGET_COLS), len(TARGET_COLS)), dtype=object)\n    for i in range(len(TARGET_COLS)):\n        for j in range(len(TARGET_COLS)):\n            count = int(cm_12x12[i, j])\n            pct = cm_norm[i, j]\n            annot_labels[i, j] = f\"{count}\\n({pct:.0%})\" if count > 0 else \"0\"\n\n    fig, ax = plt.subplots(figsize=(14, 11), facecolor=\"#121212\")\n    ax.set_facecolor(\"#121212\")\n\n    cbar_kws = {\"label\": \"Fraction of True Positives per Class (Sensitivity on Diagonal)\"}\n    sns.heatmap(cm_norm, annot=annot_labels, fmt=\"\", cmap=\"Blues\", cbar=True, ax=ax,\n                vmin=0.0, vmax=1.0, cbar_kws=cbar_kws,\n                annot_kws={\"size\": 8.5, \"fontweight\": \"bold\", \"color\": \"white\"})\n\n    cbar = ax.collections[0].colorbar\n    cbar.ax.yaxis.label.set_color(\"white\")\n    cbar.ax.yaxis.label.set_size(10.5)\n    cbar.ax.tick_params(colors=\"white\")\n\n    oof_macro, _ = compute_macro_auc(oof_targets, oof_preds)\n    ax.set_title(f\"{model_name} — 12x12 Multilabel Confusion Matrix (0.50 Threshold | OOF Macro AUC: {oof_macro:.4f})\\n\"\n                 f\"Diagonal = True Positives (Per-Class Sensitivity) | Off-Diagonal = Cross-Label Co-predictions\",\n                 color=\"white\", fontsize=12, fontweight=\"bold\", pad=14)\n\n    ax.set_ylabel(\"PREDICTED LABELS\", color=\"white\", fontsize=12, fontweight=\"bold\", labelpad=12)\n    ax.set_xlabel(\"TRUE LABELS\", color=\"white\", fontsize=12, fontweight=\"bold\", labelpad=12)\n    ax.set_yticklabels(TARGET_COLS, rotation=0, color=\"white\", fontsize=10, fontweight=\"bold\")\n    ax.set_xticklabels(TARGET_COLS, rotation=45, ha=\"right\", color=\"white\", fontsize=10, fontweight=\"bold\")\n    ax.tick_params(colors=\"white\")\n\n    plt.tight_layout()\n    safe_name = model_name.lower().replace('-', '_').replace(' ', '_')\n    plt.savefig(save_dir / f\"{safe_name}_confusion_matrix.png\", dpi=180, bbox_inches=\"tight\", facecolor=fig.get_facecolor())\n    plt.show()\n\n\ndef print_csv_results(model_name, metrics_df, save_dir=None):\n    \"\"\"Prints comprehensive Comma-Separated Values (CSV) block for instant spreadsheet pasting.\"\"\"\n    print(f\"\\n--- COMMA-SEPARATED RESULTS (CSV) FOR SPREADSHEET ({model_name}) ---\")\n    csv_str = metrics_df.to_csv(index=False)\n    print(csv_str)\n    print(f\"--- END CSV BLOCK ({model_name}) ---\\n\")\n\n    if save_dir:\n        safe_name = model_name.lower().replace('-', '_').replace(' ', '_')\n        csv_path = save_dir / f\"{safe_name}_comprehensive_metrics.csv\"\n        metrics_df.to_csv(csv_path, index=False)\n        print(f\"Saved {model_name} metrics to: {csv_path}\")","metadata":{"lines_to_next_cell":1},"outputs":[],"execution_count":null},{"id":"b58b3618","cell_type":"markdown","source":"## 9. Single-Model 5-Fold Training & Evaluation Orchestrator","metadata":{}},{"id":"ce05f7cc","cell_type":"code","source":"def train_and_evaluate_single_model(model_name, cv_samples, cv_fold_indices, test_samples, device=DEVICE):\n    \"\"\"\n    Orchestrates the complete 5-fold cross-validation and holdout test evaluation for one model.\n    Runs warmup + cosine scheduling, early stopping, and returns full predictions.\n    \"\"\"\n    disp_name = MODEL_DISPLAY_NAMES[model_name]\n    print(\"\\n\" + \"=\"*85)\n    print(f\"STARTING 5-FOLD CV: {disp_name.upper()} (Frozen Backbone + GAP+GMP + Unweighted BCE)\")\n    print(\"=\"*85)\n\n    oof_preds = np.zeros((len(cv_samples), len(TARGET_COLS)))\n    oof_targets = np.zeros((len(cv_samples), len(TARGET_COLS)))\n    fold_models = []\n    fold_histories = []\n\n    def lr_lambda(epoch):\n        if epoch < WARMUP_EPOCHS:\n            return float(epoch + 1) / float(max(1, WARMUP_EPOCHS))\n        progress = float(epoch - WARMUP_EPOCHS) / float(max(1, EPOCHS - WARMUP_EPOCHS))\n        return max(1e-3, 0.5 * (1.0 + np.cos(np.pi * progress)))\n\n    for fold in range(5):\n        val_idx = cv_fold_indices[fold]\n        train_idx = np.concatenate([cv_fold_indices[f] for f in range(5) if f != fold])\n\n        train_fold = [cv_samples[i] for i in train_idx]\n        val_fold = [cv_samples[i] for i in val_idx]\n\n        train_loader = DataLoader(Knee25DDataset(train_fold, transform=train_transforms), batch_size=BATCH_SIZE, shuffle=True)\n        val_loader = DataLoader(Knee25DDataset(val_fold), batch_size=BATCH_SIZE, shuffle=False)\n\n        model = build_backbone_model(model_name, num_classes=len(TARGET_COLS)).to(device)\n        criterion = nn.BCEWithLogitsLoss()\n        optimizer = torch.optim.AdamW(model.head.parameters(), lr=LR, weight_decay=WEIGHT_DECAY)\n        scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lr_lambda)\n\n        best_val_loss = float(\"inf\")\n        best_macro_auc = 0.0\n        patience_counter = 0\n        best_weights = None\n        history = {\"train_loss\": [], \"val_loss\": [], \"val_auc\": []}\n\n        for epoch in range(1, EPOCHS + 1):\n            tr_loss = train_one_epoch(model, train_loader, criterion, optimizer, device)\n            val_loss, val_macro_auc, _, _, _ = evaluate(model, val_loader, criterion, device)\n            scheduler.step()\n\n            history[\"train_loss\"].append(tr_loss)\n            history[\"val_loss\"].append(val_loss)\n            history[\"val_auc\"].append(val_macro_auc)\n\n            if val_loss < best_val_loss:\n                best_val_loss = val_loss\n                best_macro_auc = val_macro_auc\n                patience_counter = 0\n                best_weights = {k: v.cpu().clone() for k, v in model.state_dict().items()}\n            else:\n                patience_counter += 1\n\n            if patience_counter >= PATIENCE:\n                break\n\n        model.load_state_dict(best_weights)\n        _, fold_auc, _, val_targets, val_preds = evaluate(model, val_loader, criterion, device)\n        oof_preds[val_idx] = val_preds\n        oof_targets[val_idx] = val_targets\n        fold_models.append(model)\n        fold_histories.append(history)\n        print(f\"  [{disp_name} Fold {fold+1}/5] Best Val Loss: {best_val_loss:.4f} | Val Macro-AUC: {fold_auc:.4f}\")\n\n    oof_macro_auc, _ = compute_macro_auc(oof_targets, oof_preds)\n\n    # 5-Fold Ensemble Inference on Holdout Test Studies\n    test_loader = DataLoader(Knee25DDataset(test_samples), batch_size=BATCH_SIZE, shuffle=False)\n    test_targets = np.array([s[\"labels\"] for s in test_samples])\n    ensemble_test_preds = np.zeros((len(test_samples), len(TARGET_COLS)))\n\n    for m in fold_models:\n        m.eval()\n        with torch.no_grad():\n            preds = []\n            for imgs, _, _ in test_loader:\n                probs = torch.sigmoid(m(imgs.to(device))).cpu().numpy()\n                preds.append(probs)\n            ensemble_test_preds += np.vstack(preds) / len(fold_models)\n\n    test_macro_auc, _ = compute_macro_auc(test_targets, ensemble_test_preds)\n\n    # Clean up fold models to preserve GPU memory\n    del fold_models\n    if torch.cuda.is_available():\n        torch.cuda.empty_cache()\n    gc.collect()\n\n    return {\n        \"model_name\": model_name,\n        \"disp_name\": disp_name,\n        \"oof_targets\": oof_targets,\n        \"oof_preds\": oof_preds,\n        \"test_targets\": test_targets,\n        \"test_preds\": ensemble_test_preds,\n        \"oof_macro_auc\": oof_macro_auc,\n        \"test_macro_auc\": test_macro_auc,\n        \"histories\": fold_histories\n    }","metadata":{"lines_to_next_cell":1},"outputs":[],"execution_count":null},{"id":"51f80052","cell_type":"markdown","source":"## 10. Master Benchmark Execution Loop (Sequential per Model — No Intervening Cells)\n\nRuns all 7 backbones sequentially. For each model, outputs:\n1. Dual-panel learning curves (Figure 1)\n2. Per-class performance table (Table 1)\n3. 12x12 confusion matrix heatmap (Figure 2)\n4. Tab-separated results block (TSV)","metadata":{}},{"id":"9ed13c74","cell_type":"code","source":"all_model_results = {}\n\nfor m_idx, m_name in enumerate(MODELS_TO_BENCHMARK):\n    disp_name = MODEL_DISPLAY_NAMES[m_name]\n    print(\"\\n\" + \"#\"*90)\n    print(f\"BENCHMARKING MODEL [{m_idx+1}/{len(MODELS_TO_BENCHMARK)}]: {disp_name.upper()}\")\n    print(\"#\"*90)\n\n    # 1. Train 5 Folds & Evaluate on Test\n    res = train_and_evaluate_single_model(m_name, cv_samples, cv_fold_indices, test_samples, device=DEVICE)\n    all_model_results[m_name] = res\n\n    # 2. Output 1: Dual-Panel Learning Curves (Val Loss & Val Macro-AUC)\n    plot_learning_curves(res[\"histories\"], disp_name, OUT_DIR)\n\n    # 3. Output 2: Comprehensive Benchmark Performance Table (Side-by-Side OOF & Test)\n    metrics_df = build_side_by_side_table(disp_name, res[\"oof_targets\"], res[\"oof_preds\"], res[\"test_targets\"], res[\"test_preds\"])\n    print(\"\\n\" + \"=\"*95)\n    print(f\"COMPREHENSIVE BENCHMARK PERFORMANCE TABLE (SIDE-BY-SIDE): {disp_name.upper()}\")\n    print(\"=\"*95)\n    disp_cols = [\n        \"Pathology\", \"OOF_Pos\", \"OOF_ROC_AUC\", \"OOF_Accuracy\", \"OOF_Sensitivity\", \"OOF_Specificity\", \"OOF_Precision\", \"OOF_F1\", \"OOF_MCC\",\n        \"Test_Pos\", \"Test_ROC_AUC\", \"Test_Accuracy\", \"Test_Sensitivity\", \"Test_Specificity\", \"Test_Precision\", \"Test_F1\", \"Test_MCC\"\n    ]\n    print(metrics_df[disp_cols].to_string(index=False))\n    print(\"=\"*95)\n    print(f\"5-Fold OOF Macro ROC-AUC:      {res['oof_macro_auc']:.4f}\")\n    print(f\"Holdout Test Macro ROC-AUC:    {res['test_macro_auc']:.4f}\")\n    print(\"=\"*95)\n\n    # 4. Output 3: 12x12 Multilabel Confusion Matrix Heatmap\n    plot_multilabel_confusion_matrix(res[\"oof_targets\"], res[\"oof_preds\"], disp_name, OUT_DIR)\n\n    # 5. Output 4: Comprehensive Comma-Separated Results Block (CSV) for Spreadsheet Copy-Pasting\n    print_csv_results(disp_name, metrics_df, save_dir=OUT_DIR)\n\nprint(\"\\nAll 7 Candidate Backbones Evaluated Successfully!\")","metadata":{"lines_to_next_cell":1},"outputs":[],"execution_count":null},{"id":"4ab6a939","cell_type":"markdown","source":"## 11. Consolidated Multi-Model Macro ROC Comparison & Final Leaderboard","metadata":{}},{"id":"64ba3734","cell_type":"code","source":"def plot_consolidated_macro_roc(results_dict, save_dir):\n    \"\"\"\n    Figure: Dual-panel plot comparing Macro ROC curves of ALL 7 models simultaneously:\n    - Left Panel: 5-Fold Cross-Validation OOF Macro ROC Curves\n    - Right Panel: Holdout Test Macro ROC Curves (5-Fold Ensemble)\n    \"\"\"\n    fig, (ax_oof, ax_test) = plt.subplots(1, 2, figsize=(22, 9.5), facecolor=\"#121212\")\n    palette = sns.color_palette(\"bright\", n_colors=len(results_dict))\n\n    ax_oof.set_facecolor(\"#121212\")\n    ax_test.set_facecolor(\"#121212\")\n\n    # Sort models by OOF Macro-AUC descending for clean legend ordering\n    sorted_items = sorted(results_dict.items(), key=lambda kv: kv[1][\"oof_macro_auc\"], reverse=True)\n\n    for idx, (m_key, res) in enumerate(sorted_items):\n        color = palette[idx]\n        d_name = res[\"disp_name\"]\n\n        # OOF Macro Curve\n        fpr_oof, tpr_oof, auc_oof = compute_macro_roc_curve(res[\"oof_targets\"], res[\"oof_preds\"])\n        ax_oof.plot(fpr_oof, tpr_oof, color=color, lw=2.8, alpha=0.9,\n                    label=f\"{d_name:<18} (OOF Macro = {auc_oof:.4f})\")\n\n        # Test Macro Curve\n        fpr_test, tpr_test, auc_test = compute_macro_roc_curve(res[\"test_targets\"], res[\"test_preds\"])\n        ax_test.plot(fpr_test, tpr_test, color=color, lw=2.8, alpha=0.9,\n                     label=f\"{d_name:<18} (Test Macro = {auc_test:.4f})\")\n\n    # Random chance reference line\n    ax_oof.plot([0, 1], [0, 1], color=\"#666666\", linestyle=\"--\", lw=1.5, label=\"Random Chance (0.500)\")\n    ax_test.plot([0, 1], [0, 1], color=\"#666666\", linestyle=\"--\", lw=1.5, label=\"Random Chance (0.500)\")\n\n    ax_oof.set_title(\"5-Fold Cross-Validation: OOF Macro ROC Curves (All 7 Models)\",\n                     fontsize=13.5, color=\"white\", fontweight=\"bold\", pad=14)\n    ax_oof.set_xlabel(\"False Positive Rate (1 - Specificity)\", color=\"white\", fontsize=11.5)\n    ax_oof.set_ylabel(\"True Positive Rate (Sensitivity)\", color=\"white\", fontsize=11.5)\n    ax_oof.grid(True, linestyle=\"--\", alpha=0.25)\n    ax_oof.legend(loc=\"lower right\", facecolor=\"#1c1c1c\", edgecolor=\"#444444\", labelcolor=\"white\", fontsize=9.5)\n\n    ax_test.set_title(\"Holdout Test: 5-Fold Ensemble Macro ROC Curves (All 7 Models)\",\n                      fontsize=13.5, color=\"white\", fontweight=\"bold\", pad=14)\n    ax_test.set_xlabel(\"False Positive Rate (1 - Specificity)\", color=\"white\", fontsize=11.5)\n    ax_test.set_ylabel(\"True Positive Rate (Sensitivity)\", color=\"white\", fontsize=11.5)\n    ax_test.grid(True, linestyle=\"--\", alpha=0.25)\n    ax_test.legend(loc=\"lower right\", facecolor=\"#1c1c1c\", edgecolor=\"#444444\", labelcolor=\"white\", fontsize=9.5)\n\n    plt.tight_layout()\n    plt.savefig(save_dir / \"consolidated_multi_model_macro_roc.png\", dpi=200, bbox_inches=\"tight\", facecolor=fig.get_facecolor())\n    plt.show()\n\nplot_consolidated_macro_roc(all_model_results, OUT_DIR)\n\n\n# Print Final Multi-Backbone Leaderboard & TSV Block\nleaderboard_rows = []\nfor m_key, res in all_model_results.items():\n    leaderboard_rows.append({\n        \"Model\": res[\"disp_name\"],\n        \"5-Fold OOF Macro-AUC\": res[\"oof_macro_auc\"],\n        \"Holdout Test Macro-AUC\": res[\"test_macro_auc\"]\n    })\n\ndf_leaderboard = pd.DataFrame(leaderboard_rows).sort_values(by=\"5-Fold OOF Macro-AUC\", ascending=False).reset_index(drop=True)\ndf_leaderboard.index = df_leaderboard.index + 1\n\nprint(\"\\n\" + \"=\"*85)\nprint(\"FINAL MULTI-BACKBONE BENCHMARK LEADERBOARD (SORTED BY 5-FOLD OOF MACRO-AUC)\")\nprint(\"=\"*85)\nprint(df_leaderboard.to_string())\nprint(\"=\"*85)\n\nprint(\"\\n--- FINAL LEADERBOARD CSV FOR SPREADSHEET ---\")\nprint(\"Rank,Model,5Fold_OOF_Macro_AUC,Holdout_Test_Macro_AUC\")\ncsv_leaderboard_lines = [\"Rank,Model,5Fold_OOF_Macro_AUC,Holdout_Test_Macro_AUC\"]\nfor rank, row in df_leaderboard.iterrows():\n    line = f\"{rank},{row['Model']},{row['5-Fold OOF Macro-AUC']:.4f},{row['Holdout Test Macro-AUC']:.4f}\"\n    print(line)\n    csv_leaderboard_lines.append(line)\nprint(\"--- END FINAL LEADERBOARD CSV ---\\n\")\n\nwith open(OUT_DIR / \"final_leaderboard.csv\", \"w\", encoding=\"utf-8\") as f:\n    f.write(\"\\n\".join(csv_leaderboard_lines) + \"\\n\")\n\n# Consolidate and Save Master Side-by-Side Comprehensive Benchmark Table (All 7 Models)\nmaster_model_dfs = []\nfor m_key, res in all_model_results.items():\n    m_df = build_side_by_side_table(res[\"disp_name\"], res[\"oof_targets\"], res[\"oof_preds\"], res[\"test_targets\"], res[\"test_preds\"])\n    master_model_dfs.append(m_df)\n\nmaster_bench_df = pd.concat(master_model_dfs, ignore_index=True)\nmaster_csv_path = OUT_DIR / \"7bench_comprehensive_side_by_side.csv\"\nmaster_bench_df.to_csv(master_csv_path, index=False)\n\nprint(\"\\n\" + \"=\"*95)\nprint(f\"MASTER CONSOLIDATED BENCHMARK TABLE ({len(master_bench_df)} ROWS x {len(master_bench_df.columns)} COLUMNS)\")\nprint(f\"SAVED TO: {master_csv_path}\")\nprint(\"=\"*95)\nprint(\"\\n--- MASTER SPREADSHEET CSV (ALL MODELS CONSOLIDATED) ---\")\nprint(master_bench_df.to_csv(index=False))\nprint(\"--- END MASTER SPREADSHEET CSV ---\\n\")\n\nprint(\"Multi-Backbone Benchmark Suite Completed Successfully!\")","metadata":{},"outputs":[],"execution_count":null}]}