{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[],"dockerImageVersionId":28755,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# RSNA Knee MRI — milestone 1\nA self-contained **development notebook**: audit metadata, inspect MRI slices,\nextract frozen ImageNet ResNet-18 features, and fit 12 regularized study classifiers.\nThis tests the complete image pipeline before report supervision is introduced.\nIt does not fine-tune the image encoder or claim competitive performance.\n\n**Before Run All**\n1. Import this `.ipynb` into a private Kaggle notebook.\n2. Attach **RSNA Knee Abnormality Detection** competition data (accept its rules if required).\n3. Select an available GPU accelerator in notebook settings.\n4. Enable Internet for the first development run to obtain torchvision's pretrained\n   weights and any missing DICOM dependency, OR attach the official\n   `resnet18-f37072fd.pth` file and set `WEIGHTS_PATH` below. Dependencies must also\n   already be installed or provided as offline wheels if Internet is disabled.\n5. Run All. Download the output artifacts after completion.\n\nGPU availability and quotas depend on your account. This notebook prints the actual\ndevice. It never downloads the competition archive to your computer or sends reports\nto an external service. A saved Kaggle competition submission needs a separate,\nrule-compliant inference packaging step; `example_submission.csv` below is a format check.\n\n**Validation boundary:** use the existing 20/19/19 gold folds, not a new random split.\nReport hashes conservatively group repeated text; patient independence is unverified.\nAll 58 gold studies have a positive target. This is a small, selected development set,\nnot evidence of clinical performance. Do not repeatedly tune against these scores.\n","metadata":{}},{"cell_type":"code","source":"from pathlib import Path\nimport importlib.util\nimport subprocess\nimport sys\n\n# EDIT THESE SETTINGS BEFORE RUNNING.\nDATA_ROOT_OVERRIDE = Path('/kaggle/input/competitions/rsna-knee-abnormality-detection')  # e.g. Path('/kaggle/input/competitions/rsna-knee-abnormality-detection')\nOUTPUT = Path('/kaggle/working/milestone1')\nWEIGHTS_PATH = '/kaggle/input/datasets/sealzhao/resnet18-f37072fd-pth/resnet18-f37072fd.pth'  # Path to the official resnet18-f37072fd.pth for an offline run\nALLOW_DEPENDENCY_INSTALL = True\nALLOW_CPU = False  # True for debugging only; GPU is recommended\nSLICES_PER_SERIES = 16\nIMAGE_SIZE = 224\nENCODER_BATCH_SIZE = 16\nLOGISTIC_C = 0.1  # fixed before seeing scores; no hyperparameter search\nSEED = 20260906\n\n# Do not replace Kaggle's CUDA torch/torchvision installations.\nrequired = {'numpy': 'numpy', 'pandas': 'pandas', 'torch': 'torch',\n            'torchvision': 'torchvision', 'sklearn': 'scikit-learn',\n            'matplotlib': 'matplotlib', 'pydicom': 'pydicom>=3,<4'}\nmissing = [package for module, package in required.items() if importlib.util.find_spec(module) is None]\nif any(importlib.util.find_spec(m) is None for m in ('torch', 'torchvision')):\n    raise RuntimeError('Select a Kaggle Python GPU image with torch and torchvision installed.')\nif missing:\n    if not ALLOW_DEPENDENCY_INSTALL:\n        raise RuntimeError(f'Missing dependencies: {missing}. Supply offline wheels or enable Internet/install.')\n    subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--quiet', *missing])\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:28:17.659363Z","iopub.execute_input":"2026-09-06T08:28:17.659734Z","iopub.status.idle":"2026-09-06T08:28:17.688161Z","shell.execute_reply.started":"2026-09-06T08:28:17.659695Z","shell.execute_reply":"2026-09-06T08:28:17.687282Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import collections\nimport hashlib\nimport io\nimport json\nimport platform\nimport random\nimport time\nimport unicodedata\nimport warnings\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport pydicom\nfrom pydicom.pixels import apply_modality_lut\nimport torch\nimport torch.nn.functional as F\nimport torchvision\nfrom torchvision.models import resnet18, ResNet18_Weights\nimport sklearn\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.exceptions import ConvergenceWarning\n\nOUTPUT.mkdir(parents=True, exist_ok=True)\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntorch.manual_seed(SEED)\ntorch.backends.cudnn.benchmark = False\ntorch.backends.cudnn.deterministic = True\nDEVICE = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\nenvironment = {'python': platform.python_version(), 'torch': torch.__version__,\n               'torchvision': torchvision.__version__, 'pydicom': pydicom.__version__,\n               'numpy': np.__version__, 'pandas': pd.__version__, 'sklearn': sklearn.__version__,\n               'cuda_available': torch.cuda.is_available(), 'device': str(DEVICE)}\nif DEVICE.type == 'cuda':\n    environment.update(gpu=torch.cuda.get_device_name(0),\n                       vram_gib=round(torch.cuda.get_device_properties(0).total_memory / 2**30, 2))\n    torch.cuda.reset_peak_memory_stats()\nprint(json.dumps(environment, indent=2))\n(OUTPUT / 'environment.json').write_text(json.dumps(environment, indent=2))\nif DEVICE.type == 'cpu' and not ALLOW_CPU:\n    raise RuntimeError('No GPU detected. Enable a GPU accelerator, or explicitly set ALLOW_CPU=True for debugging.')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:30:38.024763Z","iopub.execute_input":"2026-09-06T08:30:38.025292Z","iopub.status.idle":"2026-09-06T08:30:38.037035Z","shell.execute_reply.started":"2026-09-06T08:30:38.02526Z","shell.execute_reply":"2026-09-06T08:30:38.036292Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Data discovery and metadata checks\nOnly metadata is loaded at this stage. Raw reports are not displayed or copied into\noutputs. Blank labels remain missing. The three exposed test studies are examples,\nnot the full hidden test set. UTF-8 is checked strictly rather than silently replacing\ncharacters or assuming Latin-1 from a third-party notebook.\n","metadata":{}},{"cell_type":"markdown","source":"## Load the frozen folds\nThe notebook embeds only the 58 study IDs and their existing fold assignments.\nIt verifies the training CSV fingerprint. A changed dataset stops execution rather\nthan silently creating a new experiment. Repeated report groups remain together;\n`exclude_from_training_fold` also protects held-out reports during future weak-label work.\nThe current baseline uses only gold-labeled training studies.\n","metadata":{}},{"cell_type":"code","source":"LABELS = ['ACL', 'MCL', 'Medial Meniscus', 'Lateral Meniscus', 'Medial OA',\n          'Lateral OA', 'PF OA', 'Effusion', 'Synovitis', \"Baker's\", 'Contusion', 'Fracture']\nPLANES = ['Sagittal', 'Coronal', 'Axial']\n\ndef find_data_root(override=None):\n    candidates = ([Path(override)] if override is not None else [\n        Path('/kaggle/input/competitions/rsna-knee-abnormality-detection'),\n        Path('/kaggle/input/rsna-knee-abnormality-detection')])\n    if override is None:\n        candidates += [p.parent for p in Path('/kaggle/input').glob('*/train.csv')]\n    valid = list(dict.fromkeys(p for p in candidates\n                              if all((p / f).is_file() for f in ['train.csv', 'train_series.csv', 'test.csv', 'test_series.csv', 'sample_submission.csv'])))\n    if len(valid) != 1:\n        raise RuntimeError(f'Expected one competition data root, found {valid}. Attach the competition or set DATA_ROOT_OVERRIDE.')\n    return valid[0]\n\ndef text_group(text):\n    normalized = ' '.join(unicodedata.normalize('NFKC', str(text)).casefold().split())\n    return hashlib.sha256(normalized.encode()).hexdigest() if normalized else ''\n\nDATA_ROOT = find_data_root(DATA_ROOT_OVERRIDE)\ntrain = pd.read_csv(DATA_ROOT / 'train.csv', encoding='utf-8-sig', keep_default_na=False)\ntest = pd.read_csv(DATA_ROOT / 'test.csv', encoding='utf-8-sig', keep_default_na=False)\nseries = pd.read_csv(DATA_ROOT / 'train_series.csv', encoding='utf-8-sig')\ntest_series = pd.read_csv(DATA_ROOT / 'test_series.csv', encoding='utf-8-sig')\nsample_submission = pd.read_csv(DATA_ROOT / 'sample_submission.csv')\nassert set(['StudyInstanceUID', 'Report'] + LABELS) <= set(train.columns)\nassert train.StudyInstanceUID.is_unique and test.StudyInstanceUID.is_unique\nassert series.SeriesInstanceUID.is_unique and test_series.SeriesInstanceUID.is_unique\nassert not set(train.StudyInstanceUID) & set(test.StudyInstanceUID)\nassert not set(series.SeriesInstanceUID) & set(test_series.SeriesInstanceUID)\nassert set(series.StudyInstanceUID) == set(train.StudyInstanceUID)\nassert set(test_series.StudyInstanceUID) == set(test.StudyInstanceUID)\nassert list(sample_submission.columns) == ['StudyInstanceUID'] + LABELS\nassert set(sample_submission.StudyInstanceUID) == set(test.StudyInstanceUID)\nfor label in LABELS:\n    train[label] = pd.to_numeric(train[label].replace('', np.nan), errors='raise')\n    assert train[label].dropna().isin([0, 1]).all(), f'Invalid target: {label}'\ntrain['report_group'] = train.Report.map(text_group)\ntrain.loc[train.report_group == '', 'report_group'] = train.loc[train.report_group == '', 'StudyInstanceUID']\ngold = train.loc[train[LABELS].notna().all(axis=1)].copy()\ncounts = pd.DataFrame({'positive': (train[LABELS] == 1).sum(),\n                       'negative': (train[LABELS] == 0).sum(),\n                       'missing': train[LABELS].isna().sum()})\ncounts.index.name = 'target'\ncounts.to_csv(OUTPUT / 'label_counts.csv')\ndisplay(counts)\nstudy_series_counts = series.groupby('StudyInstanceUID').size()\naudit = {'train_studies': len(train), 'gold_studies': len(gold), 'series': len(series),\n         'partial_label_studies': int((train[LABELS].notna().any(axis=1) & ~train[LABELS].notna().all(axis=1)).sum()),\n         'example_test_studies': len(test), 'test_has_report': 'Report' in test.columns,\n         'empty_reports': int(train.Report.str.strip().eq('').sum()),\n         'duplicate_report_groups': int(train.report_group.value_counts().gt(1).sum()),\n         'series_per_study': study_series_counts.describe().to_dict(),\n         'gold_studies_with_no_positive': int(gold[LABELS].sum(axis=1).eq(0).sum())}\n(OUTPUT / 'metadata_audit.json').write_text(json.dumps(audit, indent=2))\nprint(json.dumps(audit, indent=2))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:30:56.909472Z","iopub.execute_input":"2026-09-06T08:30:56.910383Z","iopub.status.idle":"2026-09-06T08:30:57.43508Z","shell.execute_reply.started":"2026-09-06T08:30:56.910353Z","shell.execute_reply":"2026-09-06T08:30:57.434182Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"EXPECTED_TRAIN_SHA256 = '8ca2203c0e9d61c080c7a314c7cdb51c1b03a1d9eb4770819f7f34af53ef4e33'\nEXPECTED_FOLDS_SHA256 = 'a7006284cea43071cf367134f31c206d7e8b29845f3c742ac2c3577a74903097'\nFROZEN_GOLD_FOLDS = {\n  \"1.2.826.0.1.3680043.8.498.10095687747295410396510538520594649149\": 2,\n  \"1.2.826.0.1.3680043.8.498.10170898615867673028696505248839028269\": 0,\n  \"1.2.826.0.1.3680043.8.498.10306159113324811538703788080836752052\": 2,\n  \"1.2.826.0.1.3680043.8.498.11287937729196958426538087439102017580\": 0,\n  \"1.2.826.0.1.3680043.8.498.11382021393803389951964005983002209238\": 0,\n  \"1.2.826.0.1.3680043.8.498.11548045715264151632153040089882701935\": 1,\n  \"1.2.826.0.1.3680043.8.498.11557620559191469069130827959098335840\": 2,\n  \"1.2.826.0.1.3680043.8.498.11771393824519892797114773408583976756\": 1,\n  \"1.2.826.0.1.3680043.8.498.11851412923016044948101698015974810604\": 2,\n  \"1.2.826.0.1.3680043.8.498.11915937982684988073644209606907169581\": 2,\n  \"1.2.826.0.1.3680043.8.498.12448079646359892252441208258836556945\": 1,\n  \"1.2.826.0.1.3680043.8.498.12505035424093604269515328931488770819\": 1,\n  \"1.2.826.0.1.3680043.8.498.12606657226568558340797193167488111973\": 1,\n  \"1.2.826.0.1.3680043.8.498.12801308844398614687904447633432197492\": 0,\n  \"1.2.826.0.1.3680043.8.498.12978510157202852202776899910529174803\": 1,\n  \"1.2.826.0.1.3680043.8.498.13267780356245120052411517053322874891\": 0,\n  \"1.2.826.0.1.3680043.8.498.13335129881737731410081002627729200903\": 2,\n  \"1.2.826.0.1.3680043.8.498.15593638897292057356864060466120253309\": 0,\n  \"1.2.826.0.1.3680043.8.498.16060119389060497136231217718921482192\": 1,\n  \"1.2.826.0.1.3680043.8.498.17844546765907321649997094867791102711\": 0,\n  \"1.2.826.0.1.3680043.8.498.18392509497170616983977319528036573378\": 2,\n  \"1.2.826.0.1.3680043.8.498.22109739962224309418874538994436903404\": 1,\n  \"1.2.826.0.1.3680043.8.498.25695966186129395049687747156570466645\": 1,\n  \"1.2.826.0.1.3680043.8.498.26790702379506190936834447203448882465\": 2,\n  \"1.2.826.0.1.3680043.8.498.27437263843446879932281897446100134924\": 1,\n  \"1.2.826.0.1.3680043.8.498.28925345859498351203477642741908452608\": 1,\n  \"1.2.826.0.1.3680043.8.498.29764868091238287072166823522853419550\": 0,\n  \"1.2.826.0.1.3680043.8.498.30246079718471552972130572444383079911\": 1,\n  \"1.2.826.0.1.3680043.8.498.32321830776739689645700555055955725945\": 0,\n  \"1.2.826.0.1.3680043.8.498.37040910196459054543310335064167212742\": 0,\n  \"1.2.826.0.1.3680043.8.498.37392127307867238524251864751713143683\": 0,\n  \"1.2.826.0.1.3680043.8.498.39500553372517290823815606829518305500\": 2,\n  \"1.2.826.0.1.3680043.8.498.41600498783384864111179175518614837125\": 1,\n  \"1.2.826.0.1.3680043.8.498.47921753480592595198052407850568677187\": 1,\n  \"1.2.826.0.1.3680043.8.498.48348615349418615469875801439348424274\": 0,\n  \"1.2.826.0.1.3680043.8.498.48946580946665031852355005294734101132\": 2,\n  \"1.2.826.0.1.3680043.8.498.54977581323931277817074741182670080450\": 2,\n  \"1.2.826.0.1.3680043.8.498.56512564464301544960446419143135447363\": 1,\n  \"1.2.826.0.1.3680043.8.498.59483999067816153759785789654766826710\": 2,\n  \"1.2.826.0.1.3680043.8.498.62465595376489211274225312453216559395\": 2,\n  \"1.2.826.0.1.3680043.8.498.62549354677638403845904556149868367236\": 1,\n  \"1.2.826.0.1.3680043.8.498.64408609256163127278435484217683272910\": 2,\n  \"1.2.826.0.1.3680043.8.498.64703506772167798469048460791472465039\": 2,\n  \"1.2.826.0.1.3680043.8.498.65708905118633339771181857781063784327\": 0,\n  \"1.2.826.0.1.3680043.8.498.67188121544063723837669983597453941774\": 0,\n  \"1.2.826.0.1.3680043.8.498.69392348385274125385290015404144639002\": 2,\n  \"1.2.826.0.1.3680043.8.498.72853333220043794904856138561095171921\": 2,\n  \"1.2.826.0.1.3680043.8.498.73527530686853911124431549317032662220\": 2,\n  \"1.2.826.0.1.3680043.8.498.73926443729786165628848843707532839995\": 0,\n  \"1.2.826.0.1.3680043.8.498.75187434248356774277526985329346125190\": 0,\n  \"1.2.826.0.1.3680043.8.498.77362718298550276679350855963451855003\": 0,\n  \"1.2.826.0.1.3680043.8.498.78512215519177850923279235346968676828\": 0,\n  \"1.2.826.0.1.3680043.8.498.82166943552764439138333504456139890254\": 0,\n  \"1.2.826.0.1.3680043.8.498.86968600239724310678905311244945464037\": 1,\n  \"1.2.826.0.1.3680043.8.498.88077418639301174409926781329613570435\": 1,\n  \"1.2.826.0.1.3680043.8.498.90283565381042081768587894596970552767\": 1,\n  \"1.2.826.0.1.3680043.8.498.94433753471890306108089557761231739312\": 0,\n  \"1.2.826.0.1.3680043.8.498.97274720257634584071500649275217521662\": 2\n}\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:31:16.188221Z","iopub.execute_input":"2026-09-06T08:31:16.188708Z","iopub.status.idle":"2026-09-06T08:31:16.196279Z","shell.execute_reply.started":"2026-09-06T08:31:16.188677Z","shell.execute_reply":"2026-09-06T08:31:16.195349Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"actual_hash = hashlib.sha256((DATA_ROOT / 'train.csv').read_bytes()).hexdigest()\nif actual_hash != EXPECTED_TRAIN_SHA256:\n    raise RuntimeError('Training CSV differs from metadata v1. Re-audit and version the folds before running.')\ngold['fold'] = gold.StudyInstanceUID.map(FROZEN_GOLD_FOLDS)\nassert gold.fold.notna().all() and set(gold.StudyInstanceUID) == set(FROZEN_GOLD_FOLDS)\ngold['fold'] = gold.fold.astype(int)\nassert gold.groupby('report_group').fold.nunique().max() == 1\ngold = gold.sort_values('StudyInstanceUID').reset_index(drop=True)\ngroup_to_fold = gold.set_index('report_group').fold.to_dict()\nall_folds = train[['StudyInstanceUID', 'report_group']].copy()\nall_folds['exclude_from_training_fold'] = all_folds.report_group.map(group_to_fold).fillna(-1).astype(int)\nall_folds['gold_fold'] = all_folds.StudyInstanceUID.map(FROZEN_GOLD_FOLDS).fillna(-1).astype(int)\nall_folds.to_csv(OUTPUT / 'folds.csv', index=False)\nfold_counts = gold.groupby('fold')[LABELS].sum().astype(int)\nfold_sizes = gold.groupby('fold').size()\nassert fold_sizes.tolist() == [20, 19, 19]\nassert (fold_counts >= 2).all().all()\nassert (fold_sizes.to_numpy()[:, None] - fold_counts.to_numpy() >= 2).all()\nprint('Frozen fold sizes:', fold_sizes.to_dict())\ndisplay(fold_counts)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:31:30.238716Z","iopub.execute_input":"2026-09-06T08:31:30.239172Z","iopub.status.idle":"2026-09-06T08:31:30.315303Z","shell.execute_reply.started":"2026-09-06T08:31:30.239141Z","shell.execute_reply":"2026-09-06T08:31:30.314595Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## MRI preprocessing and selection\nBaseline contract: one series per plane, preferring the provided fluid-sensitive flag;\nties use stable SeriesInstanceUID order, **not labels**. All slices in the chosen series\nhave their headers checked. Up to 16 geometrically spaced slices are decoded.\nPixel intensities use the modality LUT/rescale, then pooled 1st/99th percentiles across\nthe selected slices of that series, excluding DICOM padding. Full-field letterboxing\npreserves physical aspect ratio; there is no center crop or laterality flip.\n\nThis first loader deliberately rejects missing geometry, multi-frame files, mixed\norientations, or duplicate slice positions. It does not silently skip a failed study.\nCompressed DICOM decoding may require pylibjpeg plugins: install\n`pylibjpeg pylibjpeg-libjpeg pylibjpeg-openjpeg` if the decoder error requests them.\nNo raw patient identifiers are retained; geometry is checked against the supplied plane.\n","metadata":{}},{"cell_type":"code","source":"def choose_series(table, study):\n    chosen = []\n    for plane in PLANES:\n        candidates = table.loc[(table.StudyInstanceUID == study) & (table.Anatomical_Plane == plane)].copy()\n        if candidates.empty:\n            raise ValueError(f'Missing {plane} series for study {study}; v1 requires all three planes.')\n        candidates['priority'] = pd.to_numeric(candidates.Fluid_Sensitive, errors='raise')\n        chosen.append(candidates.sort_values(['priority', 'SeriesInstanceUID'], ascending=[False, True]).iloc[0])\n    return chosen\n\ndef series_headers(folder, study, series_uid):\n    files = sorted(folder.glob('*.dcm'))\n    if not files:\n        raise FileNotFoundError(f'No DICOM slices at {folder}')\n    items = []\n    reference = None\n    for path in files:\n        ds = pydicom.dcmread(path, stop_before_pixels=True)\n        if str(ds.StudyInstanceUID) != str(study) or str(ds.SeriesInstanceUID) != str(series_uid):\n            raise ValueError('DICOM UID does not match its manifest entry')\n        if int(getattr(ds, 'NumberOfFrames', 1)) != 1:\n            raise ValueError('Multi-frame DICOM needs a dedicated loader; v1 stops here')\n        orientation = np.asarray(ds.ImageOrientationPatient, dtype=float)\n        position = np.asarray(ds.ImagePositionPatient, dtype=float)\n        spacing = np.asarray(ds.PixelSpacing, dtype=float)\n        if orientation.shape != (6,) or position.shape != (3,) or spacing.shape != (2,):\n            raise ValueError('Invalid DICOM geometry dimensions')\n        if not np.isfinite(np.r_[orientation, position, spacing]).all() or (spacing <= 0).any():\n            raise ValueError('Nonfinite/invalid DICOM geometry')\n        normal = np.cross(orientation[:3], orientation[3:])\n        if not np.isclose(np.linalg.norm(normal), 1, atol=1e-3):\n            raise ValueError('Non-unit DICOM slice normal')\n        if reference is None:\n            reference = (orientation, spacing, (int(ds.Rows), int(ds.Columns)))\n        if not np.allclose(orientation, reference[0], atol=1e-3) or not np.allclose(spacing, reference[1], atol=1e-4) or (int(ds.Rows), int(ds.Columns)) != reference[2]:\n            raise ValueError('Inconsistent geometry within selected series')\n        items.append((float(position @ normal), path, ds))\n    items.sort(key=lambda item: item[0])\n    positions = np.array([item[0] for item in items])\n    if len(items) < 2 or np.any(np.diff(positions) <= 1e-3):\n        raise ValueError('Too few slices or duplicate slice positions; investigate series before modeling')\n    return items, reference\n\ndef letterbox(array, spacing, size):\n    height, width = array.shape\n    physical_height, physical_width = height * spacing[0], width * spacing[1]\n    scale = size / max(physical_height, physical_width)\n    new_h, new_w = max(1, round(physical_height * scale)), max(1, round(physical_width * scale))\n    tensor = torch.from_numpy(np.ascontiguousarray(array, dtype=np.float32))[None, None]\n    tensor = F.interpolate(tensor, size=(new_h, new_w), mode='bilinear', align_corners=False, antialias=True)\n    dh, dw = size - new_h, size - new_w\n    return F.pad(tensor, (dw//2, dw-dw//2, dh//2, dh-dh//2))[0, 0]\n\ndef load_series(folder, study, series_uid, supplied_plane):\n    items, reference = series_headers(folder, study, series_uid)\n    normal = np.cross(reference[0][:3], reference[0][3:])\n    inferred_plane = PLANES[int(np.argmax(np.abs(normal)))]  # patient x/y/z: sagittal/coronal/axial\n    if inferred_plane != supplied_plane:\n        raise ValueError(f'Plane metadata/geometry disagreement: {supplied_plane} vs {inferred_plane}')\n    selected = np.unique(np.rint(np.linspace(0, len(items)-1, min(SLICES_PER_SERIES, len(items)))).astype(int))\n    arrays, masks, photos, digests = [], [], [], []\n    for index in selected:\n        _, path, _ = items[index]\n        ds = pydicom.dcmread(path)\n        try:\n            raw = ds.pixel_array\n        except Exception as exc:\n            raise RuntimeError(f'Cannot decode transfer syntax {ds.file_meta.TransferSyntaxUID}. Check/install the required pydicom decoder plugins.') from exc\n        if raw.ndim != 2:\n            raise ValueError('Expected one grayscale image per DICOM')\n        photo = str(ds.PhotometricInterpretation)\n        if photo not in ('MONOCHROME1', 'MONOCHROME2'):\n            raise ValueError(f'Unsupported photometric interpretation: {photo}')\n        image = np.asarray(apply_modality_lut(raw, ds), dtype=np.float32)\n        valid = np.isfinite(image)\n        if hasattr(ds, 'PixelPaddingValue'):\n            lo, hi = sorted([float(ds.PixelPaddingValue), float(getattr(ds, 'PixelPaddingRangeLimit', ds.PixelPaddingValue))])\n            valid &= ~((raw >= lo) & (raw <= hi))\n        if not valid.any():\n            raise ValueError('Slice contains no valid pixels')\n        arrays.append(image)\n        masks.append(valid)\n        photos.append(photo)\n        digests.append(hashlib.sha256(str(raw.shape).encode() + raw.tobytes()).hexdigest())\n    if len(set(photos)) != 1:\n        raise ValueError('Inconsistent photometric interpretation')\n    lo, hi = np.percentile(np.concatenate([a[m] for a, m in zip(arrays, masks)]), [1, 99])\n    if not hi > lo:\n        raise ValueError('Series has no usable intensity range')\n    normalized = []\n    for array, valid, photo in zip(arrays, masks, photos):\n        image = np.clip((array-lo)/(hi-lo), 0, 1)\n        if photo == 'MONOCHROME1':\n            image = 1-image\n        image[~valid] = 0\n        normalized.append(letterbox(image, reference[1], IMAGE_SIZE))\n    gaps = np.diff([x[0] for x in items])\n    audit_row = {'StudyInstanceUID': study, 'SeriesInstanceUID': series_uid, 'plane': supplied_plane,\n                 'slice_count': len(items), 'sampled_slices': len(selected), 'height': reference[2][0],\n                 'width': reference[2][1], 'row_spacing': float(reference[1][0]), 'column_spacing': float(reference[1][1]),\n                 'median_slice_gap': float(np.median(gaps)), 'max_slice_gap': float(gaps.max()),\n                 'normal_dominance': float(np.max(np.abs(normal))),\n                 'transfer_syntax': str(items[0][2].file_meta.TransferSyntaxUID),\n                 'manufacturer': str(getattr(items[0][2], 'Manufacturer', 'unknown')),\n                 'field_strength': str(getattr(items[0][2], 'MagneticFieldStrength', 'unknown')),\n                 'sampled_pixel_hashes': digests}\n    return torch.stack(normalized), audit_row\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:32:33.564717Z","iopub.execute_input":"2026-09-06T08:32:33.565618Z","iopub.status.idle":"2026-09-06T08:32:33.585792Z","shell.execute_reply.started":"2026-09-06T08:32:33.565585Z","shell.execute_reply":"2026-09-06T08:32:33.584864Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Inspect one unlabeled training examination\nUse an unlabeled study for the first display so validation labels do not drive\npreprocessing choices. Review coverage, contrast, and physical aspect ratio.\nAnatomical orientation stays as stored in the DICOM; these are technical QC views.\nThe notebook continues automatically after displaying the montage. Stop if it reveals\na preprocessing problem, then fix that problem before interpreting model scores.\n","metadata":{}},{"cell_type":"code","source":"_inspection_prerequisites = ('train', 'gold', 'series', 'DATA_ROOT', 'OUTPUT',\n                             'choose_series', 'load_series', 'np', 'pd', 'plt')\n_inspection_missing = [name for name in _inspection_prerequisites if name not in globals()]\nif _inspection_missing:\n    raise RuntimeError(\n        'MRI inspection prerequisites are missing: ' + ', '.join(_inspection_missing)\n        + '. Run all preceding code cells in order: setup, imports/GPU check, '\n          'data discovery, frozen folds, and MRI preprocessing. '\n          'If an earlier cell fails, resolve its FIRST error before running this cell. '\n          'A restarted session does not retain variables from earlier runs.'\n    )\nunlabelled_ids = sorted(set(train.StudyInstanceUID) - set(gold.StudyInstanceUID))\nif not unlabelled_ids:\n    raise RuntimeError('No unlabeled studies available for inspection. Check the data audit and gold-set definition.')\ninspection_study = unlabelled_ids[0]\nfig, axes = plt.subplots(3, 5, figsize=(12, 8))\ninspection_rows = []\nfor plane_index, row in enumerate(choose_series(series, inspection_study)):\n    folder = DATA_ROOT / 'train_series' / inspection_study / row.SeriesInstanceUID\n    images, info = load_series(folder, inspection_study, row.SeriesInstanceUID, row.Anatomical_Plane)\n    inspection_rows.append(info)\n    for column, idx in enumerate(np.rint(np.linspace(0, len(images)-1, 5)).astype(int)):\n        axes[plane_index, column].imshow(images[idx], cmap='gray', vmin=0, vmax=1)\n        axes[plane_index, column].set_title(f'{row.Anatomical_Plane} | sample {idx+1}')\n        axes[plane_index, column].axis('off')\nfig.suptitle('Unlabeled study: geometry-sorted, normalized MRI slices')\nfig.tight_layout()\nfig.savefig(OUTPUT / 'mri_inspection.png', dpi=130)\nplt.show()\ndisplay(pd.DataFrame(inspection_rows).drop(columns=['sampled_pixel_hashes', 'StudyInstanceUID', 'SeriesInstanceUID']))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:32:56.228347Z","iopub.execute_input":"2026-09-06T08:32:56.229155Z","iopub.status.idle":"2026-09-06T08:32:59.638557Z","shell.execute_reply.started":"2026-09-06T08:32:56.22912Z","shell.execute_reply":"2026-09-06T08:32:59.637569Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Frozen image encoder\nEach slice becomes a 512-dimensional vector. Max pooling across the selected slices\nproduces one vector per plane. Concatenation yields 1,536 study features.\nResNet-18 stays in evaluation mode and is never fit on this dataset, so features may\nbe computed once for all folds. Feature scaling and logistic heads are fit separately\ninside each training fold. No report text or report-derived target enters this baseline.\nPretrained weight loading fails explicitly rather than falling back to random weights.\n","metadata":{}},{"cell_type":"code","source":"if WEIGHTS_PATH is None:\n    try:\n        encoder = resnet18(weights=ResNet18_Weights.IMAGENET1K_V1)\n    except Exception as exc:\n        raise RuntimeError('Could not load ImageNet weights. Enable Internet for development, or attach official resnet18-f37072fd.pth and set WEIGHTS_PATH.') from exc\n    weight_file = Path(torch.hub.get_dir()) / 'checkpoints' / 'resnet18-f37072fd.pth'\nelse:\n    weight_file = Path(WEIGHTS_PATH)\n    if not hashlib.sha256(weight_file.read_bytes()).hexdigest().startswith('f37072fd'):\n        raise ValueError('WEIGHTS_PATH does not match the official torchvision ResNet-18 weight fingerprint')\n    encoder = resnet18(weights=None)\n    encoder.load_state_dict(torch.load(weight_file, map_location='cpu', weights_only=True))\nweights_sha256 = hashlib.sha256(weight_file.read_bytes()).hexdigest()\nencoder.fc = torch.nn.Identity()\nencoder = encoder.to(DEVICE).eval()\nfor parameter in encoder.parameters():\n    parameter.requires_grad_(False)\nmean = torch.tensor([0.485, 0.456, 0.406], device=DEVICE)[None, :, None, None]\nstd = torch.tensor([0.229, 0.224, 0.225], device=DEVICE)[None, :, None, None]\n\ndef encode_images(images):\n    embeddings = []\n    with torch.inference_mode():\n        for batch in images.split(ENCODER_BATCH_SIZE):\n            batch = batch[:, None].repeat(1, 3, 1, 1).to(DEVICE)\n            features = encoder((batch-mean)/std)\n            embeddings.append(features.cpu().numpy())\n    return np.concatenate(embeddings).max(axis=0)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:33:34.270874Z","iopub.execute_input":"2026-09-06T08:33:34.27135Z","iopub.status.idle":"2026-09-06T08:33:34.807585Z","shell.execute_reply.started":"2026-09-06T08:33:34.271294Z","shell.execute_reply":"2026-09-06T08:33:34.806615Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"selected_image_audit = []\nfeature_start = time.perf_counter()\n\ndef extract_studies(ids, table, directory):\n    result = []\n    for number, study in enumerate(ids, 1):\n        plane_vectors = []\n        for row in choose_series(table, study):\n            images, info = load_series(DATA_ROOT / directory / study / row.SeriesInstanceUID,\n                                       study, row.SeriesInstanceUID, row.Anatomical_Plane)\n            info['dataset'] = directory\n            info['fluid_sensitive'] = int(row.Fluid_Sensitive)\n            selected_image_audit.append(info)\n            plane_vectors.append(encode_images(images))\n        result.append(np.concatenate(plane_vectors))\n        if number % 5 == 0 or number == len(ids):\n            print(f'{directory}: encoded {number}/{len(ids)} studies', flush=True)\n    result = np.asarray(result, dtype=np.float32)\n    assert result.shape == (len(ids), 1536) and np.isfinite(result).all()\n    return result\n\nX = extract_studies(gold.StudyInstanceUID.tolist(), series, 'train_series')\nX_test = extract_studies(test.StudyInstanceUID.tolist(), test_series, 'test_series')\nfeature_seconds = time.perf_counter() - feature_start\nnp.savez_compressed(OUTPUT / 'study_features.npz', X=X, X_test=X_test,\n                    study_ids=gold.StudyInstanceUID.to_numpy(dtype=str), test_ids=test.StudyInstanceUID.to_numpy(dtype=str))\n(OUTPUT / 'image_audit.json').write_text(json.dumps(selected_image_audit, indent=2))\ndisplay(pd.DataFrame(selected_image_audit).drop(columns='sampled_pixel_hashes').groupby(['dataset', 'plane']).slice_count.agg(['min', 'median', 'max']))\nprint(f'Feature extraction: {feature_seconds:.1f} seconds')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:33:50.033653Z","iopub.execute_input":"2026-09-06T08:33:50.034252Z","iopub.status.idle":"2026-09-06T08:35:37.23843Z","shell.execute_reply.started":"2026-09-06T08:33:50.034223Z","shell.execute_reply":"2026-09-06T08:35:37.2377Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Exact sampled-pixel overlap screen\nStop on cross-fold or gold/test overlap. Identical pixels can also be blank images,\nso a match requires investigation, not an automatic claim of leakage. Absence of\nmatches does not establish patient independence or rule out near-duplicates:\nonly sampled slices of chosen series are covered by this screen.\n","metadata":{}},{"cell_type":"code","source":"pixel_owners = collections.defaultdict(set)\nfor row in selected_image_audit:\n    for digest in row['sampled_pixel_hashes']:\n        pixel_owners[digest].add(row['StudyInstanceUID'])\noverlap = []\nfor digest, owners in pixel_owners.items():\n    if len(owners) < 2:\n        continue\n    partitions = {FROZEN_GOLD_FOLDS.get(uid, 'example_test') for uid in owners}\n    if len(partitions) > 1:\n        overlap.append({'pixel_hash': digest, 'studies': sorted(owners)})\n(OUTPUT / 'sampled_pixel_overlap.json').write_text(json.dumps(overlap, indent=2))\nif overlap:\n    raise RuntimeError('Exact sampled-pixel matches cross partitions. Inspect sampled_pixel_overlap.json before scoring.')\nprint('No exact sampled-pixel overlaps across folds or gold/example-test partitions. Patient independence remains unverified.')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:36:05.72545Z","iopub.execute_input":"2026-09-06T08:36:05.72597Z","iopub.status.idle":"2026-09-06T08:36:05.734728Z","shell.execute_reply.started":"2026-09-06T08:36:05.725939Z","shell.execute_reply":"2026-09-06T08:36:05.73387Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Fit the baseline and save out-of-fold predictions\nThree fixed folds; 12 logistic classifiers per fold. StandardScaler sees only that\nfold's training features. L2 regularization uses fixed C=0.1; no class balancing,\nthreshold selection, early stopping, or validation-based model selection is used.\nConvergence failures stop the run. Every study receives predictions from a model\nthat did not train on it. These are development results on the small selected gold set.\n","metadata":{}},{"cell_type":"code","source":"def fit_heads_predict(x_train, y_train, *evaluation_arrays):\n    scaler = StandardScaler().fit(x_train)\n    transformed_train = scaler.transform(x_train)\n    transformed_eval = [scaler.transform(x) for x in evaluation_arrays]\n    predictions = [np.empty((len(x), len(LABELS))) for x in evaluation_arrays]\n    heads = []\n    for j, label in enumerate(LABELS):\n        if len(np.unique(y_train[:, j])) != 2:\n            raise ValueError(f'Training partition lacks both classes for {label}')\n        head = LogisticRegression(C=LOGISTIC_C, solver='lbfgs', max_iter=3000, random_state=SEED)\n        with warnings.catch_warnings():\n            warnings.simplefilter('error', ConvergenceWarning)\n            head.fit(transformed_train, y_train[:, j])\n        for target, features in zip(predictions, transformed_eval):\n            target[:, j] = head.predict_proba(features)[:, list(head.classes_).index(1)]\n        heads.append(head)\n    return predictions, scaler, heads\n\ny = gold[LABELS].to_numpy(dtype=int)\nfolds = gold.fold.to_numpy()\noof = np.full(y.shape, np.nan, dtype=float)\nfold_test_predictions, fold_results = [], []\nfor fold in range(3):\n    held = folds == fold\n    fit = ~held\n    assert not set(gold.loc[fit, 'report_group']) & set(gold.loc[held, 'report_group'])\n    predictions, scaler, heads = fit_heads_predict(X[fit], y[fit], X[held], X_test)\n    oof[held] = predictions[0]\n    fold_test_predictions.append(predictions[1])\n    aucs = roc_auc_score(y[held], predictions[0], average=None)\n    fold_results.append({'fold': fold, 'n': int(held.sum()), 'macro_auc': float(aucs.mean()), **dict(zip(LABELS, aucs.tolist()))})\n    np.savez_compressed(OUTPUT / f'fold_{fold}_heads.npz', mean=scaler.mean_, scale=scaler.scale_,\n                        coef=np.concatenate([head.coef_ for head in heads]),\n                        intercept=np.concatenate([head.intercept_ for head in heads]))\n    print(f'Fold {fold}: macro AUC {aucs.mean():.4f}')\nassert np.isfinite(oof).all() and ((oof >= 0) & (oof <= 1)).all()\noof_frame = pd.DataFrame(oof, columns=LABELS)\noof_frame.insert(0, 'fold', folds)\noof_frame.insert(0, 'StudyInstanceUID', gold.StudyInstanceUID)\noof_frame.to_csv(OUTPUT / 'oof_predictions.csv', index=False)\ngold[['StudyInstanceUID', 'fold'] + LABELS].to_csv(OUTPUT / 'oof_targets.csv', index=False)\nfold_metrics = pd.DataFrame(fold_results)\nfold_metrics.to_csv(OUTPUT / 'fold_metrics.csv', index=False)\ndisplay(fold_metrics)\nper_label = pd.DataFrame({'target': LABELS, 'mean_fold_auc': fold_metrics[LABELS].mean().to_numpy(),\n                          'pooled_oof_auc': roc_auc_score(y, oof, average=None),\n                          'positives': y.sum(axis=0), 'negatives': (1-y).sum(axis=0)})\nper_label.to_csv(OUTPUT / 'per_label_metrics.csv', index=False)\ndisplay(per_label)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:36:34.096153Z","iopub.execute_input":"2026-09-06T08:36:34.096651Z","iopub.status.idle":"2026-09-06T08:36:34.832143Z","shell.execute_reply.started":"2026-09-06T08:36:34.096621Z","shell.execute_reply":"2026-09-06T08:36:34.830768Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Interpretation and outputs\nPrimary summary: mean of the three fold macro-AUCs. Pooled OOF AUC is secondary\nbecause probabilities from different fitted models need not share the same scale.\nFold spread is descriptive, not a confidence interval. With only 2–4 MCL positives\nper fold, small differences cannot justify a claimed improvement. A constant score\nhas AUC 0.5; use that reference to detect obvious failures, not as a clinical benchmark.\n\nExample predictions average the three fold models. They use MRI only. This notebook\nruns feature extraction and training together and is **not** a finished hidden-test\ninference submission. Save weights and freeze preprocessing for that later milestone.\nNo score has been prefilled into this delivered notebook.\n","metadata":{}},{"cell_type":"code","source":"example_predictions = np.mean(fold_test_predictions, axis=0)\nprediction_table = pd.DataFrame(example_predictions, columns=LABELS, index=test.StudyInstanceUID)\nsubmission = sample_submission[['StudyInstanceUID']].join(prediction_table, on='StudyInstanceUID')\nassert list(submission.columns) == ['StudyInstanceUID'] + LABELS\nassert submission.StudyInstanceUID.tolist() == sample_submission.StudyInstanceUID.tolist()\nassert np.isfinite(submission[LABELS].to_numpy()).all()\nassert submission[LABELS].ge(0).all().all() and submission[LABELS].le(1).all().all()\nsubmission.to_csv(OUTPUT / 'example_submission.csv', index=False)\nmetrics = {'mean_fold_macro_auc': float(fold_metrics.macro_auc.mean()),\n           'fold_macro_auc_min': float(fold_metrics.macro_auc.min()),\n           'fold_macro_auc_max': float(fold_metrics.macro_auc.max()),\n           'pooled_oof_macro_auc_secondary': float(roc_auc_score(y, oof, average='macro')),\n           'constant_score_auc_reference': 0.5, 'feature_seconds': feature_seconds,\n           'peak_cuda_allocated_gib': torch.cuda.max_memory_allocated()/2**30 if DEVICE.type == 'cuda' else None,\n           'interpretation': 'Selected 58-study development set; patient independence unverified; not clinical validation'}\nconfig = {'seed': SEED, 'encoder': 'resnet18_IMAGENET1K_V1_frozen', 'weights_sha256': weights_sha256,\n          'image_size': IMAGE_SIZE, 'slices_per_series': SLICES_PER_SERIES, 'batch_size': ENCODER_BATCH_SIZE,\n          'plane_order': PLANES, 'pooling': 'max', 'feature_dim': 1536, 'logistic_C': LOGISTIC_C,\n          'normalization': 'sampled-series pooled p1/p99 excluding DICOM padding, physical-aspect letterbox, ImageNet channel normalization',\n          'train_sha256': actual_hash, 'frozen_folds_sha256': EXPECTED_FOLDS_SHA256,\n          'labels': LABELS, 'report_supervision': False, 'encoder_finetuned': False}\n(OUTPUT / 'metrics.json').write_text(json.dumps(metrics, indent=2))\n(OUTPUT / 'run_config.json').write_text(json.dumps(config, indent=2))\ntorch.save(encoder.cpu().state_dict(), OUTPUT / 'frozen_encoder.pt')\nprint(json.dumps(metrics, indent=2))\nprint('Artifacts written to', OUTPUT)\nfor path in sorted(OUTPUT.iterdir()):\n    if path.is_file():\n        print(f'{path.name}: {path.stat().st_size:,} bytes')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-06T08:36:57.63045Z","iopub.execute_input":"2026-09-06T08:36:57.630972Z","iopub.status.idle":"2026-09-06T08:36:57.736323Z","shell.execute_reply.started":"2026-09-06T08:36:57.63094Z","shell.execute_reply":"2026-09-06T08:36:57.73551Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What to review next\n- `mri_inspection.png` and `image_audit.json`: correct slice decoding and coverage?\n- `fold_metrics.csv` and `per_label_metrics.csv`: stable across folds or driven by a few cases?\n- `oof_predictions.csv`: preserve these before changing anything.\n- `run_config.json` and `environment.json`: preserve exact settings and weight fingerprint.\n- `example_submission.csv`: valid MRI-only probability output, not a claimed leaderboard result.\n\nNext controlled experiment: keep this encoder, series selection, and folds fixed;\ncompare expert-only learning with uncertainty-aware report supervision. Prompt/rule\ndevelopment and weak-label calibration must never use the held-out fold's expert labels.\nPatient/near-duplicate checks and site/language audits remain outstanding before strong claims.\n\nReferences: [competition](https://www.kaggle.com/competitions/rsna-knee-abnormality-detection),\n[host target definitions](https://www.kaggle.com/competitions/rsna-knee-abnormality-detection/discussion/733343),\n[torchvision ResNet-18](https://docs.pytorch.org/vision/stable/models/generated/torchvision.models.resnet18.html),\n[pydicom modality correction](https://pydicom.github.io/pydicom/stable/reference/generated/pydicom.pixels.apply_modality_lut.html).\n","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}