{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"bda6234e-e54a-4e06-beea-77d1b1acd7ac","cell_type":"markdown","source":"# RSNA Knee MRI — Teacher / Student v2\n\nTwelve binary findings per study, scored by **AUC**.\n## Referred \n1. All public notebooks, previous competitions, image plus text model that is why using DINO model here\n2. https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/overview check this competition previously 2025 and 2024 also\n## What the previous run measured, and what it got wrong\n\n| | |\n|---|---|\n| CV mean AUC (vs pseudo-labels) | 0.7824 |\n| **AUC on the 58 gold studies** | **0.5998** |\n| feature extraction | 4,407 studies in 9,924 s |\n\nThe 0.78 was measured against the pipeline's *own* pseudo-labels, so it partly\nrewarded reproducing the labeler's mistakes. On audited labels the model reached\n0.600. Two concrete defects explain most of the gap:\n\n**The teacher was fitted on the wrong targets.** It was trained on the rule\nlabeler's output, so it could only ever distill the rules — and it measured\n*worse* than them (0.720 vs 0.756 accuracy). Fitting instead on the 58 audited\nstudies, with the rule scores supplied as features, is a large gain:\n\n| teacher | macro AUC on gold |\n|---|---|\n| rules alone | 0.694 |\n| text model fitted on gold | 0.715 |\n| **0.75·text + 0.25·rules** | **0.739** |\n\n**The validation leaked.** 190 studies share an identical normalised report, so\nungrouped folds put the same text on both sides of the split.\n\nThe teacher reaches 0.739 on audited labels while the student reached 0.600, so\nthe bottleneck is the image side and the supervision reaching it — not the\nlabels themselves.","metadata":{}},{"id":"26eef881-cc95-4a3b-aed1-5d67025a6a0a","cell_type":"markdown","source":"## 1. Setup","metadata":{}},{"id":"778c98e4-cd3d-436c-a45e-68e3a1d6a0c6","cell_type":"code","source":"import os, re, gc, json, math, time, random, unicodedata, warnings\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nfrom scipy import sparse\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.linear_model import LogisticRegression, Ridge\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import StratifiedKFold, GroupKFold\nfrom sklearn.metrics import roc_auc_score\n\nwarnings.filterwarnings(\"ignore\")\n\nSEED = 2026\nrandom.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)\n\nLABELS = [\"ACL\", \"MCL\", \"Medial Meniscus\", \"Lateral Meniscus\", \"Medial OA\",\n          \"Lateral OA\", \"PF OA\", \"Effusion\", \"Synovitis\", \"Baker's\",\n          \"Contusion\", \"Fracture\"]\nPLANES = [\"Sagittal\", \"Coronal\", \"Axial\"]\n\ndef find_competition_dir():\n    for p in [Path(\"/kaggle/input/competitions/rsna-knee-abnormality-detection\"),\n              Path(\"/kaggle/input/rsna-knee-abnormality-detection\"),\n              Path(r\"c:\\Users\\kumar\\kaggle\\data\\rsna\")]:\n        if (p / \"train.csv\").exists():\n            return p\n    root = Path(\"/kaggle/input\")\n    if root.exists():\n        for t in root.glob(\"*/train.csv\"):\n            if (t.parent / \"train_series.csv\").exists():\n                return t.parent\n    raise FileNotFoundError(\"competition data not found\")\n\nBASE = find_competition_dir()\nWORK = Path(\"/kaggle/working\"); WORK.mkdir(parents=True, exist_ok=True)\nDEVICE = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\ntrain = pd.read_csv(BASE / \"train.csv\")\ntrain_series = pd.read_csv(BASE / \"train_series.csv\")\ntest = pd.read_csv(BASE / \"test.csv\")\ntest_series = pd.read_csv(BASE / \"test_series.csv\")\nsample_sub = pd.read_csv(BASE / \"sample_submission.csv\")\n\nhas_gold = train[LABELS].notna().any(axis=1).to_numpy()\ngold_idx = np.flatnonzero(has_gold)\ngold_y = train.loc[train.index[gold_idx], LABELS].to_numpy(np.float32)\nprint(f\"data {BASE}\\ntrain {len(train)}  gold {len(gold_idx)}  test {len(test)}\")\nprint(\"device:\", DEVICE)","metadata":{},"outputs":[],"execution_count":null},{"id":"7496b0aa-d4b2-43ab-8170-f4b092000a11","cell_type":"markdown","source":"## 2. Report labeler (rules)\n\nCheXpert-style three stages with NegEx forward-scoped negation. Used here as a\n**feature source and a blend component**, not as the training target.","metadata":{}},{"id":"c06b0b47-769a-4ac0-b4f7-1a450e8ad057","cell_type":"code","source":"def norm(text):\n    t = unicodedata.normalize(\"NFKD\", str(text))\n    t = \"\".join(c for c in t if not unicodedata.combining(c))\n    t = t.replace(\"\\n\", \" . \")\n    return re.sub(r\"\\s+\", \" \", t.lower())\n\n\n# --- vocabularies ----------------------------------------------------------\nTEAR = (r\"(tear|torn|rupture|ruptur|rotura|roto|rota|desgarro|dechirure|dechir|\"\n        r\"scheur|riss|gerissen|lesion|laceraci|lacerazione|discontinu|\"\n        r\"disrupt|avulsion|insufficien|deficien|reconstruct|plastia|\"\n        r\"degenerativ|mucoid|fissur|fisura|macerat)\")\n# Deliberately excludes bare \"cartilage\"/\"chondral\"/\"knorpel\": those words appear\n# in every normal knee report (\"cartilage is intact\"), and including them dropped\n# OA precision to 0.40-0.47. Only terms that assert *disease* are kept.\nOA = (r\"(osteoarthrit|osteoarthros|arthrose|artrosis|artrose|gonarthros|\"\n      r\"gonartrosis|degenerative change|cambios degenerativ|chondropath|\"\n      r\"condropat|chondromalac|condromalac|osteophyt|osteofit|\"\n      r\"chondral (?:loss|thinning|ulcer|defect|fissur)|\"\n      r\"condral(?:es)? (?:de espesor|focal|difus)|ulceras? condral|\"\n      r\"cartilage (?:loss|thinning|defect|fissur)|\"\n      r\"perdida de cartilago|kraakbeenverlies|knorpelschaden|knorpeldefekt|\"\n      r\"joint space narrowing|pinzamiento|artrosi|slijtage)\")\n\nANATOMY = {\n    \"ACL\": r\"(anterior cruciate|cruzado anterior|\\blca\\b|\\bacl\\b|croise anterieur|\"\n           r\"voorste kruisband|vorderes kreuzband|crociato anteriore)\",\n    \"MCL\": r\"(medial collateral|colateral medial|colateral interno|\\blcm\\b|\\bmcl\\b|\"\n           r\"collateral interne|collateral medial|mediale collaterale|\"\n           r\"mediale band|innenband|collaterale mediale|ligament collateral medial)\",\n    \"Medial Meniscus\": r\"((medial|internal|interno|interna|interne|mediale|innen)\"\n                       r\"[^.;]{0,30}(menisc|menisque|meniskus)|\"\n                       r\"(menisc|menisque|meniskus)[^.;]{0,30}\"\n                       r\"(medial|interno|interna|interne|mediale|innen))\",\n    \"Lateral Meniscus\": r\"((lateral|external|externo|externa|externe|laterale|aussen)\"\n                        r\"[^.;]{0,30}(menisc|menisque|meniskus)|\"\n                        r\"(menisc|menisque|meniskus)[^.;]{0,30}\"\n                        r\"(lateral|externo|externa|externe|laterale|aussen))\",\n    # Compartment terms must sit next to a joint/condyle word, otherwise any\n    # clause containing \"medial\" anywhere claims medial-compartment disease.\n    \"Medial OA\": r\"((medial|interno|interna|interne|mediale|innen)[^.;]{0,30}\"\n                 r\"(compartment|compartimento|femorotibial|condyl|condil|\"\n                 r\"tibial|femoral|gelenk|kompartiment)|\"\n                 r\"femorotibial (?:medial|interno)|gonartrosis medial)\",\n    \"Lateral OA\": r\"((lateral|externo|externa|externe|laterale|aussen)[^.;]{0,30}\"\n                  r\"(compartment|compartimento|femorotibial|condyl|condil|\"\n                  r\"tibial|femoral|gelenk|kompartiment)|\"\n                  r\"femorotibial (?:lateral|externo)|gonartrosis lateral)\",\n    \"PF OA\": r\"(patellofemoral|femoropatelar|patelofemoral|retropatellar|retropatelar|\"\n             r\"rotulian|patella|patelar|trochlea|troclea|femoropatellaire|\"\n             r\"retropatellaire|kniescheibe|patellofemorale)\",\n}\n\nPATHOLOGY = {\n    \"ACL\": TEAR, \"MCL\": TEAR,\n    \"Medial Meniscus\": TEAR, \"Lateral Meniscus\": TEAR,\n    \"Medial OA\": OA, \"Lateral OA\": OA, \"PF OA\": OA,\n    \"Effusion\": r\"(effusion|derrame|epanchement|erguss|versamento|\"\n                r\"joint fluid|liquido articular|hydrops|gewrichtsvocht|\"\n                r\"vocht in het gewricht|articular fluid)\",\n    \"Synovitis\": r\"(synovit|sinovit|synovial (?:thickening|proliferat|hypertroph)|\"\n                 r\"engrosamiento sinovial|synoviale (?:verdikking|proliferat)|\"\n                 r\"sinovial (?:engros|hipertrof)|pannus)\",\n    \"Baker's\": r\"(baker|popliteal cyst|quiste popliteo|quiste de baker|kyste poplite|\"\n               r\"kyste de baker|bakerzyste|poplitealzyste|cisti di baker|\"\n               r\"cyste van baker|popliteale cyste|quistes popliteos)\",\n    \"Contusion\": r\"(contusion|contusio|bone bruise|bone marrow (?:edema|oedema)|\"\n                 r\"edema (?:oseo|de la medula osea|medular|de medula)|\"\n                 r\"knochenmarkodem|beenmergoedeem|oedeme (?:osseux|medullaire)|\"\n                 r\"botcontusie|edema osseo|medullary edema|alteracion de senal de la medula)\",\n    \"Fracture\": r\"(fracture|fractura|fractuur|fraktur|frattura|\"\n                r\"avulsion fracture|arrancamiento oseo|\\bfx\\b)\",\n}\nSELF_NAMING = {\"Effusion\", \"Synovitis\", \"Baker's\", \"Contusion\", \"Fracture\"}\n\n# --- NegEx-style cue sets --------------------------------------------------\n# Pre-cues negate forward; post-cues negate backward within the same clause.\nPRE_NEG = (r\"\\b(no|not|non|sin|senza|sans|geen|kein|keine|nicht|zonder|without|\"\n           r\"absence|ausencia|aucun|aucune|negative for|free of|libre de|\"\n           r\"vrij van|no evidence of|sin evidencia de|no signs? of|sin signos de|\"\n           r\"no hay|pas de|niet|denies|ohne|ningun|ninguna|descarta)\\b\")\nPOST_NEG = (r\"\\b(intact|intacto|intacta|intacte|intactos|intactas|\"\n            r\"normal|normales|normale|normaal|unremarkable|\"\n            r\"preserved|conservad[oa]s?|conserve|behouden|\"\n            r\"within normal limits|dentro de limites normales|\"\n            r\"sin alteraciones|zonder afwijkingen|onauffallig)\\b\")\n# Cues that look like negation but are not.\nPSEUDO_NEG = r\"\\b(no longer|nicht mehr|ya no|niet meer|not only|no significant change)\\b\"\n# Clause terminators - negation does not cross these.\nTERMINATOR = (r\"\\b(but|however|although|though|except|pero|aunque|sin embargo|\"\n              r\"mais|cependant|toutefois|aber|jedoch|maar|echter|tuttavia|\"\n              r\"with|con|met|mit|avec|showing|demonstrating|there is|hay)\\b\")\nUNCERTAIN = (r\"\\b(possible|possibly|probable|probably|suspicious|suspected|\"\n             r\"cannot exclude|no se puede excluir|posible|sospecha|\"\n             r\"questionable|equivocal|may represent|could represent|\"\n             r\"verdacht|mogelijk|moglich|dubious|dudoso)\\b\")\n\nCLAUSE_SPLIT = re.compile(r\"[.;:]|\\bthere is\\b|\\band\\b|,\")\nSECTION_RE = re.compile(\n    r\"(impression|impresion|conclusion|conclusio|conclusie|beurteilung|\"\n    r\"besluit|conclusioni|indruk)\\s*[:\\-]\", re.I)\n\n\ndef clauses(text):\n    \"\"\"Split into clauses; negation scope never crosses a clause boundary.\"\"\"\n    parts = re.split(r\"[.;\\n]+\", text)\n    out = []\n    for p in parts:\n        p = p.strip()\n        if p:\n            out.append(p)\n    return out\n\n\ndef classify_mention(clause, m_start, m_end):\n    \"\"\"Return 'pos', 'neg' or 'unc' for one pathology mention inside a clause.\"\"\"\n    before = clause[:m_start]\n    after = clause[m_end:]\n\n    if re.search(PSEUDO_NEG, clause):\n        return \"pos\"\n\n    # Pre-negation: cue appears before the mention with no terminator between.\n    for m in re.finditer(PRE_NEG, before):\n        between = before[m.end():]\n        if not re.search(TERMINATOR, between):\n            return \"neg\"\n\n    # Post-negation: \"<finding> ... is intact/normal\" inside the same clause.\n    for m in re.finditer(POST_NEG, after):\n        between = after[:m.start()]\n        if not re.search(TERMINATOR, between) and len(between) < 60:\n            return \"neg\"\n\n    if re.search(UNCERTAIN, clause):\n        return \"unc\"\n    return \"pos\"\n\n\ndef label_report(report, uncertain_as=1):\n    \"\"\"Three-stage labelling of one report -> {label: 0/1}.\"\"\"\n    text = norm(report)\n\n    # Section split: the impression, when present, overrides the body.\n    sec = SECTION_RE.search(text)\n    body, impression = (text[:sec.start()], text[sec.end():]) if sec else (text, \"\")\n\n    def scan(segment):\n        found = {}\n        for lab in LABELS:\n            path_re = PATHOLOGY[lab]\n            anat_re = ANATOMY.get(lab)\n            need_anat = lab not in SELF_NAMING\n            verdicts = []\n            for cl in clauses(segment):\n                if need_anat and not re.search(anat_re, cl):\n                    continue\n                for m in re.finditer(path_re, cl):\n                    verdicts.append(classify_mention(cl, m.start(), m.end()))\n            if verdicts:\n                if \"pos\" in verdicts:\n                    found[lab] = 1\n                elif \"unc\" in verdicts:\n                    found[lab] = uncertain_as\n                else:\n                    found[lab] = 0\n        return found\n\n    body_v = scan(body)\n    imp_v = scan(impression) if impression else {}\n\n    out = {}\n    for lab in LABELS:\n        if lab in imp_v:\n            out[lab] = imp_v[lab]          # impression wins\n        elif lab in body_v:\n            out[lab] = body_v[lab]\n        else:\n            out[lab] = 0                   # never mentioned -> absent\n    return out","metadata":{},"outputs":[],"execution_count":null},{"id":"6dcf7380-c1e3-47f0-9978-a26c301e949c","cell_type":"markdown","source":"## 3. Teacher fitted on the audited labels\n\nThe correction. Fit per-label logistic regression on the 58 gold studies over\nchar (3–5) + word (1–2) TF-IDF **plus the rule scores as features**, measure\nout-of-fold on those same 58, then blend 0.75 model + 0.25 rules.\n\n58 rows against 100k features sounds hopeless; liblinear with balanced class\nweights is the regime where it works, and the OOF numbers below say whether it\ndid rather than assuming.","metadata":{}},{"id":"e1bfc8ad-05a6-4f4c-9659-b15a07e7ed5a","cell_type":"code","source":"texts = [norm(t) for t in train[\"Report\"]]\nrule_hard = np.asarray([[label_report(t, 1)[l] for l in LABELS]\n                        for t in train[\"Report\"]], dtype=np.float32)\nrules = np.where(rule_hard > 0.5, 0.88, 0.12).astype(np.float32)\n\nchar = TfidfVectorizer(analyzer=\"char_wb\", ngram_range=(3, 5), min_df=3,\n                       max_features=120_000, sublinear_tf=True)\nword = TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=70_000,\n                       sublinear_tf=True)\nX_text = sparse.hstack([char.fit_transform(texts), word.fit_transform(texts),\n                        sparse.csr_matrix(rules)]).tocsr()\n\nteacher = np.zeros((len(train), len(LABELS)), np.float32)\noof = np.zeros((len(gold_idx), len(LABELS)), np.float32)\nrows = []\nfor j, lab in enumerate(LABELS):\n    y = gold_y[:, j].astype(int)\n    if len(np.unique(y)) < 2:\n        teacher[:, j] = rules[:, j]; continue\n    for tr, va in StratifiedKFold(4, shuffle=True, random_state=SEED).split(gold_idx, y):\n        m = LogisticRegression(C=2, class_weight=\"balanced\", solver=\"liblinear\",\n                               max_iter=700).fit(X_text[gold_idx[tr]], y[tr])\n        oof[va, j] = m.predict_proba(X_text[gold_idx[va]])[:, 1]\n    m = LogisticRegression(C=2, class_weight=\"balanced\", solver=\"liblinear\",\n                           max_iter=700).fit(X_text[gold_idx], y)\n    teacher[:, j] = 0.75 * m.predict_proba(X_text)[:, 1] + 0.25 * rules[:, j]\n    rows.append((lab, roc_auc_score(y, rules[gold_idx, j]),\n                 roc_auc_score(y, 0.75 * oof[:, j] + 0.25 * rules[gold_idx, j])))\n\ntdf = pd.DataFrame(rows, columns=[\"label\", \"rule_auc\", \"blend_auc\"])\nprint(tdf.round(3).to_string(index=False))\nprint(f\"\\nMACRO  rules {tdf.rule_auc.mean():.3f}   blended teacher {tdf.blend_auc.mean():.3f}\")\nteacher[gold_idx] = gold_y          # audited rows override","metadata":{},"outputs":[],"execution_count":null},{"id":"901b5839-fe51-48b6-965d-fd7d6202ba64","cell_type":"markdown","source":"## 4. Report-fingerprint groups\n\n190 studies share an identical normalised report. Grouping by fingerprint keeps\nduplicates on one side of a fold; without it the CV is optimistic.","metadata":{}},{"id":"c86b0fd6-36e8-4c1e-8ef2-7126ea865e3d","cell_type":"code","source":"import hashlib\nfingerprint = np.array([hashlib.md5(t.encode()).hexdigest() for t in texts])\nu, counts = np.unique(fingerprint, return_counts=True)\ndup_studies = int(counts[counts > 1].sum())\nprint(f\"distinct reports {len(u)} / {len(train)}   studies in duplicate groups: {dup_studies}\")","metadata":{},"outputs":[],"execution_count":null},{"id":"ed8af5b3-9928-4ed1-8f34-8ca3dc5514a8","cell_type":"markdown","source":"## 5. DINOv2 features\n\nExtraction cost 9,924 s last run, so the previous kernel's output is attached and\nthe **train** features are reused when the cached study IDs match exactly.\n\n**Test features are always extracted at run time.** This is a code competition:\nat scoring the notebook is re-run against a hidden test set that is a different\nsize and different studies from the three visible placeholders. A cached test\narray can never be valid then — assuming otherwise is what made the first\nsubmission throw `Notebook Threw Exception` during the rerun. The cache is keyed\non study IDs rather than row counts so a mismatch degrades to recomputation\ninstead of an error.","metadata":{}},{"id":"a03a8dea-b179-46da-8a25-402245fab535","cell_type":"code","source":"from transformers import AutoModel\nimport pydicom\ntry:\n    import cv2\nexcept ImportError:\n    cv2 = None\n\nN_SLICES, IMG_SIZE, HID = 5, 224, 384\nFEAT_BATCH, LOADER_BATCH, WORKERS = 256, 32, 8\n\ndef find_dinov2_dir():\n    root = Path(\"/kaggle/input\")\n    cands = []\n    if root.exists():\n        skip = {\"train_series\", \"test_series\", \".git\", \"__pycache__\"}\n        base = len(root.parts)\n        for r, dirs, files in os.walk(root):\n            dirs[:] = [d for d in dirs if d not in skip and len(Path(r).parts) - base < 7]\n            if \"config.json\" not in files:\n                continue\n            rp = Path(r)\n            if not any((rp / n).exists() for n in (\"model.safetensors\", \"pytorch_model.bin\")):\n                continue\n            try:\n                cfg = json.loads((rp / \"config.json\").read_text())\n            except Exception:\n                continue\n            if \"dinov2\" in str(cfg.get(\"model_type\", \"\")).lower() or \"dinov2\" in str(rp).lower():\n                cands.append((abs(int(cfg.get(\"hidden_size\", 10000)) - HID), rp))\n    if not cands:\n        raise FileNotFoundError(\n            \"DINOv2 not found. Attach the Kaggle model metaresearch/dinov2/PyTorch/small/1.\")\n    return sorted(cands, key=lambda x: x[0])[0][1]\n\nclass DinoCLS(nn.Module):\n    def __init__(self, bb):\n        super().__init__(); self.bb = bb\n    def forward(self, x):\n        return self.bb(pixel_values=x).last_hidden_state[:, 0]\n\ndef series_score(row):\n    return 2.5 * float(row.get(\"Fluid_Sensitive\", 0) or 0)\n\ndef build_plane_series(df):\n    out = {}\n    for sid, rows in df.groupby(\"StudyInstanceUID\", sort=False):\n        planes, pv = {}, rows[\"Anatomical_Plane\"].astype(str).str.lower()\n        for plane in PLANES:\n            part = rows[pv == plane.lower()]\n            if len(part):\n                planes[plane] = str(part.loc[part.apply(series_score, axis=1).idxmax()]\n                                    [\"SeriesInstanceUID\"])\n        out[str(sid)] = planes\n    return out\n\ndef slice_paths(folder, n):\n    try:\n        paths = [Path(e.path) for e in os.scandir(folder)\n                 if e.is_file() and e.name.lower().endswith(\".dcm\")]\n    except Exception:\n        return []\n    if not paths:\n        return []\n    paths.sort(key=lambda p: p.name)\n    m = len(paths); span = paths[int(0.15 * m):max(int(0.85 * m), 1)] or paths\n    idx = np.linspace(0, len(span) - 1, min(n, len(span))).round().astype(int)\n    picked = [span[i] for i in idx]\n    while len(picked) < n:\n        picked.append(picked[-1])\n    return picked\n\ndef read_dicom(path):\n    try:\n        ds = pydicom.dcmread(str(path), force=True)\n        arr = ds.pixel_array.astype(np.float32)\n    except Exception:\n        return None\n    arr = arr * float(getattr(ds, \"RescaleSlope\", 1.0) or 1.0) \\\n              + float(getattr(ds, \"RescaleIntercept\", 0.0) or 0.0)\n    if str(getattr(ds, \"PhotometricInterpretation\", \"\")).upper() == \"MONOCHROME1\":\n        arr = arr.max() - arr\n    fin = arr[np.isfinite(arr)]\n    if fin.size == 0:\n        return None\n    nz = fin[np.abs(fin) > 1e-8]\n    src = nz if nz.size >= 64 else fin\n    lo, hi = np.percentile(src, [1.0, 99.0])\n    if not np.isfinite(lo) or not np.isfinite(hi) or hi <= lo:\n        lo, hi = float(fin.min()), float(fin.max())\n    arr = np.clip(np.nan_to_num(arr, nan=lo, posinf=hi, neginf=lo), lo, hi)\n    return ((arr - lo) / max(hi - lo, 1e-6)).astype(np.float32)\n\ndef crop_resize(arr, size):\n    mask = arr > 0.03; h, w = arr.shape\n    if mask.sum() > 0.02 * h * w:\n        ys, xs = np.where(mask)\n        y0, y1, x0, x1 = int(ys.min()), int(ys.max()) + 1, int(xs.min()), int(xs.max()) + 1\n        my, mx = max(2, int(0.06 * (y1 - y0))), max(2, int(0.06 * (x1 - x0)))\n        arr = arr[max(0, y0 - my):min(h, y1 + my), max(0, x0 - mx):min(w, x1 + mx)]\n    h, w = arr.shape; side = max(h, w)\n    pad = np.zeros((side, side), np.float32)\n    pad[(side - h) // 2:(side - h) // 2 + h, (side - w) // 2:(side - w) // 2 + w] = arr\n    if cv2 is not None:\n        r = cv2.resize(pad, (size, size),\n                       interpolation=cv2.INTER_AREA if side > size else cv2.INTER_LINEAR)\n    else:\n        r = F.interpolate(torch.from_numpy(pad)[None, None], size=(size, size),\n                          mode=\"bilinear\", align_corners=False)[0, 0].numpy()\n    return np.clip(r * 255.0, 0, 255).astype(np.uint8)\n\nclass StudyImages(Dataset):\n    def __init__(self, studies, slots, root):\n        self.s, self.slots, self.root = list(studies), slots, Path(root)\n    def __len__(self):\n        return len(self.s)\n    def __getitem__(self, i):\n        sid = str(self.s[i]); sel = self.slots.get(sid, {})\n        imgs = np.zeros((len(PLANES), N_SLICES, IMG_SIZE, IMG_SIZE), np.uint8)\n        mask = np.zeros((len(PLANES), N_SLICES), np.bool_)\n        for pi, plane in enumerate(PLANES):\n            ser = sel.get(plane)\n            if ser is None:\n                continue\n            for si, p in enumerate(slice_paths(self.root / sid / ser, N_SLICES)):\n                a = read_dicom(p)\n                if a is None:\n                    continue\n                imgs[pi, si] = crop_resize(a, IMG_SIZE); mask[pi, si] = True\n        return torch.from_numpy(imgs), torch.from_numpy(mask), sid\n\n@torch.inference_mode()\ndef encode(model, imgs):\n    mean = torch.tensor([0.485, 0.456, 0.406], device=DEVICE).view(1, 3, 1, 1)\n    std = torch.tensor([0.229, 0.224, 0.225], device=DEVICE).view(1, 3, 1, 1)\n    outs = []\n    for i in range(0, len(imgs), FEAT_BATCH):\n        x = imgs[i:i + FEAT_BATCH].to(DEVICE).float().div_(255.)\n        x = ((x.unsqueeze(1).repeat(1, 3, 1, 1)) - mean) / std\n        outs.append(model(x).float().cpu())\n    return torch.cat(outs) if outs else torch.zeros(0)\n\ndef extract_features(studies, slots, root, tag):\n    from concurrent.futures import ThreadPoolExecutor\n    d = find_dinov2_dir()\n    bb = AutoModel.from_pretrained(str(d), local_files_only=True)\n    for p in bb.parameters():\n        p.requires_grad = False\n    model = DinoCLS(bb).eval().to(DEVICE)\n    print(f\"DINOv2 {d} | blocks {len(bb.encoder.layer)} | {IMG_SIZE}px | {N_SLICES} slices\")\n\n    ds = StudyImages(studies, slots, root)\n    feats = np.zeros((len(studies), len(PLANES), N_SLICES, HID), np.float16)\n    masks = np.zeros((len(studies), len(PLANES), N_SLICES), np.bool_)\n    def safe(i):\n        try:\n            return ds[i]\n        except Exception:\n            return (torch.zeros(len(PLANES), N_SLICES, IMG_SIZE, IMG_SIZE, dtype=torch.uint8),\n                    torch.zeros(len(PLANES), N_SLICES, dtype=torch.bool), str(studies[i]))\n    cur, t0 = 0, time.time()\n    with ThreadPoolExecutor(max_workers=WORKERS) as pool:\n        for st in range(0, len(ds), LOADER_BATCH):\n            items = list(pool.map(safe, range(st, min(st + LOADER_BATCH, len(ds)))))\n            im = torch.stack([x[0] for x in items]); mk = torch.stack([x[1] for x in items])\n            b = len(items)\n            flat = im.reshape(-1, IMG_SIZE, IMG_SIZE); fm = mk.reshape(-1)\n            out = torch.zeros(len(flat), HID)\n            if fm.any():\n                out[fm] = encode(model, flat[fm])\n            feats[cur:cur + b] = out.reshape(b, len(PLANES), N_SLICES, HID).numpy().astype(np.float16)\n            masks[cur:cur + b] = mk.numpy(); cur += b\n    print(f\"{tag}: {cur} studies in {time.time()-t0:.0f}s\")\n    del model, bb; gc.collect()\n    return feats, masks","metadata":{},"outputs":[],"execution_count":null},{"id":"66a14904-b122-427c-a608-d1c35e8d3e45","cell_type":"code","source":"def load_cached(tag, wanted_ids):\n    '''Reuse a cached feature array only when its study IDs match exactly.\n\n    Keyed on IDs, not row counts: at rerun the hidden test set is different, and\n    a count-based check would either mis-align silently or raise.\n    '''\n    want = set(wanted_ids)\n    roots = [WORK] + ([Path(\"/kaggle/input\")] if Path(\"/kaggle/input\").exists() else [])\n    for root in roots:\n        for p in sorted(root.rglob(f\"{tag}_feat.npy\")):\n            ids_p = p.parent / f\"{tag}_ids.npy\"\n            msk_p = p.parent / f\"{tag}_mask.npy\"\n            if not (ids_p.exists() and msk_p.exists()):\n                continue\n            try:\n                ids = [str(x) for x in np.load(ids_p, allow_pickle=True)]\n                arr = np.load(p, mmap_mode=\"r\")\n            except Exception:\n                continue\n            if set(ids) == want and arr.shape[1:] == (len(PLANES), N_SLICES, HID):\n                print(f\"reusing cached {tag} features from {p.parent}\")\n                order = {s: i for i, s in enumerate(ids)}\n                sel = np.array([order[s] for s in wanted_ids])\n                return np.asarray(arr)[sel], np.asarray(np.load(msk_p))[sel]\n    return None, None\n\ntrain_ids = train[\"StudyInstanceUID\"].astype(str).tolist()\ntest_ids = test[\"StudyInstanceUID\"].astype(str).tolist()\n\ntrain_slots = build_plane_series(train_series)\ntest_slots = build_plane_series(test_series)\n\nFtr_a, Mtr_a = load_cached(\"train\", train_ids)\nif Ftr_a is None:\n    print(\"no usable train cache - extracting\")\n    Ftr_a, Mtr_a = extract_features(train_ids, train_slots, BASE / \"train_series\", \"train\")\n\n# Test features are ALWAYS recomputed unless the cache matches these exact\n# studies. At scoring the hidden test set differs from the visible placeholders.\nFte_a, Mte_a = load_cached(\"test\", test_ids)\nif Fte_a is None:\n    print(f\"extracting test features for {len(test_ids)} studies\")\n    Fte_a, Mte_a = extract_features(test_ids, test_slots, BASE / \"test_series\", \"test\")\n\nprint(\"train\", Ftr_a.shape, \"| test\", Fte_a.shape)\nassert len(Ftr_a) == len(train) and len(Fte_a) == len(test)","metadata":{},"outputs":[],"execution_count":null},{"id":"37f406d7-9086-4113-ab79-5e986a519f36","cell_type":"markdown","source":"## 6. Series-protocol metadata\n\n`Fat_Suppression` is identical to `Fluid_Sensitive` across all 24,371 series\nrows, so only one is kept — the previous version fed both and was effectively\nduplicating a column.","metadata":{}},{"id":"a5a79cfe-2ac9-4f95-b844-b06d3885eaea","cell_type":"code","source":"ident = (train_series[\"Fluid_Sensitive\"] == train_series[\"Fat_Suppression\"]).all()\nprint(f\"Fat_Suppression identical to Fluid_Sensitive: {ident}  -> keeping one\")\n\ndef protocol_features(series_df, ids):\n    g = series_df.groupby(\"StudyInstanceUID\")\n    n_ser = g.size()\n    fluid = g[\"Fluid_Sensitive\"].mean()\n    planes = (series_df.assign(v=1)\n              .pivot_table(index=\"StudyInstanceUID\", columns=\"Anatomical_Plane\",\n                           values=\"v\", aggfunc=\"sum\", fill_value=0))\n    for p in PLANES:\n        if p not in planes.columns:\n            planes[p] = 0\n    out = pd.concat([n_ser.rename(\"n_series\"), fluid.rename(\"fluid_frac\"),\n                     planes[PLANES]], axis=1)\n    return out.reindex(ids).fillna(0.0).to_numpy(np.float32)\n\nmeta_tr = protocol_features(train_series, train[\"StudyInstanceUID\"].astype(str))\nmeta_te = protocol_features(test_series, test[\"StudyInstanceUID\"].astype(str))\nprint(\"metadata:\", meta_tr.shape)","metadata":{},"outputs":[],"execution_count":null},{"id":"0765fc1f-0ade-45cf-b97e-d45910236c6d","cell_type":"markdown","source":"## 7. Student\n\nGated attention MIL per plane, then an MLP over the concatenated plane vectors.\nGold rows are repeated `HARD_REPEAT` times at weight 1.0 while pseudo rows carry\n0.35, so 58 audited studies are not drowned by 4,349 inferred ones.","metadata":{}},{"id":"dc6388e5-26b1-4054-91d9-8d65b7ccbd46","cell_type":"code","source":"HARD_REPEAT, SOFT_W, EPOCHS, LR = 8, 0.35, 40, 1e-3\n\nclass MaskedAttnPool(nn.Module):\n    def __init__(self, dim, hidden=128):\n        super().__init__()\n        self.v, self.u, self.w = nn.Linear(dim, hidden), nn.Linear(dim, hidden), nn.Linear(hidden, 1)\n    def forward(self, x, m):\n        a = self.w(torch.tanh(self.v(x)) * torch.sigmoid(self.u(x)))\n        a = a.masked_fill(~m.unsqueeze(-1), -1e4).softmax(1) * m.unsqueeze(-1).float()\n        return (a * x).sum(1)\n\nclass Student(nn.Module):\n    def __init__(self, dim, n_planes, n_meta, n_out):\n        super().__init__()\n        self.pools = nn.ModuleList([MaskedAttnPool(dim) for _ in range(n_planes)])\n        self.norm = nn.LayerNorm(dim * n_planes)\n        self.mlp = nn.Sequential(\n            nn.Linear(dim * n_planes + n_meta, 256), nn.GELU(), nn.Dropout(0.2),\n            nn.Linear(256, 128), nn.GELU(), nn.Dropout(0.2), nn.Linear(128, n_out))\n    def forward(self, f, m, meta):\n        parts = [self.pools[p](f[:, p], m[:, p]) for p in range(f.shape[1])]\n        return self.mlp(torch.cat([self.norm(torch.cat(parts, -1)), meta], -1))\n\nclass DS(Dataset):\n    def __init__(self, rows, F, M, meta, y=None, w=None):\n        self.rows, self.F, self.M, self.meta, self.y, self.w = rows, F, M, meta, y, w\n    def __len__(self):\n        return len(self.rows)\n    def __getitem__(self, i):\n        r = self.rows[i]\n        f = torch.from_numpy(np.asarray(self.F[r], np.float32))\n        m = torch.from_numpy(np.asarray(self.M[r]))\n        mt = torch.from_numpy(self.meta[r])\n        if self.y is None:\n            return f, m, mt\n        return f, m, mt, torch.from_numpy(self.y[r]), torch.tensor(self.w[r])","metadata":{},"outputs":[],"execution_count":null},{"id":"bff9f586-f858-43af-8429-65b09bf2d274","cell_type":"markdown","source":"## 8. Grouped cross-validation\n\nFolds are grouped by report fingerprint. Reported alongside the pseudo-label CV\nis the **AUC on the gold studies inside each validation fold**, which is the\nnumber that tracks the leaderboard.","metadata":{}},{"id":"14f19383-40e9-42c1-8d83-d32be2d6b4c8","cell_type":"code","source":"Y = teacher.copy()\nW = np.where(has_gold, 1.0, SOFT_W).astype(np.float32)\nmetaF = np.concatenate([meta_tr, np.zeros((len(train), 0), np.float32)], 1)\n\ndef train_fold(tr_rows, va_rows):\n    rows = list(tr_rows) + list(np.repeat(gold_idx[np.isin(gold_idx, tr_rows)],\n                                          HARD_REPEAT))\n    model = Student(HID, len(PLANES), metaF.shape[1], len(LABELS)).to(DEVICE)\n    opt = torch.optim.AdamW(model.parameters(), lr=LR, weight_decay=1e-4)\n    sch = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=EPOCHS)\n    dl = DataLoader(DS(rows, Ftr_a, Mtr_a, metaF, Y, W), batch_size=64,\n                    shuffle=True, num_workers=0)\n    for _ in range(EPOCHS):\n        model.train()\n        for f, m, mt, y, w in dl:\n            f, m, mt, y, w = [t.to(DEVICE) for t in (f, m, mt, y, w)]\n            opt.zero_grad()\n            loss = (F.binary_cross_entropy_with_logits(model(f, m, mt), y,\n                    reduction=\"none\").mean(1) * w).mean()\n            loss.backward(); opt.step()\n        sch.step()\n    model.eval()\n    dlv = DataLoader(DS(list(va_rows), Ftr_a, Mtr_a, metaF), batch_size=256)\n    out = []\n    with torch.no_grad():\n        for f, m, mt in dlv:\n            out.append(torch.sigmoid(model(f.to(DEVICE), m.to(DEVICE),\n                                           mt.to(DEVICE))).cpu().numpy())\n    return model, np.concatenate(out)\n\noof_pred = np.zeros((len(train), len(LABELS)), np.float32)\nmodels = []\ngkf = GroupKFold(5)\nt0 = time.time()\nfor k, (tr_rows, va_rows) in enumerate(gkf.split(np.arange(len(train)),\n                                                 groups=fingerprint)):\n    mdl, pv = train_fold(tr_rows, va_rows)\n    oof_pred[va_rows] = pv; models.append(mdl)\n    g = np.intersect1d(va_rows, gold_idx)\n    if len(g) > 4:\n        pos = [j for j in range(len(LABELS))\n               if len(np.unique(gold_y[np.searchsorted(gold_idx, g), j])) > 1]\n        ga = np.mean([roc_auc_score(gold_y[np.searchsorted(gold_idx, g), j],\n                                    oof_pred[g, j]) for j in pos])\n        print(f\"fold {k}: gold-in-fold n={len(g):>3}  gold AUC {ga:.4f}\")\nprint(f\"\\ntrained in {time.time()-t0:.0f}s\")\n\npos = [j for j in range(len(LABELS)) if len(np.unique(gold_y[:, j])) > 1]\ngold_auc = np.mean([roc_auc_score(gold_y[:, j], oof_pred[gold_idx, j]) for j in pos])\nsoft_auc = np.mean([roc_auc_score((Y[:, j] > 0.5).astype(int), oof_pred[:, j])\n                    for j in pos])\nprint(f\"\\nOOF AUC vs pseudo-labels : {soft_auc:.4f}   (optimistic)\")\nprint(f\"OOF AUC on 58 gold       : {gold_auc:.4f}   <- previous run: 0.5998\")\nprint(f\"teacher AUC on 58 gold   : {tdf.blend_auc.mean():.4f}   <- the ceiling\")","metadata":{},"outputs":[],"execution_count":null},{"id":"18e819f3-a7bb-4e3a-8045-75ce84116820","cell_type":"markdown","source":"## 9. Submission","metadata":{}},{"id":"07a8a7bc-2742-4eb5-a6d6-abaac5dd9da4","cell_type":"code","source":"dlt = DataLoader(DS(list(range(len(test))), Fte_a, Mte_a, meta_te), batch_size=256)\npreds = []\nfor mdl in models:\n    mdl.eval(); out = []\n    with torch.no_grad():\n        for f, m, mt in dlt:\n            out.append(torch.sigmoid(mdl(f.to(DEVICE), m.to(DEVICE),\n                                         mt.to(DEVICE))).cpu().numpy())\n    preds.append(np.concatenate(out))\nimage_pred = np.mean(preds, 0)\n\n# Low-weight metadata ridge for cheap diversity.\nmeta_pred = np.zeros_like(image_pred)\nfor j in range(len(LABELS)):\n    r = make_pipeline(StandardScaler(), Ridge(alpha=8.0))\n    r.fit(meta_tr, Y[:, j], ridge__sample_weight=np.where(has_gold, 8.0, 1.0))\n    meta_pred[:, j] = r.predict(meta_te)\nfinal = 0.88 * image_pred + 0.12 * np.clip(meta_pred, 0.02, 0.98)\n\npred_df = pd.DataFrame(final, columns=LABELS)\npred_df[\"StudyInstanceUID\"] = test[\"StudyInstanceUID\"].astype(str).values\nsub = sample_sub[[\"StudyInstanceUID\"]].merge(pred_df, on=\"StudyInstanceUID\", how=\"left\")\nfor c in LABELS:\n    sub[c] = sub[c].fillna(float(Y[:, LABELS.index(c)].mean()))\nsub = sub[sample_sub.columns]\n\nassert len(sub) == len(sample_sub) and list(sub.columns) == list(sample_sub.columns)\nassert sub[LABELS].notna().all().all()\nassert ((sub[LABELS] >= 0) & (sub[LABELS] <= 1)).all().all()\nsub.to_csv(\"submission.csv\", index=False)\nprint(sub.to_string(index=False))\nprint(f\"\\nwrote submission.csv rows={len(sub)}\")","metadata":{},"outputs":[],"execution_count":null}]}