{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.11"},"papermill":{"default_parameters":{},"parameters":{}}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# RSNA Knee Abnormality Detection — Turbo Mosaic 2×T4\n\nThis notebook is designed for the fastest useful first submission.\n\nMain speed changes:\n\n- no scan of DICOM headers for every slice;\n- one selected series from each anatomical plane;\n- one central file per plane;\n- Sagittal, Coronal and Axial images are packed into the three channels of one image;\n- one image token per study instead of twelve 2.5D tokens;\n- only the first four pretrained DINOv2-small blocks are used;\n- the frozen encoder is distributed over both T4 GPUs;\n- one final study model is trained on all exact and report-derived labels;\n- no cross-validation ensemble in the default turbo configuration.\n\nThe trade-off is lower spatial coverage than the stronger notebook, but feature extraction should be dramatically faster.","metadata":{}},{"cell_type":"code","source":"import os, gc, re, json, math, time, random, warnings, unicodedata\nfrom copy import deepcopy\nfrom pathlib import Path\nfrom collections import defaultdict\nfrom concurrent.futures import ThreadPoolExecutor\n\nimport numpy as np\nimport pandas as pd\nfrom tqdm.auto import tqdm\n\nfrom sklearn.pipeline import FeatureUnion\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import roc_auc_score\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader, WeightedRandomSampler\n\nimport pydicom\n\nwarnings.filterwarnings(\"ignore\")\ntorch.set_float32_matmul_precision(\"high\")\nif torch.cuda.is_available():\n    torch.backends.cudnn.benchmark = True\n\n\nclass CFG:\n    seed = 20\n    debug = False\n\n    # Ultra-fast image extraction\n    img_size = 1024\n    n_slices = 1\n    n_slots = 1\n    dino_layers = 4\n    feature_batch_size = 512\n    feature_loader_batch_size = 128\n    feature_workers = 8  # threads, not multiprocessing workers\n    feature_tta_hflip = False\n    rebuild_features = False\n    feature_cache_tag = \"turbo_mosaic_dino112_l4_threaded_v2\"\n\n    # Tiny study-level classifier\n    hidden_dim = 128\n    dropout = 0.12\n    train_batch_size = 512\n    epochs = 20\n    steps_per_epoch = 24\n    lr = 8e-4\n    weight_decay = 2e-4\n    rank_loss_weight = 0.03\n    exact_sample_boost = 20.0\n    ema_decay = 0.995\n\n    # Pseudo-label confidence\n    rule_threshold = 0.15\n    text_threshold = 0.32\n    min_pseudo_weight = 0.20\n    max_pseudo_weight = 0.85\n\n    competition_dir = None\n    dinov2_dir = None\n\n    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    amp = torch.cuda.is_available()\n    n_gpus = torch.cuda.device_count()\n\n\nTARGETS = [\n    \"ACL\", \"MCL\", \"Medial Meniscus\", \"Lateral Meniscus\",\n    \"Medial OA\", \"Lateral OA\", \"PF OA\", \"Effusion\",\n    \"Synovitis\", \"Baker's\", \"Contusion\", \"Fracture\"\n]\n\nPLANES = [\"Sagittal\", \"Coronal\", \"Axial\"]\nSLOT_NAMES = [\"Sagittal\", \"Coronal\", \"Axial\"]\n\n\ndef seed_everything(seed):\n    random.seed(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n\n\ndef find_competition_dir():\n    if CFG.competition_dir is not None:\n        p = Path(CFG.competition_dir)\n        if p.exists():\n            return p\n    expected = Path(\"/kaggle/input/competitions/rsna-knee-abnormality-detection\")\n    if expected.exists():\n        return expected\n    roots = [Path(\"/kaggle/input\"), Path(\".\")]\n    for root in roots:\n        if not root.exists():\n            continue\n        for train_csv in root.glob(\"*/train.csv\"):\n            p = train_csv.parent\n            required = [\n                p / \"train_series.csv\", p / \"test.csv\", p / \"test_series.csv\",\n                p / \"train_series\", p / \"test_series\", p / \"sample_submission.csv\"\n            ]\n            if all(x.exists() for x in required):\n                return p\n    raise FileNotFoundError(\"Competition directory was not found.\")\n\n\ndef find_dinov2_dir():\n    if CFG.dinov2_dir is not None:\n        p = Path(CFG.dinov2_dir)\n        if (p / \"config.json\").exists():\n            return p\n\n    search_root = Path(\"/kaggle/input\")\n    skip_dirs = {\"train_series\", \"test_series\", \".git\", \"__pycache__\"}\n    candidates = []\n    if search_root.exists():\n        base_depth = len(search_root.parts)\n        for root, dirs, files in os.walk(search_root):\n            depth = len(Path(root).parts) - base_depth\n            dirs[:] = [d for d in dirs if d not in skip_dirs and depth < 7]\n            if \"config.json\" not in files:\n                continue\n            root_path = Path(root)\n            has_weights = any(\n                (root_path / name).exists()\n                for name in (\"model.safetensors\", \"pytorch_model.bin\")\n            )\n            if not has_weights:\n                continue\n            try:\n                cfg = json.loads((root_path / \"config.json\").read_text())\n            except Exception:\n                continue\n            model_type = str(cfg.get(\"model_type\", \"\")).lower()\n            name_hint = str(root_path).lower()\n            if \"dinov2\" in model_type or \"dinov2\" in name_hint:\n                hidden = int(cfg.get(\"hidden_size\", 10_000))\n                candidates.append((abs(hidden - 384), root_path))\n    if not candidates:\n        raise FileNotFoundError(\n            \"DINOv2 weights were not found. Add facebook/dinov2-small as a Kaggle model input \"\n            \"or set CFG.dinov2_dir to its local folder.\"\n        )\n    return sorted(candidates, key=lambda x: x[0])[0][1]\n\n\nseed_everything(CFG.seed)\nBASE = find_competition_dir()\nWORK = Path(\"/kaggle/working\")\nWORK.mkdir(parents=True, exist_ok=True)\n\nprint(\"Device:\", CFG.device)\nprint(\"Visible GPUs:\", CFG.n_gpus, [torch.cuda.get_device_name(i) for i in range(CFG.n_gpus)])\nprint(\"Competition:\", BASE)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train = pd.read_csv(BASE / \"train.csv\")\ntrain_series = pd.read_csv(BASE / \"train_series.csv\")\ntest = pd.read_csv(BASE / \"test.csv\")\ntest_series = pd.read_csv(BASE / \"test_series.csv\")\nsample_submission = pd.read_csv(BASE / \"sample_submission.csv\")\n\n\ndef normalize_column_names(df):\n    df = df.copy()\n    df.columns = [str(col).replace(\"\\ufeff\", \"\").strip() for col in df.columns]\n\n    canonical = {\n        \"studyinstanceuid\": \"StudyInstanceUID\",\n        \"patientsex\": \"PatientSex\",\n        \"report\": \"Report\",\n        \"acl\": \"ACL\",\n        \"mcl\": \"MCL\",\n        \"medial meniscus\": \"Medial Meniscus\",\n        \"lateral meniscus\": \"Lateral Meniscus\",\n        \"medial oa\": \"Medial OA\",\n        \"lateral oa\": \"Lateral OA\",\n        \"pf oa\": \"PF OA\",\n        \"effusion\": \"Effusion\",\n        \"synovitis\": \"Synovitis\",\n        \"baker's\": \"Baker's\",\n        \"bakers\": \"Baker's\",\n        \"contusion\": \"Contusion\",\n        \"fracture\": \"Fracture\",\n    }\n\n    rename_map = {}\n    for col in df.columns:\n        key = re.sub(r\"\\s+\", \" \", col.lower()).strip()\n        if key in canonical and canonical[key] not in df.columns:\n            rename_map[col] = canonical[key]\n\n    return df.rename(columns=rename_map)\n\n\ntrain = normalize_column_names(train)\ntest = normalize_column_names(test)\ntrain_series = normalize_column_names(train_series)\ntest_series = normalize_column_names(test_series)\nsample_submission = normalize_column_names(sample_submission)\n\nrequired_train_columns = [\"StudyInstanceUID\", \"Report\"] + TARGETS\nmissing_train_columns = [col for col in required_train_columns if col not in train.columns]\nif missing_train_columns:\n    raise KeyError(\n        \"Missing required columns in train.csv: \"\n        + \", \".join(missing_train_columns)\n        + f\"\\nAvailable columns: {train.columns.tolist()}\"\n    )\n\nfor df in (train, test):\n    if \"PatientSex\" not in df.columns:\n        df[\"PatientSex\"] = \"\"\n\nfor col in TARGETS:\n    train[col] = pd.to_numeric(train[col], errors=\"coerce\")\n\nexact_mask = train[TARGETS].notna().values\nfully_labeled_mask = exact_mask.all(axis=1)\nany_labeled_mask = exact_mask.any(axis=1)\n\nprint(f\"Train studies: {len(train):,}\")\nprint(f\"Test studies:  {len(test):,}\")\nprint(f\"Fully labeled studies: {fully_labeled_mask.sum():,}\")\nprint(f\"Studies with at least one exact label: {any_labeled_mask.sum():,}\")\nprint(f\"Train series: {len(train_series):,}\")\nprint(f\"Test series:  {len(test_series):,}\")\nprint(\"Optional PatientSex present in source:\", train[\"PatientSex\"].astype(str).str.len().gt(0).any())\nprint(\"Train columns:\", train.columns.tolist())\n\npreview_columns = [\"StudyInstanceUID\", \"Report\"] + TARGETS\nif train[\"PatientSex\"].astype(str).str.len().gt(0).any():\n    preview_columns.insert(1, \"PatientSex\")\n\ndisplay(train.loc[fully_labeled_mask, preview_columns].head(3))","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1. Multilingual report weak supervision\n\nAbsence of a phrase is treated as **unknown**, not automatically negative. Explicit positives, explicit negatives, uncertainty language and structure-level “normal/intact” statements are handled separately. The rule model is then blended with a character/word TF-IDF classifier trained only on available exact labels.","metadata":{}},{"cell_type":"code","source":"def normalize_text(text):\n    text = \"\" if pd.isna(text) else str(text)\n    text = unicodedata.normalize(\"NFKD\", text)\n    text = \"\".join(ch for ch in text if not unicodedata.combining(ch))\n    text = text.lower()\n    text = text.replace(\"’\", \"'\").replace(\"–\", \"-\").replace(\"—\", \"-\")\n    text = re.sub(r\"\\s+\", \" \", text)\n    return text.strip()\n\n\nNEGATION = re.compile(\n    r\"\\b(?:no|not|without|absent|absence of|negative for|free of|\"\n    r\"sin|ausencia de|no se observa|no se evidencia|\"\n    r\"sem|ausencia|nao ha|nao evidencia|\"\n    r\"kein|keine|keinen|ohne|nicht nachweisbar|\"\n    r\"pas de|sans|absence de|aucun|aucune|\"\n    r\"senza|assenza di|non si osserva|\"\n    r\"geen|zonder|afwezig)\\b\"\n)\n\nUNCERTAIN = re.compile(\n    r\"\\b(?:possible|possibly|probable|probably|suspect|suspected|suggestive|\"\n    r\"cannot exclude|may represent|questionable|equivocal|\"\n    r\"posible|probable|sugestivo|sospecha|no se descarta|\"\n    r\"possivel|provavel|suspeita|\"\n    r\"moglich|verdacht|\"\n    r\"possible|probable|suspecte|\"\n    r\"possibile|probabile|sospetto)\\b\"\n)\n\nNORMAL_WORDS = re.compile(\n    r\"\\b(?:intact|normal|preserved|unremarkable|stable|\"\n    r\"integro|integros|conservado|conservados|normal|\"\n    r\"intacto|intactos|preservado|\"\n    r\"intakt|unauffallig|erhalten|\"\n    r\"intacte|intacts|normal|\"\n    r\"integro|integri|conservato)\\b\"\n)\n\nINJURY_WORDS = re.compile(\n    r\"\\b(?:tear|torn|rupture|ruptured|sprain|injury|disruption|insufficiency|\"\n    r\"rotura|ruptura|desgarro|lesion|esguince|distension|\"\n    r\"rottura|lesione|\"\n    r\"dechirure|rupture|entorse|lesion|\"\n    r\"riss|ruptur|zerrung|verletzung|\"\n    r\"scheur|ruptuur)\\w*\"\n)\n\nMENISCUS_TEAR_WORDS = re.compile(\n    r\"\\b(?:tear|torn|rupture|ruptured|fissure|cleavage|radial|bucket handle|root tear|\"\n    r\"rotura|ruptura|desgarro|fisura|lesion|\"\n    r\"rottura|fissurazione|lesione|\"\n    r\"dechirure|fissure|rupture|lesion|\"\n    r\"riss|ruptur|einriss|\"\n    r\"scheur|ruptuur)\\w*\"\n)\n\nOA_WORDS = re.compile(\n    r\"\\b(?:osteoarthrit|osteoarthros|arthros|gonarthros|degenerative joint|\"\n    r\"cartilage loss|chondral loss|joint space narrowing|chondromalacia|\"\n    r\"artrosis|osteoartritis|gonartrosis|degeneracion condral|condropatia|\"\n    r\"osteoartrose|artrose|condropatia|\"\n    r\"arthrose|gonarthrose|knorpelschaden|\"\n    r\"arthrose|chondropathie|\"\n    r\"artrosi|gonartrosi|condropatia)\\w*\"\n)\n\nALIASES = {\n    \"ACL\": re.compile(\n        r\"\\b(?:acl|anterior cruciate ligament|ligamento cruzado anterior|\"\n        r\"ligamento crociato anteriore|ligament croise anterieur|\"\n        r\"vorderes kreuzband|voorste kruisband)\\b\"\n    ),\n    \"MCL\": re.compile(\n        r\"\\b(?:mcl|medial collateral ligament|ligamento colateral medial|\"\n        r\"ligamento collaterale mediale|ligament collateral medial|\"\n        r\"mediales kollateralband|mediale band)\\b\"\n    ),\n    \"Medial Meniscus\": re.compile(\n        r\"\\b(?:medial menisc\\w*|menisc\\w* medial\\w*|menisco medial|\"\n        r\"menisco interno|menisque medial|innenmeniskus|menisco mediale)\\b\"\n    ),\n    \"Lateral Meniscus\": re.compile(\n        r\"\\b(?:lateral menisc\\w*|menisc\\w* lateral\\w*|menisco lateral|\"\n        r\"menisco externo|menisque lateral|aussenmeniskus|menisco laterale)\\b\"\n    ),\n    \"Medial OA\": re.compile(\n        r\"\\b(?:medial compartment|compartimento medial|compartimento interno|\"\n        r\"femorotibial medial|mediales kompartiment|compartiment medial)\\b\"\n    ),\n    \"Lateral OA\": re.compile(\n        r\"\\b(?:lateral compartment|compartimento lateral|compartimento externo|\"\n        r\"femorotibial lateral|laterales kompartiment|compartiment lateral)\\b\"\n    ),\n    \"PF OA\": re.compile(\n        r\"\\b(?:patellofemoral|patello femoral|femoropatellar|femoro patelar|\"\n        r\"femororotulian|retropatellar|femoropatellaire|femoropatellare)\\w*\"\n    ),\n    \"Effusion\": re.compile(\n        r\"\\b(?:joint effusion|effusion|hydarthrosis|derrame articular|derrame|\"\n        r\"liquido articular|derrame articular|erguss|gelenkerguss|\"\n        r\"epanchement|versamento articolare)\\b\"\n    ),\n    \"Synovitis\": re.compile(\n        r\"\\b(?:synovitis|sinovitis|synovite|synovial inflammation|\"\n        r\"synoviale entzundung)\\b\"\n    ),\n    \"Baker's\": re.compile(\n        r\"\\b(?:baker'?s? cyst|popliteal cyst|quiste de baker|quiste popliteo|\"\n        r\"cisto de baker|kyste de baker|baker zyste|cisti di baker)\\b\"\n    ),\n    \"Contusion\": re.compile(\n        r\"\\b(?:bone contusion|bone bruise|marrow contusion|traumatic marrow edema|\"\n        r\"contusion osea|contusao ossea|edema ose\\w*|edema de medula|\"\n        r\"contusion osseuse|oedeme osseux|knochenmarkodem|knochenkontusion)\\b\"\n    ),\n    \"Fracture\": re.compile(\n        r\"\\b(?:fracture|fractura|fratura|fraktur|frattura)\\w*\"\n    ),\n}\n\nGROUP_NORMAL = {\n    \"ACL\": re.compile(r\"\\b(?:cruciate ligaments?|ligamentos cruzados|kreuzbander)\\b\"),\n    \"MCL\": re.compile(r\"\\b(?:collateral ligaments?|ligamentos colaterales|kollateralbander)\\b\"),\n    \"Medial Meniscus\": re.compile(r\"\\b(?:menisci|meniscos|menisken|menisques)\\b\"),\n    \"Lateral Meniscus\": re.compile(r\"\\b(?:menisci|meniscos|menisken|menisques)\\b\"),\n}\n\n\ndef split_report(text):\n    text = normalize_text(text)\n    parts = re.split(r\"(?<=[\\.\\!\\?;:])\\s+|\\n+\", text)\n    return [p.strip() for p in parts if p.strip()]\n\n\ndef local_window(sentence, start, end, radius=90):\n    lo = max(0, start - radius)\n    hi = min(len(sentence), end + radius)\n    return sentence[lo:hi]\n\n\ndef target_evidence(sentence, target):\n    alias = ALIASES[target]\n    evidences = []\n\n    for m in alias.finditer(sentence):\n        window = local_window(sentence, m.start(), m.end())\n        negated = bool(NEGATION.search(window))\n        uncertain = bool(UNCERTAIN.search(window))\n        normal = bool(NORMAL_WORDS.search(window))\n\n        if target in (\"ACL\", \"MCL\"):\n            pathology = bool(INJURY_WORDS.search(window))\n        elif target in (\"Medial Meniscus\", \"Lateral Meniscus\"):\n            pathology = bool(MENISCUS_TEAR_WORDS.search(window))\n        elif target in (\"Medial OA\", \"Lateral OA\", \"PF OA\"):\n            pathology = bool(OA_WORDS.search(window))\n        else:\n            pathology = True\n\n        if negated or (normal and not pathology):\n            evidences.append((0.03, 0.95))\n        elif pathology:\n            if uncertain:\n                evidences.append((0.68, 0.45))\n            else:\n                base_conf = 0.82 if target == \"Contusion\" else 0.95\n                evidences.append((0.97, base_conf))\n\n    if target in GROUP_NORMAL:\n        group = GROUP_NORMAL[target]\n        for m in group.finditer(sentence):\n            window = local_window(sentence, m.start(), m.end())\n            if NORMAL_WORDS.search(window) or NEGATION.search(window):\n                evidences.append((0.03, 0.85))\n\n    if target in (\"Medial OA\", \"Lateral OA\", \"PF OA\"):\n        tri = re.search(r\"\\b(?:tricompartmental|tri-compartmental|tricompartimental)\\b\", sentence)\n        if tri and OA_WORDS.search(sentence):\n            evidences.append((0.97, 0.90))\n\n    return evidences\n\n\ndef report_rule_scores(report):\n    sentences = split_report(report)\n    probs = np.full(len(TARGETS), 0.5, dtype=np.float32)\n    confs = np.zeros(len(TARGETS), dtype=np.float32)\n\n    for j, target in enumerate(TARGETS):\n        candidates = []\n        for i, sentence in enumerate(sentences):\n            for p, c in target_evidence(sentence, target):\n                # Findings near the end often occur in the impression/conclusion.\n                position_bonus = 0.05 * (i / max(1, len(sentences) - 1))\n                candidates.append((min(1.0, c + position_bonus), p))\n        if candidates:\n            candidates.sort(reverse=True)\n            confs[j], probs[j] = candidates[0]\n    return probs, confs\n\n\nreports = train[\"Report\"].fillna(\"\").astype(str).tolist()\nrule_probs = np.zeros((len(train), len(TARGETS)), dtype=np.float32)\nrule_conf = np.zeros_like(rule_probs)\n\nfor i, report in enumerate(tqdm(reports, desc=\"Rule pseudo-labels\")):\n    rule_probs[i], rule_conf[i] = report_rule_scores(report)\n\nrule_rows = []\nfor j, target in enumerate(TARGETS):\n    known = train[target].notna().values\n    covered = known & (rule_conf[:, j] >= 0.50)\n    if covered.sum() > 0:\n        pred = (rule_probs[covered, j] >= 0.5).astype(int)\n        truth = train.loc[covered, target].astype(int).values\n        acc = float((pred == truth).mean())\n    else:\n        acc = np.nan\n    rule_rows.append({\n        \"target\": target,\n        \"exact_n\": int(known.sum()),\n        \"rule_covered\": int(covered.sum()),\n        \"rule_accuracy_on_covered\": acc\n    })\n\ndisplay(pd.DataFrame(rule_rows))","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"normalized_reports = [normalize_text(x) for x in reports]\n\ntext_vectorizer = FeatureUnion([\n    (\"char\", TfidfVectorizer(\n        analyzer=\"char_wb\", ngram_range=(3, 5), min_df=2,\n        max_features=80_000, sublinear_tf=True, dtype=np.float32\n    )),\n    (\"word\", TfidfVectorizer(\n        analyzer=\"word\", ngram_range=(1, 2), min_df=1,\n        max_features=60_000, sublinear_tf=True, dtype=np.float32\n    )),\n])\n\nX_text = text_vectorizer.fit_transform(normalized_reports)\nprint(\"TF-IDF shape:\", X_text.shape)\n\n\ndef fit_text_probabilities(exclude_indices=None, compute_oof=False):\n    exclude = np.zeros(len(train), dtype=bool)\n    if exclude_indices is not None:\n        exclude[np.asarray(exclude_indices, dtype=int)] = True\n\n    text_probs = np.zeros((len(train), len(TARGETS)), dtype=np.float32)\n    text_oof = np.full_like(text_probs, np.nan)\n    text_auc = {target: np.nan for target in TARGETS}\n\n    for j, target in enumerate(TARGETS):\n        idx = np.where(train[target].notna().values & (~exclude))[0]\n        y = train.loc[idx, target].astype(int).values\n\n        if len(idx) == 0:\n            text_probs[:, j] = 0.5\n            text_auc[target] = np.nan\n            continue\n\n        if len(np.unique(y)) < 2:\n            text_probs[:, j] = float(y.mean())\n            text_auc[target] = np.nan\n            continue\n\n        if compute_oof:\n            n_splits = min(3, int(np.bincount(y).min()))\n            if n_splits >= 2:\n                skf = StratifiedKFold(\n                    n_splits=n_splits,\n                    shuffle=True,\n                    random_state=CFG.seed + j\n                )\n                for tr_local, va_local in skf.split(idx, y):\n                    tr_idx = idx[tr_local]\n                    va_idx = idx[va_local]\n                    model = LogisticRegression(\n                        C=4.0, solver=\"liblinear\", class_weight=\"balanced\",\n                        max_iter=2000, random_state=CFG.seed + j\n                    )\n                    model.fit(\n                        X_text[tr_idx],\n                        train.loc[tr_idx, target].astype(int).values\n                    )\n                    text_oof[va_idx, j] = model.predict_proba(X_text[va_idx])[:, 1]\n\n                valid_oof = ~np.isnan(text_oof[idx, j])\n                if valid_oof.sum() and len(np.unique(y[valid_oof])) == 2:\n                    text_auc[target] = roc_auc_score(\n                        y[valid_oof],\n                        text_oof[idx[valid_oof], j]\n                    )\n                else:\n                    text_auc[target] = np.nan\n            else:\n                text_auc[target] = np.nan\n\n        final_model = LogisticRegression(\n            C=3.5, solver=\"liblinear\", class_weight=\"balanced\",\n            max_iter=8000, random_state=CFG.seed + j\n        )\n        final_model.fit(X_text[idx], y)\n        text_probs[:, j] = final_model.predict_proba(X_text)[:, 1]\n\n    return text_probs, text_oof, text_auc\n\n\ndef make_supervision(exclude_indices=None, compute_oof=False):\n    text_probs, text_oof, text_auc = fit_text_probabilities(\n        exclude_indices=exclude_indices,\n        compute_oof=compute_oof\n    )\n\n    rule_mix = (0.25 + 0.65 * rule_conf).clip(0.25, 0.90)\n    pseudo_probs = rule_mix * rule_probs + (1.0 - rule_mix) * text_probs\n\n    distance = np.abs(pseudo_probs - 0.6)\n    pseudo_mask = (\n        ((rule_conf >= 0.6) & (distance >= CFG.rule_threshold)) |\n        ((rule_conf < 0.6) & (distance >= CFG.text_threshold))\n    )\n\n    pseudo_weight = (\n        CFG.min_pseudo_weight\n        + 0.55 * (2.0 * distance)\n        + 0.25 * rule_conf\n    ).clip(CFG.min_pseudo_weight, CFG.max_pseudo_weight).astype(np.float32)\n\n    targets = pseudo_probs.astype(np.float32)\n    mask = pseudo_mask.astype(np.float32)\n    weight = (pseudo_weight * mask).astype(np.float32)\n\n    exact_values = train[TARGETS].fillna(0.5).values.astype(np.float32)\n    targets[exact_mask] = exact_values[exact_mask]\n    mask[exact_mask] = 1.0\n    weight[exact_mask] = 1.0\n    return targets, mask, weight, text_oof, text_auc\n\n\ntargets_all, target_mask_all, target_weight_all, text_oof, text_auc = make_supervision(\n    exclude_indices=None,\n    compute_oof=False\n)\n\nif any(np.isfinite(text_auc.get(target, np.nan)) for target in TARGETS):\n    display(pd.DataFrame({\n        \"target\": TARGETS,\n        \"text_oof_auc\": [text_auc.get(target, np.nan) for target in TARGETS]\n    }))\nelse:\n    print(\"Text OOF AUC skipped in turbo mode (compute_oof=False).\")\n\ncoverage = target_mask_all.mean(axis=0)\nsummary = pd.DataFrame({\n    \"target\": TARGETS,\n    \"exact_count\": exact_mask.sum(axis=0),\n    \"training_coverage\": coverage,\n    \"soft_positive_rate\": [\n        np.average(\n            targets_all[:, j],\n            weights=np.maximum(target_weight_all[:, j], 1e-6)\n        )\n        for j in range(len(TARGETS))\n    ]\n})\ndisplay(summary)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fully_idx = np.where(fully_labeled_mask)[0]\nprint(\"Fully labeled studies used in final training:\", len(fully_idx))\nprint(\"Turbo mode trains one final model on all available exact and pseudo labels.\")\n","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Turbo DICOM mosaic and truncated dual-GPU DINOv2\n\nThe previous loader opened the metadata header of every DICOM in every selected series before choosing slices. That is accurate but dominated runtime on Kaggle storage.\n\nTurbo mode deliberately uses an approximate filename order and reads only one file from each selected plane. The three normalized grayscale planes become the R/G/B channels of one 112×112 image. This preserves all three anatomical planes while requiring only one encoder pass per study.","metadata":{}},{"cell_type":"code","source":"try:\n    import cv2\nexcept ImportError:\n    cv2 = None\n\n\ndef series_score(row):\n    fluid = float(row.get(\"Fluid_Sensitive\", 0) or 0)\n    fat = float(row.get(\"Fat_Suppression\", 0) or 0)\n    return 2.5 * fluid + 1.5 * fat\n\n\ndef build_plane_series(df_series):\n    result = {}\n    for study_id, rows in tqdm(\n        df_series.groupby(\"StudyInstanceUID\", sort=False),\n        total=df_series[\"StudyInstanceUID\"].nunique(),\n        desc=\"Selecting plane series\"\n    ):\n        study_planes = {}\n        plane_values = rows[\"Anatomical_Plane\"].astype(str).str.lower()\n\n        for plane in PLANES:\n            part = rows[plane_values == plane.lower()]\n            if len(part) == 0:\n                continue\n\n            scores = part.apply(series_score, axis=1)\n            best = part.loc[scores.idxmax()]\n            study_planes[plane] = str(best[\"SeriesInstanceUID\"])\n\n        result[str(study_id)] = study_planes\n    return result\n\n\ndef fast_middle_path(folder):\n    try:\n        paths = [\n            Path(entry.path)\n            for entry in os.scandir(folder)\n            if entry.is_file() and entry.name.lower().endswith(\".dcm\")\n        ]\n    except Exception:\n        return None\n\n    if not paths:\n        return None\n\n    paths.sort(key=lambda p: p.name)\n    return paths[len(paths) // 2]\n\n\ndef read_dicom_plane(path):\n    if path is None:\n        return None\n\n    try:\n        ds = pydicom.dcmread(str(path), force=True)\n        arr = ds.pixel_array.astype(np.float32)\n    except Exception:\n        return None\n\n    slope = float(getattr(ds, \"RescaleSlope\", 1.0) or 1.0)\n    intercept = float(getattr(ds, \"RescaleIntercept\", 0.0) or 0.0)\n    arr = arr * slope + intercept\n\n    if str(getattr(ds, \"PhotometricInterpretation\", \"\")).upper() == \"MONOCHROME1\":\n        arr = arr.max() - arr\n\n    finite = arr[np.isfinite(arr)]\n    if finite.size == 0:\n        return None\n\n    nonzero = finite[np.abs(finite) > 1e-8]\n    source = nonzero if nonzero.size >= 64 else finite\n    lo, hi = np.percentile(source, [1.0, 99.0])\n\n    if not np.isfinite(lo) or not np.isfinite(hi) or hi <= lo:\n        lo, hi = float(finite.min()), float(finite.max())\n\n    arr = np.nan_to_num(arr, nan=lo, posinf=hi, neginf=lo)\n    arr = np.clip(arr, lo, hi)\n    arr = (arr - lo) / max(hi - lo, 1e-6)\n    return arr.astype(np.float32)\n\n\ndef crop_resize_plane(arr, size):\n    mask = arr > 0.03\n    h, w = arr.shape\n\n    if mask.sum() > 0.02 * h * w:\n        ys, xs = np.where(mask)\n        y0, y1 = int(ys.min()), int(ys.max()) + 1\n        x0, x1 = int(xs.min()), int(xs.max()) + 1\n        my = max(2, int(0.06 * (y1 - y0)))\n        mx = max(2, int(0.06 * (x1 - x0)))\n        arr = arr[\n            max(0, y0 - my):min(h, y1 + my),\n            max(0, x0 - mx):min(w, x1 + mx)\n        ]\n\n    h, w = arr.shape\n    side = max(h, w)\n    padded = np.zeros((side, side), dtype=np.float32)\n    oy = (side - h) // 2\n    ox = (side - w) // 2\n    padded[oy:oy + h, ox:ox + w] = arr\n\n    if cv2 is not None:\n        interpolation = cv2.INTER_AREA if side > size else cv2.INTER_LINEAR\n        resized = cv2.resize(padded, (size, size), interpolation=interpolation)\n    else:\n        tensor = torch.from_numpy(padded)[None, None]\n        resized = F.interpolate(\n            tensor, size=(size, size), mode=\"bilinear\", align_corners=False\n        )[0, 0].numpy()\n\n    return np.clip(resized * 255.0, 0, 255).astype(np.uint8)\n\n\nclass StudyDicomDataset(Dataset):\n    def __init__(self, studies, plane_series, series_root):\n        self.studies = studies.reset_index(drop=True)\n        self.plane_series = plane_series\n        self.series_root = Path(series_root)\n\n    def __len__(self):\n        return len(self.studies)\n\n    def __getitem__(self, idx):\n        row = self.studies.iloc[idx]\n        study_id = str(row[\"StudyInstanceUID\"])\n        selected = self.plane_series.get(study_id, {})\n\n        mosaic = np.zeros((3, CFG.img_size, CFG.img_size), dtype=np.uint8)\n        valid_planes = 0\n\n        for channel, plane in enumerate(PLANES):\n            series_id = selected.get(plane)\n            if series_id is None:\n                continue\n\n            folder = self.series_root / study_id / series_id\n            path = fast_middle_path(folder)\n            arr = read_dicom_plane(path)\n            if arr is None:\n                continue\n\n            mosaic[channel] = crop_resize_plane(arr, CFG.img_size)\n            valid_planes += 1\n\n        images = mosaic[None, None]\n        token_mask = np.asarray([[valid_planes > 0]], dtype=np.bool_)\n\n        sex = str(row.get(\"PatientSex\", \"\")).strip().lower()\n        sex_id = 1 if sex.startswith(\"m\") else 2 if sex.startswith(\"f\") else 0\n\n        return (\n            torch.from_numpy(images),\n            torch.from_numpy(token_mask),\n            torch.tensor(sex_id, dtype=torch.long),\n            study_id\n        )\n\n\ntrain_slots = build_plane_series(train_series)\ntest_slots = build_plane_series(test_series)\n\nplane_counts = {\n    plane: sum(plane in item for item in train_slots.values())\n    for plane in PLANES\n}\ndisplay(pd.DataFrame({\n    \"plane\": list(plane_counts),\n    \"train_studies_with_plane\": list(plane_counts.values())\n}))\n","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from transformers import AutoModel\n\n\nclass DinoCLS(nn.Module):\n    def __init__(self, backbone):\n        super().__init__()\n        self.backbone = backbone\n\n    def forward(self, pixel_values):\n        out = self.backbone(pixel_values=pixel_values)\n        return out.last_hidden_state[:, 0]\n\n\ndef load_dinov2():\n    model_dir = find_dinov2_dir()\n    backbone = AutoModel.from_pretrained(\n        str(model_dir),\n        local_files_only=True,\n        trust_remote_code=False\n    )\n    hidden = int(backbone.config.hidden_size)\n\n    original_layers = None\n    if hasattr(backbone, \"encoder\") and hasattr(backbone.encoder, \"layer\"):\n        original_layers = len(backbone.encoder.layer)\n        keep = min(CFG.dino_layers, original_layers)\n        backbone.encoder.layer = nn.ModuleList(\n            list(backbone.encoder.layer)[:keep]\n        )\n        backbone.config.num_hidden_layers = keep\n\n    for p in backbone.parameters():\n        p.requires_grad = False\n\n    model = DinoCLS(backbone).eval().to(CFG.device)\n    if CFG.n_gpus > 1:\n        model = nn.DataParallel(model, device_ids=list(range(CFG.n_gpus)))\n\n    print(\"DINOv2:\", model_dir)\n    print(\"Hidden size:\", hidden)\n    print(\"DINO blocks:\", CFG.dino_layers, \"/\", original_layers)\n    print(\"Feature extraction GPUs:\", max(1, CFG.n_gpus))\n    print(\"Turbo input:\", CFG.img_size, \"one 3-plane mosaic per study\")\n    return model, hidden\n\n\n@torch.inference_mode()\ndef encode_chunks(model, images):\n    mean = torch.tensor(\n        [0.485, 0.456, 0.406], device=CFG.device\n    ).view(1, 3, 1, 1)\n    std = torch.tensor(\n        [0.229, 0.224, 0.225], device=CFG.device\n    ).view(1, 3, 1, 1)\n\n    outputs = []\n    for start in range(0, len(images), CFG.feature_batch_size):\n        x = images[start:start + CFG.feature_batch_size].to(\n            CFG.device, non_blocking=True\n        ).float().div_(255.0)\n        x = (x - mean) / std\n\n        with torch.autocast(\n            device_type=\"cuda\",\n            dtype=torch.float16,\n            enabled=CFG.amp\n        ):\n            feat = model(x)\n\n        outputs.append(feat.float().cpu())\n    return torch.cat(outputs, dim=0)\n\n\ndef extract_features(studies, slots, series_root, prefix, model, hidden_size):\n    stem = f\"{prefix}_{CFG.feature_cache_tag}\"\n    feat_path = WORK / f\"{stem}_features.npy\"\n    mask_path = WORK / f\"{stem}_token_mask.npy\"\n    sex_path = WORK / f\"{stem}_sex.npy\"\n    ids_path = WORK / f\"{stem}_ids.npy\"\n    done_path = WORK / f\"{stem}.done\"\n\n    feature_dim = hidden_size\n    expected_shape = (len(studies), CFG.n_slots, CFG.n_slices, feature_dim)\n    expected_mask_shape = expected_shape[:-1]\n\n    cache_files = (feat_path, mask_path, sex_path, ids_path)\n    cache_ok = all(p.exists() for p in cache_files) and done_path.exists()\n\n    if cache_ok and not CFG.rebuild_features:\n        try:\n            cached = np.load(feat_path, mmap_mode=\"r\")\n            cached_mask = np.load(mask_path, mmap_mode=\"r\")\n            cached_sex = np.load(sex_path, mmap_mode=\"r\")\n            cached_ids = np.load(ids_path, allow_pickle=True)\n\n            valid_cache = (\n                cached.shape == expected_shape\n                and cached_mask.shape == expected_mask_shape\n                and cached_sex.shape == (len(studies),)\n                and len(cached_ids) == len(studies)\n            )\n            if valid_cache:\n                print(f\"Using completed cached {prefix} features:\", cached.shape)\n                return feat_path, mask_path, sex_path, ids_path\n        except Exception as exc:\n            print(f\"Ignoring invalid {prefix} cache:\", repr(exc))\n\n    # A shape-correct .npy can still be incomplete after a crashed DataLoader.\n    # Only the .done marker makes a cache reusable.\n    done_path.unlink(missing_ok=True)\n\n    ds = StudyDicomDataset(studies, slots, series_root)\n    workers = max(1, min(CFG.feature_workers, os.cpu_count() or CFG.feature_workers))\n    batch_size = CFG.feature_loader_batch_size\n\n    print(\n        f\"{prefix}: thread-based DICOM loading \"\n        f\"({workers} threads, no multiprocessing queues)\"\n    )\n\n    features = np.lib.format.open_memmap(\n        feat_path, mode=\"w+\", dtype=np.float16, shape=expected_shape\n    )\n    masks = np.lib.format.open_memmap(\n        mask_path, mode=\"w+\", dtype=np.bool_, shape=expected_mask_shape\n    )\n    sexes = np.lib.format.open_memmap(\n        sex_path, mode=\"w+\", dtype=np.int8, shape=(len(studies),)\n    )\n    ids = np.empty(len(studies), dtype=object)\n\n    def safe_get(index):\n        try:\n            return ds[index]\n        except Exception as exc:\n            row = studies.iloc[index]\n            study_id = str(row[\"StudyInstanceUID\"])\n            images = torch.zeros(\n                CFG.n_slots, CFG.n_slices, 3,\n                CFG.img_size, CFG.img_size,\n                dtype=torch.uint8\n            )\n            token_mask = torch.zeros(\n                CFG.n_slots, CFG.n_slices, dtype=torch.bool\n            )\n            return images, token_mask, torch.tensor(0, dtype=torch.long), study_id\n\n    cursor = 0\n    starts = range(0, len(ds), batch_size)\n\n    with ThreadPoolExecutor(max_workers=workers) as pool:\n        for start in tqdm(\n            starts,\n            total=math.ceil(len(ds) / batch_size),\n            desc=f\"{prefix} threaded turbo features\"\n        ):\n            stop = min(start + batch_size, len(ds))\n            batch_indices = list(range(start, stop))\n\n            if workers == 1:\n                items = [safe_get(i) for i in batch_indices]\n            else:\n                items = list(pool.map(safe_get, batch_indices))\n\n            images = torch.stack([item[0] for item in items], dim=0)\n            token_mask = torch.stack([item[1] for item in items], dim=0)\n            sex_id = torch.stack([item[2] for item in items], dim=0)\n            study_ids = [item[3] for item in items]\n\n            b, s, n = token_mask.shape\n            valid_flat = token_mask.reshape(-1).bool()\n            batch_feat = torch.zeros(\n                b * s * n, feature_dim, dtype=torch.float32\n            )\n\n            if valid_flat.any():\n                flat_images = images.reshape(\n                    b * s * n, 3, CFG.img_size, CFG.img_size\n                )\n                encoded = encode_chunks(model, flat_images[valid_flat])\n                batch_feat[valid_flat] = encoded\n\n            end = cursor + b\n            features[cursor:end] = batch_feat.reshape(\n                b, CFG.n_slots, CFG.n_slices, feature_dim\n            ).numpy().astype(np.float16)\n            masks[cursor:end] = token_mask.numpy()\n            sexes[cursor:end] = sex_id.numpy().astype(np.int8)\n            ids[cursor:end] = np.asarray(\n                [str(x) for x in study_ids], dtype=object\n            )\n            cursor = end\n\n            if CFG.debug and cursor >= 32:\n                break\n\n    features.flush()\n    masks.flush()\n    sexes.flush()\n    np.save(ids_path, ids)\n\n    del features, masks, sexes\n\n    if cursor == len(studies):\n        done_path.write_text(\n            f\"rows={cursor}\\nshape={expected_shape}\\n\",\n            encoding=\"utf-8\"\n        )\n        print(f\"Completed {prefix} feature cache:\", expected_shape)\n    else:\n        print(\n            f\"{prefix} cache is partial ({cursor}/{len(studies)}) \"\n            \"and will not be reused.\"\n        )\n\n    return feat_path, mask_path, sex_path, ids_path\n\n\ndinov2, dinov2_hidden = load_dinov2()\n\n\ntrain_feature_files = extract_features(\n    train, train_slots, BASE / \"train_series\",\n    \"train\", dinov2, dinov2_hidden\n)\ntest_feature_files = extract_features(\n    test, test_slots, BASE / \"test_series\",\n    \"test\", dinov2, dinov2_hidden\n)\n\ndel dinov2\ngc.collect()\nif torch.cuda.is_available():\n    torch.cuda.empty_cache()\n","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Tiny study MLP\n\nThere is only one pretrained feature vector per study, so target-token attention is unnecessary. A small normalized MLP predicts all twelve labels. Exact labels are strongly oversampled, while report-derived soft labels provide broad weak supervision.","metadata":{}},{"cell_type":"code","source":"class FeatureDataset(Dataset):\n    def __init__(\n        self, feature_files, indices,\n        labels=None, label_mask=None, label_weight=None,\n        exact_cells=None, train_mode=False\n    ):\n        feat_path, mask_path, sex_path, ids_path = feature_files\n        self.features = np.load(feat_path, mmap_mode=\"r\")\n        self.token_mask = np.load(mask_path, mmap_mode=\"r\")\n        self.sex = np.load(sex_path, mmap_mode=\"r\")\n        self.ids = np.load(ids_path, allow_pickle=True)\n        self.indices = np.asarray(indices, dtype=np.int64)\n        self.labels = labels\n        self.label_mask = label_mask\n        self.label_weight = label_weight\n        self.exact_cells = exact_cells\n        self.train_mode = train_mode\n\n    def __len__(self):\n        return len(self.indices)\n\n    def __getitem__(self, i):\n        idx = int(self.indices[i])\n        feat = torch.from_numpy(\n            np.asarray(self.features[idx], dtype=np.float32).copy()\n        )\n        token_mask = torch.from_numpy(\n            np.asarray(self.token_mask[idx], dtype=np.bool_).copy()\n        )\n        sex = torch.tensor(int(self.sex[idx]), dtype=torch.long)\n\n        if self.train_mode:\n            y = torch.from_numpy(self.labels[idx].astype(np.float32))\n            ym = torch.from_numpy(self.label_mask[idx].astype(np.float32))\n            yw = torch.from_numpy(self.label_weight[idx].astype(np.float32))\n            ex = torch.from_numpy(self.exact_cells[idx].astype(np.bool_))\n            return feat, token_mask, sex, y, ym, yw, ex, idx\n\n        return feat, token_mask, sex, str(self.ids[idx]), idx\n\n\nclass FastStudyMLP(nn.Module):\n    def __init__(self, feature_dim, n_targets=len(TARGETS)):\n        super().__init__()\n        h = CFG.hidden_dim\n        self.norm = nn.LayerNorm(feature_dim)\n        self.sex_embed = nn.Embedding(3, h)\n        self.backbone = nn.Sequential(\n            nn.Linear(feature_dim, h),\n            nn.GELU(),\n            nn.Dropout(CFG.dropout),\n            nn.Linear(h, h),\n            nn.GELU(),\n            nn.Dropout(CFG.dropout),\n        )\n        self.head = nn.Linear(h, n_targets)\n\n    def forward(self, features, token_mask, sex):\n        x = features[:, 0, 0]\n        valid = token_mask[:, 0, 0].float().unsqueeze(1)\n        x = self.backbone(self.norm(x))\n        x = x + self.sex_embed(sex)\n        x = x * valid\n        return self.head(x)\n\n\ndef compute_pos_weight(indices, targets, label_mask, label_weight):\n    y = targets[indices]\n    m = label_mask[indices]\n    w = label_weight[indices]\n    pos = (y * m * w).sum(axis=0)\n    neg = ((1.0 - y) * m * w).sum(axis=0)\n    pw = np.sqrt((neg + 1.0) / (pos + 1.0))\n    return torch.tensor(np.clip(pw, 0.75, 4.0), dtype=torch.float32, device=CFG.device)\n\n\ndef masked_bce(logits, targets, mask, weight, pos_weight):\n    loss = F.binary_cross_entropy_with_logits(\n        logits, targets, reduction=\"none\", pos_weight=pos_weight\n    )\n    effective = mask * weight\n    return (loss * effective).sum() / effective.sum().clamp_min(1.0)\n\n\ndef pairwise_ranking_loss(logits, targets, exact_cells):\n    losses = []\n    hard = targets > 0.5\n    for j in range(logits.shape[1]):\n        valid = exact_cells[:, j]\n        pos = logits[valid & hard[:, j], j]\n        neg = logits[valid & (~hard[:, j]), j]\n        if len(pos) and len(neg):\n            diff = pos[:, None] - neg[None, :]\n            losses.append(F.softplus(-diff).mean())\n    if not losses:\n        return logits.new_tensor(0.0)\n    return torch.stack(losses).mean()\n\n\n@torch.no_grad()\ndef evaluate(model, loader):\n    model.eval()\n    preds, truths = [], []\n    for feat, tm, sex, y, _, _, _, _ in loader:\n        feat = feat.to(CFG.device, non_blocking=True)\n        tm = tm.to(CFG.device, non_blocking=True)\n        sex = sex.to(CFG.device, non_blocking=True)\n        with torch.autocast(\"cuda\", dtype=torch.float16, enabled=CFG.amp):\n            logits = model(feat, tm, sex)\n        preds.append(torch.sigmoid(logits).float().cpu().numpy())\n        truths.append(y.numpy())\n    p = np.concatenate(preds)\n    y = np.concatenate(truths)\n    aucs = {}\n    for j, target in enumerate(TARGETS):\n        if len(np.unique(y[:, j])) == 2:\n            aucs[target] = roc_auc_score(y[:, j], p[:, j])\n        else:\n            aucs[target] = np.nan\n    macro = float(np.nanmean(list(aucs.values())))\n    return macro, aucs, p\n\n\n@torch.no_grad()\ndef update_ema(ema_model, model, decay):\n    ema_params = dict(ema_model.named_parameters())\n    model_params = dict(model.named_parameters())\n    for name, p in model_params.items():\n        ema_params[name].mul_(decay).add_(p.detach(), alpha=1.0 - decay)\n    ema_buffers = dict(ema_model.named_buffers())\n    model_buffers = dict(model.named_buffers())\n    for name, b in model_buffers.items():\n        ema_buffers[name].copy_(b)\n\n\nfeature_dim = dinov2_hidden\nprint(\"Study token feature dimension:\", feature_dim)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"all_indices = np.arange(len(train))\nexact_sample = exact_mask.any(axis=1)\n\ntrain_ds = FeatureDataset(\n    train_feature_files,\n    all_indices,\n    labels=targets_all,\n    label_mask=target_mask_all,\n    label_weight=target_weight_all,\n    exact_cells=exact_mask,\n    train_mode=True\n)\n\nsample_weights = np.ones(len(all_indices), dtype=np.float64)\nsample_weights[exact_sample] = CFG.exact_sample_boost\n\nsampler = WeightedRandomSampler(\n    sample_weights,\n    num_samples=max(len(all_indices), CFG.train_batch_size * CFG.steps_per_epoch),\n    replacement=True,\n    generator=torch.Generator().manual_seed(CFG.seed)\n)\n\ntrain_loader = DataLoader(\n    train_ds,\n    batch_size=CFG.train_batch_size,\n    sampler=sampler,\n    num_workers=0,\n    pin_memory=True,\n    drop_last=True\n)\n\nmodel = FastStudyMLP(feature_dim).to(CFG.device)\nema_model = deepcopy(model).eval()\nfor p in ema_model.parameters():\n    p.requires_grad = False\n\noptimizer = torch.optim.AdamW(\n    model.parameters(),\n    lr=CFG.lr,\n    weight_decay=CFG.weight_decay\n)\n\ntotal_steps = CFG.epochs * len(train_loader)\nscheduler = torch.optim.lr_scheduler.OneCycleLR(\n    optimizer,\n    max_lr=CFG.lr,\n    total_steps=total_steps,\n    pct_start=0.15,\n    anneal_strategy=\"cos\",\n    div_factor=10.0,\n    final_div_factor=30.0\n)\n\nscaler = torch.cuda.amp.GradScaler(enabled=CFG.amp)\npos_weight = compute_pos_weight(\n    all_indices,\n    targets_all,\n    target_mask_all,\n    target_weight_all\n)\n\nfor epoch in range(1, CFG.epochs + 1):\n    model.train()\n    running = 0.0\n\n    for step, batch in enumerate(train_loader):\n        feat, tm, sex, y, ym, yw, ex, _ = batch\n        feat = feat.to(CFG.device, non_blocking=True)\n        tm = tm.to(CFG.device, non_blocking=True)\n        sex = sex.to(CFG.device, non_blocking=True)\n        y = y.to(CFG.device, non_blocking=True)\n        ym = ym.to(CFG.device, non_blocking=True)\n        yw = yw.to(CFG.device, non_blocking=True)\n        ex = ex.to(CFG.device, non_blocking=True)\n\n        optimizer.zero_grad(set_to_none=True)\n\n        with torch.autocast(\"cuda\", dtype=torch.float16, enabled=CFG.amp):\n            logits = model(feat, tm, sex)\n            bce = masked_bce(logits, y, ym, yw, pos_weight)\n            rank = pairwise_ranking_loss(logits, y, ex)\n            loss = bce + CFG.rank_loss_weight * rank\n\n        scaler.scale(loss).backward()\n        scaler.unscale_(optimizer)\n        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n        scaler.step(optimizer)\n        scaler.update()\n        scheduler.step()\n        update_ema(ema_model, model, CFG.ema_decay)\n\n        running += float(loss.item())\n\n        if CFG.debug and step >= 2:\n            break\n\n    print(\n        f\"Epoch {epoch:02d}/{CFG.epochs} \"\n        f\"| loss={running / max(1, step + 1):.4f} \"\n        f\"| lr={optimizer.param_groups[0]['lr']:.2e}\"\n    )\n\nmodel_path = WORK / \"knee_turbo_mosaic_mlp.pt\"\nbest_state = {\n    key: value.detach().cpu().clone()\n    for key, value in ema_model.state_dict().items()\n}\n\ntorch.save({\n    \"state_dict\": best_state,\n    \"feature_dim\": feature_dim,\n    \"targets\": TARGETS,\n    \"cfg\": {\n        \"img_size\": CFG.img_size,\n        \"dino_layers\": CFG.dino_layers,\n        \"feature_cache_tag\": CFG.feature_cache_tag\n    }\n}, model_path)\n\nfold_models = [model_path]\nprint(\"Saved:\", model_path)\n\ndel model, ema_model, optimizer, scheduler, scaler\ngc.collect()\nif torch.cuda.is_available():\n    torch.cuda.empty_cache()\n","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"@torch.no_grad()\ndef predict_model(model, loader):\n    model.eval()\n    logits_all, ids_all = [], []\n    for feat, tm, sex, study_ids, _ in tqdm(loader, leave=False):\n        feat = feat.to(CFG.device, non_blocking=True)\n        tm = tm.to(CFG.device, non_blocking=True)\n        sex = sex.to(CFG.device, non_blocking=True)\n        with torch.autocast(\"cuda\", dtype=torch.float16, enabled=CFG.amp):\n            logits = model(feat, tm, sex)\n        logits_all.append(logits.float().cpu().numpy())\n        ids_all.extend(list(study_ids))\n    return np.concatenate(logits_all), ids_all\n\n\ntest_ds = FeatureDataset(\n    test_feature_files,\n    np.arange(len(test)),\n    train_mode=False\n)\ntest_loader = DataLoader(\n    test_ds,\n    batch_size=CFG.train_batch_size,\n    shuffle=False,\n    num_workers=0,\n    pin_memory=True\n)\n\nfold_logits = []\nprediction_ids = None\n\nfor fold_path in fold_models:\n    checkpoint = torch.load(fold_path, map_location=\"cpu\")\n    model = FastStudyMLP(checkpoint[\"feature_dim\"]).to(CFG.device)\n    model.load_state_dict(checkpoint[\"state_dict\"])\n    logits, ids = predict_model(model, test_loader)\n    fold_logits.append(logits)\n    if prediction_ids is None:\n        prediction_ids = ids\n    del model\n    gc.collect()\n    if torch.cuda.is_available():\n        torch.cuda.empty_cache()\n\nmean_logits = np.mean(fold_logits, axis=0)\ntest_probs = 1.0 / (1.0 + np.exp(-mean_logits))\n\npred_df = pd.DataFrame(test_probs, columns=TARGETS)\npred_df.insert(0, \"StudyInstanceUID\", prediction_ids)\n\nsubmission = sample_submission[[\"StudyInstanceUID\"]].merge(\n    pred_df, on=\"StudyInstanceUID\", how=\"left\"\n)\nfor target in TARGETS:\n    submission[target] = submission[target].fillna(0.5).clip(1e-5, 1 - 1e-5)\n\nsubmission.to_csv(WORK / \"submission.csv\", index=False)\nsubmission.to_csv(\"submission.csv\", index=False)\n\nprint(\"submission.csv:\", submission.shape)\ndisplay(submission.head())","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Runtime profile\n\nThe slow v3 pipeline used 12 image tokens per study, and every token required a 2.5D triplet. Before that, it opened the headers of every DICOM in each selected series to recover physical ordering.\n\nTurbo v4 instead performs:\n\n- three DICOM pixel decodes per study;\n- zero full-series header scans;\n- one encoder image per study;\n- four DINOv2 blocks instead of twelve;\n- one final MLP instead of a three-fold ensemble.\n\nThis is intended as a rapid iteration baseline. The next quality step is to increase `dino_layers` to 6 or use two mosaics per study after confirming the pipeline and submission format.\n\n## Stability update in v4.2\n\nKaggle's Python 3.12 environment can occasionally corrupt multiprocessing\n`DataLoader` queues, producing `Bad file descriptor` and semaphore errors.\nThis version uses `ThreadPoolExecutor` for parallel DICOM I/O and\n`num_workers=0` for the lightweight memmap loaders. No multiprocessing queue,\nsemaphore, or persistent worker is created.\n\nFeature caches are accepted only when a matching `.done` marker exists, so a\npartially written cache from an interrupted run cannot silently be reused.\n","metadata":{}}]}