Experiments Log

Running record of experiments on the living project — trigger, method, result, interpretation, confidence. Newest first.

2026-08-23 — South Africa (PRJNA723337): signal INVERSION (AUC 0.320, verified)

Trigger. Third plaque cohort, completing the geographic sweep (Africa gap — the only public African oral 16S T2DM data).

Method. PRJNA723337, 88 subgingival-plaque samples (57 diabetic [DM+Known_DM] / 31 control [Normal]; Pre-DM excluded). Single-end, deep (~227k reads/sample). Required --fastq_qmax 50 (SA reads hit Q43, above vsearch's default 41 — a real pipeline fix, now in the generic processor). Labels verified against NCBI BioSample isolate (spot-checked: DM_13→diabetic, Known_DM_23→diabetic, etc. — all correct). Same 139-feature → rank → CatBoost(existing pool) transfer test.

Result (verified inversion).

Four-cohort gradient (all new cohorts this session):

Interpretation. A real, verified systematic inversion — the extreme end of cross-population failure: the model's T2DM-associated features anti-correlate with T2DM in South Africa. The gradient above roughly tracks population distance from the US+Asian training set. BUT the cause is entangled and must not be overclaimed as biological: SA differs on four axes at once — population (African), specimen (plaque), sequencing depth (227k reads → far denser genus detection → a very different rank-profile distribution than the shallow training cohorts), and clinic composition. The compressed, uniformly-high predictions (0.83–0.88) point to a severe distribution shift where the rank transform interacts badly with the depth difference.

Confidence. Inversion is verified and systematic (labels correct, flip=0.68). Attribution is NOT established — do not claim "African oral T2DM signal is inverted."

Implication. (1) Naive cross-population application isn't just unreliable, it can be actively wrong — a strong argument for mandatory population-specific calibration before any deployment. (2) New concrete suspect: sequencing-depth harmonization. Testable next step — rarefy/normalize SA to comparable depth before the rank transform and re-run; if the inversion softens, depth (not population) is a major driver.

2026-08-23 — Mexico (PRJNA1208116) transfer test REFUTES the specimen hypothesis

Trigger. India (plaque) transferred only partially and naive pooling hurt; the leading hypothesis was that specimen mismatch (plaque vs saliva) was the culprit. Mexico is saliva — a specimen-matched, balanced, new-continent test to isolate population effect from the plaque confound.

Method. PRJNA1208116, 87 saliva samples (41 T2DM / 46 control; labels recovered from the paper's Supplementary Table S1, CC-BY). Same VSEARCH → 139-feature → per-sample rank → CatBoost(existing pool) transfer test. scripts/integrate_cohort.py.

Result.

Mexico had the best feature overlap (109/139) yet transferred worst — near random.

Interpretation (hypothesis refuted). Specimen match did NOT rescue transfer: saliva Mexico (0.564) transferred worse than plaque India (0.621). So specimen type is not the dominant barrier. BUT the result is confounded: Mexico is a 100%-periodontitis cohort (Severe 48 / Moderate 31 / Mild 21), so the T2DM contrast is within periodontitis — where the diabetes signal is subtle relative to the dominant periodontitis-driven microbiome shift — and this differs from the general-population (NHANES-heavy) training set. Two entangled explanations remain: (a) population distance (Latin American), (b) cohort composition (all-periodontitis clinic case-control vs general-population survey). One cohort cannot separate them.

Confidence. Result solid (near-random, n=87, balanced). Interpretation graded: specimen-hypothesis refutation is well-supported; population-vs-periodontitis remains open.

Implication (reframes the roadmap). The clinic-recruited, periodontitis-focused cohorts (India, Mexico) test a different, harder, confounded task than general-population T2DM screening. To measure pure population transfer, prioritize general-population survey cohorts (like NHANES itself, or the Qatar Biobank) over periodontitis clinic case-controls. This is now the sharper dataset-search criterion.

2026-08-23 — Does pooling India into training help the existing holdouts? (NO)

Trigger. After the India transfer test, ask the complementary question: does adding India to the training pool improve generalization to the current holdouts?

Method. Baseline CatBoost (existing pooled cohorts) vs. the same with India's 60 plaque samples appended to training (rank-transformed into the 139-feature space). Re-scored the Portugal and 3-cohort holdouts. scripts/experiment_india_in_pool.py.

Result (negative).

Interpretation. Naive pooling of the India plaque cohort degrades transfer to the (saliva) holdouts by 0.05–0.07 AUC. The specimen mismatch (plaque vs saliva) plus partial feature overlap acts as a domain shift that adds noise rather than useful diversity. Consistent with the project's recurring finding that naive cross-study pooling hurts generalization (prior domain-adaptation attempts also degraded holdout).

Confidence. Clear directional result — both holdouts drop materially and in the same direction. Caveat: single cohort, n=60, specimen-confounded.

Implication. Don't naively pool across specimen types. India (plaque) needs site-matching or explicit domain handling; specimen type is a confound to control before added geography can help. Motivates seeking a saliva cohort for new geography.

2026-08-23 — India/Mumbai (PRJNA1240053) cross-population transfer test

Trigger. Living-project goal "close the cross-population transfer gap": test whether the existing pooled model generalizes to a never-seen population.

Method. Downloaded the India/Mumbai plaque 16S cohort (60 samples, labels from SRA aliases: 40 diabetic [T2DM + T2DM+perio] / 20 non-diabetic [healthy + perio]). Processed through the same VSEARCH pipeline (merge → maxee 1.0/minlen 200 filter → 97% closed-ref vs HOMD v16.03 → genus). Mapped to the 139 model genera, applied the identical leakage-free per-sample rank transform, and scored with CatBoost (depth 4, iters 200, lr 0.1, balanced, seed 42) trained on the existing pooled cohorts. India was never seen in training — a clean LOSO-style holdout. Scripts: scripts/process_india_vsearch.py, scripts/integrate_india.py.

Result.

Only 76/139 model genera were observed in India (plaque community + region differences); the 63 absent genera tie at the bottom rank and add little signal.

Interpretation. Partial transfer — above the near-random cross-continental failure floor (0.541) and above chance, but below the European holdouts. The model retains some signal on a new population, which is notable given three stacked disadvantages: (1) never-seen population, (2) plaque, not saliva (specimen mismatch), (3) only 55% feature overlap. The plaque mismatch likely understates population transfer, so 0.621 is a conservative floor for "does the signal reach India."

Confidence. Suggestive, not definitive — n=60, single cohort, specimen mismatch. Not strong enough to claim India generalization; strong enough to justify adding saliva Indian data and testing whether site-matching lifts it.

Next. (a) Get a saliva Indian cohort to separate site effect from population effect. (b) Add India to the training pool and re-check whether it helps other holdouts (diversity gain). (c) Pending Mexico/South Africa labels for more geography.