Open, Reliable & Repeatable

name: title
layout: true
class: center
---
layout: false
count: false

.center[
<a href="https://oesteban.github.io/talks/202504-CDT-UCL/">
 <object type="image/svg+xml" data="images/qr-talk-url.svg" style="width: 20%"></object>
 
 https://oesteban.github.io/talks/202504-CDT-UCL/
</a>

## Open, Reliable, and Repeatable

### Foundations for Breakthroughs in Network Analyses

Oscar Esteban <<code><a href="mailto:phd@oscaresteban.es">phd@oscaresteban.es</a></code>>

#### i4health CDT Seminar April 2nd, 2025
]

???

---
name: newsection
layout: true

.perma-sidebar[

<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0; height: 20px; padding-top: 6px;" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a>
 Open, Reliable & Repeatable :: CDT 2/04/2025

]

---

# About me

.large[
**Data scientist** who plays with brain data & **open science** advocate.
]

.right-column3.center[
<a href="https://oesteban.github.io/talks/202504-CDT-UCL/">
 <object type="image/svg+xml" data="images/qr-talk-url.svg" style="width: 80%"></object>
 
 Link to slides
</a>
]

.pad-top.left-column3[
.people-table.larger[

| | |
|---:|---|
| ![oscar-esteban](https://www.axonlab.org/images/teampic/Oscar_HQD-1-square.jpg) | **Oscar Esteban** Research & Teaching FNS Fellow Head of [AxonLab](https://www.axonlab.org) Dept. of Radiology, CHUV |
]

.larger[
* PD (2020) @ Stanford University
* Ph.D. (2015) @ Universidad Politécnica de Madrid ESKAS (2012) @ EPFL
]

]

---

.boxed-content.large.no-bullet[
* .larger[**Context**]
 .indent[.gray-text[Structural and functional networks]]

* .larger[**The Human Connectome PHantom (HCPh)**]
 .indent[.gray-text[A New Standard for Dense, Multi-Modal Neuroimaging]]

* .larger[**Quality assurance & control (QA/QC)**]
 .indent[.gray-text[Opening windows into your data]]

* .larger[**Standardization**]
 .indent[.gray-text[Registered reports, SOPs & NiPreps]]
]

???

---
count:false

.boxed-content.large.no-bullet[
* .larger[**Context**]
 .indent[.gray-text[Structural and functional networks]]

* .larger.dim[**The Human Connectome PHantom (HCPh)**]
 .indent[.dimgray-text[A New Standard for Dense, Multi-Modal Neuroimaging]]

* .larger.dim[**Quality assurance & control (QA/QC)**]
 .indent[.dimgray-text[Opening windows into your data]]

* .larger.dim[**Standardization**]
 .indent[.dimgray-text[Registered reports, SOPs & NiPreps]]
]

???

---

# The research workflow of network analyses

.boxed-content[
<object type="image/svg+xml" data="../assets/neuroimaging-workflow-large.svg" style="width: 100%; padding-top: 20pt;"></object>

.center[
[Esteban et al., (2020)](http://doi.org/10.1038/s41596-020-0327-3);
[Niso et al., (2022)](https://doi.org/10.1016/j.neuroimage.2022.119623)
]
]

???

In broad strokes, this is the MRI-based network analysis pipeline.

Interestingly, it looks like any machine learning pipeline: we acquire data, perform quality control, preprocess it, define features, and then apply statistical modeling to extract insights.

For a deeper dive into the details of the neuroimaging worflow, I've participated in several efforts such as the two references given below.

---
count:false

# The research workflow of network analyses

.boxed-content[
<object type="image/svg+xml" data="../assets/neuroimaging-workflow-1.svg" style="width: 100%; padding-top: 20pt;"></object>

.center[
[Esteban et al., (2017)](https://doi.org/10.1371/journal.pone.0184661);
[Provins et al., (2023)](https://doi.org/10.3389/fnimg.2022.1073734); 
[Provins et al., (2024, *Stage 2 RR under review*)](https://doi.org/10.1101/2024.10.11.617777)
[Hagen et al., (2024, *under review*)](https://doi.org/10.1101/2024.10.21.619532)
]

.no-bullet[
* .larger[ Run the **MRI experiment** following Standard Operating Procedures (SOPs)]

* .larger[**Standardized data structure** (BIDS—Brain Imaging Data Structure)]

* .larger[ **QA/QC** (Quality assurance / control)]

]

???

The pipeline starts with data acquisition, management and quality assurance and control (QA/QC).

My early work on QA/QC applied machine learning to automatically assess MRI images of the brain, and raised awareness over issues like “site effects”—where data from different scanners or labs can systematically vary, much like “batch effects” in genomics.

One key principle in AI is that models are only as good as their data and neuroimaging is no exception.

This is why my group and I have been deeply involved in BIDS, produced version-controlled, machine-readable standard operating procedures, developed comprehensive QA/QC protocols and created ready-to-use applications to detect suboptimal data early.

---
count:false

# The research workflow of network analyses

.boxed-content[
<object type="image/svg+xml" data="../assets/neuroimaging-workflow-2.svg" style="width: 100%; padding-top: 20pt;"></object>

.center[
[Esteban et al., (2019)](https://doi.org/10.1038/s41592-018-0235-4);
[Ciric et al., (2022)](https://doi.org/10.1038/s41592-022-01681-2);
[Adebimpe et al., (2022)](https://doi.org/10.1038/s41592-022-01458-7)
]

.no-bullet[
* .larger[ Detection of **nuisance sources**]

* .larger[ Spatiotemporal **location** of signals]

* .larger[ Definition of **brain units** of analysis (regions)]

]
]

???

Next in the pipeline is preprocessing, which transforms raw data into something models can reliably interpret.

For example, it identifies signals of no interest for their cleaning or accounting within modeling.

In neuroimaging, preprocessing also deals with accurately locating signals and objects in space and time, and often the definition of relevant brain regions that focus the analysis.

A decade ago, we might have called it “feature engineering.”

Regardless of the term, it’s crucial: “preprocessing” is now a keyword in job posts at major players like Anthropic, DeepMind, and OpenAI.

---

# fMRIPrep: despite minimal, a complex workflow

.boxed-content.center[
<img src="https://github.com/oesteban/fmriprep/raw/f4c7a9804be26c912b24ef4dccba54bdd72fa1fd/docs/_static/fmriprep-21.0.0.svg" style="width: 100%; padding-top: 20pt" />
]

???

Here I'm showing a very much simplified overview of fMRIPrep's design.

Despite it being just "minimal" preprocessing and just one step of the pipeline, its complexity is evident.

Rather than delving into each of these steps, I'll just say that there's a current need to update and improve computer vision algorithms and approaches (segmentation, image registration, dimensionality reduction, etc.) with AI/ML all across the dataflow.

---

# The research workflow of network analyses

.boxed-content[
<object type="image/svg+xml" data="../assets/neuroimaging-workflow-3.svg" style="width: 100%; padding-top: 20pt;"></object>

.center[
[Thompson et al., (2021, *under review*)](https://doi.org/10.1101/2021.01.16.426941);
[Rodrigue et al., (2021)](https://doi.org/10.1016/j.bpsc.2020.12.002);
[Li et al., (2024)](https://doi.org/10.1038/s41562-024-01942-4)
]
]

.pull-right[
<img src="../assets/matrix-fc.png" alt="matrix-fc" style="width: 20%" />
 
**Functional Connectivity (FC)**
 
.gray-text[Synchronized BOLD co-variation between brain regions]
]

.pull-left.align-right[
<img src="../assets/matrix-sc.png" alt="matrix-fc" style="width: 20%" />
 
**Structural Connectivity (SC)**
 
.gray-text[Tracked water diffusion pathways between brain regions]

]

???

Once data are ready for analysis, we formalize connectivity as undirected acyclic graphs, represented with symmetric matrices.

In the case of structural connectivity, we use diffusion MRI to track pathways of white-matter bundles connecting brain regions.

For functional connectivity we measure synchronized functional MRI signal fluctuations sensitive to the dynamic oxygen content in blood.

I’ve been involved in several works applying structural and functional connectivity.

For instance, in Rodrigue 2021, we explored ML approaches to identify clinical biomarkers.

Unfortunately, we had to conclude that, despite promising data quality and reliability remain insufficient for robust biomarker discovery.

---

# Despite active research, scarce clinical application

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/neuroimaging-workflow-large.svg" style="width: 100%; padding-bottom: 10pt; padding-top: 20pt;"></object>
]

.no-bullet[
* .large[ Alarming **analytical variability**:]
 * .larger[Diffusion MRI [[Maier-Hein et al., (2015)](https://doi.org/10.1038/s41467-017-01285-x)]]
 * .larger[Functional MRI [[Botvinik-Nezer et al., (2020)](https://doi.org/10.1038/s41586-020-2314-9)]]

* .large[ **Like genomics** between HGP (2003) and NGS (2015)?]
]
]

???

Therefore, although simple on the surface, the workflow is highly-complex and each step introduces potential variability.

Complicating matters further, every lab around the world might use a slightly different pipeline.

Different tools, different parameters—lead to inconsistent results.

This fragmentation hinders progress and precludes between-study comparisons.

The consequence is an acute scarcity of real-world applications of connectivity analyses.

Let's compare to genomics.

Neuroimaging is today in an early stage of standardization akin to genomics at the conclusion of the Human Genome Project in 2003.

One key aspect for the remarkable advance of genomics is precisely the comprehensive standardization process of the pipeline that culminated some 10 years ago.

For that reason, I advocate for replicating this success experience in neuroscience.

---

# Analytical variability in dMRI

<figure style="width: 70%">
![:img ismrm-challenge, 100%](images/tractography-challenge.png)
<figcaption>From <a href="https://doi.org/10.1038/s41467-017-01285-x">Maier-Hein et al. 2017</a>;
doi:![:doi](10.1038/s41467-017-01285-x)</figcaption>
</figure>

---
count:false

# Analytical variability in dMRI

.vertical-center[
.center.white-bg[
.large[
***Tractography-based connectomes are dominated by false-positive connections***
]

Draft title (2015) - doi:[10.1101/084137](https://doi.org/10.1101/084137)

]
]
<div style="background-image: url(images/tractography-challenge.png); background-size: cover; height: 80%; width: 80%; margin-left: 180px; opacity: 0.1">
</div>

---

# Analytical variability in fMRI

<figure style="width: 65%">
![:img narps, 100%](images/narps-paper.png)
<figcaption>From <a href="https://doi.org/10.1038/s41586-020-2314-9">Botvinik-Netzer et al. 2020</a>;
doi:![:doi](10.1038/s41586-020-2314-9)</figcaption>
</figure>

---
count: false

# Despite active research, no real-world application

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/neuroimaging-workflow-large.svg" style="width: 100%; padding-bottom: 10pt; padding-top: 20pt;"></object>
]

* .large[ Insufficient **test-retest reliability** at the finer scales of brain units]
 * .larger[[Nakuci et al., (2023)](https://doi.org/10.1038/s41598-023-33441-3)]]
]

???

On top of all that methodological variability, there’s also inherent unreliability in measuring connectivity from session to session.

If you scan the same participant twice, back-to-back, you can still get different connectivity measurements, indicating limited test-retest reliability.

Despite all these concerns, more than 20 thousand papers were published focusing on functional connectivity and 10 thousand on structural connectivity in 2024, as per Pubmed.

---

.center[
<object type="image/svg+xml" data="../assets/reproducible-definition-grid.svg" style="width: 68%"></object>
.small[
*The Turing Way project* illustration by Scriberia. Used under a CC-BY 4.0 licence. doi:<a href="https://doi.org/10.5281/zenodo.3332807">10.5281/zenodo.3332807</a>.]
]

---

# More reliable measurements

.center[
.boxed-content[
<object type="image/svg+xml" data="../assets/reproducible-definition-grid-axes-0.svg" style="width: 65%;"></object>

[Esteban, (2025)](https://doi.org/10.1007/978-1-0716-4260-3_8) [[Methods for analyzing large neuroimaging datasets](https://doi.org/10.1007/978-1-0716-4260-3)]
]
]

---
count: false

# More reliable measurements

.center[
.boxed-content[
<object type="image/svg+xml" data="../assets/reproducible-definition-grid-axes-1.svg" style="width: 65%;"></object>

[Esteban, (2025)](https://doi.org/10.1007/978-1-0716-4260-3_8) [[Methods for analyzing large neuroimaging datasets](https://doi.org/10.1007/978-1-0716-4260-3)]
]
]

---
count: false

# More reliable measurements

.center[
.boxed-content[
<object type="image/svg+xml" data="../assets/reproducible-definition-grid-axes-3.svg" style="width: 65%;"></object>

[Esteban, (2025)](https://doi.org/10.1007/978-1-0716-4260-3_8) [[Methods for analyzing large neuroimaging datasets](https://doi.org/10.1007/978-1-0716-4260-3)]
]
]

---
count: false

# More reliable measurements

.center[
.boxed-content[
<object type="image/svg+xml" data="../assets/reproducible-definition-grid-axes-2.svg" style="width: 65%;"></object>

[Esteban, (2025)](https://doi.org/10.1007/978-1-0716-4260-3_8) [[Methods for analyzing large neuroimaging datasets](https://doi.org/10.1007/978-1-0716-4260-3)]
]
]

---
count:false

.boxed-content.large.no-bullet[
* .larger.dim[**Context**]
 .indent[.dimgray-text[Structural and functional networks]]

* .larger[**The Human Connectome PHantom (HCPh)**]
 .indent[.gray-text[A New Standard for Dense, Multi-Modal Neuroimaging]]

* .larger.dim[**Quality assurance & control (QA/QC)**]
 .indent[.dimgray-text[Opening windows into your data]]

* .larger.dim[**Standardization**]
 .indent[.dimgray-text[Registered reports, SOPs & NiPreps]]
]

???

Having deeply focused on the standardization to address analytical variability, in 2020 I started to work on the reliability problem of connectivity analysis.

Tat that moment, I identified that the neuroimaging community was lacking an ultra-dense, calibration-grade dataset to address the problem.

With the support of an Ambizione project, I launched the Human Connectome Phantom project.

The idea is to have a single subject—myself—scanned intensively within a very short time window to minimize brain changes (and hence, serve as a "phantom" reference).

The dataset needed to combine sophisticated functional and structural MR acquisitions, and a wealth of physiological tracking.

---

## HCPh: An ultra-dense, calibration dataset

.right-column3[
Avoid **between-subject** variability.

Short span of time to minimize brain changes (~8 weeks)

* Wave 1: **within-scanner variability**
* Wave 2: **between-scanner variability**

]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-00.svg" style="width: 90%;"></object>
]

???

As mentioned, I lent myself as the phantom given the hefty commitment the project involved and the total absence of antecedents of a human being scanned so frequently.

---
count:false

## HCPh: An ultra-dense, calibration dataset

.right-column3[
**Anatomy**:
 * T1-weighted & T2-weighted

**Diffusion MRI** (HARDI):

* 1.6 × 1.6 × 1.6 [mm3]
 * 6 × b = 0 s/mm2
 * 274 DWIs 137 DWIs non-collinear
 b={700,1000,2000,3000} s/mm2

**Functional MRI** (BOLD): (next slide)

***B0* fieldmaps**:

* GRE (2min 38s)
  * 4 × EPI/dMRI (54s)
  * 4 × EPI/fMRI (27s)
]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-01.svg" style="width: 90%;"></object>
]

???

First, our study focused on a single scanner at CHUV.

We collected a very comprehensive MRI protocol with technical sophistication to provide a solid foundation to secondary methodological developments.

---
count:false

## HCPh: An ultra-dense, calibration dataset

.right-column3[
**Functional MRI**:

* 2.2 × 2.2 × 2.2 [mm3]
 * TR = 1.6s
 * multi-echo (4 echos)
 * three tasks:
 * QC task (3min 7s)
 * naturalistic movie resting (20min 9s)
 * breath-holding task (6min 0s)

<video width="100%" autoplay loop muted>
 <source src="../assets/mundaka-clip.webm" type="video/webm">
 Your browser does not support the video tag.
</video>

]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-03.svg" style="width: 90%;"></object>
]

???
We collected eye-tracking to assess arousal and analyze correlation with other facets such as head motion.
We also obtained a wealth of signals to investigate physiological variability, including expired oxygen and CO2 (which is far from standard practice but needed).

---
count:false

## HCPh: An ultra-dense, calibration dataset

.right-column3[
Collected within **4 weeks**

Following stringent **Standard Operating Procedures (SOPs)**

.center[
<a href="https://www.axonlab.org/hcph-sops">
 <object type="image/svg+xml" data="images/qr-talk-url.svg" style="width: 80%"></object>
 
 Link to **public** SOPs
</a>
]

]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-04.svg" style="width: 90%;"></object>
]

???

This first wave of data was collected in about 3 weeks to ensure minimal changes on the human brain phantom.

---
count:false

## HCPh: An ultra-dense, calibration dataset

.right-column3[
<a href="https://www.axonlab.org/hcph-sops/data-management/intro/">Data Management (SOPs)</a>

* BIDS Conversion & Validation

* QA/QC with *MRIQC*

* DataLad: version control

]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-05.svg" style="width: 90%;"></object>
]

???

Data were then standardized, properly stored under version control, and quality-controlled.

---
count:false

## HCPh: An ultra-dense, calibration dataset

.right-column3[
<a href="https://www.axonlab.org/hcph-sops/processing/preprocessing/">Preprocessing (SOPs)</a> with *fMRIPrep* and *dMRIPrep*

]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-06.svg" style="width: 90%;"></object>
]

???

We preprocessed the data with NiPreps

---
count:false

## HCPh: An ultra-dense, calibration dataset

.right-column3[
<a href="https://www.axonlab.org/hcph-sops/processing/functional-connectivity/">Network extraction (SOPs)</a> (mostly *NiLearn*)
]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-07.svg" style="width: 90%;"></object>
]

???

Next comes connectivity extraction.

---
count:false

## HCPh: An ultra-dense, calibration dataset

.right-column3[
We collected 36 sessions
]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-08.svg" style="width: 90%;"></object>
]

???

And this process yielded at least 36 pairs of structural and functional matrices.

---
count:false

## HCPh: An ultra-dense, calibration dataset

.right-column3[
Objective: **testing generalization**

* Clinical-use scanners

* Homogenized protocol

]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-09.svg" style="width: 90%;"></object>
]

???

But we also wanted to test agreement between scanners.

To do so, we modified the protocol so that it would be suitable for the two clinical scanners with more limited performance.

We collected 12 more sessions on the research scanner with the homogenized protocol.

---
count:false

## HCPh: An ultra-dense, calibration dataset

.right-column3[
We collected 36 sessions: 12 sessions each scanner (x3)
]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-10.svg" style="width: 90%;"></object>
]

???

And 12 more sessions per scanner on the two clinical scanners.

---
count:false

## HCPh: An ultra-dense, calibration dataset

.right-column3[
Data will be released fully open (CC-BY), including derivatives.

]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-full.svg" style="width: 90%;"></object>
]

???

Finally, we committed to release all materials (data, code, operating procedures, etc.) open.

---

## HCPh: Registered-report driven

.right-column3[
.larger.center[
.large[**Stage 1 RR**] 
Nature Methods

<a href="https://doi.org/10.17605/OSF.IO/VAMQ6"><object type="image/svg+xml" data="../202405-ISMRM/images/qr-RR.svg" style="width: 85%"></object>
Provins et al., (2023)</a>
]
]

.left-column3[
<object type="image/svg+xml" data="../assets/hcph-full.svg" style="width: 90%;"></object>
]

???

The study was accepted in October 2023 as a Stage 1 Registered Report at Nature Methods.

The registered report, data collection, and the research was carried out by a large team, with the outstanding role of the Ph.D. student funded by the Ambizione (Celine Provins), and is accessible with the QR code.

By pre-registering, we locked in our hypotheses and methods up front and addressed design issues raised by three reviewers before data collection started.

---

# HCPh: Study Outcomes

.boxed-content.no-bullet.pad-top[

* .large.pad-top[ One-of-a-kind, open **dataset**]
 .indent[
 .gray-text[Thoroughly documented, with open [Standard Operating Procedures (SOPs)](https://www.axonlab.org/hcph-sops)]
 ]

* .large[ Within-scanner **reliability characterization**]
 .indent[
 .gray-text[PCA-based modeling of FC residuals]

.gray-text[Bayesian modeling of SC]
  ]

* .large[ Between-scanner **agreement**]
 .indent[
 .gray-text[One research scanner vs. two clinical scanners]
 ]

]

???

We are currently concluding the final Stage 2 reporting the results, to be submitted to the journal within the next 3-4 months.

We have collected and preprocessed a unique dataset and publicly released its comprehensive standard operating procedures.

Following our pre-registered commitments, we are currently executing our promised analysis involving principal components analyses of functional connectivity and bayesian modelling of structural connectivity.

Finally, we will investigate how structural and functional connectivity measurements agree if obtained with different devices.

This will yield a baseline measure of how stable connectivity metrics can be if the protocol is maximally controlled.

---

# HCPh: Significance

.boxed-content.no-bullet[

* .large.pad-top[ Complement standardization with **calibration**]
 .indent[
 .gray-text[Provide normative distributions of SC and FC (prior knowledge)]

.gray-text[An approach to design *calibrated* studies, potentially with the same or new **human phantoms**]
  ]

* .large[ Methodological **advance**]
 .indent[
 .gray-text[Removal of physiological variability, artifacts, etc.]

.gray-text[Between-scanner agreement of measurements (other than networks)]
  ]

* .large[ A **foundational dataset**]
 .indent[
 .gray-text[SC & FC training dataset]

.gray-text[Fully-open, ideal for courses and training]
  ]

]

???

But the impact of HCPh goes far beyond the analyses committed within the Stage 1 report.

We hope to see new “human phantom” datasets at multiple sites, enabling direct cross-site calibration.

HCPh can also serve as ground truth data for AI-driven motion-correction algorithms, generative modeling, or even real-time scanning protocols.

Essentially, we’re complementing standardization efforts (like NiPreps) with a calibration resource that addresses test-retest and multi-site reproducibility.

---
count:false

.boxed-content.large.no-bullet[
* .larger.dim[**Context**]
 .indent[.dimgray-text[Structural and functional networks]]

* .larger.dim[**The Human Connectome PHantom (HCPh)**]
 .indent[.dimgray-text[A New Standard for Dense, Multi-Modal Neuroimaging]]

* .larger[**Quality assurance & control (QA/QC)**]
 .indent[.gray-text[Opening windows into your data]]

* .larger.dim[**Standardization**]
 .indent[.dimgray-text[Registered reports, SOPs & NiPreps]]
]

???

---

# QA/QC of the neuroimaging worflow

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/neuroimaging-workflow-large.svg" style="width: 90%; padding-bottom: 55.5pt; padding-top: 120pt;"></object>
]

.align-right[
([Provins et al., 2023](http://doi.org/10.3389/fnimg.2022.1073734))
]
]

???

---
count:false

# QA/QC of the neuroimaging worflow

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/provins-qaqc-protocol-00.svg" style="width: 100%; padding-bottom: 5pt; padding-top: 40pt;"></object>
]

.align-right[
([Provins et al., 2023](http://doi.org/10.3389/fnimg.2022.1073734))
]
]

???

---
count:false

# QA/QC of the neuroimaging worflow

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/provins-qaqc-protocol-01.svg" style="width: 100%; padding-bottom: 5pt; padding-top: 40pt;"></object>
]

.align-right[
([Provins et al., 2023](http://doi.org/10.3389/fnimg.2022.1073734))
]
]

???

---
count:false

# QA/QC of the neuroimaging worflow

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/provins-qaqc-protocol-02.svg" style="width: 100%; padding-bottom: 5pt; padding-top: 40pt;"></object>
]

.align-right[
([Provins et al., 2023](http://doi.org/10.3389/fnimg.2022.1073734))
]
]

???

---
count:false

# QA/QC of the neuroimaging worflow

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/provins-qaqc-protocol-03.svg" style="width: 100%; padding-bottom: 5pt; padding-top: 40pt;"></object>
]

.align-right[
([Provins et al., 2023](http://doi.org/10.3389/fnimg.2022.1073734))
]
]

???

---
count:false

# QA/QC of the neuroimaging worflow

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/provins-qaqc-protocol-04.svg" style="width: 100%; padding-bottom: 5pt; padding-top: 40pt;"></object>
]

.align-right[
([Provins et al., 2023](http://doi.org/10.3389/fnimg.2022.1073734))
]
]

???

---
count:false

# QA/QC of the neuroimaging worflow

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/provins-qaqc-protocol-05.svg" style="width: 100%; padding-bottom: 5pt; padding-top: 40pt;"></object>
]

.align-right[
([Provins et al., 2023](http://doi.org/10.3389/fnimg.2022.1073734))
]
]

???

---
count:false

# QA/QC of the neuroimaging worflow

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/provins-qaqc-protocol.svg" style="width: 100%; padding-bottom: 5pt; padding-top: 40pt;"></object>
]

.align-right[
([Provins et al., 2023](http://doi.org/10.3389/fnimg.2022.1073734))
]
]

???

---

# QA/QC protocols: 'Swiss-cheese security model'

.boxed-content[
.center[
<object type="image/svg+xml" data="../assets/provins-qaqc-protocol.svg" style="width: 60%; padding-bottom: 10pt; padding-top: 10pt;"></object>

.small[
[BenAveling @ wikipedia](https://en.wikipedia.org/wiki/Swiss_cheese_model#/media/File:Swiss_cheese_model_textless.svg)
]
]

]

???

For those with knowledge about security protocols, this approach will surely evoke the Swiss cheese model.

The model assumes that all QC checkpoints will have holes through which data progresses toward analysis.

By layering several QC checkpoints looking at the data in different ways, we make sure that images with potential to bias the results do not make all the way through the workflow.

---

# Does defacing influence QA/QC decisions?

.right-column3[
.larger.center[
.large[**Stage 2 RR**] 
(nearly accepted)

Provins et al., (2025)</a>
]
]

.left-column3[
<img src="../assets/defacing-RR-figure1.png" style="width: 90%; padding-top: 25pt" /> 
]

???

---

# Does defacing influence QA/QC decisions?

.boxed-content[

.pull-right.center[
]

.pull-left[

.large[**IXI dataset** (N = 581; 3 sites)]
* **Manual QA/QC**: N = 185 (Hammersmith Hospital, 3T) 4 raters
* **Automated QA/QC**: Full dataset IQMs (image quality metrics)

.large[**Processing**:] 
MRIQC reports (**manual**) and IQMs (**automated**)
* before defacing (**nondefaced**)
* after defacing (**defaced**)

([Provins et al., 2025](https://doi.org/10.1101/2024.10.11.617777))
]

]

---

# Does defacing influence **manual** QA/QC decisions?

.boxed-content[

.pull-right.center[
<object type="image/svg+xml" data="../assets/defacing-RR-ba-traditional-minimal.svg" style="width: 100%; padding-top: 17pt"></object>
]

.pull-left[

.large[**4 raters** assessed 185 images x2]

.large[**Bland-Altman** analysis]

([Provins et al., 2025](https://doi.org/10.1101/2024.10.11.617777))
]

]

---
count: false

# Does defacing influence **manual** QA/QC decisions?

.boxed-content[

.pull-right.center[
<object type="image/svg+xml" data="../assets/defacing-RR-ba-traditional.svg" style="width: 100%; padding-top: 17pt"></object>
]

.pull-left[

.large[**4 raters** assessed 185 images x2]

.large[**Bland-Altman** analysis]

.larger[
* Large, asymmetric **variability** LoA = [-0.91, 0.81]

* **Negative**—albeit small—bias 
 𝔼[Δndef-def] = -0.05
]

([Provins et al., 2025](https://doi.org/10.1101/2024.10.11.617777))
]

]

---
count: false

# Does defacing influence **manual** QA/QC decisions?

.boxed-content[

.pull-right.center[
<img src="../assets/defacing-RR-ba-animated.gif" alt="defacing-RR" style="width: 100%;" />
]

.pull-left[

.large[**4 raters** assessed 185 images x2]

.large[**Bland-Altman** analysis]

.larger[
* Large, asymmetric **variability** LoA = [-0.91, 0.81]

* **Negative**—albeit small—bias 
 𝔼[Δndef-def] = -0.05
]

([Provins et al., 2025](https://doi.org/10.1101/2024.10.11.617777))
]

]

---

# Does defacing influence **manual** QA/QC decisions?

.boxed-content[

.pull-right.center[

]

.pull-left[

.large[
* **Overoptimistic** defaced

* concentrated at **lowest nondefaced**,

* **more extreme** &

* **more numerous**
]
]

]

---

# Does defacing influence QA/QC decisions?

.boxed-content.center[
<img src="../assets/defacing-RR-figure2.png" style="width: 90%; padding-top: 15pt;" />
]

???

---
count:false

.boxed-content.large.no-bullet[
* .larger.dim[**Context**]
 .indent[.dimgray-text[Structural and functional networks]]

* .larger.dim[**The Human Connectome PHantom (HCPh)**]
 .indent[.dimgray-text[A New Standard for Dense, Multi-Modal Neuroimaging]]

* .larger.dim[**Quality assurance & control (QA/QC)**]
 .indent[.dimgray-text[Opening windows into your data]]

* .larger[**Standardization**]
 .indent[.gray-text[Registered reports, SOPs & NiPreps]]
]

???

---

# **The paradox** (Prof. C. Chambers)

## Results determine careers

and

## Resarchers SHOULD NOT determine results.

---

# What are Registered Reports?

---

# What are Registered Reports?

---

# What are Registered Reports?

---

.vertical-center[
.center.white-bg[
.large[
**Results do not matter**
]
]
]

---
count: false

.vertical-center[
.center.white-bg[
.large[
**Results** [*whether positive or negative*] **do not matter**
]
]
]

---

# Why the Registered Report Approach

.boxed-content.large.no-bullet[

* .larger[**Pre-Approved Study Design**]
 .indent[.gray-text[Hypotheses, protocols, and analyses peer-reviewed before data collection]]

* .larger[**Transparency & Rigor**]
 .indent[.gray-text[Minimizes biases, “p-hacking,” and the garden of forking paths]]

* .larger[**Challenges**]
 .indent[.gray-text[Pre-registration report preparation & slow review process]]
 .indent[.gray-text[Deviations from the canonical confirmatory analysis]]

]

???

Registered Reports reduce biases and ensure transparency as it removes incentives to drive the results and interpretation.

Why? Because with the acceptance of the Stage 1, journals commit to publish the final work (or Stage 2) regardless of the final outcomes.

Given that the HCPh is intended as a foundational reference dataset, it was crucial that the design itself was peer-reviewed.

Registered reports are not free of caveats.

With the communal improvement of scientific endeavors, we will be able to shorten the preparation time of pre-registered analysis and ensure that peer-review does not excessively delays the onset of data collection.

---

# HCPh's SOPs

.boxed-content[
<iframe src="https://www.axonlab.org/hcph-sops/" width="100%" height="510px" style="border: 0; margin-top: 15px"></iframe>
.center.small[https://www.axonlab.org/hcph-sops/]
]

---

# Standardization in neuroimaging—NiPreps

.boxed-content[
 
.larger.center[
"*analysis-grade*" data data **directly consumable by analyses**
]

.pull-left[
 
 
*Analysis-grade* data is an analogy to the concept of "*sushi-grade (or [sashimi-grade](https://en.wikipedia.org/wiki/Sashimi)) fish*" in that both are:

.large[ **minimally preprocessed**,]

and

.large[ **safe to consume** directly.]
]

.pull-right.center[
<img src="../assets/nipreps-transparent.png" width="100%" />
<a href="https://www.nipreps.org"><object type="text/xml+svg" data="../assets/qr-nipreps-url.svg" style="width: 40%"></object> 
www.nipreps.org</a>
]
]

???

In computational neuroscience I'm recognized by my standardization efforts.

NiPreps is a framework for standardized preprocessing pipelines in neuroimaging.

I often use the “sushi-grade” analogy

just like sushi-grade fish is minimally processed but safe to consume,

NiPreps yields “analysis-grade” data, with minimal processing interventions but ready for direct machine learning ingestion.

---

# Tracking *fMRIPrep*'s adoption

.center[
<img src="images/20250204_weekly.png" style="width: 90%; padding-top: 10pt" /> 
]

---

# Tracking *fMRIPrep*'s usage

.larger[
 It's critical to report **exact versions**

Semantics inform about compatibility: **24.0.1**

Long-term support (LTS) program.

]

.center[
<img src="images/20250204_versionstream.png" style="width: 90%; padding-top: 10pt" /> 
]

---

# Standardization in neuroimaging—impact

.boxed-content.larger.no-bullet[

* .large[ **Not just wide adoption**]
 .indent[.gray-text[Used in large neuroscientific endeavors: ENIGMA]

.gray-text[Industry: [Flywheel's gears for *MRIQC* and *fMRIPrep*](https://flywheel.io/insights/blog/comprehensive-support-bids#:~:text=The%20Flywheel%20Gear%20Exchange%20currently,to%20download%20your%20entire), [Amazon Web Services featured *fMRIPrep*](https://aws.amazon.com/blogs/opensource/fmri-data-preprocessing-aws-fmriprep/#:~:text=itself,efforts%20on%20preprocessing%20fMRI%20data), [QMENTA's RS fMRI tool derived from *fMRIPrep*](https://catalog.qmenta.com/tool/resting-state-fmri#:~:text=Catalog%20catalog,Networks%3B%20%E2%80%A2%20Salience%20Network)
  ]
  .gray-text[Relevant works: [Caucheteux, Gramfort & King, (2023)](https://doi.org/10.1038/s41562-022-01516-2), [Tozzi et al., (2024)](https://doi.org/10.1038/s41591-024-03057-9), [Hyon et al., (2020)](https://doi.org/10.1073/pnas.2013606117), [Yamashita et al., (2020)](https://doi.org/10.1371/journal.pbio.3000966).
  ]
  ]

* .large[ **Built a Lasting Community**]
 .indent[
 .gray-text[100+ researchers engaged in the community worldwide] 
 .gray-text[Community is critical to provide a footing for standardization (following BIDS' experience)]
 ]

* .large[ **DevOps-inspired standardization**]
 .indent[
 .gray-text[Reducing the gap between research and application reversing the transference flow]
 ]
]

???

Beyond citations and the staggering usage figures I showed in my first presentation, NiPreps is a major achievement by DORA principles.

NiPreps is used in large consortia like ENIGMA, given a foundation to pioneering clinical applications such as Tozzi 2024, and is offered by major industry players like Flywheel, QMENTA, and even AWS workflows.

The NiPreps framework is driven by a community involving more than 100 scientist globally and their adoption continues growing.

That’s the real impact: it’s changing how people conduct imaging studies and ensuring higher data quality across labs.

One distinct reason for success is that we contributed to bridge the gap between research and real-world applicationi by reversing the typical knowledge transference flow:

We integrate software engineering best practices from devops and industry for the standardization and solidification of the neuroimaging workflow.

---

.center[
<img src="../assets/nipreps-chart.png" width="63%" /> 
NiPreps (<a href="https://doi.org/10.31219/osf.io/ujxp6">Esteban et al., 2020</a>)
]

???

Here’s the basic architecture.

It’s an ecosystem of modular tools at different levels of abstraction, offering space for AI/ML innovation all accross individual computer vision tasks.

---
count: false

# Thank you!

.right-column3.center[
<a href="https://oesteban.github.io/talks/202504-CDT-UCL/"><object type="image/svg+xml" data="images/qr-talk-url.svg" style="width: 60%"></object></a>

Oscar Esteban <<code><a href="mailto:phd@oscaresteban.es">phd@oscaresteban.es</a></code>>

**i4health CDT Seminar April 2nd, 2025**
]

.left-column3[
.center[
<a href="https://www.axonlab.org"><img style="width: 200px;" src="../assets/axonlab_logo.png" alt="AxonLab website" /></a>
<img src="https://www.axonlab.org/images/logopic/logo_unil_chuv_fbm.png" style="width:120px; padding-left: 20px" alt="CHUV & UNIL" />

The **HCPh team** 
The **NiPreps Community**

<img src="../assets/snsf_logo.png" style="width:180px;" alt="SNSF" />  
<img src="../assets/nimh_logo.png" style="width:180px;" alt="NIMH" />  
<img src="../assets/czi_logo.png" style="width:100px;" alt="CZI" />   
]
]

???