¹Institute of Psychology, University College of Professional Education, Wroclaw, Poland, ²Department of Human Sciences, VIZJA University, Warsaw, Poland, ³Institute of Psychology, University of Wroclaw, Wroclaw, Poland

Background: Systematic reviews with meta-analyses (SR-MAs) of psychedelic-assisted psychotherapy (PAP) guide clinical and policy decisions. Their trustworthiness depends, in part, on how consistently different SR-MAs assess risk of bias (RoB) in the same underlying randomized controlled trials (RCTs), and how transparently those judgments are justified.

Aims & Objectives: To summarise (i) RoB assessment practices across included PAP SR-MAs and (ii) preliminary cross-review disagreement in published RoB judgments for overlapping psilocybin and 3,4-methylenedioxymethamphetamine (MDMA) PAP trials.

Method: We are conducting a systematic review of SR-MAs of adult PAP RCTs. At this stage, 17 SR-MAs have been included. We extracted whether each SR-MA performed trial-level RoB assessment, which tool version was used, and whether supporting comments or justifications were provided. For preliminary cross-review consistency analyses, we collated domain and overall RoB judgments for selected psilocybin and MDMA RCTs that appeared in multiple SR-MAs. Discordance was defined as the presence of ≥2 distinct judgment categories among SR-MAs for the same trial-by-domain or overall assessment, excluding “not reported” entries. These preliminary results are before double independent data extraction.

Results: Fourteen of 17 SR-MAs (82%) reported trial-level RoB assessments; 8 used RoB 1, 4 used RoB 2 (including one SR-MA that stated RoB 2 but applied RoB 1-like domains), and 2 used other tools. Supporting comments for RoB judgments were present in 2 of 14 SR-MAs (14%). In a preliminary subset of 8 MDMA PAP RCTs assessed with RoB 1 across up to five SR-MAs, seven trials had ≥2 ratings for most domains. Among 48 RCT-by-domain comparisons with ≥2 ratings, 22 (46%) were discordant. Discordance was most frequent in RoB 1 domain 6 (7/7 trials with ≥2 ratings), domain 7 (4/6), and domain 3 (4/7). For psilocybin, across 9 RCTs assessed with RoB 2 in three SR-MAs, one SR-MA rated all nine as low overall, while another SR-MA rated one RCT as high and one as some concerns. Three psilocybin RCTs had overlapping overall RoB 2 judgments from ≥2 SR-MAs; all three showed disagreement, including one trial rated low in one SR-MA and high in another.

Discussion & Conclusions: Preliminary findings indicate an under-recognized methodological vulnerability in PAP evidence syntheses: RoB is assessed but rarely justified, and published RoB judgments for the same RCT can differ substantially across SR-MAs. To our knowledge, this is the first PAP-focused overview to quantify cross-review discordance in trial RoB judgments extracted directly from SR-MAs. Full appraisals and independently verified cross-review RoB comparisons across all included SR-MAs will be presented at CINP 2026.

Funding: This research was funded by the National Science Centre, Poland (grant number 2021/41/B/HS6/02844).