AutoMates Sign in

Appraise · Automate · Archive

AutoMates

An online platform for appraising how completely meta-analyses are reported, built on MATES, the published 14-item appraisal tool. Complete the checklist by hand or automatically, with every answer linked to the passage in the paper that supports it.

Open the platform Read the paper ↗
14
checklist items, each answered Yes, No or N/A
1,000
most cited meta-analyses in the corpus
100
papers in the reference set, appraised by hand
2026
Environment International, 207, 109935

The paper

Open access

MATES: A tool for appraising the completeness with which a meta-analysis has been reported. Morrison et al. (2026), Environment International.

doi.org/10.1016/j.envint.2025.109935

The protocol

Coming soon

The published protocol for the AutoMates study: the 1,000-paper corpus, the 100-paper reference set appraised by hand, and the automated filler validated against it.

The code

Private during development

The annotation workbench and the AutoMates pipeline, to be opened with the protocol.

github.com/mahizakir

01The platform

What AutoMates does

Three modules, one standard: each reads the same 14-item MATES specification, so a checklist filled by hand, by model, or found in the database is always directly comparable.

  1. 1

    An appraisal

    The 14 items answered Yes, No or Not applicable: by a reviewer in the workbench, or by the AutoMates model.

  2. 2

    The evidence

    Every answer linked to the passage that supports it: file, page and quotation.

  3. 3

    The database

    Every checklist searchable by item, answer, journal and year, and comparable across fillers.

Apply Live

Manual filler

Appraisers complete the MATES checklist by hand for any paper of interest. They read the manuscript, supplementary materials, code and data within a single interface, record Yes, No, or Not applicable for each of the 14 items, and link every decision to a highlighted passage stored with its file, page and quotation. This is the module used to generate the 100-paper reference set, and it will remain available for anyone wishing to appraise further papers manually.

Automate In development

Automatic filler

This module will run the finalised AutoMates pipeline on any uploaded paper, producing a model-generated, fully evidenced checklist in the same format as the manual output. Users will be able to submit their own papers, published or unpublished, for automated appraisal, and will receive a completed checklist along with the supporting evidence and confidence scores underlying each decision.

Database In development

Open checklist archive

This module will hold the completed checklists for the 1,000 most cited meta-analyses, alongside the 100-paper reference set, all openly accessible and searchable by item, journal, year or answer. An upload feature will let users who have completed their own MATES appraisals, whether manually or via the Automate module, contribute their checklists to the database, so the evidence grows over time and reporting completeness can be tracked across the field.

02The checklist

The 14 items

Each item is answered Yes, No or Not applicable, and each answer is tied to the passage that supports it. The wording below is the wording reviewers work against, read from the shared specification.

  1. 01

    Effect size statistic formula reporting

    Do the authors report the formula (or reference for the formula) for each effect size statistic/measure used (e.g., Zr, odds ratio, risk ratio, Hedges' g, Cohen's d, or log response ratio) in the manuscript or supplementary material?

  2. 02

    Model type reporting

    Do the authors report the type of the primary or first meta-analytic/meta-regression model (e.g., fixed, random, or mixed-effects models), either descriptively or in statistical notation in the manuscript or supplementary material?

  3. 03

    Effect size dependence (non-independence) reporting

    Do the authors report the dependency structure (e.g., multiplicity or repeated measure) of effect size estimates (e.g., acknowledge multiple effect sizes were taken from a single study) in the primary or first meta-analytic/meta-regression model in the manuscript or supplementary material?

  4. 04

    Software/programming language reporting

    Do the authors report the name or a reference to the software or programming language used to conduct the analyses (i.e., used to conduct the computations, statistical models, and inferences comprising the meta-analysis) in the manuscript or supplementary material?

  5. 05

    Number of included primary studies reporting

    Do the authors report the total number of unique primary studies (i.e., separate individual articles) included in the primary or first meta-analytic/meta-regression model in the manuscript or supplementary material?

  6. 06

    Number of included effect sizes reporting

    Do the authors report the total number of effect size estimates synthesised in the primary or first meta-analytic/meta-regression model in the manuscript or supplementary material?

  7. 07

    Pooled mean effect reporting

    Do the authors report the pooled mean effect estimate from the primary or first meta-analytic/meta-regression model and its associated error or precision (e.g., standard error, confidence intervals, prediction intervals) using a plot, table, or within text in the manuscript or supplementary material?

  8. 08

    Effect size weighting strategy reporting

    Do the authors report the strategy used to assign effect size weights (e.g., the sampling variance, sampling error, or equivalent) used in the primary or first meta-analytic/meta-regression model in the manuscript or supplementary material?

  9. 09

    Heterogeneity reporting

    Do the authors report an assessment of heterogeneity (e.g., Q-statistic, τ, I²) between effect size estimates for the primary or first meta-analytic/meta-regression model in the manuscript or supplementary material?

  10. 10

    Publication bias/small-study effect reporting

    Do the authors report a statistical or graphical assessment of publication bias/small-study effects in the manuscript or supplementary material?

  11. 11

    Sensitivity analysis reporting

    Do the authors report a sensitivity analysis or subset analysis in addition to the publication bias assessment (e.g., leave-one-out analysis, use of an alternative model, or within-study risk of bias assessment) in the manuscript or supplementary material?

  12. 12

    Code reporting

    Do the authors report any relevant code used to conduct the analyses (i.e., used to conduct the computations, statistical models, and inferences comprising the meta-analysis) in the manuscript, supplementary material, or in a referenced external repository?

  13. 13

    Data reporting/archiving

    Do the authors report the original source statistics (i.e., used to calculate effect sizes and conduct the meta-analysis) extracted from each eligible primary study (i.e., research paper) in the manuscript, supplementary material, or in a referenced external repository?

  14. 14

    Variable description reporting

    Do the authors report descriptions of the variables extracted from each eligible primary study (i.e., research paper) in the manuscript, supplementary material, or in a referenced external repository?

03How to cite

Morrison, K., Pottier, P., Pollo, P., Ricolfi, L., Williams, C., Yang, Y., et al. (2026). MATES: A tool for appraising the completeness with which a meta-analysis has been reported. Environment International, 207, 109935.

doi.org/10.1016/j.envint.2025.109935