Drawing on Kane’s argument-approach to assessment validity and Toulmin’s later work on cosmopolitanism and diversity, this paper asks whose validity arguments and evidence count, and how they are reconciled and assembled. We present a case study of the OECD’s ‘PISA for Development’, to demonstrate that validity arguments are assembled, negotiated and transformed by the network of international actors. The case study is based on an International Large-Scale Assessment with multiple actors and contexts, but our argument has wider relevance across other large-scale language assessments. We claim that the challenge of assessment validity should not be to establish a single authoritative argument through the displacement of plural interpretations and uses. We argue that one of the tasks of an argument-based approach to validity should be to create a democratic space in which legitimately diverse arguments and intentions can be recognised, considered, assembled and displayed. We therefore suggest that 1) this socio-material practice of assembling validity should be integrated into validity theory and practice, and 2) the task of assembling validity should be informed by democratic principles of diversity and inclusion.