Nachiket Midha

Despite growing interest in using LLMs to add robustness or reduce data-collection costs in social science experiments, their efficacy in conjoint design — an increasingly popular method in political science — remains underexplored. This paper addresses that gap by investigating whether synthetic agents can reproduce the multi-dimensional preference patterns that conjoint is designed to capture. It replicates published conjoint studies and systematically compares the results generated by synthetic agents with original human data along three dimensions: representational correspondence, inferential correspondence, and procedural stability. Our analysis evaluates the alignment of choice distributions as well as the statistical and substantive similarity of estimates. We find that synthetic agents often approximate marginal attribute-level distributions and sometimes recover the direction of AMCE/MM estimates, but they perform poorly on full joint profile distributions, individual-level choice alignment, precise effect magnitudes, subgroup heterogeneity, and stability across models. These findings suggest that the validity of synthetic participants should be considered claim-dependent and hierarchical: reproducing a published figure or obtaining strong sign agreement is evidence of aggregate output similarity, but not sufficient evidence of respondent replacement.

Conversational AI systems are capable of shifting political attitudes through persuasion. Recent large-scale experiments across multiple countries and hundreds of political issues show effect sizes exceeding those of traditional campaign advertising. This report diagnoses the structural reasons why current regulatory frameworks fail to address this threat. The EU's Artificial Intelligence Act relies on compute as its primary trigger for identifying systemically risky models, but for political persuasion, this is not a defensible proxy: persuasive capacity does not track model size, and the risks that matter most arise at the post-training and deployment stage, not during pretraining. We propose a benchmark-triggered, context-sensitive regulatory framework that assesses outputs directly and scales obligations with deployment contexts.

A follow-on project to earlier work on synthetic agents in conjoint experiments, examining how LLMs arrive at their responses when standing in for human participants in social-science conjoint designs. By replicating established conjoint experiments and probing the reasoning processes behind LLM-generated responses, this project asks what kind of reasoning — if any — underlies synthetic-agent behaviour, and what that implies for using LLMs as substitutes for human respondents.

Political and policy questions — concerning democratic reform, institutional breakdown, and crisis management — frequently hinge on counterfactual reasoning in settings where experimentation is infeasible. Existing methods face a persistent trade-off: qualitative approaches offer rich causal narratives but cannot scale, while quantitative methods generalise well but struggle with rare or path-dependent outcomes. This paper develops a methodological framework that treats LLM-based synthetic agents as controlled experimental environments for approximating counterfactual political realities, and proposes criteria for assessing the credibility and limits of simulation-generated counterfactuals.