Over a five year period, 34 in-depth surveys were conducted with over 90,000 survey respondents (though some individuals were respondents in more than one survey). Most of the policy positions considered were based on proposed legislation. Others were derived from proposals made by the president, or were items in budget proposals.
The surveys were not simply standard polls used by the media or in political polling. For many issues, average Americans have a lack of information and in some cases misinformation on key issues. They also have not heard key arguments on each side of the issues. Research shows that in these circumstances poll responses are not highly stable and subject to even minor variations in the wording of the questions. Thus, they are not a reliable source of direction for policymakers.
Therefore, the Program for Public Consultation developed a unique survey form for this project that goes well beyond standard polls. The aim of the process is to put respondents in the shoes of a policymaker; thus they are called policymaking simulations. In this process, respondents are:
- provided a briefing on the issue and the policy proposals under consideration;
- presented and asked to evaluate arguments for and against the policy proposal; and
- finally, asked for their recommendations.
In some cases, the final recommendations are simply a binary question, such as whether the respondent favors or opposes a proposed legislative action. In other cases, the respondent is not only given the option of making a change or not, but is given the ability to specify their level of preferred change on a continuum, such as raising a tax rate or a benefit level.
In some cases, respondents are also given feedback about the impact of their choices. For example, in making choices about budget items, they are immediately shown the impact of their choices on the budget deficit.
The content of the policymaking simulations are reviewed in advance by proponents and opponents of the policy options to ensure that the briefings are accurate and balanced and that the arguments presented are indeed the strongest ones being made by proponents and opponents.
The surveys were conducted online with samples of 2,000 or more provided by Nielsen Scarborough from their larger probability-based panel that was recruited by phone and mail.