TIMELINE of RANKING PROCESS
Nominations: Expert Panel – April 10 to May 31, 2013
In preparation for the 2013 Global Go To Think Tank Indexing process, a call for nominations was issued for qualified individuals to serve on the Regional, Functional, and Special Areas of Distinction Panels.
Round I: Nominations – August 9 to September 30, 2013
A call for Nominations is sent to 6500 plus think tanks and approximately 7500 plus journalists, public and private donors, and policy-makers from around the world. These nominations are tabulated and institutes with 10 or more nominations are included in the next step of the 2013 Think Tank Indexing process.
Round II: Peer/Expert Rankings October – November 2013
Think tanks with 10 or more nominations will be placed in an electronic ranking survey. A letter announcing the second round is emailed to all the think tanks, journalists, public and private donors, and policy maker groups who have agreed to participate in the process. The rankings are tabulated and the list of finalists is generated for the Expert Panel to review and make final selections. This year, Regional and Functional Expert Panels have been created for every category. These specialists will be consulted to help assure the quality and accuracy of the nominations before they are placed on the final rankings survey.
Round III: Expert Panel Selects 2013 Go To Think Tanks November – December 2013
The members of the Expert Panel receive information packets by email in order to facilitate the final selection process. Individuals who served on last year’s Expert Panel as well as those who have been nominated this year will be invited to serve on the 2013 Expert Panel. Experts from every region and functional area will be represented on the Expert Panel. Panelists will submit their rankings and recommendations by Friday, November 15, 2013.
Launch: 2013 Global Go To Think Tank Index Announced January 2014
The 2013 Global Go To Think Tank Index is announced at the United Nations in New York and at selected organizations in every region of the world.
DEFINING AND EVALUATING THINK TANKS
Think tanks are public policy research, analysis, and engagement organizations. They are organizations that generate policy-oriented research, analysis, and advice on domestic and international issues that enable policymakers and the public to make informed decisions about public policy issues. Think tanks may be affiliated with political parties, governments, interest groups, or private corporations or constituted as independent nongovernmental organizations (NGOs). These institutions often act as a bridge between the academic and policymaking communities, serving the public interest as an independent voice that translates applied and basic research into a language and form that is understandable, reliable, and accessible for policymakers and the public.
Structured as permanent bodies, in contrast with ad hoc commissions or research panels, think tanks devote a substantial portion of their financial and human resources to commissioning and publishing research and policy analysis in the social sciences: political science, economics, public administration, and international affairs. The major outputs of these organizations are books, monographs, reports, policy briefs, blogs, conferences, seminars, web-based reports and commentary, formal briefings and informal discussions with policymakers, government officials, and key stakeholders.
In an effort to help make sense of this highly diverse set of institutions we have created a typology that takes into consideration the comparative differences in political systems and civil societies around the world. While think tanks may perform many roles in their host societies, not all think tanks do the same things to the same extent. Over the last 85 years, several distinctive organizational forms of think tanks have come into being that differ substantially in terms of their operating styles, their patterns of recruitment, their aspirations to academic standards of objectivity and completeness in research and their engagement of policy makers, the press and the public. We believe, despite these differences, most think tanks tend to fall into the broad categories outlined below.
Think Tank Typology/Think Tank Affiliations
AUTONOMOUS AND INDEPENDENT: Significant independence from any one interest group or donor and autonomous in its operation and funding from government.
QUASI INDEPENDENT: Autonomous from government but an interest group (i.e. unions, religious groups, etc.), donor or contracting agency provides a majority of the funding and has significant influence over operations of the think tank.
UNIVERSITY AFFILIATED: A policy research center at a university.
POLITICAL PARTY AFFILIATED: Formally affiliated with a political party.
GOVERNMENT AFFILIATED: A part of the structure of government.
QUASI GOVERNMENTAL: Funded exclusively by government grants and contracts but not a part of the formal structure of government.
FOR PROFIT: Public policy research unit located within a corporation or operating as a free standing for-profit think tank.
Nominations and Ranking Criteria
It is essential that you consider a variety of criteria in making your decisions. These may include, but are not limited to:
2013 Global Go To Think Tank Index Nomination and Ranking Criteria
● The quality and commitment of the think tank’s leadership (chief executive and governing body). This involves effectively managing the mission and programs of the think tank, mobilizing the financial and human resources necessary to fulfill the mission and monitoring the quality, independence and impact of the think tank;
● The quality and reputation of the think tank’s staff. Ability to assemble a critical mass of highly skilled, experienced and productive scholars and analysts who are recognized as either emerging or established experts in their respective area of research;
●The quality and reputation of the research and analysis produced. The ability to produce high quality, rigorous, policy oriented research that is accessible to policymakers, media and the public;
● Ability to recruit and retain elite scholars and analysts;
● Academic performance and reputation. This involves the academic rigor associated with the research conducted. This includes formal accreditation of a think tank’s scholars and analysts, the number and type of scholarly publications produced such as: books, journals and conference papers and the number of presentations delivered at scholarly and other professional meeting and the number and type of citations of the think tanks scholars’ research in scholarly publications produced by other scholars;
● The quality, number, and reach of its publications;
● The impact of a think tanks research and programs on policymakers and other policy actors. Policy recommendations considered or actually adopted by policymakers, civil society or policy actors;
● Reputation with policymakers (name recognition associated with specific issues or programs, number of briefings and official appointments, number of policy briefs and white papers produced, legislative testimony delivered);
● A demonstrated commitment to producing independent research and analysis. This involves standards and policies for producing rigorous evidence based research and analysis that are posted and monitored by the organization, research teams and individual researchers. This includes disclosure of conflict of interest (financial, institutional or personal) and a commitment to nonpartisanship and established professional standards for research in the social sciences;
● Access to key institutions. The ability to reach and connect with key audiences and personnel such as government officials (elected and appointed), civil society, traditional and new media, and academia;
● Ability to convene key policy actors and to develop effective networks and partnerships with other think tanks and policy actors;
● Overall output of the organization (policy proposals, web visits, briefings, publications, interviews, conferences, staff nominated to official posts);
● Utilization of research, policy proposal and other products. The effective transmission and utilization of policy briefs, reports, policy recommendations and other products by policymakers and the policy community, number of current and former staff serving in advisory roles to policymakers, advisory commissions, etc., awards given to scholars for scholarly achievement or public service;
● Usefulness of organization’s information in public engagement, advocacy work, preparing legislation or testimony, preparing academic papers or presentations, conducting research or teaching;
● Ability to use electronic, print and the new media to communicate research and reach key audiences;
● Media reputation (number of media appearances, interviews and citations);
● Ability to use the Internet including social media tools, to engage with policymakers, journalists and the public;
● Web Site and Digital presence. The quality, accessibility, effective maintenance of the organization’s web presence, as well as, the quality and level Digital traffic and engagement. (quality, accessibility and navigability of web site, number of website visitors, page views, time spent on pages, “likes” or followers);
● Level, diversity and stability of funding.The ability of an organization to mobilize the necessary financial resources to support and sustain the think tank over time (endowment, membership fees, annual donations, government and private contracts, earned income);
● Effective management and allocation of financial and human resources. The ability of a think tank to effectively manage its money and people so that they produce high quality outputs that achieve maximum impact;
● Ability of the organization to effectively fulfill the terms of the gifts, grants and contracts from government(s), individuals, corporations and foundations who have provided the financial support to the think tank (financial stewardship);
● The organization’s ability to produce new knowledge, innovative policy proposals or alternative ideas on policy;
● Ability to bridge the gap between the academic and policymaking communities;
● Ability to bridge the gap between policymakers and the public;
● Ability to include new voices in the policymaking process;
● Ability of organization to be inscribed within issue and policy networks;
● Success in challenging the traditional wisdom of policymakers and in generating innovative policy ideas and programs.
● The impact on society. Direct relationship between the organization’s efforts in a particular area to a positive change in societal values such as significant changes in the quality of life within respective country (amounts of goods and services available to citizens, state of physical and mental health, quality of environment, quality of political rights, access to institutions)
Think Tank Assessment Tool
Clearly, assessing the impact of think tanks is not an easy endeavor to undertake given the various and conflicting actors, events, and politics involved in the policy making process. Despite the significant challenges in establishing a causal relationship between knowledge and policy, it is necessary for think tanks to understand and effectively respond to the growing chorus of questions being raised by donors, journalists, and the public about the role and influence of think tanks in civil societies and governments around the world.
Think tanks can employ a variety of metrics to assess the impact including such measures as an increase in research and analysis they produce as well as to account for their contributions to the policymaking environment and civil society. McGann’s recent (2008) research has focused on developing a comprehensive assessment tool for evaluating a think tank’s impact. The impetus for this research, in part, was the apparent confusion that exists about the differences between outputs and impacts. In various studies and surveys that McGann has conducted over the years, researchers and think tanks responded curiously when asked about impact on public policy and how they measure it. The overwhelming response was to provide a list of research outputs (number of books published, conference held, web hits, media appearances, etc.). Outputs, however, are not the only way to measure impact.
The metric provided below is designed to serve as a catalyst for a discussion on how to effectively measure the impact of think tanks. It is provided here as background for the think tank ranking process in the hopes that it will help clarify the distinction between outputs and impacts and provide a useful tool as you prepare your rankings. We ask that you consider the following indicators when contemplating the impact of think tanks:
• RESOURCE INDICATORS: Ability to recruit and retain leading scholars and analysts; the level, quality, and stability of financial support; proximity and access to decision-makers and other policy elites; a staff with the ability to conduct rigorous research and produce timely and incisive analysis; institutional currency; quality and reliability of networks; and key contacts in the policy academic communities, and the media
• UTILIZATION INDICATORS: Reputation as a “go-to” organization by media and policy elites in the country; quantity and quality of media appearances and citations, web hits, testimony before legislative and executive bodies; briefings, official appointments, consultation by officials or departments/agencies; books sold; reports distributed; references made to research and analysis in scholarly and popular publications and attendees at conferences and seminars organized
• OUTPUT INDICATORS: Number and quality of: policy proposals and ideas generated; publications produced (books, journal articles, policy briefs, etc.); news interviews conducted; briefings, conferences, and seminars organized; and staff who are nominated to advisory and government posts
• IMPACT INDICATORS: Recommendations considered or adopted by policymakers and civil society organizations; issue network centrality; advisory role to political parties, candidates, transition teams; awards granted; publication in or citation of publications in academic journals, public testimony and the media that influences the policy debate and decision-making; listserv and web site dominance; and success in challenging the conventional wisdom and standard operating procedures of bureaucrats and elected officials in the country
Beyond this quantitative assessment, an effective evaluation of impact should also involve NGOs, as well as members of the government and policymakers, to ascertain the degree to which they have utilized the grantee’s research output. This participation can be obtained through interviews, surveys, questionnaires, and focus group meetings, utilizing the Outcome Mapping which “moves away from assessing the products of an activity or a program to focus on changes in behaviors and relationships (outcomes) which can lead to changes.” Impact can be viewed as positive if it “changes the behavior, relationships, activities, or actions of the people, groups, and organizations with whom a program works directly.”
Although this qualitative assessment is essential because it recognizes that policy impact can be successfully achieved even if policy prescriptions are not directly translated into actual policy, we recommend that this assessment should be translated into numerical rankings, thereby allowing comparisons with baseline data for effective monitoring and evaluation in the future.