Business leaders and public policymakers worldwide continue to have high interest in identifying the conditions, investments, infrastructure, and policies likely to promote and hasten the pace of innovation by businesses and other economic actors. Increasingly, surveys of innovation are being developed and used to gauge the incidence of innovation across national economies and to identify the driving variables and relationships. The European Union has run its Community Innovation Survey (CIS) among member countries regularly since 1992. CIS-based surveys have also been implemented in many of the member countries of the Organisation for Economic Co-operation and Development (OECD) and, more recently, in China and some African countries. In the United States, the National Science Foundation?s new Business R&D and Innovation Survey (BRDIS), first fielded in 2009, includes questions on business innovation drawn from the CIS approach.
This is an opportune time to review the CIS approach from several perspectives: what are the CIS questions actually measuring; how well can these measures, transferred to the U.S and other national economies, yield robust statistical results and produce indicators that can be compared over time and with those of other countries using the CIS model; and what might also usefully be measured were new questions to be added.
Project Description
The proposal outlines a project to conduct an in-depth review of the questions in the CIS and other existing innovation surveys and to develop new and improved questions. The project would be conducted over two years, as part of an ongoing OECD taskforce on business R&D and innovation survey re-design.
The work program would include cognitive testing of existing questions in English and selected languages, as well as the development and testing of new questions in areas of growing interest in the study of innovation, such as the measurement of intangible assets for innovation. The result would be a set of carefully tested questions and guidelines for their use. These results could be used to extend the coverage of existing surveys, or for new surveys, to produce internationally and inter-temporally comparable indicators of innovation.
The project would be managed by the Secretariat of the OECD Working Party of National Experts on Science and Technology Indicators (NESTI) -- which includes the European Union and the United States and has observers from the African Union, Brazil, China, and the Russian Federation, the Ibero-American Network on Science and Technology Indicators (RICYT) and the UNESCO Institute of Statistics (UIS). Participants in this proposed activity would be invited to bring together present work on question testing and methodology, as applied to questions in innovation surveys, and to contribute to an agenda for new work.
Importance and Broader Significance
The intellectual merit of this work arises from the use of leading edge cognitive testing and questionnaire development methods and the application of such methods in the domain of innovation survey questions. Survey design techniques will also benefit from work on a survey instrument that is used in a wide variety of economies and cultures.
The work will give rise to a community of practice that will continue well beyond the end of the project -- as part of the NESTI membership, supported by the OECD. The impact of the project would be considerable. A better understanding of innovation and its impacts will provide input to policy development directed at fostering business innovation and entrepreneurship. The project?s final report will include detailed assessments of existing questions and will offer new or improved questions to advance the capture of information on innovation.
For NSF?s BRDIS, adding rigorously tested questions and implementing a reviewed structure will contribute a better understanding of business survey methods. And with wide dissemination, survey results will support public debate of the role of innovation in the U.S. economy and inform the development of better designed policies. The report will be also available for use internationally, including for survey programs in developing countries.
Understanding how innovation happens in firms and how it contributes to economic growth and prosperity is a major policy preoccupation. The worldwide statistical community, led by the OECD and Eurostat, agreed in the early 1990s to develop a statistical approach to support the measurement of innovation in firms. This framework, the Oslo Manual, has been used in more than 80 different countries. The framework has proved its value in the context of microeconomic analysis of business performance and plays an important role in the official statistics of several countries in the OECD and beyond. However, there are significant measurement and interpretation problems with the data, often resulting in some paradoxical results, especially at a macro level. Innovation can be a fuzzy and highly subjective concept. It therefore needs to be operationalized in a meaningful way for the users of the information but also to ensure that survey respondents understand what they are being asked about and provide the requested information in a consistent manner, so that appropriate comparisons and benchmarks can be made. For this particular reason, the Manual has been in constant evolution to ensure the robustness of the approach and the validity and internationally comparability of the results, building on a wide range of research and validation methodologies. This project has focused on the use of cognitive interviewing and testing methods as the research tool with which to carry out a diagnosis of the framework and make recommendations on how it should be improved. The use of cognitive testing techniques is not new to the field of innovation statistics. What is novel to this OECD-led project is that cognitive testing has been applied for the first time to both European and non-European countries within a same exercise. Another key novelty of this research project is that the discipline of cognitive interviews has been applied not only to specific questions but also to the basic concepts and definitions traditionally promoted by OECD and Eurostat. The OECD has coordinated a project involving consultants and representatives from volunteer OECD member countries who have co-developed survey protocols and applied them to firms within their countries. The project has helped retrieve valuable information which countries have hitherto kept to themselves. From a prior assumption that testing had been very limited, it was possible to show there were plenty useful experiences to draw upon, meriting further dissemination, particularly among new practitioners in the field. Through the process of developing and implementing in-depth interviews with business, the project has contributed to an improved understanding of how business managers, the target of innovation surveys, perceive and report innovation. The findings indicate that the framework has succeeded at some levels at capturing how firms perceive innovations but also highlight some significant degree of rejection and confusion about some concepts and their implementation in surveys. The project has also assessed a number of approaches for dealing with some of the limitations identified in the scoping interviews, postulating the use of new questions approach and even suggesting the use of vignette as possible mechanisms for addressing the inherent subjectivity of responses. Among the key findings, it is possible to highlight the following: The need to place more emphasis on new-to-market innovations, identifying the appropriate way of dealing with changes that are only new to the firm, which most interviewees did not describe as innovations. The need to address, and possibly, avoid the use of "significantly improved" as a qualifier of innovations. The need to address the artificial –in the view of managers– separation between some forms of process and organisational innovations. This appears to be achievable with a reconsolidation of some of the existing categories. The limited scope for collecting reliable data on innovation expenditures, especially on an itemised basis, and the case for redirecting efforts towards measurement of investment in knowledge while pursuing more in-depth qualitative reporting on those activities. The scope for introducing meaningful questions on the use of design, and drawing lessons about which are the attributes that define the functional, experiential, or aesthetic novelty of products, thus addressing one of the topics raised in the Oslo Manual which had not been implemented in surveys. While some of the conclusions and proposed future steps will be subject to re-interpretation in different contexts, it is expected that the outcomes of this project will be of high relevance to the international statistical community and in particular the organisations and authorities in charge of reviewing and updating official guidelines on the collection and reporting of innovation statistics. The findings contained in this report are intended as a contribution to future joint OECD/NESTI and Eurostat work on revision of the Oslo Manual. The opinions expressed in this report do not necessarily reflect those of the OECD or of the governments of its member countries.
