Appendix 1

Methodology

The value of the CEO Survey is that the methods are largely invariant over time, thus making it possible to uncover substantive changes in executive opinions. If the methodology changed each year, it would be unclear whether changes in findings were due to real changes in executive opinions or to changes in methodology. For that reason, much of the material in this appendix is very similar or identical to that contained in Appendix 1 of the 2008 report.

Qualitative Research

In fall the of 2006, MBA students at the Butler University College of Business conducted the qualitative research needed to focus and develop the study. These students analyzed two CEO weblogs hosted by Inside INdiana Business, observed a quarterly CEO round table discussion, and conducted in-depth interviews with Indiana CEOs. The information gathered was used to identify issues of concern to Indiana executives.

Simultaneously, students gathered secondary information about the Indiana business environment from a variety of governmental and private sources. This information included descriptive information of companies (e.g., number of employees, revenues, etc.); as well as information about industry classifications and geographic distribution of organizations.

Quantitative Research

A total of 1,913 CEOs and executives of Indiana-based organizations were identified and comprised the sample frame for this project. Potential respondents were identified from a variety of sources including a database maintained by Ice Miller, as well as a list of other potential respondents identified by members of the project steering committee. Inside INdiana Business contacted each potential respondent by e-mail, requested their participation in the project, and provided a link to the online survey.

Of the original sample frame, 181 of the e-mails distributed were returned as "undeliverable" for a wide variety of reasons including a non-functioning e-mail address, a full inbox, a "spam" filter bounce, etc. Additional contacts were directed to those who originally had undeliverable e-mails with functioning e-mail addresses. Of the 1,732 usable e-mail addresses, we obtained 360 responses yielding a response rate of 21 percent.

Most of the 360 who responded to the request to participate actually completed the survey. Three hundred and forty-six respondents answered all questions.

Several caveats are necessary for interpreting survey results. First, the original sample frame was not a complete and accurate listing of all CEOs of Indiana-based organizations and thus the resulting convenience sample may not provide an accurate representation of all CEOs of Indiana-based organizations. As the project continues to expand, the sampling frame will become more complete and thus the sample should become more representative over time.

Second, because the sample may not be technically representative in a statistical sense, computations of statistical significance are presented for illustrative purposes only. A formal discussion of statistical significance in this context appears below.

Finally, we have not conducted a formal assessment of nonresponse bias associated with the obtained sample, so those who responded may be systematically different from those who did not.

Despite the limitations of this third iteration of the Indiana CEO Survey, we believe the results will prove useful in multiple regards. First, the project provides a "snapshot" of issues of concern to Indiana’s corporate leaders. Thus, it can provide a platform for discussion and analysis of a wide variety of topics critical to the economic future of the state.

Second, while there may be large error ranges around reported parameter estimates, the relative rankings of key variables are probably accurate. For example, while the reported mean importance rating of 6.71 for "corporate reputation" may actually be higher or lower in the total population, it is clear that "corporate reputation" has a higher perceived importance level than does "natural resource prices" to Indiana CEOs and other executive officers.

Third, this is the third in a series of annual reports focused on Indiana CEOs. One key strength of this endeavor is the ability to track changes over time and thus to focus future discussions on trends rather than on one-time observations. This should make the project more valuable to policymakers and strategists.

Technical Notes on Statistical Significance

The term "statistical significance" is often misunderstood by managers and other policymakers. This misunderstanding seems to be rooted in two sources: confusion about the technical statistical meaning of the term and confusion surrounding the word "significance."

First, the term statistical significance simply refers to information obtained from a sample which we have reason to believe is different from information that we may have obtained by chance alone. For example, if we say that high revenue companies are statistically significantly more likely than are lower revenue companies to pursue alternative energy sources (Q2 in the survey), we are saying two things: 1) in our sample, the mean response for high revenue companies is higher than it is for low revenue companies; and 2) that this difference is likely due to a REAL difference between high and low revenue companies. In other words, we are not just "unlucky" in our choice of who to talk to in each group and therefore have obtained results which are really not true. Several factors influence statistical significance including how sure we want to be that we are finding real differences or real relationships, how large the sample is, and the actual survey results.

Statistical significance is thus about making inferences from a sample to a population. To make such inferences accurately, we need to have a randomly selected sample from a population of interest. In this project, our sample is not random because we did not have access to an accurate listing of the e-mail addresses of all Indiana CEOs (an accurate population listing). Thus, it is not possible to determine statistical significance in a formal sense. However, we have reported "statistical significance" for our results as if we had obtained a truly random sample in order to highlight results which we believe have a higher likelihood of being "real."

Second, there is a difference between "statistical significance" and "managerial significance." Managers often think that if a difference is "statistically significant," it is somehow necessarily important and deserving of managerial attention. This may or may not be true. We use the term "managerial significance" to mean that a manager ought to consider the information in making some decision. While a piece of information must be statistically significant before it can be considered to be managerially significant, not all statistically significant information is of managerial significance. For example, if we were using an extremely large sample, even small and relatively meaningless differences would be "statistically significant." In many circumstances, it would be a mistake for managers to make decisions based on such information because although it is "real," it is too small to be of practical value. In short, a piece of information needs to be statistically significant before it can be considered managerially significant, but not all statistically significant information is managerially significant. Managers must exercise judgment in deciding when to use or ignore statistically significant information.