51Quant. Methods 1 (M. Messner)

Concepts and their measurement

Pixabay.com

52Quant. Methods 1 (M. Messner)

Concepts and their measurement

• Research is usually interested in relationships between different phenomena or concepts.

– E.g., Intelligence, organizational performance, customer satisfaction, strategic position, etc.

– Some phenomena are directly observable and can thus be (rather) easily measured (e.g., age, industry affiliation, revenues, etc.)

– Other phenomena are much harder to capture.

• Construct validity is an important concern in quantitative research

53Quant. Methods 1 (M. Messner)

Concepts and their measurement

Concept measurement depends on the method of data collection:

• Archival research: Use of secondary data obtained from firms, public sources, databases, etc.

• Survey studies: Collection of data through questionnaires

• Experiments: Collection of data through laboratory replication of a real-world situation

54Quant. Methods 1 (M. Messner)

Concepts and their measurement

Examples for archival proxies:

• “To proxy and control for the firm’s innovative capability or capacity, we include the variable cumulative number of patents, which is the total number of patents that the firm has generated since the year of its establishment until the year before the observation year.” (Jia et al. 2019, AMJ)

• “As a proxy for project complexity, I used the total number of electrical engineering hours on a project”. (Young-Hyman, 2017, ASQ)

55Quant. Methods 1 (M. Messner)

Concepts and their measurement

Using surveys to measure concepts

 Example

56Quant. Methods 1 (M. Messner)

Concepts and their measurement

• Surveys often rely on so-called Likert scales to measure the extent to which the respondent agrees or disagrees with a statement.

– Likert-scales can have different degrees of detail, but most common are 7-point scales.

– End-points represent the most extreme positions on a question, e.g.:

• 1 = definitely false; 7 = definitely true; 1 = strongly disagree, 7 = strongly agree, etc.

• The different questions represent “items“. The construct (representing the concept of interest) is then a combination of these items.

– Items either taken from prior research or newly developed scale

– # of items: balance length of questionnaire with measurement quality; at least 3 items for a (reflective) construct

57Quant. Methods 1 (M. Messner)

Concepts and their measurement

Compare the two regressions:

Manifest variables (items) x1-x5:

y = ß0 + ß1* x1 + ß2 * x2 + ß3 * x3 + ß4 * x4 + ß5 * x5 + u

One latent construct z1:

y = ß0 + ß1* z1 + u

58Quant. Methods 1 (M. Messner)

Concepts and their measurement

There are two types of multi-item (latent) constructs:

• Reflective: Items represent manifestations of the construct

• Formative: Items represent elements (components) of the construct

This distinction is important, because construct validity and reliability of these constructs must be assessed differently.

59Quant. Methods 1 (M. Messner)

Concepts and their measurement

Monitoring difficulty

It is not possible to supervise the salesperson’s activities closely.

It is difficult for us to evaluate how much effort this salesperson really puts into her/his job.

It is relatively easy for this salesperson to turn in falsified sales call reports.

Our evaluation of this salesperson cannot be based on his/her activity and sales call reports.

Salesperson’s ability

This salesperson has a high degree of competence in tailoring his/her sales approach to the specific situation on hand.

This salesperson has been very creative in designing relevant solutions to customers’ problems.

This salesperson is a skilled and persuasive negotiator.

This salesperson is capable of closing a deal in a tough selling situation.

This salesperson is able to learn from past experiences and adapt them to current circumstances.

This salesperson is skilled in extracting the unique problems faced by and the requirements of his/her customers.

Lo et al. (2011, JMR)

60Quant. Methods 1 (M. Messner)

Concepts and their measurement

Construct validity (Do we really measure what we seek to measure?) and reliability (Do we measure reliably?) are a frequent concern in survey research.

Construct validity and reliability should be approached from a judgmental and a technical perspective:

• Judgmental (for both reflective and formative constructs):

– Do the questions really reflect what I think the phenomenon is about? (often referred to as “face validity”)

61Quant. Methods 1 (M. Messner)

Concepts and their measurement

• Technical (for reflective constructs only):

– Internal consistency and convergent validity: do the items converge on the construct?

– Discriminant validity: are the constructs sufficiently different from each other?

62Quant. Methods 1 (M. Messner)

Concepts and their measurement

Reflective constructs Formative

constructs

(composite

constructs)

Face validity Items should all reflect the underlying construct (in a similar way)

Items should together add up to the construct (in a comprehensive way)

Internal consistency & convergent validity

Factor analysis:• High factor loadings of each item• High Average Variance Extracted

(AVE)• High composite reliability• Good goodness-of fit criteria

Discriminant validity Factor analysis:• Small cross-loadings• Square root of AVE >bivariate

correlations between constructs (Fornell-Larcker Test)

63Quant. Methods 1 (M. Messner)

Concepts and their measurement

The basis for assessing construct validity and reliability of reflective

constructs is typically factor analysis.

• Factor analysis:

– Identifies (or confirms) the relationship between items and underlying constructs (factors)

– Assumes that the correlation between items can be explained by underlying factors (of which the items are manifestations)

– Exploratory factor analysis (EFA): no ex ante specification of factor structure: see which factors emerge from the data

– Confirmatory factor analysis (CFA): factors and their relationships to items are specified a priori and then tested. Confirm if items load as they should.

• Different uses in studies:

– Only EFA or only CFA

– First an EFA, then a CFA

64Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Example

We are interested in 4 (reflective) constructs and, to this end, collected 16 variables through a survey.

Our hypothesized factor structure is as follows:

Factor1: v2 + v9 + v13 + v16 + v17Factor2: v6 + v7 + v8 + v11Factor3: v1 + v3 + v5 + v20Factor4: v4 + v12 + v14

65Quant. Methods 1 (M. Messner)

Exploratory factor analysis

v1 v2 v3 v4 v5 v6 v7 v8 v9 v11 v12 v13 v14 v16 v17 v20v1 1,00v2 -0,05 1,00v3 0,44 0,08 1,00v4 0,00 -0,07 0,10 1,00v5 0,41 0,00 0,40 0,13 1,00v6 0,02 0,02 0,07 0,02 0,19 1,00v7 0,01 -0,09 0,00 0,13 0,08 0,34 1,00v8 0,05 -0,02 0,03 0,02 0,09 0,31 0,37 1,00v9 -0,05 0,25 -0,06 0,03 -0,03 -0,12 -0,06 -0,04 1,00v11 0,04 -0,10 0,08 0,04 0,16 0,47 0,48 0,37 -0,10 1,00v12 0,11 0,06 0,08 0,29 0,17 0,12 0,03 0,06 0,03 0,03 1,00v13 -0,13 0,31 -0,03 0,05 0,05 -0,08 0,02 -0,09 0,32 -0,08 0,08 1,00v14 0,02 -0,04 0,04 0,34 0,11 -0,05 0,00 0,02 0,11 -0,07 0,25 0,10 1,00v16 -0,05 0,28 0,03 0,08 0,00 -0,13 0,00 -0,04 0,36 -0,15 0,08 0,38 0,03 1,00v17 -0,01 0,28 0,01 -0,04 -0,01 -0,14 -0,06 -0,07 0,36 -0,18 0,04 0,33 -0,01 0,29 1,00v20 0,35 0,01 0,40 0,04 0,36 0,00 0,09 -0,06 -0,05 0,00 0,10 0,00 0,03 0,03 0,07 1,00

Let‘s first look at the correlations between the variables

66Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Fa

Fb

x1

x2

x3

x4

Latent factors Manifest variables

a1

a2

b1

b4

X1 = a1*Fa + b1*Fb + u1

X2 = a2*Fa + b2*Fb + u2

etc

All relationships possible  Exploratory factor analysis:

a1, a2, b1, etc.: factor loadings

Must specify either:- Number of factors (in line with your theory)- Minimum explanatory value of each factor (typically:

Eigenvalue > 1).

Two main “extraction methods“ for factors:- Principal component analysis: factors try to explain

entire variance of variables- Principal axis factoring: factors explain only shared

variance of variables

EFA: finding a good factor structure

67Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Illustration of factor structure

Factor 1

Factor 2

1

0.5

– 0.5

– 0.5 10.5

– 1

– 1

Variable x1 = 0.6 * F1 + 0.6 * F2 + u

68Quant. Methods 1 (M. Messner)

Exploratory factor analysis

SPSS: Analyze  Dimension Reduction  Factor

69Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Output from SPSS

70Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Output from SPSS

71Quant. Methods 1 (M. Messner)

Exploratory factor analysis

• Eigenvalues: an indicator for how much variance in the variables the factor explains

• E.g., when Eigenvalue of each Factor = 1  each Factor explains the same amount of variance

• Mathematically: the sum of the squared loadings across all variables for a given factor (see next page)

• Variance Extracted: how much % of the shared variance in all variables is explained by the factor.

• E.g., when Eigenvalue (Factor 1) = 1, and 5 Factors  Factor 1 explains 20% of variance

• Factor loadings for each variable-factor pair: correlations

Output from EFA

72Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Output from SPSS

Factor Matrixa

Factor1 2 3 4

v1 0,336 0,419 -0,350 -0,130v2 -0,281 0,292 0,207 -0,220v3 0,319 0,508 -0,280 -0,133v4 0,120 0,259 0,122 0,559v5 0,388 0,499 -0,135 -0,036v6 0,497 -0,018 0,330 -0,103v7 0,444 0,016 0,438 -0,032v8 0,387 -0,026 0,338 -0,049v9 -0,389 0,304 0,280 -0,044v11 0,622 -0,070 0,424 -0,126v12 0,107 0,303 0,100 0,308v13 -0,370 0,364 0,345 -0,048v14 -0,002 0,244 0,049 0,515v16 -0,363 0,380 0,270 -0,066v17 -0,388 0,342 0,175 -0,180v20 0,226 0,447 -0,257 -0,118Extraction Method: Principal Axis Factoring.a. 4 factors extracted. 12 iterations required.

73Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Factor analysis first provides an “unrotated” solution.

• First factor is defined in such a way that it explains most of the shared variance of all variables  almost any variable will load on this factor

• Residual variance to be explained then by second, third, etc. factor

Rotation changes the factor structure such that variance is explained more equally among factors.

• The most frequently used method is “Varimax“, which defines factors in such a way that they have high or low loadings, but not medium ones.

Unrotated and rotated factor structure

74Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Unrotated and rotated factor structure

Hair et al., p. 111

75Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Output from SPSS

76Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Output from SPSS:

rotated solution

Rotated Factor Matrix

Factor1 2 3 4

v1 -0,002 -0,089 0,648 0,012v2 -0,019 0,496 0,045 -0,083v3 0,033 0,009 0,672 0,051v4 0,065 -0,007 0,040 0,635v5 0,166 0,013 0,604 0,165v6 0,593 -0,095 0,078 -0,007v7 0,619 -0,008 0,010 0,083v8 0,514 -0,047 0,006 0,031v9 -0,073 0,553 -0,073 0,084v11 0,750 -0,141 0,058 -0,022v12 0,082 0,083 0,140 0,418v13 -0,016 0,613 -0,051 0,114v14 -0,063 0,027 0,020 0,567v16 -0,060 0,584 0,003 0,093v17 -0,123 0,559 0,039 -0,042v20 -0,016 0,034 0,572 0,038Extraction Method: Principal Axis Factoring. Rotation Method: Varimax with Kaiser Normalization.a. Rotation converged in 5 iterations.

77Quant. Methods 1 (M. Messner)

Exploratory factor analysis

Output from SPSS:

rotated solution

Rotated Factor Matrix

Factor1 2 3 4

v1 -0,002 -0,089 0,648 0,012v2 -0,019 0,496 0,045 -0,083v3 0,033 0,009 0,672 0,051v4 0,065 -0,007 0,040 0,635v5 0,166 0,013 0,604 0,165v6 0,593 -0,095 0,078 -0,007v7 0,619 -0,008 0,010 0,083v8 0,514 -0,047 0,006 0,031v9 -0,073 0,553 -0,073 0,084v11 0,750 -0,141 0,058 -0,022v12 0,082 0,083 0,140 0,418v13 -0,016 0,613 -0,051 0,114v14 -0,063 0,027 0,020 0,567v16 -0,060 0,584 0,003 0,093v17 -0,123 0,559 0,039 -0,042v20 -0,016 0,034 0,572 0,038Extraction Method: Principal Axis Factoring. Rotation Method: Varimax with Kaiser Normalization.a. Rotation converged in 5 iterations.

Guide:

Loadings > | 0.5 | or | 0.6 |

Cross-loadings < | 0.4 | or | 0.5 |

79Quant. Methods 1 (M. Messner)

Confirmatory factor analysis

Fa

Fb

x1

x2

x3

x4

Latent factors Manifest variables

a1

a2

b3

b4

X1 = a1*Fa + u1

X2 = a2*Fa + u2

X3 = b3*Fb + u3

X4 = b4*Fb + u4

Relationship specified  Confirmatory factor analysis:

a1, a2, b3, etc.: factor loadings

CFA tests how well the specified model fits the data

80Quant. Methods 1 (M. Messner)

Confirmatory factor analysis

CFA produces one hypothesis test and various “fit indicators“

Indicator Description “good“ fit

GFI Goodness of fit GFI >= 0.95

RMSEA Root mean square error of approximation

RMSEA < 0.08

CFI Comparative fit index CFI >= 0.90

SRMR (Standardized) root meansquare residual

(S)RMR < 0.08

Hypothesis test (chi-squared):

H0: “The model fits perfectly.”P-value > 0.05 so as not to reject this H0

See also: https://www.cscu.cornell.edu/news/Handouts/SEM_fit.pdf

Typically these indicators are calculated for each factor separately and for the entire model as a whole

81Quant. Methods 1 (M. Messner)

Confirmatory factor analysis

Confirmatory factor analysis output

Model Test User Model:

Test statistic 95.847

Degrees of freedom 98

P-value (Chi-square) 0.543

cfi rmsea gfi rmr1.000 0.000 0.954 0.113

82Quant. Methods 1 (M. Messner)

Confirmatory factor analysis

Confirmatory factor analysis output

Latent Variables:Estimate Std.Err z-value P(>|z|) Std.lv Std.all

factor1 =~ v2 1.000 0.771 0.474v9 1.216 0.229 5.302 0.000 0.938 0.579v13 1.259 0.233 5.395 0.000 0.971 0.605v16 1.361 0.254 5.363 0.000 1.050 0.596v17 1.123 0.215 5.219 0.000 0.866 0.559

83Quant. Methods 1 (M. Messner)

Additional validity indicators

Additional checks for convergent and discriminant validity

Convergent validity (for each factor separately):

• High Average Variance Extracted (guide: > 0.5)• How much of the variance in the items (averaged across all items of the given

factor) is explained by the given latent factor• E.g. AVE = 0.4  Factor explains 40% of the variation in item values

• High Composite reliability (guide: > 0.7)• Compares the variance extracted to the error variance (not explained).

• [Cronbach’s alpha / Coefficient alpha (guide: > 0.6 or 0.7)]

84Quant. Methods 1 (M. Messner)

Additional validity indicators

AVE and Composite Reliability

Squared loading = % variance of this variable explained by the factor

Sum of squared loadings = 0.5792 + 0.6052 + 0.5962 + 0.5592 = 1.36

Average Variance extracted = 1.36 / 4 = 0.34

factorloadings

squaredloadings

1- squared loadings

0.579 0.335 0.6650.605 0.366 0.6340.596 0.355 0.6450.559 0.312 0.6882.339 1.369 2.631

Composite reliability = 2.3392 / (2.3392 + 2.631) = 0.68

85Quant. Methods 1 (M. Messner)

Additional validity indicators

Additional checks for convergent and discriminant validity

Discriminant validity of factors:

Is a given factor more strongly correlated with its own items than with other factors?

Criterion: √ AVE > cross-correlation with other factors (Fornell-Larcker test)

Why?

• Remember: correlation2 = R2 = % of variance explained

• AVE = average variance of items explained (by the given factor)

86Quant. Methods 1 (M. Messner)

Concepts and their measurement

Typical procedure for technical reliability and construct validity (for reflective

constructs)

1. Exploratory factor analysis:

– Number of factors with Eigenvalue > 1: in line with theory?

– Sufficiently high factor loadings and low cross-loadings

– Possibly delete items that do not load correctly (and re-do EFA)

2. Confirmatory factor analysis:

– Test hypothesized factor structure: Different Goodness of fit criteria

3. Further criteria to ascertain convergent validity of factors: Composite Reliability, Average Variance Extracted

4. Further criterion to ascertain discriminant validity of factors: √ AVE > cross-correlation with other factors (Fornell-Larcker test)

87Quant. Methods 1 (M. Messner)

Example

Ramani & Kumar (2008), Journal of Marketing

“Composite reliability is an indicator of the shared variance among the set of observed variables used as indicators of a latent construct (Fornell and Larcker1981; Kandemir, Yaprak, and Cavusgil 2006). As Table 2, Panel A, shows,construct reliabilities for all the seven latent constructs ranged from .82 to .97, well above the recommended value. In addition, the coefficient alpha values were well above the threshold value of .7 that Nunnally (1978) recommends. The seven-factor CFA model exhibited a good fit with the data (χ2 = 253.95, d.f. = 131; CFI = .95; GFI = .82; TLI = .93; IFI = .95; and SRMR = .07). The standardized factor loadings ranged from .62 to greater than .90 and were statistically significant at the α = .95 level (see Table 2, Panel B). This provided the necessary evidence that all the constructs exhibited convergent validity. We examined discriminant validity using a procedure suggested by Fornell and Larcker (1981) and widely followed by other researchers (e.g., De Wulf, Odekerken- Schroder, and Iacobucci 2001; Kandemir, Yaprak, and Cavusgil 2006). We computed the average variance extracted by the indicators corresponding to each of the seven factors and compared it with the highest variance that each factor shared with the other factors in the model. The average variance extracted for each factor was always greater than the highest shared variance (see Table 2, Panel A).“

88Quant. Methods 1 (M. Messner)

Example (cont’d)

Ramani & Kumar (2008), Journal ofMarketing

89Quant. Methods 1 (M. Messner)

Concepts and their measurement

How to aggregate the item responses to a value of the construct?

• When used in further regression analysis: • Add up the values from the items (“summated scale”) or average

them• Use so-called “factor scores” (there are different ways to calculate

such factor scores), which take into account the loadings

• Example: “We construct our composite variables using factor scores. The correlations between our variables using factor scores and summated scales range between 0.98 and 0.99 for our four composite variables (not tabulated).” (Abernethy, Bouwens, & Kroos, 2017)

• When part of a structural equations model:• Measurement model and structural model are estimated at the same

time  “weights” used depend on structural model