EDUR 9131
Doctoral Research Methods

Spring 2015

Instructor/Facilitator: Bryan W. Griffin
My personal web pages can be found at 

Updates for future sections of EDUR 9131

a. Review sample project 1 manuscript and revise to include more attention to instrumentation (validity reliability present sample items etc), add regression component, maybe use sample proposal written for 7130 as background for new example manuscript report. Also, make cristal clear that regression analysis is required to be included. Also make clear that results must include APA styled tables for correlations and regression, and table of qualitative results. Make clear summary section must be present with research questions or hypotheses.

b. revise current activities not updated (e.g. sampling, effect size calcualtion, etc)

c. add instructions for composite scores calculation in exel and spss


Added as needed.

  Instructor note - add chat notes prior to meeting but without answers

Course Index (check frequently for alterations)

A. Course Administration

  1. Syllabus 
  2. Course Calendar and Reading Assignments -- see D below.

B. Activities that count toward course grade

  1. Test 1: Due date to be determined; expect Test 1 after about  coverage of Coding Open-ended Responses (after about 7th class session)
  2. Test 2: Made available after the final class session, responses are due May 4  April 30 (so as not to conflict with comprehensive tests).
  3. Activity 1:  Research Project: This is a group activity and will be due as a PDF attachment by 9:00pm on May 4; see timeline and instructions on the linked page.

C. Activities that do not count toward course grade

  1. Activity 2 (Individual, non-graded) Instrument Construction Steps (complete and bring responses to second class session)
  2. Activity 3 (Individual, non-graded) Internal Consistency (Answers to this activity can be found here: Internal Consistency Answers; Note to Instructor -- update data with reversed items)
  3. Activity 4 (Individual, non-graded) Coding Open-ended Responses (due date to be determined). 
  4. Activity 5 (Individual, non-graded) Sample Size Calculations (Do not attempt this activity yet, it may be revised Spring 2015; due date to be determined)
  5. Activity 6 (Individual, non-graded) Composite Scores (to be developed; due date to be determined)
  6. Activity 7 (Individual, non-graded) Factor Analysis (to be developed; due date to be determined)
  7. Activity 8 (Individual, non-graded) Effect Sizes (to be developed; due date to be determined)

D. Course Calendar and Content

Sessions Date Topic



Covered via Discussion Forum posts


Reading assignments are noted in the grey box below.



1. Syllabus Review

2. Big Picture -- what is goal of this course?

3. Activity 1: Research Project -- Form Groups for Activity 1

4. Activity 2: Instrument Construction Steps 

5. Refresher review of educational research basics (variables, hypotheses, validity, reliability, etc.)

6. Collect data for later in-class analysis: (private link to raw data here: Per Control, Per Competence, Goal Internal Menon 2001)


Read (or skim as a refresher):  


Review material from EDUR 7130, Introduction to Educational Research


2 1/24 On-campus Class 9am Room 3165 College of Education

In-class Presentation Outline

Reading assignments are noted in the grey box below.

(Note: May need to briefly review previous week exercises on Variables and Hypotheses)

1. Questionnaire Item Development - introductory review


Instructor note - see Questionnaire Development Steps and Questionnaire Develop Steps in /content folder

2. Literature Searches

  • Brief review using Google and Google Scholar

  • Literature Review Presentation Google Doc

  • (Instructor note: add literature review writing, list vs. integration, early vs. late citations, transitions)

3. SPSS Introduction and Review of Descriptive Statistics and Inferential Statistics (entering data, frequencies, M, Md, Mo, R, SD, Variance, Alpha, Beta, Power, Type 1 & 2 errors, p-value, r)

  • Review of Basic Descriptive Concepts (Google Doc presentation in class)
    • Frequencies, Central Tendency, Variability
    • SPSS Data Entry
    • Hypothesis Testing: Alpha, Beta, Power, Type 1 & 2 Errors, p-values (Instructor Note: draw from ANOVA workshop material)
  • Logic of Hypothesis Testing; Coin flop comparing obtained p vs. alpha
  • Hypothesis Testing with p-values: Decision rule with alpha (Decision Rule: If p <= alpha reject Ho; if p > alpha fail to reject Ho )

4. Pearson Correlation, r

5. Group mean comparison with t-test

Read (or skim as a refresher as needed):

 Read if additional review is needed:


1. Review of Educational Research (linked readings below were taken from EDUR 7130, Introduction to Educational Research)

2. Review of Descriptive Statistics and Inferential Statistics


3 1/26 Live Chat Monday 6:30pm with Adobe Connect

Resume coverage of Saturday material.



Live Chat Monday 6:30pm with Adobe Connect


Chat Presentation Outline

Reading assignments are noted in the grey box below.

1. Questionnaire Development 

2. Creating Electronic Questionnaires in Google Docs, Qualtrics

3. Brief review of t-tests if not already covered

4. Effect Size d - standardized mean difference between groups.  

Read for Questionnaire Development:

  • de Vaus, Chapter 4 Developing Indicators for Concepts

  • de Vaus, Chapter 7 Constructing Questionnaires (note section on Wording Questions for useful questions to consider when developing/selecting items)

  • de Vaus, Chapter 11 Building Scales 

Supplemental Reading on Questionnaire Development

  • Brief Presentations on Questionnaire Development:

Review for Creating Questionnaires in Google Docs and Qualtrics

Read for Effect Size d





Live Chat Monday 6:30pm with Adobe Connect


Chat Presentation Outline

Reading assignments are noted in the grey box below.

1. Reliability: Stability of Scores with Test-Retest  (scores must be correlated, and means should be similar; correlation and correlated samples t-test)

Instructor Note: Prepare data early for merging data file on respondent id. Can be lengthy process, so check Excel merging options.

  • Example data from class responses to this questionnaire:  (EDUR 9131 Employment Thoughts 2nd)
  • Test-retest Reliability (note include ICC, mean comparisons with r) with SPSS (warning -- rough draft): appropriate items, how performed; single item score [e.g. income] vs. composite scores [construct]; scores must be linked (Note add kappa, etc. other measures to assess test-retest reliability)
    • Correlation may not be best since ignores mean differences
    • Intra-class correlation (ICC) is better since it takes into account mean differences
      • Using page 1 data in my notes above, r = .958 and ICC = .961 (absolute agreement) and ICC = .958 (consistency)
      • if +3 is added to time 2 for the first data set, r remains .958, but ICC = .435 (absolute agreement) and ICC = .958 (consistency)
      • Use absolute agreement to test whether means differ, consistency ignores item variance and mean differences; absolute tests variability due to raters while consistency does not; absolute tests whether same scores are provided by each rater, consistency test whether relative scores show similar patterns
      • Single rater vs. multiple rater ICC results -- use single if assessing possible reliability of a single rater from several, use multiple if one plans to average ratings across several judges; multiple ICC tests whether multiple raters would provide similar mean ratings across several items
    • Krippendorff's alpha
      • Ordinal results from page 1 data K alpha = .938 (t1 and t2)
      • Ordinal results from page 1 data with +3 added to t2 K alpha = -.17 (shows no agreement), note Krippendorff measures agreement, not consistency
  • Check Mean Equivalence: Correlated samples t-test
  • Published Examples:
    • Kush & Watkins 1996, see Table 2 for test-retest correlations, repeated measures ANOVA to test mean differences p. 317
    • Sapountzi-Krepia 2005, see Table 3 for use of Kappa to assess stability of categorical responses; Table 4 for intra-class correlation and rho (Pearson r) for stability and mean differences (also, nice discussion of correlation limitation, alternatives)
  • Single Item Measure Assessment for Reliability and Validity

2. Equivalent forms (parallel forms) -- statistical assessment executed in same way as test-retest reliability

  • No additional readings

3. Reliability: Internal Consistency with Cronbach's Alpha

  • See material for 4 below

4. Item Analysis, Composite Score, and Reverse Scoring

  • Logic of Internal Consistency (use Table 3 and 4 examples: )
  • Cronbach's alpha conceptualized:  "the percent of variance the observed scale would explain in the hypothetical true scale composed of all possible items in the universe. Alternatively, it can be interpreted as the correlation of the observed scale with all possible other scales measuring the same thing and using the same number of items." Source: G. David Garson
  • Cronbach's alpha with SPSS
    • SPSS reports two Cronbach's alpha values if correlations are requested
      • Cronbach's alpha = calculated on raw data items that usually have different variances, unequal variances
      • Cronbach's alpha based on standardized items = estimate of reliability if all items have equal variances; called Spearman-Brown stepped-up reliability coefficient; value of alpha obtained if all variables are standardized to have equal variances.
  • Example 1:  Employment Thoughts Questionnaire (use 1st administration data)
    • Compare reliability estimates -- both alpha and test-retest to those reported by Menon 2001 (p. 171 alpha and test-retest reliabilities reported on 9 item scale)
  • SPSS analysis of Example 1 data
  • Dimensionality
    • Cronbach's alpha is not a measure of unidimensionality (does not assess internal structure, use factor analysis instead)
    • should not be used as overall internal consistency measure on instruments with diverse constructs
    • Use Employment Thoughts data
  • Reverse Scoring Items and Forming Composite Scores
    • Formula:   Reversed Score = (minimum score) + (maximum score) – actual score  
    • Source: , see first comment
    • Calculation Check: Correlate original and reversed item, r = -1.00
    • Composite: Sum vs. Mean
    • Item Weighting Assumption: Equal contribution (simple sums/means) vs. Unequal Contribution (factor scores)
  • Second Example:  Leisure Activities (raw data, private access)
  • Published Examples
    • Fassinger 1994: Development and Testing of the Attitudes Toward Feminism and The Women's Movement (FWM) Scale. Item-total correlation, Table 1 p. 395  
    • Menon 2001: Reported in text format; p. 164 Cronbach's alpha and test-retest reliabilities reported on 15 item scale; p. 171 Cronbach's alpha and test-retest reliabilities reported on 9 item scale
    • Kanning, Böttcher & Herrmann 2012:  Reported in table format; see Table 2 p. 145 (alpha and test-retest)
    • Frey & Bos 2012: Reported in table format; see Table 4 p. 34 (alpha, item-total correlations minimum and maximum)

(e) Scorer-rater; inter-judge agreement -- Discussed below with coding open-ended responses

Read for Reliability:

1. William Gabrenya's Reliability Overview

2. de Vaus text

5th Edition

  • Chapter 11 pages 180 to 186
  • Reliability pages 52 to 53
  • Scaling Checklist p. 1959

6th Edition

  • Chapter 11 pages 179 to 185
  • Reliability pages 48 to 51
  • Scaling Checklist p. 194+

3. Joseph A. Gliem and Rosemary R. Gliem (2003) Calculating, Interpreting, and Reporting Cronbach’s Alpha Reliability Coefficient for Likert-Type Scales.

4. Tavakol and Dennick (2011). Making Sense of Cronbach's Alpha, International Journal of Medical Education.

5. SPSS: Cronbach's Alpha

6. Test-retest Reliability: Hopkins 2011 New View of Statistics: Measures of Reliability - simple yet detailed introduction to test-retest reliability

Supplemental Reliability Readings

Hopkins, 2000. Measures of Reliability in Sports Medicine and Science. Excellent sources for details of reliability estimates, how to design studies to assess reliability, uses of reliability, etc.

Klaas Sijtsma (2009) On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha. Psychometrika. 74(1): 107–120.





Live Chat Monday 6:30pm with Adobe Connect


Chat Presentation Outline

Reading assignments are noted in the grey box below.


1. Resume coverage of Reliability Material

2. Validity

  • Content Validity (Logical Validity) - brief review
  • Structural Validity
  • Construct Validity Overview (seeking anticipated/hypothesized patterns among structured response scores from instruments)
    • McKenna & Kear 1990 p. 15 of PDF (predicted behavior with other variables, factor analysis?)
    • Fassinger 1994 Convergent and divergent/discriminate validity examples , Table 2 p. 397
    • Ragheb & Beard 1982 (correlation with similar measures; factor analysis)
    • Menon 2001 (various)
  • Examples of how to report questionnaire development results in journal publications:


Readings for Validity:

1. Validity Evidence

Source: US Department of Education Office of Special Education Programs 

Summary notes:

2. Table Outlines of Validity Evidence 

 Floyd, Phaneuf, & Wilczynski 2005; scan article and note particularly Table 1

Goodwin & Leech 2003; scan article and study Table 1 (2 MB file, may be slow download)

Review again the checklist from Holmbeck and Devine 

Possible future readings to incorporate:





Live Chat Monday 6:30pm with Adobe Connect


Chat Presentation Outline

Reading assignments are noted in the grey box below.


1. Coding open-ended questionnaire responses (conceptual analysis discussed here; see link in readings section to Palmquist 2008 for relational analysis example).

(a) Qualitative Data Analysis

  • Brief review of process
    • Study, read data carefully
    • Begin coding data
      • Inductive: codes, categories, themes are derived from raw data, emergent
      • Deductive: codes, categories, theme developed prior to analysis of data
      • Mixed: initial codes developed or expected prior to analysis, but revisions, deletions, additions occur during analysis
    • Identify categories or groupings of codes, develop codebook and codesheet
    • Identify patterns and connections among and within categories
    • Repeat process until one believes little more can be gained from repeating steps above
    • Independent coders should attempt to code data or samples of data and reliability of coding assessed
    • Develop report of results/findings - tables of codes/categories, quotations, descriptions

(b) Codebooks/Code sheets

(c) Reliability -- inter-rater and intra-rater agreement


Readings for Open-ended Responses:

1. Qualitative Data Analysis (QDA)

Many of you probably have taken a Qualitative Research course so the readings below are brief and highlight the basic steps in QDA./p>

Supplemental Readings on QDA -- Explore these if interested

2. QDA Codebooks

Supplemental Readings on Codebook Development -- Explore these if interested

3. Reliability of Coders

Supplemental Reading on Reliability of Coders -- Explore these if interested

Instructor Note: Other material to consider





8 2/28

On-campus Class 9am Room 3165 College of Education

In-class Presentation Outline

Reading assignments are noted in the grey box below.


Interesting Validity/Reliability Assessment Dissertation: Kroupin (2011) VALIDITY AND RELIABILITY OF THE POWER/CONTROL SCALES. U. of Minn. 58 pages.

Shows that dissertations need not be long and can focus on assessing validity of scores obtained from a scale.


1. Class Example of Activity 1 Data Collection and Analysis

 To be revised


Social Attitudes Scale:

Social Attitudes Data: 

Social Attitudes Items:   

R 11, 12, 15; 1-8 FA, 9-12 EPI, 13-16 SPI


Note to instructor:

  • Revise in Google to provide numeric responses -- Illustrative Questionnaire: EDUR 9131 Research Methods Review or  (note, items worked poorly fall 09)
  • Data file to above questionnaire:
  • (Note: 1-6 instructional rating; 7-10 value; 11-13 methods efficacy/confidence [r 12]; 14-17 effort regulation [r 14 15])
  • Poor results for last two constructs
  • Maybe use Leisure example, but no reversed items
  • Develop new example

2. Sample Size Determination for Categorical Variables (Chi-square tests)


To find sample sizes for studies involving simple proportions, such as responses to Yes/No type questions (assumes binomial distribution)

3. Sample Size Determination and Effect Sizes for Linear Models (t-test, Pearson r, Regression, ANOVA, ANCOVA): 

Group Comparisons: t-tests, ANOVA

To find sample sizes for group comparisons (e.g., t-tests, ANOVA type analyses)


Sample sizes for Pearson's correlations



  • Calculating adjusted effect sizes for ANCOVA - material to be added for

Sample Size Calculators

  • G Power -- free software for sample size determination

  • On-line sample size calculators:

Readings for Sampling and Sample Size:



9 3/2 Live chat if needed


Chat Presentation Outline

Reading assignments are noted in the grey box below.


1. Finish Sample Size Determination


10 3/9

Live Chat Monday 6:30pm with Adobe Connect


No Chat Held



  3/16 No Chat -- Spring Break

11 3/23

Live Chat Monday 6:30pm with Adobe Connect


Chat Presentation Outline

Reading assignments are noted in the grey box below.


1. Inter-coder Agreement for Ordinal, Interval, and Ratio Data (to be covered 4/13 or 4/20)


2. Regression Analysis  

3. Brief review of Factor Analysis

4. Study Guide for Factor Analysis (to be posted later this week) 


Readings for Regression:

de Vaus

  • 4th edition -- Chapter 11: Bivariate Analysis: Alternative Methods, p. 173-184
  • 5th edition -- Chapter 15: Bivariate Analysis for Interval-level Variables, p. 279-289
  • 6th edition -- Chapter 15: Bivariate Analysis for Interval-level Variables, p. 278-285

Online Notes from EDUR 8131 (Introduction to Statistics)

Burns, W. (1997). Spurious Correlations.  Explains necessity of "multiple" or partialed regression by illustrating mis-interpretation of correlational evidence.


Supplemental Material -- Review if Needed

Notes 8a Simple Regression (linked above) Videos

Notes 8b Multiple Regression (linked above) Videos



  3/28 No On-campus Class for EDUR 9131 -- B Griffin unavailable

  3/30 No Chat -- B Griffin unavailable

  4/6 No Chat -- B Griffin unavailable


Live Chat Monday 6:30pm with Adobe Connect


Chat Presentation Outline

Reading assignments are noted in the grey box below.


1. Finish Regression Presentation

2. Inter-coder Agreement for Ordinal, Interval, and Ratio Data (to be covered 4/13 or 4/20)




Live Chat Monday 6:30pm with Adobe Connect


Chat Presentation Outline

Reading assignments are noted in the grey box below.


1. Inter-coder Agreement for Ordinal, Interval, and Ratio Data (to be covered 4/13 or 4/20)

2. Factor Analysis


Readings for Factor Analysis:

de Vaus

  • 4th edition -- Chapter 15: Factor Analysis, p. 257-275
  • 5th edition -- Chapter 11: Factor Analysis, p. 186-192
  • 6th edition -- Chapter 11: Factor Analysis, p. 185-191

Field (2005). Factor Analysis Using SPSS 

Author unknown What is Factor Analysis? A Simple Explanation   

Kootstra (2004) Exploratory Factor Analysis    

Neill (2008) Writing Up A Factor Analysis -- provides example of how to reported factor analysis details.

Jackman (2005) An Introduction to Factor Analysis   A more challenging introduction 



  4/25 On-campus Class 9am Room 3165 College of Education

In-class Presentation Outline

Reading assignments are noted in the grey box below.

1. Factor analysis
2. Meta-analysis

  4/27 Chat if needed

1. Factor analysis
Data 1:
Data 2:
Alpha Data Questions:
2. Meta-analysis

  5/4 Finals week (Test 2 due April 30 so not to conflict with EdD comprehensive tests)


E. SPSS Tutorials (obtained from various on-line sources: Dr. McKnight at U. Oklahoma,  Dr. Elvers at U. of Dayton, Guang-Hwa Change at Youngstown State U.)

  1. Entering data into SPSS (variable labels, value labels, setting variable as numeric, etc.)
            Data Entry A, Data Entry B
  2. Frequencies/Explore Commands (including descriptive statistics, box plots, histograms, stem-and-leaf, etc.)
            Frequencies A, Frequencies B, Stem and Leaf, Various Graphical Displays
  3. t-tests (one sample, independent sample, and correlated samples)
            t-tests, independent samples t-tests
  4. Correlation 
            Correlation A, Correlation B, For scatter plots see Various Graphical Displays above
  5. Regression 
            Regression A, Regression B
  6. ANOVA
            ANOVA A (using ONEWAY command), ANOVA B (using ONEWAY command), ANOVA C (using UNIVARIATE command)
            ANCOVA A

F. PDF File Creation

Since groups will be required to submit this project, I will only accept PDF attachments from one individual in each group. See detailed instructions on this in the Research Project link above. If you do not know how to create a PDF file, use one of the following sources:

Here are a few free web page that convert files to PDF over the internet: 

I use the following free software to create my PDF files (it leaves no watermark): 

I also use OpenOffice to create some free PDF files. Open Office is a free Office Suite similar to Microsoft Office (Open Office leaves no watermark): 

If you want further tips and links for converting to free PDFs, read this site or this .



Note: The material indented with grey background are comments for instructor about content to review for possible incorporation into course. Students should ignore this information.

Add links to on-line statistical programs, e.g., 

Reading Research Reports --- t-tests and ANOVA

Note -- Links to possible Reading Research report tables












Copyright Bryan W. Griffin