EDUR 8331
Applied Measurement

Course Index, Syllabus, and Calendar
Fall
2018

Instructor: Bryan W. Griffin
Personal web page: http://www.bwgriffin.com


Announcements:  

Update Nov 18  

* Added Activity 2 (which is now optional). See Graded Activities below.
* Added Test 3 in Folio.

Update Nov 17 

* Added additional reading for Classroom Test Development materials (week 10)
* Added presentation notes for Classroom Test Development (week 10)
* Added presentation note for different test item formats (week 12)
* Added presentation note on Item Analysis (week 13)


Course Index

A. Course Administration

  1. Syllabus 
  2. Course Calendar and Reading Assignments -- see D below.
  3. Creating PDFs Files

B. Graded Activities

  1. Test 1: Posted in Folio two weeks before the due date of  to be determined - about week 5 or 6; topics include the following:
  2. Test 2: Posted in Folio two weeks before the due date of  to be determined - about week 10 or 11; topics include the following:.
  3. Test 3: Posted in Folio two weeks before the due date of Dec 4; topics include all content covered in the course.
  4. Activity 1: Scale Development and Assessment      
  5. Activity 2: Test Item Analysis
  6. Activity 3: Test Item Development and Assessment (future semester addition)

C. Course Calendar and Content

Note: Calendar topics may be altered or relocated to reflect realized course pace during Spring 2018.

Week Date Topic

1

8/13

Note - chat dates differ from week-start dates located on the left. Optional chats will occur Thursday each week.

 

1. Syllabus

2. Course Content Overview

3. Topics and readings for weeks 1 and 2

a. Measurement: Van Blerkom, Chapter 1 Introduction to Measurement

b. Variables, Graphical Displays, and Descriptive Statistics:

* Van Blerkom,

- Chapter 18 The Language and Logic of Statistics
- Chapter 19 Frequency Distributions and Graphs
- Chapter 20 Central Tendency: What is a Typical Score?
- Chapter 21 Variability: How Spread Out Are the Scores?

* Additional material from EDUR 8131 Introduction to Educational Statistics: Basics Review

c. Pearson Correlation:

* Van Blerkom, Chapter 22 Correlation

* Additional material from EDUR 8131 Introduction to Educational Statistics: Correlation

d. Two-group t-test:

* Van Blerkom, t-test not covered by text

* Material from EDUR 8131 Introduction to Educational Statistics: t-test

e. SPSS introduction:

 * See presentation video below for SPSS basics with descriptive statistics, graphical displays, correlation, and t-test.

* Material from EDUR 8131 Introduction to Educational Statistics: SPSS Tutorials

4. Presentation

a. Notes:

* Course Overview, Measurement, and Variables

* Basic Statistical Concepts and SPSS

* Pearson Correlation and SPSS

* Two-group t-test

b. Video:

* Overview, Measurement, and Variables (48 minutes)

* Basic Statstics Concepts and SPSS (39 minutes)

* Pearson Correlation and SPSS (21 minutes)

* Two-group t-test (to be added)

 

5. Non-graded Activity - complete if you need practice with correlation and t-test. Perform analyses of data found below to ensure your findings match those provided in the answers.

 

2

8/20

1. Topics and readings: same as week 1

2. Presentation: same as week 1 

 

3 8/27

1. Topic and readings for week 3: Reliability

 

* Van Blerkom, Chapter 3 (my apologies, this should have been Chapter 4 Reliability)

* Conceptual Introduction to Reliability (from EDUR 7130 Introduction to Educational Research)

* Tavakol and Dennick (2011). Making Sense of Cronbach's Alpha, International Journal of Medical Education.

* Gliem and Gliem (2003). Calculating, Interpreting, and Repoting Cronbach's Alpha Reliability Coefficient for Likert-Type Scales .

* Koo and Li (2016). A Guideline for Selecting and Reporting Intraclass Correlation Coefficients for Reliabiloty Research. Journal of Chiropractic Medicine, 15, 155-163 (This publication is statistical in nature; read it for conceptual understanding and study Figure 1 to identify different types of ICC; the ICC will be explained in this week's presentation and video).

* Cronbach's Alpha with SPSS 

* Add examples from published studies and standardized tests.

2. Data for illustrations

Data: Employment Thoughts Merged Data

Leisure Activities Questionnaire: http://goo.gl/forms/JrMtZmVHF5

3. Presentation

a. Notes:

* Review of Reliability and Validity
* Test-retest and Parallel Forms Reliability

* Internal Consistency and Item Analysis

b. Video:

* Review of Reliability and Validity
* Test-retest and Parallel Forms Reliability (63 minutes)

* Internal Consistency and Item Analysis  (71 minutes)

 

4. Non-graded Activity: Internal Consistency

 

4

9/3

1.Topic and readings for week 4: Validity

 

a. Van Blerkom, Chapter 4 (my apologies, this should have been Chapter 5 Validity)

b. Current Views on Validity

* Brown, T. (2010) Construct Validity: A Unitary Concept for Occupational Therapy Assessment and Measurement. Hong Kong Journal of Occupational Therapy, 20, 30-42. Very good overview of traditional and current views of validity. Study Table 1 carefully.

* Cook, DA, & Beckman, TJ (2006). Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Applicaiton. The American Journal of Medicine, 119, 166.e7 - 166.e16. Anoher good review of traditional and current views of validity.

c. Validity Evidence Based Upon Content or Content Validity 

* Haynes, S.N., Richard, D.C.S., & Kubany, E.S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7, 238-247. (See especially Content Validation Guidelines starting page 244.)

* Review Table 1, p. 35-36 in the Brown (2010) Construct Validity article linked above, with focus on "1. Test Content Evidence."

* Add Lawsche article on quantitative content validity

d. Validity Evidence Based Upon Internal Structure and Relation to Other Variables

* Review Table 1, p. 35-36 in the Brown (2010) Construct Validity article linked above; study the two subjects in Table 1 entitled "4. Internal Structure Evidence" and "5. Relations to Other Variables Evidence."

2. Example publications to showing various validity evidence as discussed by readings above.

a. Ragheb, M. G., & Beard, J. G. (1982). Measuring leisure attitude. Journal of Leisure Research, 14, 155-167.
b. Menon, S.T. (2001). Employee empowerment: An integrative psychological approach. Applied Psychology: An International Review, 50, 153-180.
c. Add standardized test validity assessment.

3. Data for illustrations

Data: Employment Thoughts Merged Data

Leisure Activities Questionnaire: http://goo.gl/forms/JrMtZmVHF5

4. Presentation

a. Notes: Validity Evidence  

b. Video: to be added

 

5 9/10

1. Topic and readings for week 5: Composite Scores, Reverse Scoring, and Questionnaires/Scales Selection

a. Composite Scores and Reverse Scoring - see presentation notes below
a. Questionnaires/Scales Selection - see presentation notes below

2. Presentation

a. Notes:

* Composite Scores and Reverse Scoring    
* Searching for Developed Questionnaires/Scales 

b. Video:

* Composite Scores and Reverse Scoring (50 minutes)

* Searching for Developed Questionnaire Scales (41 minutes) 

 

3. Non-graded Activity: Composite Scores

 

 
6 9/17

1. Topics and readings for week 6: Reading Factor Scores for Item Selection and Questionnaire/Scale Construction

a. Reading Factor Analysis Results for Item Selection

* For a simple introduction see Presentation Notes and Video below on Reading Factor Analysis Scores

* Kootsa: Factor Analysis Explained (this is briefly reviewed in Factor Analysis video below; focus on interpretation of results)

b. Questionnaire/Scale Development Construction

* Frary (1996) Hints for Designing Effective Questionnaries

* Fanning (2005) Formatting a Paper-based Survey Questionnaire: Best Practices 

* de Vaus (2002) Chapter 7: Questionnaire Construction

* Folz (1996) Chapter 4: Survey Design and Implementation  

* Oskamp & Schultz (2005) Explicit Measures of Attitudes   

* Siniscalco and Auriat (2005) Module 8 Questionnaire Design (supplemental material; multi-national perspective)

c. Questionnaire Development Checklist

* Holmbeck, G. N., & Devine, K. A. (2009). Editorial: An author's checklist for measure development and validation manuscripts. Journal of Pediatric Psychology, 1-6, 2009.

* Note: This file contains only Holmbeck and Devine's checklist

d. Likert-scaled Response Options

* List of Step Descriptions for Five-point Likert Responses 

* Dr. Siegle offers these: Likert Response Options 

* Dr. Vagias provides these: Likert Response Options 

2. Presentation

a. Notes:

* Reading Factor Scores     
* Questionnaire/Scale Construction  

b. Video:

* Reading Factor Scores (58 minutes)    
* Questionnaire/Scale Construction, Part 1 (44 minutes) 
* Questionnaire/Scale Construction, Part 2 (48 minutes) 

 

3. Non-graded Activity: Instrument Construction Steps 

 
7 9/24 1. Continue readings on Questionnaire Development above
2. Instructor note - add readings on Single-item measures, validity and reliability
3. Electronic Questionnaire Development

4. Presentation

a. Notes:

* Creating Electronic Questionnaire with Google Docs
* Google Spreadsheet with Data from Electronic Questionnaire (to be added)

* Using vlookup with Google Spreadsheet to change text to numbers. Youtube: https://www.youtube.com/watch?v=YMASsjPkngg

b. Video:

* No videos 


8 10/1

1. Topics and readings for week 8: Qualitative Data Analysis, Codebooks and Codesheets,and Assessing Coder Agreement

a. Qualitative Data Analysis

* Taylor-Powell and Renner (2003) Analyzing Qualitative Data; An easy to read, brief (10 pages) summary of Qualitative Data Analysis   

* LeCompte (2000) Analyzing Qualitative Data. Theory into Practice, 39, 146-154; A more theoretical introduciton to Qualitative Data Analysis

* Kent Lofgren (2013) Qualitative analysis of interview data: A step-by-step guide. His video provides basic introduction to coding textual data. His focus is on interview data, but his presentation applies equally to written responses to open-ended items. 

Youtube: https://www.youtube.com/watch?v=DRL4PF2u9XA

* Hilary Nixon (2014). Analyzing Qualitative Data. Her video provides a general introduction for the first 6 mintues, then starting at minute 7 she begins providing multiple examples of coding real text. Feel free to skip to minute 7.

Youtube: https://www.youtube.com/watch?v=BnDUARfEu5I

b. Codebooks

c. Inter-coder Agreement 

* Viera & Garrett (2005) Understanding Interobserver Agreement: The Kappa Statistic.

* Joyce (2013) Blog Entry: Picking the Best Intercoder Reliability Statistic for Your Digital Activism Content Analysis (PDF version of page)

 

2. Presentation

a. Notes:

* Qualitative Data Analysis of Open-ended Item Responses, Qualitative Code Sheets and Codebooks     

* Codebooks for Quantitative Data: Creating a codebook using a Questionnaire (to be added after data collection)

* Assessing Coder/Rater Agreement for Nominal Data 
* Assessing Coder/RAter Agreement for Ordinal and Interval/Ratio Data

b. Video:

* Qualitative Data Analysis - see youtube videos linked above
* Code Sheets and Codebooks (to be added)

* Assessing Rater Agreement for Nominal Data
* Assessing Rater Agreement for Ordinal and Interval/Ratio Data

 

3. Non-graded Activity:

Non-graded Activity: Coding Open-ended Responses

 


9 10/8 1. Continue readings from Qualtative Data Analysis, Code Books and Sheets above

* Instructor note: review and possibly add Siniscalco and Saito (2005) Module 10 Data Preparation and Management



10 10/15

1. Topic and readings for week 10: Classroom Measurement

 

 Van Blerkom Text and John Hills notes on Standard Scores

* Chapter 2 Frames of Reference: Interpreting Test Scores

* Chapter 3 Developing Objectives

* Chapter 5 Validity - please re-read this chapter

* Chapter 10 Producing and Administering Tests

* Chapter 16 Standardized Tests

* Chapter 17 Alternative Ways to Report Test Scores

Hills 1983 Grade Equivalent Scores

Hills 1983 Percentile Scores 

Hills 1984 Normal Curve Equivalent (NCE) Scores 

Hills 1983 Stanines  

Supplemental Reading on Standard Scores

Hills 1984 IQ Scores

Hills 1984 SAT and ACT Scores (may be dated material)

2. Presentation

a. Notes:

Chapter 2 Frames of Reference - Normed vs Criterion Referenced

* Chapter 3 Performance Objectives

* Chapter 5 Test Validity

* Chapter 10 Producing Tests

* Chapter 16 Standadized Tests

* Chapter 17 Alternate Test Score Formats

b. Video:

* No Videos Planned (may add videos as brief review)

 

11 10/22 1. Continue readings from week 10 above

12 10/29

1. Topic and readings for week 12: Writing Test Items

 

 Van Blerkom

* Chapter 6 Completion and Short-Answer Items

* Chapter 7 Essay Items

* Chapter 8 Multiple-choice Items

* Chapter 9 True-false Items (and Variations)

2. Presentation

a. Notes:

* Chapter 6 Completion and Short-Answer Items

* Chapter 7 Essay Items

* Chapter 8 Multiple-choice Items

* Chapter 9 True-false Items (and Variations)

b. Video:

* No Videos Planned (may add videos as brief review)


13 11/5

1. Topic and readings for week 13: Item Analysis

 

Van Blerkom

* Chapter 11 Analyzing Tests

* Grades: Reporting Student Performance (see presentation below)

* Additional readings to be added

2. Presentation

a. Notes:

* Chapter 11 Analyzing Tests and Item Analysis 

* Grades: Reporting Student Performance  

b. Video:

* Youtube Video: Introduction to Item Analysis and Distractor Analysis  


e. Non-graded activity: Item Analysis of Test Responses (to be added)
14 11/12 a. Rasch Models Logic
b. Rasch Models for Tests
c. Rasch Models for Scales
  11/19 Thanksgiving break

15 11/26 a. Rasch Models Guide for Analysis of Scales/Tests
b. Item Response Theory
16 12/3 Finals week - half week