EDUR 9131
Doctoral Research Methods
Note:
single measure for overall construct
measure
 place in clendar
 place in instructions for gropu progject
 place in notes
reversed score = (minimum score) + (maximum score) – actual score souorce: www.theanalysisfactor.com/easyreversecode
Summer 2012
Instructor/Facilitator: Bryan W. Griffin
My personal web pages can be found at http://coe.georgiasouthern.edu/foundations/bwgriffin/
Announcements:
Activity 1 due date extended to June 21.
Course Index (check frequently for alterations)
A. Course Administration
B. Activities that count toward course grade
Andy Field, of the University of Sussex, has students complete a similar project. He provides useful insight into the steps one must take to develop a questionnaire so I recommend you review his project instructions: Design a Questionnaire
C. Activities that do not count toward course grade
D. Course Notes and Assigned Readings
1. Review of Educational Research (linked readings below were taken from EDUR 7130, Introduction to Educational Research)
2. Review of Descriptive Statistics and Inferential Statistics
3. Questionnaire Development
4. Reliability
(a) Read: de Vaus chapter 11; pp 5253; pp. 180186; review "Scaling Checklist" p. 195; William Gabrenya's Reliability
(b) Stability with testretest (scores must be correlated, and means should be similar; correlation and correlated samples ttest)
(c) Equivalent forms, brief overview
(d) Internal Consistency with Cronbach's alpha
(e) Scorerrater; interjudge agreement  Discussed below with coding openended responses
5. Sample Size Determination
(a) Read: de Vaus chapter 6; and Cohen 1992
(b) Excel spreadsheet to calculate sample size or power, or to convert among effect size measures: http://www.bwgriffin.com/samplesize
(c) Yes/No Items (categorical variable status; use for Chisquare type analyses)
To find sample sizes for studies involving simple proportions, such as responses to Yes/No type questions (assumes binomial distribution)
 sample_size_proportion.xls (Excel [or OpenOffice] file download)
 D. A. de Vaus (2004). Surveys in social research (5th ed.). Routledge press. See Tables and pages 80 through 83.
 http://www.raosoft.com/samplesize.html
(d) Effect Size d
 Effect size d: Thalheimer & Cook 2002 Becker 2000 (see also Notes 5 ttests pp. 1213)
 Calculation and Interpretation of d
 pooled vs. control group sd for calculation of d
 Distributional overlap illustration:
 Cohen's recommended effect sizes (refer to Cohen's article linked above)
 Effect size conversions: see Excel spreadsheet to calculate sample size or power linked above in (b), or see content\sample_size_effect_size_conversion.xls
(e) Group Comparisons
To find sample sizes for group comparisons (e.g., ttests, ANOVA type analyses)
 sample_size_in_ANOVA_Maxwell_Delaney.pdf (per group)
 Cohen 1992 Table 2 (per group)
(f) Correlation
Sample sizes for Pearson's correlations
 sample_size_correlation.xls (Excel [or OpenOffice] file download)
 (Source for formula used in above Excel file: Dell, Holleran, & Ramakrishnan 2002)
 An online calculator for sample size for correlation can be found here: http://www.biomath.info/power/corr.htm
(g) Regression  presented below after coverage of regression
(h) Other software or websites for sample size calculations
G Power  free software for sample size determination
 Free sample size software: http://www.psycho.uniduesseldorf.de/abteilungen/aap/gpower3/
 Also downloadable from course web site (9.7MB): content\GPower3Windows.zip
 Online sample size calculators:
6. Coding openended questionnaire responses (conceptual analysis discussed here; see link below by Palmquist 2008 for relational analysis example).
(a) Read: de Vaus chapter 9, and the following works by Le Compte, Palmquist, Glenn, and Tobacco Control:
 Le Compte 2000 Analyzing Qualitative Data. Theory into Practice, 39, 146154
 Palmquist 2008 Content Analysis
 Glenn 2007 Analyzing Openended Questions (blog or discussion board post of simple steps to follow)
 Tobacco Control Evaluation (no date) Content Analysis: A Method for Analyzing Qualitative Data
 Stemler 2001 An Overview of Content Analysis (brief discussion of rater reliability)
 Supplemental reading:
 http://www.bwgriffin.com/gsu/courses/edur9131/readings/content_analysis_tobacco.pdf
 http://www.bwgriffin.com/gsu/courses/edur9131/readings/content_analysis_acell.pdf
 http://www.bwgriffin.com/gsu/courses/edur9131/readings/content_analysis_analyzingqualdata.pdf (within framework of qual. analysis)
 http://www.bwgriffin.com/gsu/courses/edur9131/content/PASS_coding_data.pdf
 http://tobaccoeval.ucdavis.edu/files/ReadyTalk/Analyzing_Qualitative_Data/lib/playback.html (analyzing qualitative data webinar)
(b) Codebooks/Codesheets
 Development: http://www.bwgriffin.com/gsu/courses/edur9131/readings/content_analysis_codebook.pdf
 Samples: http://academic.csuohio.edu/kneuendorf/content/hcoding/hcindex.htm
 Sample Questionnaire and Coding Sheet:
 Sample questionnaire: sample_survey_open_ended_authorship.pdf
 Sample code sheet: sample_code_sheet_open_ended_authorship.pdf
 Coded response #420: open_ended_coding_example_420.pdf
 Coded response #1140: open_ended_coding_example_1140.pdf
 Published version of questionnaire and results presented above: Moore & Griffin 2006
(c) Reliability  interrater and intrarater agreement
 Rating Scales are Categorical/Nominal (angry, fearful, contempt, disgust) or Few Ranked Categories (poor, good, very good)
 Notes on Interrater Agreement with Categorical Data
 Review of interrater agreement: http://www.bwgriffin.com/gsu/courses/edur9131/content/Inter_Rater_agreement.pdf
 Percentage Agreement
 Simple count of agreement/disagreement with percentage agreement calculated for two raters
 Index of concordance = A / (A+D) where A=agreement and D=disagreement
 Mean level of agreement if more than two raters
 Kappa:
 Useful for two raters with unordered rating scale or few ordered options
 Discussion: Viera & Garrett 2005
 Example of kappa use: Uiters et al 2006, see page 4 and table
 SPSS  see Interrater Agreement with Categorical Data
 Requires symmetrical scores  all scores must be present for each rater (Rater A uses 1 to 5, Rater B uses 1 to 4, no kappa)
 Likely not useful if number of categories is large (many themes to coding responses, good for overall judgments [excellent, pass, fail])
 SPSS notes
 http://www.stattutorials.com/SPSS/TUTORIALSPSSInterraterReliabilityKappa.htm
 http://www.uccs.edu/~faculty/lbecker/SPSS/ctabk.htm
 http://www.sma.org.sg/smj/4412/4412bs1.pdf  page 617
 (problem, all categories must be present for both raters): http://www.ats.ucla.edu/stat/spss/faq/kappa.htm (can 3+ raters be used?)
 Supplemental reading on kappa: http://www.childrensmercy.org/stats/definitions/kappa.htm
 Kappa for more than two raters
 See Interrater Agreement with Categorical Data
 Note to instructor: http://www.ccitonline.org/jking/homepage/interrater.html (Excel files for Fleiss kappa; large excel may be problematic)
 Rating Scales are Ordinal with Several Steps, or Interval/Ratio Scale
 Notes on Interrater Agreement with Ranked/Interval Data
 Two Raters
 Correlation between pairs of scores
 Check for mean differences/similarities
 More than Two Raters
 Intraclass Correlation (check description of single vs. multiple judge intraclass correlation estimates from SPSS  definition)
 Cronbach's alpha
 Note to instructor: http://www.bwgriffin.com/gsu/courses/edur9131/content/interrater_agreement_vs_reliability.pdf add this discussion to Interrater Agreement notes.
 Instructor note (students may ignore): Spearman rho; Kendall’s coefficient concordance (W)  mean of all Spearman rho among raters; Cronbach's alpha among raters (yes)? Supplemental notes http://tinyurl.com/47ens9j
 Instructor note: See Liao Hunt Chen 2010 Comparison between interrater reliability and interrater agreement in performance assessments. Annals of Academy of Medicine
 Instructor note: http://www.bwgriffin.com/gsu/courses/edur9131/content/interrater_agreement_vs_reliability.pdf add this discussion to Interrater Agreement notes.
7. Validity  Providing Evidence
(a) Read: de Vaus chapter 4
(b) Summary notes: http://www.bwgriffin.com/gsu/courses/edur9131/content/Validity_According_to_Standards.pdf
(c) Read the following articles:
(d) Illustrative Examples of Validity Evidence  for each, in what ways is validity evidenced?
8. Factor Analysis
(a) Read:
(b) Notes on Conducting Factor Analysis
(c) Reporting Factor Analysis
E. SPSS Tutorials (obtained from various online sources: Dr. McKnight at U. Oklahoma, Dr. Elvers at U. of Dayton, GuangHwa Change at Youngstown State U.)
F. PDF File Creation
Since groups will be required to submit this project, I will only accept PDF attachments from one individual in each group. See detailed instructions on this in the Research Project link above. If you do not know how to create a PDF file, use one of the following sources:
Here are a few free web page that convert files to PDF over the internet:
I use the following free software to create my PDF files (it leaves no watermark): http://www.primopdf.com/
I also use OpenOffice to create some free PDF files. Open Office is a free Office Suite similar to Microsoft Office (Open Office leaves no watermark): http://www.openoffice.org/
If you want further tips and links for converting to free PDFs, read this site http://www.pruittfamily.com/paul/freepdf.htm or this http://en.wikipedia.org/wiki/List_of_PDF_software .
G. Reporting Statistical Outcomes
APA style for ttest, correlation, regression, ANOVA, chisquare: Reporting Statistics (Word document with sample APA styled tables and written presentations).
Add factor analysis.
Course Calendar (tentative plan of course coverage)
Session  Date  Topic 
1 
5/21 
1. Syllabus 2. Research Project  Review Activity 1 Assignment 3. Questionnaire Development
4. Literature Searches  brief review using Google Scholar and GALILEO (search for reading interest measure; also add literature review writing, list vs. integration, early vs. late citations, transitions). 5. Review of basics of educational research 6. Review of Descriptive Statistics and Inferential Statistics 7. SPSS Introduction


5/27 
1. Activity 2 Assignment due Possible videos will be posted for viewing


5/28 
1. Discuss Activity 2 : Instrument
Construction Steps 2. Questions on Activity 1  Brief review of questionnaire item wording and development; guidelines to questionnaire construction (practical tips)
3. Review of independent and correlated samples ttest (read: de Vaus chapter 15)
4. Review of Pearson's Correlation (read: de Vaus chapter 14)
5. Reliability (Spring 2011  covered briefly tips on questionnaire construction, independent samples ttest, correlation, Cronbach's alpha)


5/31 
1. First draft questionnaire you plan to use for the Activity 1 is due
Possible video of Sample Size Determination posted by 6/1 (see de Vaus chapter 6; Cohen 1992)

3 
6/4 
1. Note to students: Send revised group research project questionnaire to instructor for final review. 2. Work through internal consistency exercise: http://www.bwgriffin.com/gsu/courses/edur9131/content/cronbachs_alpha_exercise_data.sav 3. Correlated samples ttest (read: de Vaus chapter 15)
4. Reliability: Testretest
5. Coding openended questionnaire responses (conceptual analysis discussed here; see Palmquist 2008 for relational analysis example). Read de Vaus chapter 9, and the following works by Le Compte, Palmquist, Glenn, and Tobacco Control: 6. Begin Regression, see de Vaus' text pp 280288 Other readings from my course notes in EDUR 8131 (introductory statistics): SPSS Regression Videos
(Spring 2011: Covered all above except correlated ttest, started simple regression)

6 
2/28 
1. Resume Regression
2. Review sample size determination for regression; see Cohen 1992
3. Validity  Providing Evidence (read de Vaus chapter 4) 4. Factor Analysis (read de Vaus chapter 11) (Spring 2011: through regression sample size )

7  3/7 
1. Resume material from chat session 6 2. Illustrate group project analysis steps from data collection to final regression analysis Note to instructor:

Note: The material indented with grey background are comments for instructor about content to review for possible incorporation into course. Students should ignore this information.
Assessing reliability and validity of single item measures:
Validity Standards: Validity evidence (Standards Validity Evidence), and these standards (note also steps for questionnaire development): http://nces.ed.gov/StatProg/2002/std2_6.asp
National standards on validity discussion, useful table: http://www.bwgriffin.com/gsu/courses/edur9131/content/validity_new_standards_11539590.pdf
Link between Research Questions and Methods  (Note to instructor: develop examples; consider design issues)
Add links to online statistical programs, e.g.,
Reading Research Reports  ttests and ANOVA
From "Instrument Construction Steps" activity: Conley, D. T., & Goldman, P. (1995). Reactions from the field to state restructuring legislation. Educational Administration Quarterly, 31, 512538.
Note  Links to possible Reading Research report tables
http://coe.georgiasouthern.edu/foundations/bwgriffin/edur7130/RR_JER_rating_and_learning.pdf
http://coe.georgiasouthern.edu/foundations/bwgriffin/edur7130/RR_backgroundmusic.pdf
http://coe.georgiasouthern.edu/foundations/bwgriffin/edur9131/infantmentalhealthjournal.pdf
http://coe.georgiasouthern.edu/foundations/bwgriffin/edur9131/infantandchilddeveopment.pdf
http://coe.georgiasouthern.edu/foundations/bwgriffin/edur9131/journalofchildneurology.pdf
http://coe.georgiasouthern.edu/foundations/bwgriffin/edur9131/journalofsocialpsychology.pdf
http://coe.georgiasouthern.edu/foundations/bwgriffin/edur9131/ceprittschofgriffin.pdf
Copyright © 2005, Bryan W. Griffin
Last revised on 18 February, 2015 01:26 PM