A Cautionary Paper on Complex Models

Thanks to Paul Barrett for alerting us to this newly published paper: Saylors, R., & Trafimow, D. (2020). Why the increasing use of complex causal models is a problem: On the danger sophisticated theoretical narratives pose to truth. Organizational Research Methods (https://doi.org/10.1177/1094428119893452 ), In Press, , 1-14. [paywall]

As pointed out by the authors, “As use of complex models increases, the joint probability a published model is true decreases.”
The paper comes with a calculator to compute said probability:
https://practiceoftheory.weebly.com/a-causal-models-probability-of-being-true.html

An analogous concern in OOM is that as a path model increases in complexity, fewer and fewer individuals will be traceable through the model. It is easy to imagine a complex path model in which not a single person can be accurately traced through all of the links. What use would such a model be as an explication of causes and effects? Of course, this information can only be known if the researcher attempts to perform such person-centered analyses.

2019 Posts

  • October 27th, 2019. A special thanks to Chris Cunningham at the University of Tennessee-Chattanooga for his invitation to present at the 15th Annual River Cities I-O (RCIO) Psychology Conference, October 25-26, 2019. I was to present a talk titled “Person-centered data analyses: Observation Oriented Modeling as an alternative and rational data analytics approach.” Unfortunately, due to illness I was not able to attend, and our attempts to present via the internet were not successful. The Powerpoint slides are nonetheless available upon request.
  • Congratulations (!) to Valentine, Buchanan, Scofield, and Beauchamp on the publication of their paper “Beyond p values: Ultilizing multiple methods to evaluate evidence” published in Behaviormetrika, 46(1), 121-144. They compare NHST, Bayes, and OOM methods for analyzing repeated measures data.  
  • March 20th, 2019. A special thanks (!) to David Trafimow and the faculty and students at New Mexico State Universityfor hosting a talk on OOM. It was a pleasure and an honor to visit, especially given my affinity for the Desert Southwest. Godspeed to Dr. Trafimow and several of his colleagues as well as they continue to fight the good fight against NHST.  
  • January 22nd, 2019. Congratulations to Craig and Abramson on their recent publication covering Ordinal Pattern Analysis! It can be found in the International Journal of Comparative Psychology. Craig, D. P, & Abramson, C. I. (2018). Ordinal pattern analysis in comparative psychology – A flexible alternative to null hypothesis significance testing using an observation oriented modeling paradigm. International Journal of Comparative Psychology, 31. Retrieved from https://escholarship.org/uc/item/08w0c08s.

2018 Posts

  • October 15th, 2018. A special thanks to Kevin Weinfurt and the faculty and staff at Duke University Medical School, Department of Population Health Sciences for hosting two talks and a workshop on OOM! I am also particularly grateful for the commentary of Daniel Mark and Rick Hoyle, and the discussion time with Bryce ReevePatrick Curran, and others. With his permission, I’ve uploaded Rick Hoyle’s Powerpoint slides here for viewing.
     
  • July 17th, 2018. A special thanks to Elizabeth McClure from the Lego Foundation. She visited my lab at Oklahoma State University this week to study OOM with the hopes of bringing more person-centered methods to projects sponsored by the foundation.
     
  • May 15th, 2018. Congratulations to David Trafimow and many other authors for their paper titled: Manipulating the alpha level cannot cure significance testing online in Frontiers in Psychology (Quantitative Psychology and Measurement). We published this paper in response to arguments for lowering the standard p-critical value from .05 to .005 as a way to improve psychological research and to help cure the replication problem in psychology. As we have argued extensively in our OOM papers, NHST should largely be abandoned and replaced with integrated modeling, non-parametric (pattern-based) types of analyses, randomization tests, and — of course — exact replication.  
     
  • April 14th, 2018. Congratulations to Eliwid Medellin who presented a poster at the 64th annual meeting of the Southwestern Psychological Association in Houston, TX! Here’s his poster titled Logical Hypothesis Testing. He reports results from two studies showing how logical combinations can be evaluated in OOM.  Eliwid was also selected as the Student of the Month for May of 2018. Congratulations, Eliwid!
     
  • April 14th, 2018. Congratulations to Meggie Baker who presented a poster at the 64th annual meeting of the Southwestern Psychological Association in Houston, TX! Here’s her poster titled Person-Centered Methods for Personality Profiling. She reports results from her summer project from OSU’s American Indians Into Psychology program at OSU. Her study examined the use of the Ordinal Analysis option in OOM to detect group personal profiles.
     
  • March 23rd, 2018. A sincere thank-you to the faculty, students, and staff of Baylor University! A particular word of gratitude to Alex Beaujean, who orchestrated my visit during which I delivered a talk and short workshop on the OOM software. It was a pleasure meeting the wonderful faculty, and it was an honor to visit Baylor University and Waco, Texas!
     
  • March 3rd, 2018. I presented a brief talk titled: Methodological Incorrigibility and its Cost to Innovation. I presented at the 9th Midwinter Meeting of the Society for Theoretical and Philosophical Psychology, Phoenix, AZ. March 3rd, 2018. My talk was part of a symposium titled: Scientific Psychology’s Troubling Methodological Incorrigibility: Its Nature, Sources, and Possible Solutions, James Lamiell, Chair.

2017 Posts

  • December 23rd, 2017. Congratulations to Evan Jordan and David Thomas who published a paper titled Contagious Positive Affective Responses to Laughter in Infancy in Archives of Psychology (Vol 1, no. 2). The paper is open access and demonstrates use of the Ordinal Pattern (Concatenated Orderings) analysis in the OOM software.
     
  • December, 2017. Congratulations to Lisa D. Cota, newly minted OSU Ph.D.! She defended her dissertation (in my absence, due to a medical issue) titled An Examination of Alternatives to Null Hypothesis Significance Testing. She conducted simulations comparing OOM to Bayesian statistics. Here’s the summary
     
  • October 29th, 2017. All of the dated instructional videos have been deleted. New videos are included under the Instructional Videos link to the right. With any luck, I’ll build a large set of videos in the coming months so that a person can learn the software through independent study.  
     
  • September 8th, 2017. I’ve published a short comment regarding NHST in Basic and Applied Social PsychologyComment on Locascio’s Results Blind Manuscript Evaluation Proposal.
     
  • September 4th, 2017. Congratulations to Sebastian Sauer who has published a paper titled Observation Oriented Modeling Revised from a Statistical Point of View. The paper appears in the latest online pre-prints at Behavior Research Methods. Here’s an interesting quote from the paper: “In sum, our refined OOM approach is optimal in terms of the regression of indicator matrices and can be seen as a naive Bayes classifier, methods that are well known in statistical learning (Hastie et al., 2009). Unless stated otherwise, we have used this revised version of OOM throughout the paper.” In the Build/Test Model feature of the OOM software, Sauer’s refined approach can be accessed by selecting the “Conforming Only” normalization option. As pointed out by Sauer, the results will be equivalent to a Bayesian classifier.   
     
  • August 15th, 2017. We’ve published a paper titled Four Bad Habits of Modern Psychologists in Behavioral Sciences. I will be posting the data sets and instructional videos on how we analyzed the data in OOM in the near future. Citation: Grice, J., Barrett, P., Cota, L., Felix, C., Taylor, Z., Garner, S., Medellin, E., & Vest, A. (2017). Four Bad Habits of Modern Psychologists. Behavioral Sciences, 7(3), 53. doi:10.3390/bs7030053
     
  • February 14th, 2017. Congratulations to Michaluk, L. M., DeVore, S., Stewart, G. B., and Stewart, J. C. for publishing their recent book chapter New Directions in Educational Research Methodology and Analytical Techniques [Chapter 4, pp. 90-131, in D’Souza, M. J., SJ (Editor) Teaching and Learning in Higher Education: Emerging Trends, United Scholars Publications, USA, 2016]. They compared OOM with traditional statistics in analyses of gender differences in test scores and self-efficacy.
     

2016 Posts

  • August 2nd, 2016. We have published a paper demonstrating the new Threshold Analysis in OOM which can be used as an alternative to logistic regression analysis. The Threshold Analysis also permits the automatic construction of logical combinations of orderings. Here’s the paper: A Simple and Transparent Alternative to Logistic Regression. Advances in Social Science Research Journal, 3(7), 147-165.
     
  • July 7th, 2016. Congratulations to Gatobu, Arocha, and Hoffman-Goetz on their recent paper Numeracy, Health Numeracy, and Older Immigrants’ Primary Language: An Observation-Oriented Exploration published in Basic and Applied Social Psychology. Here’s a link to the abstract: http://www.tandfonline.com/doi/full/10.1080/01973533.2016.1197129
     
  • May 27th, 2016. An updated version of the OOM software was posted along with an updated version of the technical manual (in MS Word format). Please follow the link to the right to download the program and manual for free.
     
  • March 30th, 2016. Congratulations to Lübke and Sauer who presented a poster at the annual DAGStat meeting in Göttingen, 16 March 2016. Here’s a link to the poster: https://www.researchgate.net/publication/301364374_Observation_Oriented_Modeling_OOM_as_a_Data_Analytic_Tool   and a link to the data: https://osf.io/3j4xr/. They conclude: “Observation Oriented Modelling [Grice, 2011] was proposed to overcome some of the problems in the application of statistical inference methods in human and behavioural sciences. We review and refine this approach and show connections to methods well known in statistical learning. From this point of view it can be shown that Observation Oriented Modellng can indeed be appropriate for some of the tasks in the analysis of observed data. For a practical example, the revised method is demonstrated by analysing the effect of mindfulness training on attentional processes.”
  • January 11th, 2016. Craig et al. have published yet another paper using OOM! Here’s the citation and link: Craig, D. P. A., & Abramson C. I. (2015). A need for individual data analyses for assessments of temporal control: Invertebrate fixed interval performance. International Journal of Comparative Psychology – Special Issue on Timing and Time Perception, 28, 1-39.

2015 Posts

  • September 10th, 2015. Our paper on an alternative to ANOVA has been published in Sage Open. Here the citation and link: Grice, J. W., Craig, D. P. A., & Abramson, C. I. (2015). A simple and transparent alternative to repeated measures ANOVA. Sage Open, July-September, 1-13. doi: 10.1177/2158244015604192
  • August 30th, 2015. David Craig and co-authors have published another study on learning in which they used OOM. Here’s the citation: Craig, D. P. A., Varnon, C. A., Pollock, K. L., & Abramson, C. I. (2015). An assessment of horse (Equus ferus caballus) responding on fixed interval schedules of reinforcement: An individual analysis. Behavioural Processes, 120, 1-13. Here’s a link to the article’s abstract.
  • August 10th, 2015. We have published an OOM article in Basic and Applied Social Psychology in a special issue devoted to critiques of modern statistical mediation analysis. All of the papers in this special issue are worth reading and seriously considering. Here’s the link to the abstract of our paper. BASP is not open access; consequently, if you have trouble locating the article, please contact me at james.grice@okstate.edu.
  • July 24th, 2015. I have published an OOM article in Frontiers demonstrating how to diagram and analyze an integrated model. Personally, I have found this diagramming task to be both challenging and rewarding because it forces me to sit down and really think about the structures and processes underlying the data. Check out this article and see what you think: http://journal.frontiersin.org/article/10.3389/fpsyg.2015.01007/abstract
  • May 6th, 2015. Congratulations to Charles Abramson and his co-authors on their recent publication: Abramson, C. I., Craig, D. P. A., Varnon, C. A., & Wells, H. (2015). The effect of ethanol on reversal learning in honey bees (Apis mellifera anatolica): Response inhibition in a social insect model. Alcohol, 49, 245-258. They employed OOM in their analyses. Personally, this is one of my favorite quotes from the article: “These analyses are concerned with model fit and are not concerned with rejecting or failing to reject a hypothesis. If no null hypothesis is made, it is impossible to be correct, or incorrect, about rejecting something that does not exist” p. 248-249. If only a majority of psychologists could demonstrate such fortitude and understanding, we might actually make some progress as a science. 
  • April 16th, 2015. I presented a talk on OOM to the faculty and students at The University of Tennessee Chattanooga. It was a pleasure meeting the faculty and students who mainly hailed from the department of psychology, and I owe a special thanks to Chris Cunningham for orchestrating the visit.
  • April 14th, 2015. I presented a talk to researchers at the Center for Military Psychiatry and Neuroscience, Walter Reed Army Institute of Research, in which I highlighted the many problems with the p-value in psychology. I also demonstrated OOM briefly. It was a pleasure and an honor to meet everyone at this historic and noble facility. I’ve posted the annotated Powerpoint slides for this talk to the left under Presentations (NHST Critique and OOM).
  • April 8th, 2015. The editors of Basic and Applied Social Psychology have banned NHST! OOM does not employ NHST procedures as emphasis is placed squarely on pattern recognition and accuracy (the PCC index). When a probability statistic is requested, it is most often a simple, distribution-free randomization test; otherwise, a simple binomial can be used on occasion.
  • March 19th-20th, 2015. I presented a talk and OOM workshop to the faculty and students at The University of Central Arkansas. It was a pleasure meeting the faculty and students in the department of psychology and counseling, and I owe special thanks to Elson Bihm for orchestrating the visit.

2014 Posts

  • On October 10th, 2014, I presented a talk to the faculty and students at Georgetown University entitled Observation Oriented Modeling: A Common Sense Alternative to Modern Data Analysis. I also delivered a workshop demonstrating the OOM software later that afternoon. It was a pleasure meeting the faculty and students in the department of psychology, as well as George Henriques and Mark Fox who drove in to hear the talk. 
  • On September 19th, 2014, I presented a talk entitled Preparing Students for Research in the 21st Century with Observation Oriented Modeling. It was presented at the 8th annual conference of the Oklahoma Network for the Teaching of Psychology, Stillwater, OK.
  • July 28th, 2014. Congratulations to Craig et al. on the publication of their latest honeybee learning paper published in PlosOne. Graphs and simple statistical analyses in OOM are all you need!
  • July 27th, 2014. We have submitted two papers for review comparing OOM to repeated measures ANOVA and to logistic regression. If you are interested in examining these papers please send me an e-mail: james.grice[att]okstate.edu.
  • July 27th, 2014. While working on a paper comparing OOM to repeated measures ANOVA, my co-author (David Craig) fortunately found the work of Warren Thorngate, Ph.D. Dr. Thorngate created a method of data analysis he named Ordinal Pattern Analysis. You can read about it in JASSSAdvances in Psychology or in Valsiner’s book The Individual Subject and Scientific Psychology. The Ordinal Pattern Analysis in OOM accomplishes most of the goals in Dr. Thorngate’s technique, but using a visual tool for defining the expected ordinal patterns. Credit is certainly due to Dr. Thorngate regardless, as he first published the basic idea behind the analysis.
  • June 9th, 2014. Lisa Cota presented OOM at the 4th annual meeting of the Association for the Assessment of Learning in Higher Education (AALHE) in Albuquerque, NM. Her Powerpoint slides are linked to the left. OOM appeared to be well received, particularly given the nonparametric nature of assessment data and the desire to focus on individual student outcomes. Interestingly, David Eubanks, an expert in assessment analysis, advocated for the same sorts of pattern-based, visual and nonparametric analyses found in OOM in his talk From Data to Understanding. Personally, I’m excited to see what new features I can add to the software to make it even more useful for assessment work.
  • March 25th, 2014. The Innovative Teaching paper, with data sets and videos, is now published in the newest journal offered by Ammons Scientific. The links to the paper, data sets, and videos are to the left!
  • March 16th, 2014. Lisa Cota, a graduate student at OSU, and myself presented OOM to faculty at OSU as part of assessment training. The video was shot and produced by the outstanding folks of ITLE at OSU. You can watch the video here. Sorry about the coughing! I think it was mild allergies or a cold.
  • March 10th, 2014. I’ve added a link to a recent paper by Dinges et al., 2013, published in the Journal of Experimental Biology. Congratulations to Chris, a graduate student at OSU, and his co-authors! 
  • March 3rd, 2014. The OOM software is now being offered for free! You can download the latest version of the software using the link to the right. The software is still under copyright protection and should not therefore be altered or used for profit without written permission from Elsevier. 

2013 Posts

  • November 21st, 2013. Version 2 is ready! The latest version has several new analysis features, and the output across analyses is streamlined and more consistent. The technical manual has also been completely updated.
  • August 4th: A graduate student at OSU has been running OOM on his Mac using Winebottler (http://winebottler.kronenberg.org/). This might work as an alternative to booting a Mac in PC mode or using other software like VMWare Fusion.  CodeWeaver’s Crossover is a commercial product that will also permit you to run OOM on a Mac. 
  • July 26th: Added video showing new import feature. Data from SPSS, SAS, Excel, and other programs can quickly be imported through .csv files. Value labels can also be imported from SAS and SPSS.
  • July 11th: Katy Valentine’s and Erin Buchanan’s paper in which they used OOM has been published in the Journal of Cognitive Psychology. Valentine, K.D., & Buchanan, E.M. (2013). JAM-boree: An Application of Observation Oriented Modeling to Judgments of Associative Memory. Journal Of Cognitive Psychology, 25(4), p. 400-422. Congratulations! 
  • April 9th, 2013: Brian Haig’s invited presentation for the Annual Meeting of the Southwestern Psychological Association (Fort Worth, TX, April 5-7) was a smashing success. The title of his talk was, How Research Methods Textbooks Mislead Psychologists about the Scientific Method. You can view his Powerpoint slides here.
  • March 17th, 2013. Updated several links. Added link for Reviews of the OOM book.
  • Feb 3rd, 2013. Beta testing Version 2 of the OOM software is ready for release. If you own a copy of Version 1, please e-mail me if you are interested in Beta testing Version 2 (james.grice]aatt[okstate.edu).
  • New paper published by Erika Brown demonstrating mediation analysis in OOM.