Airport ruminations: Glass on evaluating scholarly e-pubs

Thoughts from the Mpls, MN airport…three hours to kill.

Why should anyone trust the data analyses and interpretation I have posted to date regarding my Carroll Analyses of 50 CHC-designed tests (3-17-05 and 3-18-05 )? I’ve asked this question of myself many times and had to resolve it to my satisfaction before launching this blog. The tipping point came when I recently read an interview in the Educational Researcher (Vol 33, No 3, April 2004 ) with one of the leading and respected educational researchers of our times—Dr. Gene Glass.

First---who is Gene Glass and why are his thoughts worth merit?
As stated in this interview article:

• Gene V. Glass is a Regents’ Professor of both Educational Leadership & Policy Studies and Psychology in Education at Arizona State University. He has won the Palmer O. Johnson Award for best article in the American Educational Research Journal (AERJ) not once, but twice (1968 and 1970). He has served as President of the American Educational Research Association (AERA) in 1975, co-editor of AERJ (1984–1986), editor of Review of Educational Research (1968–1970) and Psychological Bulletin (1978–1980), is executive editor of the International Journal of Education and the Arts (since 2000), and serves as editor of Education Policy Analysis Archives (since 1993) and Education Review (since 2000).
• Dr. Glass has also served on the editorial boards of 13 journals and has published approximately 200 books, chapters, articles, and reviews. Dr. Glass is perhaps best known for his role in the development of the quantitative research synthesis technique known as meta-analysis. Not shabby! And, Dr. Glass has been a vocal advocate of on-line e-journal publications, and, more recently, the publication and posting of non-peer reviewed manuscripts (he has written) for dissemination to other scholars.
  

I am in no way claiming to be anywhere near the same caliber of a researcher/scholar as Dr. Glass, but I find his recent thoughts on publishing manuscripts on the web of interest. His thoughts and logic are largely responsible for my current experiment in dissemination of select research that suffers from the “languishing or banned to forgotten sectors of the hard drive” syndrome.

So…what did he say and why should it make you consider the results and interpretations I post as credible? Or, as stated by Glass in his ER interview (note – all italics are emphasis added by me):

• "Now, the questions in most people’s mind are: Won’t there be chaos if everybody 'publishes' anything they want? How will we be able to separate wheat from chaff, truth from error, pure gold from garbage? There are answers, of course, and more questions."

Glass on reaching a wider audience
According to Glass (in the ER interview), and consistent with my belief in the need for more timely dissemination of scholarly insights:

• “Some papers that I have made public through my own Website are downloaded a half dozen times a day, often from places that have no access to the traditional journals. That’s reaching a wider, larger audience than paper journals reach.”
  

Glass on open access to knowledge
Furthermore, my beliefs and experiences coincide with those of Dr. Glass:

• "Let’s get down to brass tacks. Scholarly communications are not essentially about paper vs. Internet “packets”; they are about commercialization vs. open access to knowledge. My experiences with publishing e-journals over the last decade have taught me that many more people than we ever dreamed want access to educational research: parents, teachers, professionals of many types, students and scholars far from the United States who cannot afford our books and journals.
  

Glass on self-empowered reviewing/filtering in your area(s) of expertise
I resonate to Dr. Glass’s observed paradox in the importance of referred publications. According to Glass:

• "There’s an irony hiding in this business of “refereed publication.” If I am vaguely interested in a topic that isn’t at the center of my work, then I’m prone to pick up a peer-reviewed journal that rejects 90% of everything sent to it. I simply don’t have the background to plow through tons of stuff and judge it for myself; trust the experts."

This makes eminent sense to me. I also must rely on journal reviewers and editors to bring to my attention the best reviews and empirical studies in areas outside of my professional research sandbox. I simply do not have the time.

However, if someone is going to play in my professional sandbox (i.e., present research regarding CHC theory and measurement), I believe I am best served by being my own filter of all available information. As stated by Glass:

• "But if you are talking about things that come to the core of my own research (let’s say, the re-segregating effects of school choice policies), then please don’t filter out anything for me; I want to see whatever anyone is writing on the subject and I want it right away. I’ll judge it for myself. When we interviewed a sample of “hard scientists” about their reading habits, they said exactly that."

Bottom line
It is my belief that the readers of this blog must serve as their own filters, judges, and reviewers of any data-based results I present. Those knowledgeable in the areas where I present data and/or hypothesis are in the best position to evaluate the integrity of the information presented. In hope that the Blogs “comments” feature will provide an outlet for those who want to question or critique what is presented. In the pursuit of knowledge that (ultimately) may impact the lives of individuals (e.g., individuals referred for comprehensive psycho-educational assessments), I believe in open access and the timely dissemination of results to the widest audience possible. Thus, this blog, which will be judged by the readers and users.

Finally, like Dr. Glass, I still believe in, and continue to pursue, the publication of scholarly results via the mechanism of peer-reviewed journals. There is more than one means by which to contribute to our respective professional domains of knowledge.

Ok….within the next few days I will return to interpretation of the Carroll Analysis of 50-CHC designed tests…starting first with the most interesting Gf factor. And, in the spirit of accountability for the results I present, when done, I will post the correlation matrix analyzed for independent analysis by others.