On Quality of OER

Introduction

This is really just a post to try to get some ideas down about quality in relation to learning resources, perhaps specifically OER. This is prompted by the work of Royce Kimmons, who I don’t think I’ve mentioned on this blog before. In 2019 I saw Royce putting together an Ed Tech Books library which is a fabulous resource. If you teach in ID or Ed Tech I strongly encourage you to go check it out (I say this while leering at a 2003 distance ed book I own while wishing this library had been around back then).

Anyway, Royce tweeted this out recently,

Version 0.1a of a quality assurance system for https://t.co/HRroewyzrb. Authors will go through an extensive checklist with various automated features. Then, once all reqs are met, the author can submit the textbook for certification. Thoughts? #oer #opentextbooks pic.twitter.com/TgSPhmHfGr

— Royce Kimmons (@roycekimmons) November 20, 2019

I took a quick look at the images included (on my phone) and my first thought was that it looked like a list of items that make a resource “look like a textbook”. This is where I admit that on my phone I really mostly saw things like the book cover and media items. My initial reaction thus was that it looked like it reinforced what appear to make a “quality” resource from a traditional course materials perspective. Quality (the kind of quality that is defined as “good content” and “nice graphic design”) has been called out as a red herring in OER circles. Most often in conversations, but I did manage to cycle back and find this 2013 post from David Wiley.

Before being able to address whether or not the items in the quality indicators list are complete or needed, we would have to define what quality is (a clear and concise definition is often missing). JISC attempts to address this lack of definition,

[quote cite=”JISC. (2016)” url=”https://www.jisc.ac.uk/guides/open-educational-resources/quality-considerations”]It is difficult to specify precisely what ‘quality’ means in the context of OER, where discoverability, accessibility and availability are at least as important as the production values they embody. There is a difference in emphasis with OER release in that third parties are actively encouraged to re-use, re-purpose and remix the resources. This, OER advocates claim, leads to higher standards when a long view is taken. However, the issue remains that the quality of learning resources is usually determined using the following lenses: – Accuracy – Reputation of author/institution – Standard of technical production – Accessibility – Fitness for purpose[/quote]

but do not come to one concise definition, and fall back on what look like fairly traditional perspectives on quality learning resources.

Looking more closely (aka on a bigger screen) at the list of items, it looks like there is quite a bit of overlap with the JISC identified lenses. So with those lenses in mind maybe I’ll jot down some initial thoughts in each category.

Accessibility

Image descriptions is often a starting place to look at accessibility (oft overlooked during writing!). Off the top of my head, the most overlooked accessibility item comes before this, styles. Over the last decade, working with SMEs who submit content via GDocs or Word have had a variety of approaches to organize the heirarchy of their content. I can probably count on one hand the number of SMEs that properly used stylized headings in those documents after 100ish course developments. As the books at ed tech books could be presented in a variety of formats as well, links should be considered (for example, I’ve heard it’s better from an accessibility standpoint to have links open in the same tab, not a new one). Overall on accessibility, without making this QI list too large I might include the following in the check list based on BC Campus’s Accessibility Toolkit:

• Content Organization
• Images
• Links
• Tables
• Formulas
• Font-Size
• Colour Contrast

Book Formatting/Information

I might collapse these two together into just “Book Information”.

Chapters (Content and Formatting)

The reading level item is a novel inclusion I haven’t seen on other lists. I know in Pressbooks for example you can identify target audiences. Perhaps this reading level could somehow be aligned with/checked against that? I see Headings under formatting, which I think addresses the “content organization” I included under accessibility instead.

Instructional Guides

This is one where I’m not sure these should be required items per se. There are plenty of quality open and digital texts that do not have Learning Objectives at all, let alone at the chapter level. I can think of An Urgency of Teachers, Creating Online Learning Experiences, or iDesign’s iDEA Book. The inclusion of Learning Objectives are not an indication of quality in and of themselves. After all, even in published textbooks that cost over $100 I see objectives like “After reading this chapter, students will be able to: understand the role-and limits-of criminal law.” (colour me skeptical).

While I agree that there is value in ways for learners to practice, and that quizzes can be a tool for that, presenting “practice quizzes” as the quality indicator is too narrow. Teaching in a Digital Age by Tony Bates is one example of a quality open textbook (translated into multiple languages and over 500K downloads) that does NOT include quizzes. It still has activities though. As a side note, it’s not the chapters that include Learning Objectives, although he includes them for each “part”, if you consider the architecture of Pressbooks. British Columbia in a Global Context is another example of a highly adopted (and therefore I’d assume high quality) resource that doesn’t use quizzes, but instead includes a bunch of different learning activities. The Instructional Designer on the project once mentioned that the approach used was meant to lean towards constructivism above other learning approaches. The number of practice items per word count seems a bit arbitrary. I like the phrasing of “each full length chapter” for some of the items. In contrast, if it had to be quizzes for example, then the qualifier from my perspective wouldn’t be about word count for how many quizzes were needed, but that each learning objective was aligned to the quiz/quiz questions. Finally, the Professional Communications open textbook from Olds College includes both quizzes (i.e. check your understanding) and other instructional guide elements including in class activity write ups, assignments, and rubrics. (Maybe instead of “Quality Indicators” the list could be called “What’s Inside” or “What’s Included” or “Book Information Overview”)

Perhaps this would be a section that has a branching element, where if it’s more traditional textbook style with instructor guides then clicking “instructional guides included” then there are further options provided. With my instructional design hat on, I would say that I’m looking for alignment here more than anything else. If there aren’t objectives, does the intro list the topics or key concepts to be presented in a chapter? Did those actually appear in the chapter? Was there some way for the reader to reflect back on that or check their understanding? A more general approach would be more inclusive of different teaching, learning, and writing perspectives.

Media

For image use, requiring at least one image once chapters reach 500 words does not seem like it is necessary. Again I can point back to the Urgency of Teachers or Creating Online Learning Experiences books where longer blocks of text are used (or text and tables, or text and video, etc.) and they communicate just fine. I generally look at “image use” as the purpose, not the frequency. Many courses I’ve worked on have had loads of images placed in the materials. This is especially heavy with courses that have a lot of PPTs. Are those course materials better because they have images? Largely no. I’ve seen so many PPTs, for example, jammed full of images that distract from the learning. The images are meant to be decorative and “engaging” but end up just looking awful and being distracting. The four purposes of images outlined in the Professional Communications Open Textbook are:

• decorative
• representational
• analogical
• organizational

An indication of quality for me then would be in aligned with JISC’s lens of “fit for purpose”. For example, organizational images included in the book clearly provide structure to information, visually define relationships, or illustrate connections. Even taking a step back from that, just looking at the distribution of images into those four purposes would indicate quality use in some way. If I have learning materials that only use decorative images, that doesn’t spell quality for me.

For Graphic Quality, perhaps a scale would work better than a tick box. “Images are consistent stylistically”, clear, and convey the desired meaning. I might have to think more about how this could be parsed.

Conclusion

That’s my quick, throw ideas at the wall, reflection on Quality Indicators for open textbooks. What are yours?