As we continue talking about the organizational practices that support and reinforce the six principles of CBTE, our focus for today will be on continuous improvement. When a community of people (i.e., a school) is following the principles and practices we have been describing over the past few weeks, the opportunity to evaluate and improve the school’s various programs and processes is much greater than it was in the conventional approaches that have come to be the norm.
For example, in Kairos there are thousands of participants (i.e., students or mentors) and dozens of partners – all working together in light of the principles of CBTE. As a result, there is a greater amount of data from which we can glean insights on what is working and what isn’t and greater visibility into the processes and systems creating the data. With the blessing of new data, we are presented with an opportunity to think differently about how we improve learning, programs, and the institution. Rather than simply cataloging data in assessment reports that are reviewed every few years with the goal of doing macro-level program reviews, institutions engaged in CBTE are able to engage in ongoing change thereby creating a culture of continuous improvement.
The concept of continuous improvement is not new. In our experience, the best way to learn about it is to study the product development process utilized by several contemporary software companies. We find CBTE allows institutions to combine new opportunities for observation and evaluation with the approach to software development trumpeted by people like Jason Fried, founder and CEO of Basecamp and a leading voice in contemporary software development practices. With this approach we can improve programs and student learning at both the micro (individual student) and macro (entire programs and institutions) levels – often closing feedback loops in real time as we respond to the data created when we follow the principles and practices of CBTE.
In practice, this means CBTE creates space for at least three continuous improvement accelerants: 1) data-driven micro and macro improvements, 2) empowered feedback loops, and 3) distributed ideation.
Data-Driven Macro & Micro Improvements
In a conventional system, a school of 150 students might get input from a few points of connection: the students, the faculty who teach courses in the program’s curriculum, a few engaged board members, and the staff who are connected to the assessment process. There may also be mechanisms for gathering a smattering of data from “external” constituencies. This is a good description of where Sioux Falls Seminary was in 2013.
We launched the Kairos Project in October of 2014, which was guided by the principles and practices of CBTE. After a few years, about 50% of the students in the school were part of the Kairos Project and our connection points expanded exponentially. With 75 students, the school had well over 200 engaged mentors, 60 to 75 different ministry contexts, new/additional faculty who are serving as mentors, mission-minded partners who helped to create the program, more staff members who interacted with students along the way, new stakeholders who were more engaged in the Project, and a fresh array of learning experiences that were as unique as the 75 students.
Each student represented a network of connections to our system of assessment. We knew more about student progress and learning, program effectiveness, institutional vitality, mission fulfillment, and stakeholder engagement than we thought was possible. Because it is possible to be overwhelmed by the data created or made visible by a CBTE program, we had to find a way to capture, synthesize, and respond to it. This is why CBTE required the practice of continuous improvement.
A commitment to continuous improvement lessens the burden of this exponential increase in assessment data because it empowers institutions to make data-driven decisions at both the micro (individual student) and macro (program and institutional) levels. Rather than collecting mounds of data for future use, we respond to these streams of data in real time. For example, incremental changes can be made to seminar schedules, curriculum documents, resources within an LMS, even the educational path of particular students. These micro changes are driven by data and made without the need for wide-ranging conversations. Over the course of a few months or a semester, data can be aggregated to suggest macro-level changes (e.g. adjustments to the user interface of an LMS, new language within a competency or outcome statement, refreshed orientation material for new students, curricular enhancements, etc.). With the increased presence of timely and actionable data, a school engaged in CBTE can make ongoing change that consistently improves the program. These changes are what close the feedback loop.
Empowered Feedback Loops
Well-designed and well-meaning assessment systems can come to a grinding halt when confronted with the conventional decision-making processes of higher education. The assessment data may reveal the need to adjust the curriculum and this begins a six-month conversation on what to change. After we decide what to change, we develop a plan to implement the new curriculum only to find that another change is needed (because the previous data is now over a year old). Another example could be the one a friend of mine shared with me regarding the process for appointing faculty in the teaching hospital where she worked. It served as a stark reminder of the reality that well-meaning processes can push against the need for real-time action. During the early parts of the pandemic in March 2020, a teaching hospital was overwhelmed with the need for doctors. While several doctors were available, willing, able, and qualified to serve, they could not do so because each of them had to be approved as faculty in addition to being approved as doctors – a process that normally took 4 to 6 months. It is quite easy for the process of decision-making and implementation to derail any momentum that might be created by assessment data or on-the-ground feedback.
We believe CBTE creates an opportunity to approach this challenge from a fresh perspective. With its focus on ends rather than means, CBTE invites institutions to hold lightly many things which were once sacrosanct. It is not that we completely abandon those things we once held dear. Rather we give them appropriate weight within a system that says outcomes, demonstrated by observable behavior and/or data, are the guide. By adjusting this weight distribution, we are able to develop faculty, staff, mentors, partners, board members, and administrators who embrace and embody this new reality thereby empowering people to make decisions without the need for complex and cumbersome systems of oversight.
Let’s take curriculum adjustments as an example. In a conventional system, conversations about curricular adjustments will no doubt be based on good assessment data and will take into account the program goals and outcomes. Eventually, however, the conversation will turn to a focus on means. We will talk about which courses, assignments, and activities students must complete in order to address the issue noted in the data. After several conversations about the means, we will develop a plan for implementing the idea and a way to assess the change. Again, this is all well-meaning and focused on enhancing quality. The challenge with this approach, however, is that it assumes: 1) the means will deliver the intended outcome, 2) that each student will respond to the means in the same way, and 3) that a small group of people who are not connected to the day-to-day life of each student are the right group to make this decision.
In a CBTE system, the mentor team, who is tasked with holistic and general assessment of integrated outcomes, is trained to conduct assessment in a particular way in light of program goals, rubrics, discrete competencies, and institutional mission. As a result, when they notice a student is not adequately demonstrating competency, they have the power, wisdom, and competence needed to make changes in real time. They can adjust the learning pathway to account for the areas in which the student needs more development. The team gathered data through a consistent process of assessment, reflected on the data in light of their “close-to-the-ground” knowledge of the student and her context, worked with the student to make adjustments in real time, and closed the feedback loop by revisiting the same assessment process that surfaced the need for change. In a conventional system we gather feedback and slowly close the loop. In CBTE, we gather feedback and close it in real-time – all the while gathering data that can be aggregated to inform large-scale institutional/programmatic assessment.
Now, imagine how that plays across all of the Kairos community. With students, mentors, and partners spread out around the world, data is being constantly generated thereby creating feedback loops that are opened and closed on a daily basis. It also creates a distributed ideation network.
Distributed Ideation
With a distributed community of people who have on-the-ground access to real-time qualitative and quantitative data about what is working, where things need to be enhanced, and how students are progressing, one should expect ideas for how to improve the program or various aspects of the organization to come from anywhere – and this must be encouraged! Each of those additional connections points represents a different perspective, a different source of feedback, and another engaged voice. Improvements that come from a program director, vice president, board member, office manager, student, and partner are all valid. The source of the idea does not determine its value. If we confine ourselves to the expectations of those with the most power, we will undermine the new data provided by CBTE.
The challenge is to create mechanisms for gathering that feedback, synthesizing it into actionable ideas, and responding to it. Many of the improvements we have made to Kairos have come from this extended network of connections. They do not always come from our full-time faculty and staff. In one case, it was a software engineer from a partner organization that suggested the layout for a learning experience library. In another instance, it was a denominational connection that suggested a different approach to clinical pastoral education. The key is to recognize that this vast array of connections creates not only a dispersed network of learning but also a distributed system of ideation.
Over the years, as any school moves more fully into the principles and practices of CBTE, they will find that it becomes easier, even natural, to focus on continuous improvement. Given our experience with Kairos, it is hard to imagine any other way. While we did not set out to develop a system of continuous improvement, we learned very quickly that we need to expect – even embrace – ongoing, unending, and adaptive change.
This post originally appeared on the Kairos University blog.
