Taking a Deep Dive Into Data to Improve CTE Program Quality

How a two-time Pacesetter winner is making data work for its teachers and students

By Kirsten Sundell, SREB

Teachers and school leaders are awash in student data, but many struggle with how to assess its quality and validity, store it safely and harness it to improve curriculum and instruction.

At Penta Career Center in Perrysburg, Ohio — winner of a Gene Bottoms Pacesetter School Award in 2018 and 2020 — career and technical education teachers and leaders are improving student and program outcomes using a step-by-step process to collect, organize and analyze assessment data in their CTE programs.

The suburban career-tech center serves 16 school districts in five counties with six sophomore exploratory programs, eight “level one” CTE programs, 26 on-campus CTE programs for juniors and seniors, and two senior-only credentialing programs. The center averages more than 4,000 students on campus each year in grades 10-12. About 41% of students are economically disadvantaged, and 40% of students have special needs.

Penta’s Six-Year Assessment Literacy Initiative

When the Ohio Department of Education launched its teacher evaluation system in 2013, Penta needed a way to capture student growth. Leaders and teachers saw that students were performing well on teacher-created assessments, but not so well on state end-of course tests, called WebXams. Staff recognized that the two types of assessments were not aligned at the same level of rigor

Penta’s staff knew they needed to not only determine how well they were preparing career-ready students — students equipped with the skills needed to think critically and solve problems versus simply completing tasks — but also gauge teacher effectiveness, score high marks on the state’s CTE report card and earn compliance on CTE program reviews.

With these needs in mind, Penta’s Curriculum and Instruction Supervisor Christina Kerns and her CTE program administrators came together to build consistency and rigor across assessments in a schoolwide assessment literacy initiative that centered on aligning curricula to assessments. The initiative is now in its sixth year of implementation.

Following a train-the-trainer model, Kerns worked with administrators on gauging the level of rigor of standards, creating assessment items and activities aligned to specific levels of rigor, and designing “stretch” activities that allow students to demonstrate growth. CTE administrators then repeated the same steps with CTE program instructors.

Tools and Steps of Penta’s Assessment Literacy Initiative

Kerns and her staff use powerful tools to help assess levels of rigor in their instruction. Every teacher received a Critical Thinking for Life Wheel, which features the Depth of Knowledge levels on one side and the revised Bloom’s taxonomy levels on the other.

Kerns asserts that Hess’s Cognitive Rigor Matrix (download a PDF copy) is one of the most valuable tools districts and teachers can use to drill down to a granular level of student knowledge and understanding. Other tools commonly used by Penta teachers are the Levels of Thinking in Bloom’s Taxonomy and Webb’s Depth of Knowledge and the Depth of Knowledge Thinking Cap.

The first two years of Penta’s initiative were used to prepare leaders and teachers to collect valid and reliable data by applying the DOK levels to course standards, aligning instruction and assessments to those levels, and creating assessment maps.

Years 3 and 4 focused on collecting and conducting initial analyses of good data, implementing rigorous assessments, reflecting on the effectiveness of assessment items and refining as needed. Leaders and teachers dug deeper into WebXam test data and adopted data tracking systems for individual students and whole classes. Teachers and leaders also began rethinking their lesson plans to target three areas of significant course standard deficiencies.

In Years 5 and 6, teachers and leaders created a Success Plan for managing, analyzing and using data to take Penta’s CTE programs to the next level. This has involved creating a shared vision for improvement, identifying strengths and weaknesses, setting objectives and establishing a plan for implementing and refining data analysis methods. Staff conduct root cause analyses, design CTE data dashboards, study WebXam outcome reports and blueprints, and analyze achievement gaps.

CTE Data Spotlight: Penta’s Small Animal Care Program

Nicole Costello and Jennifer Rigali, instructors in Penta’s Small Animal Care program, share how they analyze data on instruction, assignments and assessments in their classes using five key strategies:

• best practices of project-based learning
• spiderweb discussions
• project assessment maps
• student data dashboards
• pre-post assessments aligned to industry credentials

Well-designed project-based learning units give students voice and choice and allow them to model and design their own solutions, says Costello. Students work with peers to critique and revise their work, unlike old-school pass-fail unit tests.

Students are “used to the teacher driving the lesson. They just sit back, take down some notes, and regurgitate the material back to you,” explains Costello. “I was trying to have them to be independent thinkers. I was doing research trying to help myself and help them, and I stumbled upon spiderweb discussions.”

Drawn from Alexis Wiggins’ The Best Class You Never Taught, spiderweb discussions structure classroom discourse to promote student-led discussions that engage every member of the class. Following this model, teachers remain hands off, more outgoing students encourage quiet students to talk, and distracted students and those who tend to dominate discussions are held accountable.

Project assessment maps (download a PDF sample), introduced in the first two years of Penta’s initiative, help Costello stay on track, ensure her students are learning what they need to, and track the DOK levels of her assignments.

A student data dashboard helps Costello stay accountable to herself, too. An Excel spreadsheet evaluates student scores against benchmarks, turning green, pink or red to indicate whether students are scoring above, at or below benchmark. Pink or red signal that students may need specialized interventions — or a rethink of her instructional approach.

The first time she saw her columns turn pink she was “flabbergasted… I was a ‘least effective teacher’… I thought, those students didn’t study. [But] when I looked back, I was like, there’s something wrong with what I did because even my really good students didn’t do well. So, I went back, revamped the lesson, differentiated and taught it again… I now had a lot more greens and I scored a four… It’s not the highest, but I was OK with that,” she explains.

Instructor Jennifer Rigali uses Easy TestMaker to create online pre-post assessments that align with state standards and her program’s industry-recognized credentials. Pretest data “give me a starting point for instruction,” says Rigali. “The post assessment not only analyzes student knowledge, but it gives me the opportunity to evaluate whether the lesson needs improvement. Do I need to change my instruction to improve results?”

Rigali uses Google Forms to empower students to reflect on skills they know or are unsure of. Self-assessments help Rigali determine the extra help students need to prepare for industry credentials.

Suggestions and Next Steps

As Penta’s assessment literacy initiative moves forward, leaders and teachers continue to monitor the growth of Penta’s on- and off-campus programs. Says Kerns, “This is working for us… we couldn’t be where we are today without those first two years of assessment literacy where we really took the time to align the curriculum to the assessment items.”

For CTE programs considering their own data collection models, Kerns and her team share a process you can follow in the full video of their 2021 Making Schools Work Conference session, which we’re making available to Promising Practices Newsletter subscribers along with resources shared in their session, which we link to below. In the video, Heating & Air Conditioning Technology Instructor Jeff Miller explains how his student self-assessments and data dashboard helped 100% of seniors and 65% of juniors pass Ohio’s Mechanical, Electrical and Plumbing exams. CTE program supervisors Jerry Gladieux and Tonya Kessinger also share how they used data to create an action plan for an IT pathway. Download Penta’s resources to adapt for your programs:

• Analyzing Student Data Question Prompts (Word)
• CTE Project Assessment Map (PDF)
• Detailed Analysis of Student Work (Word)
• Penta’s 2017-18 Student Data Tracking Packets (PDF)
• HVAC Sample Data Template (Excel)
• Sample HVAC CTE Teacher Dashboard (Excel)
• HVAC Student Daily Production Report (Word)  

Contacts: Christina Kerns, ckerns@pentacc.org, @ChristiKerns, @pentacareerctr; Nicole Costello, ncostello@pentacc.org; Jennifer Rigali, jrigali@pentacc.org; Jeff Miller, jmiller@pentacc.org; Jerry Gladieux,  ggladieux@pentacc.org; and Tonya Kessinger, tkessinger@pentacc.org.

This article was featured in the October 2021 issue of SREB School Improvement’s Promising Practices Newsletter.