Phase 4: Analyze & Revise

Campus Spotlights: Analyzing & Sharing Direct Assessment Results

• Middlesex Community College: Digging Deeper into Assessment Data

  When the Middlesex Community College team received their initial VALUE assessment reports, they were not surprised to see that a majority of the student work from their Information Technology (IT) course scored at the level 1 benchmark or level 2 milestone performance levels—the course from which the artifacts were drawn was an introductory, first-semester course. So, this scoring was anticipated.

  However, when the data was further broken down and disaggregated by race/ethnicity, a different story emerged. Overall, the team found that work from BIPOC students outperformed work from the White students in the IT program. At the level 1 benchmark performance level, 42 percent of White students’ work met the criteria, while 38 percent of BIPOC students’ work met the criteria. In the level 2 and 3 milestone performance levels, however, 62 percent of BIPOC students’ work met the criteria, while 58 percent of White students’ work met the criteria.

  This revelation caused the Middlesex team to dig deeper into the assessment data and consider an additional perspective: gender. The team found that in the IT program, women of color did exceedingly well on the assignment submitted for scoring. The team will utilize the findings to help recruit more women of color into the IT program at Middlesex, by sharing the narrative and data that demonstrate their success in IT, a field in which women of color are grossly underrepresented.

  As a result of their project, the faculty within the IT pathway thought more intentionally about equity and what it means to be an equity-minded practitioner within their courses, within programs, at the institution, and within the surrounding community.
  

• Northeast Wisconsin Technical College: Analyzing & Sharing VALUE Data

  After receiving VALUE reports and scores, the Northeast Wisconsin Technical College (NWTC) project team organized a data debrief session to share the goals, process, and data results with colleagues and important stakeholders on campus.

  The project team’s data debrief session included reflection on the process of their project, and reflected on the following questions:

  • Where did we make the biggest change?
  • Why does it matter?
  • What did we anecdotally see because of the change?

  The team discussed what was learned after submitting student work to the VALUE Scoring Collaborative, and how assignments were intentionally redesigned and revised to consider the lessons learned from the VALUE results. For this team, these changes included:

  • Linking a service-learning assignment to the philosophy of nursing assignment
  • Mediasite recording of updated service-learning requirements
  • Updating the philosophy assignment and course rubric
  • Reevaluating the written assignment in alignment with the VALUE rubric and collect/score artifacts internally, using online scorer training knowledge

  Finally, the session helped faculty interpret and understand the new results by walking through the VALUE scoring scale and reading through aggregate results for the 100+ artifacts scored in using the Written Communication VALUE Rubric. The project team presented next steps, which included:

  • Further data evaluation
  • Separating the two VALUE submissions and comparing the results
  • Evaluating the writing levels of NWTC students compared to those at other institutions
  • Collaborating with colleagues on rubric development

  The session included opportunities for participating stakeholders to engage in breakout groups to discuss next steps and practice. Prompts for breakout groups to consider included:

  • What are three steps you will put into practice?
  • What will you need?
  • Who will you work with?
  • How will you share your learnings with other faculty?
  • As we enhance employability skills and equity, what role would you like to have?

Campus Spotlights: Revising Institutional Practices

• Miami Dade College: Student Inquiry Model

  After receiving their VALUE Scoring Collaborative results, the Miami Dade College team worked collaboratively to create a flexible, institution-informed model to enhance and revise common practices at the college. To prepare for scaling the work, the project team focused on implementing and revising practices across various institutional areas, including embedded assignments, assessment/evaluation, qualitative inquiry, data dialogues, and decision-making.

  The table below shows some of the practices that were revised and improved, based on the results received from direct assessment of student learning in the School of Justice at Miami Dade College:

  

  
  

• Salt Lake Community College: Revising & Strengthening HIPs for Equitable Student Learning

  For the Salt Lake Community College (SLCC) team, VALUE Scoring Collaborative results demonstrated that most student work in their Critical Thinking sample received scores across Milestones 2 and 3. However, disaggregated VALUE data revealed persisting gaps in the learning outcome among student groups. After analyzing these disaggregated data, the SLCC team used their VALUE report to highlight the need to close these gaps and revise their existing HIPs, which had been implemented earlier in the project.

  For SLCC, this included enhancing the use of the signature assignment requirement in ePortfolios by making General Education (Gen Ed) and student learning outcomes more transparent and intentional. This practice would include scaffolding the signature assignment earlier and throughout the semester.

  Revisions also included the use of Gen Ed maps to create a rubric for evaluating signature assignments in Gen Ed courses and to assess how they align with specific learning outcomes. By providing structured criteria and focusing on specific designations, the Gen Ed maps will allow SLCC more effectively and efficiently to assess student learning outcomes across pathways and across student demographic groups.
  

Key Takeaways

• Reviewing Direct Assessment Results

  • Be intentional about the process of reviewing your direct assessment results—collaborate with other faculty, staff, leaders, and students to make sense of the data.
  • Start to identify strategies and guiding questions for addressing the results through improving assignments, pedagogies, future assessment practices, etc.
  • Utilize cross-sectional data, intersectional disaggregation, and triangulated data to paint a clear picture of your results. Make sure to pay attention to any data outliers and what they may imply on pedagogical and systemic levels.

• General Assessment Reflections

  • Consider the implications that direct assessment data have for your courses, programs, and institution as a whole—think about the implications for resources, policies, assignment design, teaching practices, and future assessment.
  • Consider what your assessment results tell you about students who are either earlier or later in their college careers. Reflect on what needs to be learned as well as on areas where improvement is needed, where students excelled, and where students did not meet expectations.
  • Think about how your results fit into the larger institutional environment for assessment and the implications that this has for accountability and transparency.

• Demographic & Equity Implications

  • Compare demographic data from your assessment sample to your institution’s overall demographics to measure whether your sample is representative.
  • Pinpoint any noticeable disparities or patterns across student demographic groups including sex, race/ethnicity, and Pell eligibility.
    • Consider any other evidence on campus that might also point out existing equity gaps among student groups.
  • Reflect on the implications of these equity gaps and how they affect course, program, and institutional teaching practices, learning environments, and culture.

• Result Sharing

  • Consider the key campus stakeholders that would be most impacted by your assessment data—consider their position on campus, their influence on campus decision-making, and how they may disseminate the data themselves.
  • Be intentional about how you are sharing your assessment data—consider how you will communicate and deliver the results, which results will be most effective to share, and how you will convey next steps for creating change.
  • Consider hosting a data debrief session to break down your assessment results into digestible, understandable information for your intended audience. Show disaggregated data and visuals to help your audience understand any disparities or equity issues that exist.

• Revising Institutional Practices

  • Utilize direct assessment data to interrogate, revise, and scaffold course-, program-, and institution-level practices.
  • Examine disaggregated data by student demographics to better understand where equity issues still exist and how these affect student learning—utilize these data for a deeper inquiry into why and how these inequities in student learning exist and persist. Then, focus on revising and enhancing classroom practices, assignments, HIPs, assessment methods, and professional development/training to address the inequities.
  • Consider creating an informed decision-making framework, situated within the context and culture of your individual institution—involve campus leaders and stakeholders in conversations about direct assessment data and their relevance to student learning in order to inform future decision-making and policies.