AI Formative Feedback

Ensightful is an innovative platform that provides a structured approach to collaborative learning experiences. We assist educators in scaling scaffolding and support, while promoting equity and inclusion. In line with our mission, we developed an AI tool to streamline the formative assessment process, assisting educators in providing personalized and timely feedback at scale.

Don’t have time to read the whole case study? Get the TL;DR ↓

TIMELINE

June - August 2023

TEAM

Erin Forbes, Lead Product Designer

Chris Du, Product Owner

Mingyu Gao, Full-Stack Engineer

Brandon Lee, Full-Stack Engineer

Timothy Chan, QA Test Engineer

MY ROLE

I was the only designer working on this project and was responsible for the end to end design process including UX Design, UI Design, accessibility testing and creating marketing and support content.

Students wanted to complete their assignments more efficiently and improve the final outcome, but Ensightful’s features were not effectively supporting these goals.

While students valued the long-term benefits of using Ensightful, like career readiness and improved collaboration, the lack of support for their immediate goals led to low satisfaction and increased abandonment throughout the semester.

“I didn't rank Ensightful that high because, while it is an amazing organizational tool, it sometimes feels like it is more of a chore than an asset.”

Anonymous Student

We developed an AI-powered feedback loop to help students get more feedback and iterate on their assignments more effectively.

Ensightful’s AI leverages contextual information provided by instructors to deliver personalized feedback specific to each course and individual assignment. This empowers students to continuously iterate and enhance their work, fostering ongoing improvement throughout the assignment.

Instructors create a customized feedback loop

Ensightful makes prompt design easy. By uploading a rubric, assignment description, and creating a graded baseline, a tailored framework is created to guide our AI in providing feedback. This approach is personalized to each assignment and its specific grading criteria.

Students gain access to 24/7 on-demand feedback, tailored to the assignment requirements

Rather than turning to ChatGPT or another LLM with unspecified prompt designs, students can immediately access tailored feedback aligned with assignment requirements and instructor expectations. This not only enhances their learning experience but also promotes equity by ensuring that all students have equal access to personalized guidance and support.

Two methods to harness the power of AI feedback

Instructors have the option to conduct either a manual review of AI-generated feedback before sharing it with students, or students can generate instant feedback with the choice to request instructor review if necessary, ensuring continuous human oversight.

Many students perceived the integration of Ensightful into their courses as just one more task adding to their workload.

Student satisfaction surveys highlighted that while students value Ensightful for its long-term benefits like enhanced career readiness and improved collaboration, their immediate priorities are efficiency and the assignment outcome. Ensightful’s current features were not effectively supporting these priorities, leading students to abandon the platform and miss out on the long-term benefits.

We began to explore how we could streamline student’s assignment workflows while still championing student learning and success.

Business Goals

  1. Improve the alignment of Ensightful’s features with student needs

  2. Expand our market by extending support to include both group projects and individual projects

  3. Establish Ensightful as a leader in educational technology innovation by integrating AI solutions to boost product competitiveness

  4. Expedite these initiatives to launch this new feature for the Fall 2023 Term

Feedback from interviews with students and instructors highlighted a significant opportunity: improving the formative feedback loop. Students wanted more feedback to help them improve their assignments before submission, but often struggled to access it promptly. Instructors wanted to offer more feedback but lacked the capacity to provide multiple rounds of feedback on assignment drafts.

Looking at the connection between user needs and our business goals, we hypothesized that an AI feedback feature would serve as a catalyst for eliminating this disconnect. This feature would effectively streamline a process pivotal to students' desire to rapidly and effectively improve their assignments. Simultaneously, it would alleviate the burden on instructors and enhance their capacity to provide feedback at scale.

Check out our hypotheses and key metrics here 👀

Initially we explored the possibility of providing feedback and grades for student drafts so students would have a quantifiable measure of improvement across different versions. AI final submission grading was something we were considering developing for instructors, and this seemed like it could be a natural precursor.

While most key stakeholders expressed an overall openness to AI grading in the longterm, they had significant concerns about accuracy, equity and ethics that posed significant barriers to the immediate widespread adoption of graded drafts. Given the magnitude of these concerns, we decided to prioritize an MVP focused exclusively on feedback.

We began to strategize how to cultivate trust by demonstrating both the capabilities and limitations of Ensightful’s AI in an auxiliary feedback-only capacity, offering instructors complete transparency and control of AI generated feedback. This was a strategic precursor to address these concerns and create a viable path towards the integration of AI grading in the future.

Redefining the feedback loop with tailored, automated, formative feedback.

I understood that building user trust in the AI generated feedback was going to be one of the most important aspects of this design. With this in mind, I focused on design elements that promoted transparency, and human touch-points.

Initially, I explored the idea of creating a separate space for feedback generation within the platform, however I quickly realized that this would make it difficult to visualize the relationship between iterative drafts and the final deliverable. Ultimately, I believed that this would contribute to student’s perception of the platform as ‘one more task’ if they had to go to multiple areas to refine, store and submit their work. This led me to explore methods to integrate feedback generation into existing workflows, keeping the iteration cycle all in one place.

Milestone Integration

Currently, students submit deliverables to Ensightful’s Milestones. I started to brainstorm a solution that would allow students to submit documents for feedback within the Milestone. I also wanted to include the ability finalize an iteration, and make it a final submission. This allows to students visualize their progress from the first draft all the way to the final iteration.

Looking Ahead

Recognizing the future potential for grading features, many of my early iterations focused on creating synergy between feedback and grading to ensure a seamless experience when grading was eventually implemented.

Calibrating the Results

We believed that the best way to test this feature, and the accuracy of the feedback it delivered was to follow how this is currently done in higher education - through calibration meetings.

  • We had instructors go through the setup process to create a series of mock courses modelled on real data from past courses. We then ran anonymized past student assignments through the feedback system and collaborated with them on calibration meetings. Unlike traditional calibration meetings where the comparison is between instructors, we compared the AI feedback delivered to existing instructor feedback and grading¹ data.

    ¹ While we were primarily testing feedback accuracy, looking at qualitative data meant this was a subjective assessment. We tested with grades to ensure that we had an quantitative measure of accuracy

  • By facilitating these mock calibration sessions, we tested the instructor process end-to-end, from setup to feedback review. Repeating this process with small prompt changes based on instructor feedback helped us to reach a variance of +/- 1 from the grades instructors had given to papers.

In one case where there was significant variance, the instructor reviewed their grading, and concluded that the grade the AI had provided was actually closer to the score the student should have received.

Arizona State University

Testing the output required instructors to explore the nuances of entire setup process, which helped us to rapidly identify and resolve core usability issues on the instructor side.

Adding Manual Review

Our research indicated that most instructors and institutions were skeptical about AI grading but were more receptive to automated feedback. Instructors felt comfortable delivering on-demand feedback without their direct involvement, provided students knew the feedback was AI-generated and could request instructor review if necessary.

However, after seeing prototypes of this feature, instructors began to reconsider. Many requested the ability to briefly review and edit feedback before it was published to students, highlighting the need for a choice between prioritizing speed and human oversight.


As a result I designed a secondary workflow that allows instructors to review and edit AI-generated feedback before it is published to students.

THE TL;DR

Automated formative feedback for instructors and students

I get it, this was a lot to read! If you didn’t have time to read this whole case study, it’s ok. Here are two videos that will show you what the feature is about in under 5 minutes, enjoy!

Business Outcomes

  • Increased lead generation: Successfully generated over 20 new leads

  • Successful Pilot Adoption: Secured 9 new pilots in the first semester it was released

  • Market Expansion: Cultivated interest from K-12 educators, tapping into a previously untapped market segment that was challenging to reach due to the platform's complexity

This highlights a great foundation from which we will continue to test and iterate on the version of this feature that was developed for the Fall 2023 Term. Given the interest in Phase 1, we are moving ahead with Phase 2 and 3 which will focus on grading, and iterative submission analysis. Our next steps are to monitor the success criteria mentioned earlier in this case study, and conduct A/B testing with pilots this fall who are using the original version of the platform compared to those who have enabled formative feedback.

Personal Outcomes

As a designer working in EdTech to enhance student learning, I think a lot about my own learning as well. At the end of every project I take time to reflect on what I learned. This time around:

Always validate!

In theory I know this, but one of the things that I love the most about UX design is that it reminds me time and time again to be open to having my hypotheses changed. I thought instructors would be excited to bring AI grading into their workflow, as the process is one of their biggest pain points. In interviews, I was surprised by how resistant most instructors were to implementing AI in this way. The lack of precedent of AI in higher-ed (Fall Semester 2023) when compared to the high level of importance/risk associated with grades meant that they were actually more anxious, mistrustful and pessimistic. This informed our feature prioritization and provided us with important considerations for when AI grading was implemented in the future, re-emphasizing to me how crucial it is to continually validate your assumptions.

Complexity does not necessary equate to value.

I tend to be a big thinker when it comes to developing new features. I find that I jump to thinking about how great this feature could be because it might have X, Y, Z, X.1, Y.1, Z.1 (you get the picture). Both the timeline of this project, and the “unknown” of integrating AI into the higher-ed space required a straightforward interface that focused on delivering a high-quality, simple experience with the goal of easing AI adoption in higher-ed and increasing trust in our tool. Oftentimes reducing complexity can enhance the user experience, which was certainly true in this case.

(We’ll get to Z.1 down the road).