Pathwright AI Research & Data Science (example case study)

Role: AI research and data science intern
Status: Active portfolio case study template
Tools: Python, notebooks, structured evaluation tables, stakeholder summaries

Summary

At Pathwright, Tram is helping explore how AI-assisted instructional tools can support more personalized learning. The work sits between data science, product research, and education: define the question clearly, collect comparable evidence, and explain model behavior in a way that helps the team make a decision.

Problem

AI agents can produce useful instructional support, but their behavior needs to be evaluated consistently. A single good response is not enough; the team needs repeatable ways to compare outputs, spot failure modes, and decide what should change in prompts, product flows, or evaluation criteria.

Tram's Role

  • Help translate broad product questions into measurable research tasks.
  • Organize model outputs into structured observations.
  • Compare results across prompts, user contexts, or evaluation criteria.
  • Summarize findings for teammates who need clear decisions instead of raw transcripts.

Approach

1. Frame the Evaluation

Start with a narrow question, such as whether an instructional agent identifies the learner's actual blocker or whether it gives generic advice.

2. Capture Comparable Outputs

Collect responses in a format that makes them easy to compare across the same dimensions: accuracy, specificity, helpfulness, tone, and next-step clarity.

3. Tag Failure Modes

Group recurring issues into practical categories, such as vague guidance, missed context, overconfident claims, or weak follow-up questions.

4. Report the Decision

Turn the analysis into a concise recommendation: keep, revise, test again, or investigate further.

Example Artifact

A useful artifact for this case study would be a one-page evaluation summary:

  • Research question.
  • Sample size and source.
  • Scoring dimensions.
  • Top 3 failure modes.
  • Recommended product or prompt change.

What This Shows

This project shows applied data thinking in an AI product setting: defining evidence, structuring messy outputs, and communicating what the team should do next.

Next Steps

  • Add a sanitized screenshot or table from an evaluation run.
  • Include one before/after example showing how a prompt or workflow changed.
  • Add a short metric summary once there is enough comparable data.