> Source URL: /pathwright.casestudy
# Pathwright AI Research & Data Science (example case study)

**Role:** AI research and data science intern  
**Status:** Active portfolio case study template  
**Tools:** Python, notebooks, structured evaluation tables, stakeholder summaries

---

# Summary

At Pathwright, Tram is helping explore how AI-assisted instructional tools can support more personalized learning. The work sits between data science, product research, and education: define the question clearly, collect comparable evidence, and explain model behavior in a way that helps the team make a decision.

---

# Problem

AI agents can produce useful instructional support, but their behavior needs to be evaluated consistently. A single good response is not enough; the team needs repeatable ways to compare outputs, spot failure modes, and decide what should change in prompts, product flows, or evaluation criteria.

---

# Tram's Role

- Help translate broad product questions into measurable research tasks.
- Organize model outputs into structured observations.
- Compare results across prompts, user contexts, or evaluation criteria.
- Summarize findings for teammates who need clear decisions instead of raw transcripts.

---

# Approach

## 1. Frame the Evaluation

Start with a narrow question, such as whether an instructional agent identifies the learner's actual blocker or whether it gives generic advice.

## 2. Capture Comparable Outputs

Collect responses in a format that makes them easy to compare across the same dimensions: accuracy, specificity, helpfulness, tone, and next-step clarity.

## 3. Tag Failure Modes

Group recurring issues into practical categories, such as vague guidance, missed context, overconfident claims, or weak follow-up questions.

## 4. Report the Decision

Turn the analysis into a concise recommendation: keep, revise, test again, or investigate further.

---

# Example Artifact

A useful artifact for this case study would be a one-page evaluation summary:

- Research question.
- Sample size and source.
- Scoring dimensions.
- Top 3 failure modes.
- Recommended product or prompt change.

---

# What This Shows

This project shows applied data thinking in an AI product setting: defining evidence, structuring messy outputs, and communicating what the team should do next.

---

# Next Steps

- Add a sanitized screenshot or table from an evaluation run.
- Include one before/after example showing how a prompt or workflow changed.
- Add a short metric summary once there is enough comparable data.


---

## Backlinks

The following sources link to this document:

- [Read more](/index.path.llm.md)
- [Open case study](/case-studies.index.llm.md)