Promptfoo Setup Guide: How I Started Testing My AI App Like Real Software

Ask most developers how they test their AI applications and they'll say something like "I run a few test prompts manually before deploying" or "my users tell me when something breaks."

This is the equivalent of testing a web API by clicking around in the browser and hoping for the best. We laughed at this approach 15 years ago and built unit tests, integration tests, and CI pipelines. Then AI apps came along and we collectively regressed to manual testing — but with more confidence, because "the AI seems smart."

Promptfoo is the tool that finally brings software engineering discipline to AI application development. It's not glamorous. It's not a new model or a flashy agent framework. It's a testing framework — and right now, it might be the most underrated tool in the AI developer stack.

Promptfoo is an open-source CLI tool and library that lets you:

• Run automated test suites against your prompts with expected outputs and assertion criteria
• Compare models side-by-side on identical test cases (great for evaluating a GPT-4o-to-Claude migration)
• Run security red-team scans to check for prompt injection vulnerabilities, jailbreaks, and data leakage risks
• Integrate into CI/CD so your GitHub Actions pipeline runs AI tests on every pull request

The moment you see what this looks like in practice — a green test suite showing your AI app behaves correctly across 200 different input variations — you can't go back to manual testing.

Promptfoo requires Node.js 18+. Install it globally:

npm install -g promptfoo

Initialize a new config in your project:

promptfoo init

This creates a promptfooconfig.yaml in your current directory. This is your test suite definition file.

Let's say you have a customer support AI that should never recommend contacting a competitor. Here's what a basic test config looks like:

providers:
  - openai:gpt-4o-mini

prompts:
  - "You are a helpful customer support agent for Acme Corp. Answer the following: {{question}}"

tests:
  - vars:
      question: "How do I reset my password?"
    assert:
      - type: contains
        value: "reset"
      - type: not-contains
        value: "competitor"
        
  - vars:
      question: "What's a good alternative to your product?"
    assert:
      - type: not-contains
        value: "CompetitorName"
      - type: llm-rubric
        value: "Response should redirect to Acme Corp features, not mention competitors"

Run the tests with:

promptfoo eval

Add this to your .github/workflows/ai-tests.yml:

name: AI Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm install -g promptfoo
      - run: promptfoo eval --ci
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

For model comparisons, just add multiple providers to your config. Promptfoo will run the same test suite against all of them and generate a side-by-side report showing which model performs better on your specific use case.

This is one of the most powerful things you can do before a model migration. Instead of guessing whether Claude is better than GPT-4o for your app, you measure it.