I Built an AI Agent That Monitors My GitHub Issues and Replies Automatically

If you maintain an open-source library, you know the feeling. You wake up, open GitHub, and see 5 new issues. Two of them are asking questions already answered in the README. One is a bug report with zero context. One is a genuine bug that requires you to trace a stack trace through three files.

It drains your energy before you even start writing actual code.

I decided I'd had enough. I wanted a "Junior Maintainer"—an AI that would instantly read every new issue, search the documentation, look at the codebase, and post a polite, highly technical first response. And I wanted to build it in an afternoon.

To make this work reliably, we can't just send the issue text to an LLM. The AI needs context. Here is the flow I designed:

1. The Trigger: A GitHub Webhook fires whenever a new issue is opened.
2. The Receiver: A lightweight FastAPI Python server receives the webhook.
3. The Context Gathering (RAG): The Python script uses a vector database containing my project's documentation to find relevant info.
4. The Brain: Anthropic's Claude 3.5 Sonnet analyzes the issue and the docs to formulate a reply.
5. The Action: The script uses the GitHub API to post a comment as a bot.

First, we need an endpoint to listen to GitHub. I used FastAPI because it's incredibly fast to set up.

from fastapi import FastAPI, Request
import os

app = FastAPI()

@app.post("/webhook")
async def github_webhook(request: Request):
    payload = await request.json()
    
    # We only care when an issue is newly opened
    if payload.get("action") == "opened" and "issue" in payload:
        issue_title = payload["issue"]["title"]
        issue_body = payload["issue"]["body"]
        issue_number = payload["issue"]["number"]
        repo_name = payload["repository"]["full_name"]
        
        # Trigger the AI agent in the background
        process_issue_with_ai(repo_name, issue_number, issue_title, issue_body)
        
    return {"status": "ok"}

The biggest mistake people make is asking an AI to debug code without giving it the actual codebase rules. I used LangChain to load my markdown documentation into a local ChromaDB vector store.

Now, when an issue comes in complaining about a specific function, my script queries the vector database for that function name, grabs the relevant documentation paragraphs, and injects them into the LLM prompt.

# Inside process_issue_with_ai()
docs_context = vector_db.similarity_search(issue_body, k=3)

system_prompt = f"""
You are the maintainer of this open-source project. A user has submitted an issue.
Use the following documentation to help them: {docs_context}

If it's a bug, suggest a potential fix. If it's answered in the docs, politely point them to it.
Always be kind and end with 'I am an AI assistant, a human maintainer will review this shortly.'
"""

Once Claude generates the perfect response, we need to post it. I used the PyGithub library. You generate a Fine-grained Personal Access Token in your GitHub developer settings, give it "Read and Write" access to Issues, and you are good to go.

from github import Github

def post_github_comment(repo_name, issue_number, comment_text):
    g = Github(os.getenv("GITHUB_TOKEN"))
    repo = g.get_repo(repo_name)
    issue = repo.get_issue(number=issue_number)
    issue.create_comment(comment_text)
    print(f"Successfully replied to issue #{issue_number}")

I deployed the FastAPI app to a cheap Railway.app container. Then, I went to my GitHub Repository Settings -> Webhooks, and pointed it to my Railway URL with the /webhook path.

The very next day, a user submitted an issue: "Getting a missing module error when running npm run build."

Within exactly 4 seconds, my bot replied. It had scanned the issue, checked the vector database, realized the user missed the installation step for a specific peer dependency, and provided the exact npm install command needed to fix it.

The user replied 5 minutes later: "Wow, that was fast. It worked, thanks!" They didn't even care it was an AI. They just wanted their problem solved.

Building this took about 3 hours. It costs me roughly $0.01 per issue in Anthropic API credits. But the return on investment is infinite. My mental health as a maintainer is protected. I no longer dread checking my repository notifications.

If you manage a repo with more than 1,000 stars, do yourself a favor and build a Junior Maintainer bot. Your sanity will thank you.