Join Our Free Masterclass & Learn from Industry Experts! 🚀 | Register For FREE🚀 Get ₹100 OFF on E-Learning Premium Plan | Code: | Enroll Now

GenAI Applications for QA Automation Engineers: How to Build Generative AI Apps

October 26, 2026MicroDegree

Cover with the title GenAI Applications for QA Automation Engineers on a dark blue background and a card on the right headed test_plan.json with three passed checks (Valid login, Empty story rejected, Non-JSON reply rejected) and one warning (Duplicate titles)
A test plan card: three passing checks and one warning for the generator's output.

GenAI applications for QA automation engineers suit the job better than they sound. An application built on a language model is a component whose output changes from run to run, and deciding whether that output is acceptable is a problem testers already know. You write scripts, you build harnesses and you distrust anything that has not been checked. Most first AI projects lack exactly those habits.

This guide builds one small example of GenAI applications for QA automation engineers: a test-case generator in Python that turns a user story into structured test cases. It includes the code, the pytest tests that check the application itself, a small evaluation set, the usual failure modes and a pay comparison from named sources.

Table of Contents

â–¼

Why QA Engineers Build These Applications Well

Most developers building their first AI feature test it by trying a few prompts and deciding it looks fine. A QA automation engineer starts somewhere else: what is the expected result, how will I know when it is wrong, and how do I run that check every time the prompt changes? Test oracles, regression sets, flaky behaviour and test data are daily work for you, and they carry straight over.

What you add is the model call and a few new ideas: prompts as code, structured output and evaluation sets. None needs a machine learning background. Our post on whether manual testers can become AI engineers covers the wider career question.

Explore MicroDegree FREE E-learning programs

Pick a Small Application You Understand: A Test-Case Generator

Choose something where you can judge the output yourself. A test-case generator is a good first project for a tester: you give it a user story with acceptance criteria and it returns test cases as JSON. You already know what good test cases look like, so you can tell quickly whether the model is helping.

The flow has five steps, split between your code and the model. The model appears once, and everything around it is ordinary code you can test.

A two-lane diagram. Lane one, Your code: Story in, Build prompt, Check the reply, Save the plan. Lane two, The model: Model replies. Arrows go from Build prompt down to Model replies and back up to Check the reply.
The model is one step in a flow. The steps around it are normal code you can test.

Step 1: Make the Model Return Structured Output

A chat reply is text, and text is hard to test. Ask for JSON in a fixed shape and check it against a schema. The code below uses the OpenAI Python SDK as in OpenAI's quickstart, with the model name read from an environment variable, and Pydantic to validate the reply. You can swap in another provider by changing only the call_model function.

import os

from openai import OpenAI
from pydantic import BaseModel

PROMPT = """You are a QA engineer. Read the user story below, including its acceptance criteria.
Return only JSON in this shape:
{"cases": [{"title": "...", "steps": ["..."], "expected": "...", "kind": "happy"}]}
"kind" is one of: happy, negative, boundary.
Cover each acceptance criterion at least once. Use only what the story says.
If something is unclear, add a case titled "Clarify: ..." instead of guessing.

Story:
"""


class TestCase(BaseModel):
    title: str
    steps: list[str]
    expected: str
    kind: str


class TestPlan(BaseModel):
    cases: list[TestCase]


def call_model(prompt: str) -> str:
    client = OpenAI()  # reads OPENAI_API_KEY from the environment
    response = client.responses.create(model=os.environ["MODEL_NAME"], input=prompt)
    return response.output_text


def generate_plan(story: str) -> TestPlan:
    if not story.strip():
        raise ValueError("story is empty")
    raw = call_model(PROMPT + story)
    return TestPlan.model_validate_json(raw)

Three details come from testing habits. The empty-story check stops a bad input before it costs a model call. The prompt tells the model what to do when information is missing, a "Clarify" case, instead of letting it guess. And the reply is validated, so a model that answers in prose fails loudly instead of passing a broken plan downstream.

Step 2: Test the Application, Not the Model

You cannot get a stable answer from a live model on every run, so test your own code against a stand-in. Replace call_model with a fake that returns a fixed string. Then pytest checks the parts you control: parsing a good reply, rejecting a bad one and refusing empty input.

import pytest
from pydantic import ValidationError

import testgen

GOOD = (
    '{"cases": [{"title": "Valid login", "steps": ["Open the login page", "Enter a valid user"],'
    ' "expected": "The dashboard opens", "kind": "happy"}]}'
)


def test_valid_reply_is_parsed(monkeypatch):
    monkeypatch.setattr(testgen, "call_model", lambda prompt: GOOD)
    plan = testgen.generate_plan("As a user I can log in")
    assert plan.cases[0].kind == "happy"


def test_non_json_reply_is_rejected(monkeypatch):
    monkeypatch.setattr(testgen, "call_model", lambda prompt: "Sure! Here are your tests:")
    with pytest.raises(ValidationError):
        testgen.generate_plan("As a user I can log in")


def test_empty_story_never_reaches_the_model(monkeypatch):
    def boom(prompt):
        raise AssertionError("the model was called")

    monkeypatch.setattr(testgen, "call_model", boom)
    with pytest.raises(ValueError):
        testgen.generate_plan("   ")

These three tests ran in a few seconds in a local environment with a stubbed model and passed. They cost nothing to run, so they can run on every commit. If you add a new rule to the prompt or the schema, add a test for it here.

Step 3: Build a Small Evaluation Set

The fake model cannot tell you whether a real model writes useful test cases. For that you need an evaluation set: a handful of user stories where you know what a good plan must contain. Start crude. The script below checks that required words appear in the cases and counts duplicate titles.

from testgen import generate_plan

STORIES = [
    {
        "story": "As a user I can reset my password by email. The reset link expires after 30 minutes.",
        "must_mention": ["email", "expire"],
    },
]


def check(item):
    plan = generate_plan(item["story"])
    text = " ".join(c.title + " " + c.expected for c in plan.cases).lower()
    missing = [word for word in item["must_mention"] if word not in text]
    titles = [c.title.lower() for c in plan.cases]
    duplicates = len(titles) - len(set(titles))
    return missing, duplicates


if __name__ == "__main__":
    for item in STORIES:
        missing, duplicates = check(item)
        print(item["story"][:40], "| missing:", missing, "| duplicate titles:", duplicates)

A keyword check is a weak test, but it beats none: it catches a plan that forgot the 30-minute expiry in the example story. Grow the set as you find failures: every bad output you see in real use becomes a new story in the set. Thoughtworks' QA team measured things like acceptance-criteria coverage, duplication and ambiguity when they compared AI-written and manual test cases, which gives you a list of what to measure as the set matures.

Where These Applications Fail

The sources behind this post point to the same weak spots:

  • Vague or invented steps. Thoughtworks reported vague steps and an ambiguity average of about 27% across its stories, so check steps for specifics.
  • Narrow coverage. The same team found the tools leaned towards functional tests and struggled with techniques such as decision tables, so ask for non-functional and boundary cases and check that they appear.
  • Too much in one prompt. A ZenML case study found that putting several features in one request degraded the output, so give the model one feature at a time.
  • Duplicates and drift. Two runs on the same story can give different plans, so count duplicates and compare runs against your set rather than trusting one.
  • Input that tries to steer the model. If story text comes from outside your team, treat it as untrusted input. This is general good practice, not something the sources above measured.

One way to organise the work is in three layers: pytest runs on every commit, the evaluation set runs before a release and a person reads a sample of real outputs, because no script catches everything. It is a working rule, not a standard. Our guide to measuring retrieval quality in RAG systems shows what an evaluation set looks like when the application searches documents.

Three Projects to Build After This One

  1. A flaky-test triage helper. Feed it the log of a failed run and ask it to suggest likely causes in a fixed JSON shape. Test it with logs you have already diagnosed.
  2. A question-answering tool over your team's test documentation. It adds retrieval, and with it a new set of things to evaluate.
  3. A script drafter. Given a manual test case, it drafts an automated test in the framework you use, and your pipeline runs the draft before anyone reads it.

Put the finished project in a repository with a README that states what it does, how to run the tests and the evaluation set, and what you saw fail. Wire the pytest step into a pipeline; our guide to CI/CD projects for software testers covers one way to do it. For the wider skills list behind these projects, see generative AI skills every software engineer should learn.

Pay: Test Automation Engineer vs Generative AI Engineer

Flexiple has no QA automation engineer page, so its test automation engineer page is used as the closest match. For the AI side, its generative AI engineer page is used. Both publish self-reported annual pay as cost to company, in lakh rupees a year (LPA). The figures are medians from pages updated in August 2026, read in October 2026. The first three columns are India-wide medians by experience, which Flexiple does not re-cut to Bengaluru, and only the last column is specific to the city.

Role0 to 3 years3 to 6 years6+ yearsBengaluru, all levels (middle 50%)
Test automation engineer (closest to QA automation)4.96.314.27.1 (4.9 to 12.5)
Generative AI engineer78.8289.2 (6 to 22.3)

Treat these as estimates that vary by company, skills and city. The generative AI engineer page has only 59 records for Bengaluru and 35 for 6+ years, so its senior figure of ₹28L is thin. Each page's own FAQ also gives different level medians from the main figures used here (₹6.7L and ₹16.4L for test automation engineer, ₹6.5L at entry level for generative AI engineer). The direction is still consistent: the generative AI engineer median is higher at 0 to 3 years (by 2.1 LPA) and at 3 to 6 years (by 2.5 LPA). Those describe the pay of people already in each job, not what a move from QA will pay. For more on one route, see Top Automation and AI Tools Every Tester Should Learn.

Showing It to an Employer

An interviewer will want to hear how you know your application works. Be ready to say what your tests cover, what your evaluation set caught, what still fails and what a person must review. Keep the numbers you quote to the ones your own set produced. If you want a guided order alongside your job, MicroDegree's learning programs are one option to compare.

Conclusion

You do not have to leave testing to build with generative AI. Start with a small application whose output you can judge, wrap the model call in code you can test, and grow an evaluation set from every failure you see. The skill you already have, proving that something works, is the one these projects need most.

FAQs

1. Do I need machine learning knowledge to build these applications?â–¼

Not for the application in this guide. It calls a hosted model, validates the reply and tests the surrounding code. Machine learning theory helps later, but it is not the first step.

2. Which language and tools should a QA engineer use?â–¼

Python is the easiest start here because the SDKs and test tools are mature, and pytest and Pydantic cover the checks in this guide. If your team uses another language, the same structure applies.

3. Can I test a language model with exact-match assertions?â–¼

Rarely. Output varies between runs, so test the shape of the reply with a schema, check required content with an evaluation set and review samples by hand.

4. Will generated test cases replace manual test design?â–¼

The sources read for this post say human review stays essential. Thoughtworks found vague steps and weak coverage of some techniques, so treat generated cases as drafts.

5. Will this move raise my salary?â–¼

Flexiple's medians are higher for generative AI engineers than for test automation engineers at the first two experience bands, but they are averages across people, based on few records for the AI role, not a promise for your own move.

You already know how to break things on purpose, which is half of building AI applications that hold up. MicroDegree's Generative AI & Agentic AI Certification Program is one option for the other half.

Ready to see one working end to end?Click here to take a free Generative AI demo

Also Read Blogs

Loading related blogs...