Accelerating software delivery with agentic QA automation using Amazon Nova Act – Part 3

As quality assurance (QA) automation adoption grows inside an organization, a new problem appears that has nothing to do with writing tests. A platform team stands up test automation, then a second team wants to join, then a dozen more. Soon hundreds of test cases belong to different products and squads, and everyone can see everyone else’s tests. No one can answer a basic question at a glance: is this application’s test coverage healthy right now? Test authoring stops being the bottleneck. Organizing tests and controlling who can touch them becomes the work. A second, related question follows close behind: who owns this application and its tests, and who is accountable when coverage slips? Without a clear owner, health problems have no one to land on. In the first post , we introduced QA Studio, a reference solution built with Amazon Nova Act . Teams use it to define tests in natural language and run them through AI-powered visual navigation. In the second post , we added test suites and a CLI for batch regression testing in continuous integration and continuous delivery (CI/CD) pipelines. In this post, we cover two capabilities that help QA Studio scale across an organization. The first, applications , organizes tests and reports health metrics. The second, team-based user management , controls who can see and change what. Organizing tests with applications An application in QA Studio represents the system under test: a website, a web app, a service front end. It’s the container that gives a collection of tests, a home, and an owner. Use cases (the individual tests from Part 1) and test suites (the regression collections from Part 2) belong to an application, and they inherit its ownership. Instead of one flat list of every test in the deployment, each team works inside the applications they own. Creating an application takes a name, a base URL, and an optional set of environments such as development, staging, and production. From the application page, teams create new use cases directly in context, add existing ones, and group them into test suites. This is also where QA Studio reports on the health of the tests it runs. Figure 1: The application detail page, showing pass rate, executions, and failures with the use cases and test suites tabs Understanding application health with metrics Tests produce results. Understanding those results over time is what tells a team whether the development lifecycle for an application is healthy, not only whether a single build is ready for release. QA Studio records the outcome of every execution as Amazon CloudWatch custom metrics, dimensioned by application and by environment. It turns these metrics into a health view on the application page. The top of the page answers the immediate question with three figures: the pass rate over the selected window, the total number of executions, and the current failure count. Alongside them, QA Studio computes a health score of 0–100. The score starts from the pass rate and subtracts a penalty for flaky tests. That way, an application whose tests pass but flip between pass and fail doesn’t present a falsely perfect picture. A team scanning a portfolio of applications can tell the stable ones from the ones that need attention without opening each result. The metrics tab moves from the current snapshot to trends. Teams choose a 7-day window with hourly granularity or a 30-day window with daily granularity, and QA Studio charts executions, successes, failures, and average duration across that period. Because the underlying metrics carry an environment dimension, teams filter the view to a single environment. For example, they can compare how staging behaves against production for the same set of tests. Two views target the tests that cost teams the most time. The flaky tests view ranks use cases by how often they flipped between passing and failing over the last seven days. This surfaces the intermittent failures that erode trust in a suite before they turn into a production miss. The recent failures view lists the latest failed executions for quick triage. Each one links back to the Amazon Nova Act trajectory logs and session recordings described in Part 1. That way, moving from “this test is failing” to “here is exactly what the agent saw” takes one click. Figure 2: The metrics tab: pass rate, executions, and average duration over the selected window, with the daily success-and-failure chart, failure hotspots, and the flaky use case ranking Controlling access with teams and roles Metrics tell a team how their applications are doing. Team-based user management decides which applications are theirs in the first place. QA Studio organizes access around teams, and an application belongs to exactly one team. Members of that team work with its applications. Everyone else doesn’t see them. Every team member holds one of five roles, and each role includes the permissions of the ones below it: Admin : Studio-wide access to every team, application, and user, including creating teams and assigning applications. Admin is a studio-wide role rather than a per-team assignment. Team Lead : Manages their own team members and creates applications for the team. Editor : Creates and edits use cases and test suites within the team’s applications. Executor : Runs use cases and suites but doesn’t modify them. Viewer : Read-only access to the team’s applications and results. Because roles are cumulative, a Team Lead has everything an Editor has, and so on down to Viewer. Access to a use case or a test suite follows the application that contains it. If a member can work with the application, they can work with its tests, bounded by their role. This keeps permission management at the application level instead of on thousands of individual tests. An administrator sees the whole studio. The Users page lists every account with its team and role in one place. An admin can tell who holds Team Lead on which team and who carries Studio-wide Administrator rights. In the reference deployment shown here, four users span two teams: two Team Leads, one Viewer, and one Studio-wide Administrator. The Teams page rolls the same data up by team, showing each team’s member count and application count side by side. An admin can see which teams are expanding before opening any one of them. Figure 3: The studio-wide Users page: every account with its team and role, filterable by team and by role The same membership data, rolled up by team, appears on the Teams page. Figure 4: The Teams page: each team with its member and application counts Switching from teams to applications, the studio-wide Applications list shows every application across all teams. Figure 5: The studio-wide Applications list: every application with its base URL, owning team, use-case count, and last execution status Sharing applications between teams Teams rarely work in complete isolation. A platform team might own an application that a feature team needs to run against, without handing over the ability to change it. QA Studio supports this with shared access: the owning team grants another team access to an application at a view, execute, or full level. Shared access follows a ceiling rule. A user’s effective permission on a shared application is the lower of their own role and the level of the grant. A Viewer who is given full access to a shared application remains view-only, because their role is the ceiling. This way, an owning team can share widely without granting more than a recipient’s role allows. What a team lead sees A team lead works inside a single team rather than the whole studio. Opening My Team presents four tabs that cover everything the lead controls: Members, Applications, Use Cases, and Shared Access. The Members tab lists the people on the team. The lead can change any member’s role inline or add a new one, so day-to-day access management stays with the person closest to the work. In the AWS Team shown here, the lead manages