Evaluate Agent Performance

Once your agent has trained, you'll want to know how it performed - how well did it control the system? Was it successful? Did it beat benchmarks?

To determine this, after training is finished AMESA runs "benchmark episodes." In these episodes, the agent is controlling the process, with the simulation giving feedback, but unlike in experimental training episodes, the agent is always making what it has learned is the best decision. The default is one benchmark episode; you can set the number of benchmark episodes under Advanced Configuration when you start your training job.

Benchmark and KPI

When you set up your use case, you set the KPI and the benchmark.

In this case, the benchmark is a quality rating of 87 - that is the number that your agent needs to beat to succeed.

Performance Graphs

On the Benchmarks page, you will see multiple visualizations of your agents' performance. This bar chart compares every agent orchestration within the use case to each other and to the benchmark, represented by a dotted line. You can see that for the Material Fusion use case, both the single agent systems and the multi-agent system beat the benchmark by similar margins.

Investigate Agent Behavior

Further down the Benchmarks page, additional plots allow you to visualize agent behavior in detail.

This plot shows the quality rating in green and the material fusion metric variable in red. The important thing to note is that at the beginning, the material fusion metric is high, leading to a quality rating that is low. But because the agent has learned well, it quickly adjusts its control so that the material fusion metric and quality rating are more favorable.

Think back to activity clusters. What you see in this plot is the agent moving from an unsuccessful cluster to a successful one.

When you see results like this, you know you have a successful agent and you can begin thinking about testing it for deployment.

Dig Deeper

To dig deeper into agent performance, AMESA allows you to export a benchmark.json file with detailed information about every decision made during the benchmarking episodes. You can analyze this using your preferred data analysis tool. If you have AMESA's Assist Agents, you can also use the Analyst to gain insight about what happened during the benchmarking episodes and how your agent performed.