Joule Studio Part 7: Evaluation, Managed Runtime and Observability

📌 KEY POINTS
✅ Evaluating an agent means testing multi-step scenarios end to end, not just individual LLM responses — an agent can reason correctly at each step yet still reach the wrong overall outcome.
✅ Agents built and deployed within Joule Studio's low-code path run on SAP's managed runtime; Bring Your Own Agent means you operate the runtime yourself (scaling, patching, availability).
✅ Observability for agents means tracing which tools were called, in what order, with what inputs, and why the agent chose that path — plain request/response logging isn't enough to debug agentic behavior.
✅ Retrying an unresolved failure without understanding why it failed can compound problems — an agent that keeps calling the same failing tool needs its failure-handling logic examined, not just more retries.
⚠️ Treating agent evaluation the same as evaluating a single generative AI hub prompt misses the failure modes unique to multi-step orchestration — plan/tool-selection errors that a single-call evaluation would never surface.

━━━━━━━━━━━━━━━━━━━━
▶ WHERE WE ARE
Part 6 covered business context. Now running agents responsibly in production.

━━━━━━━━━━━━━━━━━━━━
▶ 1. MULTI-STEP EVALUATION

• Test full scenarios, not single calls.

━━━━━━━━━━━━━━━━━━━━
▶ 2. MANAGED VS SELF-MANAGED RUNTIME

• Low-code path is managed; Bring Your Own Agent is not.

━━━━━━━━━━━━━━━━━━━━
▶ 3. OBSERVABILITY

• Tool call tracing, decision logging.

━━━━━━━━━━━━━━━━━━━━
▶ 4. FAILURE HANDLING

• Understand why before retrying.

━━━━━━━━━━━━━━━━━━━━
▶ 5. OPERATIONAL OWNERSHIP

• Know which runtime you're responsible for.

━━━━━━━━━━━━━━━━━━━━
▶ STEP BY STEP

1. Write a multi-step evaluation scenario for a real agent use case.
2. Run it and trace exactly which tools were called, in order.
3. Identify whether managed or self-managed runtime applies, and check the relevant operational responsibilities.

✅ CHECKPOINT: you can show the full tool-call trace for a scenario, not just its final answer.

An agent kept retrying the same unresolved purchase-order lookup failure five times before giving up, each retry adding latency without changing the outcome. Tracing the tool calls showed the underlying skill was failing for a data reason no retry would fix; adding a distinct error path (rather than blind retries) and fixing the root cause in the skill resolved it.

• Evaluating only single LLM calls, missing multi-step orchestration failures.
• Confusing managed and self-managed runtime responsibilities.
• Debugging with plain logs instead of tool-call traces.
• Retrying a failure blindly without understanding its cause.
• Not checking idempotency before allowing retries on consequential tools.

• Evaluate full multi-step scenarios, not isolated calls.
• Know which runtime (managed vs self-managed) you're responsible for.
• Use tool-call tracing for observability, not just request/response logs.
• Diagnose failure causes before adding retry logic.
• Verify idempotency for any tool subject to retries.

Questions: "How is evaluating an agent different from evaluating a single LLM prompt?", "What's the difference in operational responsibility between the managed runtime and Bring Your Own Agent?", "What does good observability look like for a multi-step agent?"
Strong answer: multi-step scenario evaluation, managed vs self-managed operational ownership, tool-call tracing observability, failure diagnosis before retry.