Randy Rice's Software Testing Page

05/26/2026

I recently wrote a blog post called "Crossing the Line" about something that has been bothering me: Can Agentic AI really be tested?

Traditional software crosses a line from testing to production. That line is called a release. Before release, we test. After release, users experience the real thing.

But Agentic AI seems much different.

Agentic AI is being "released" with limited (or no) pre-release testing because its behaviors can change, adapt, and emerge in ways we may not fully predict.

Then, once the agent is taking actions in the real world, we often aren't really testing anymore. We're monitoring. And monitoring is not the same as testing.

By the time an AI agent makes a bad decision, leaks information, takes an unexpected action, or creates business risk, the "test" has already become reality. This is not speculation. These things have already happened, some with major consequences.

So where is the line now?

Can we move testing approaches forward enough to deal with Agentic AI, or are we accepting a world where production becomes the experiment? What defines a "defect" versus a "behavior"?

I explore this question in the newest episode of The Value of Testing podcast.

https://youtu.be/gZTvgCEj10s

I'd love to hear your thoughts: Can Agentic AI really be tested, or are we entering an era where we can only manage risk and monitor outcomes?

I received some great comments after the blog post, but I'm still gathering input and opinions on this.

There is a line that exists not only in software, but also in other domains such as writing, music, and movies, just to name a few. That line is the release ...

05/22/2026

You can give a team better tools, more automation, and even AI...

But you still can't automate good thinking.

Just wrapped up a live virtual software testing class for a great client this week, and one thing stood out:

The biggest testing challenges today are rarely tool problems.

They're often:

> Unclear requirements
> Misunderstood risks
> Assumptions that nobody questioned
> Pressure to move faster without understanding the tradeoffs

Tools can help. AI can help. Automation can help.

But quality still depends on people asking the right questions.

After many years of teaching software testing, I still enjoy seeing that moment when a group realizes that testing isn't just about finding defects — it's about reducing risk and improving decisions.

Way back in the mid-90's I started asking testers about their greatest challenges in testing. The result was a book I co-wrote with Bill Perry, "Surviving the Top Ten Challenges of Software Testing." (You can still get it on Amazon!)

What I learned was that the great majority (over 90%) of testing problems are actually human in origin. So, the book focuses on the human issues in testing. As far as I know, it remains the only book to focus on the human issues in testing.

Over the last 30 years, I have continued to do this research, and the results remain the same!

What do you think is the biggest challenge facing software testing teams today?

05/06/2026

Most teams are doing test automation, but only a few are doing it well.

I’ve seen too many automation suites that:

– Break constantly
– Deliver little value
– Create more maintenance than insight

That’s not a tooling problem…It’s an architecture problem.

If you’ve already earned your ISTQB Foundation Level certification, the next step isn’t just “more automation” - It’s learning how to design, build, and optimize automation that actually works and is sustainable.

That’s exactly what I cover in my new course:

ISTQB Advanced Test Automation Engineer (v2)

You will learn

- Real-world automation strategies
- Designing maintainable test solutions
- Building automation solutions you can trust and scale

All taught by me, someone who has been working with test automation since the late 1980's! (I've seen a lot!)

Plus, test automation skills are in the top 5 most desirable skills right now by employers.

If you're ready to move beyond the basics, then let's go!

https://mysoftwaretesting.com/ISTQB-Advanced-Test-Automation-Engineer-e-Learning-Course-V2-2024_p_13.html

ISTQB Advanced Test Automation Engineer e-Learning course with access to mentoring by Randy Rice, holder of all 5 ISTQB Advanced Certifications, including the Advanced Test Automation Engineer (CTAL-TAE).

04/27/2026

Meaningless Metrics…And Some That Matter

I learned early on in my 38-year QA and Testing career that measurement and metrics are important parts of evaluating quality.

I have also learned that it’s tempting to measure so many things that the really important things get lost in the noise. I have seen spreadsheets with dozens of metrics sliced and diced so many ways that it would make my head spin. I would ask, “How in the world do you keep up with all of that?”

In testing software, there are some core metrics that are often taught, but when you peel back the layers, they have little or no real meaning. Things like test coverage, number of test cases, number of defects, etc.

Yes, I know those metrics seem essential at first glance, but they're not. For example, I might report that I have 100% coverage of all user stories. I can achieve that with one test case per user story! But most of us know that it takes more than just one test case to verify a user story.

In the most recent episode of my Value of Testing podcast, I present my top 5 meaningless metrics. Then I present seven meaningful metrics and explain why you should capture and report them, especially in an age of AI.

In this episode, you’ll learn:
- Why many testing metrics fail to deliver real insight
- The difference between activity metrics and value-driven metrics
- How metrics can mislead teams and decision-makers
- What makes a metric truly meaningful
- Practical examples of metrics that actually drive better outcomes

You can check out the podcast on YouTube at https://youtu.be/IP48Rf_c6Ts

By the way, I would be interested to hear your take on this. Which metrics have you found to be of little value? Or, do you even care about measuring anything in testing?

Many Test Metrics Are Useless (Here’s What Actually Matters in Testing)

04/20/2026

From The Hacker News - Why Most AI Deployments Stall After the Demo.

I guess I could add my own somewhat humorous but very real observation, Why Most Test Automation Efforts Stall During the Demo, but that's another story for another time!

Key quotes

"Everything moves quickly. Prompts land cleanly. The system produces impressive outputs in seconds. It feels like the beginning of a new era for your team. But most AI initiatives don't fail because of bad technology. They stall because what worked in the demo doesn't survive contact with real operations. The gap between a controlled demonstration and day-to-day reality is where teams run into trouble."

"Teams that successfully move beyond the demo tend to share a few habits. They test AI against real workflows rather than idealized scenarios, using real data, real processes, and real constraints. They evaluate performance under realistic conditions, measuring accuracy under load, monitoring latency, and understanding how the system behaves when inputs vary. They prioritize integration depth, because AI operating in isolation rarely has much impact. And they pay close attention to the cost model, since AI usage can scale quickly and without visibility into consumption, costs can become a blocker."

"Perhaps most importantly, they invest in governance early. Clear policies, guardrails, and oversight mechanisms help teams avoid delays and build confidence in their deployments."

Read the full article here -

AI projects stall when clean demos meet messy operations, latency, and governance gaps, limiting enterprise impact.

04/17/2026

You’ve earned your ISTQB Foundation certification.

Now—are you ready to level up?

The ISTQB Advanced Test Analyst course is here.
> Built on the 2025 syllabus
> Designed for real-world application
> Self-paced with direct access to me for support

This isn’t just about passing an exam—it’s about becoming a more effective, analytical tester.

Ready to take your testing to the next level? Enroll here: https://mysoftwaretesting.com/ISTQB-Advanced-Test-Analyst-e-Learning-Course--2025-Syllabus-Version-V-40_p_21.html

Direct support from me is included throughout the course.

I'm Curious - What was the biggest challenge you faced after earning your ISTQB Foundation certification?

For many, it’s making the jump from learning concepts to applying them in real-world testing.

Feel free to comment!
hashtag hashtag hashtag

Learn the deeper techniques in test analysis and design in the ISTQB Advanced Test Analyst 2025 e-Learning course, presented by Randy Rice

04/16/2026

The Unexpected Power of Test Automation and the Human Touch

I recently interviewed my good friend Greg Pascal, a seasoned test automation expert, on The Value of Testing Podcast. What struck me most was his perspective on the evolving role of automation in quality assurance.

I consider Greg to be an innovator. When I asked him what's got his attention these days, he shared how he's deploying automation on tiny microcontrollers — like the ESP32 — turning them into powerful IoT monitoring devices.

It’s a reminder that automation isn't just about big systems; sometimes, it’s about small, clever solutions that bring massive value.

One topic we explored was "Are the basic principles of test automation still the basic principles?" Things like only automating highly repeatable and predictable features.

What really resonated was his emphasis on the importance of understanding what makes a good test. We discussed the difference between just automating a test and automating a good and meaningful test.

Speed and green checks are not enough. The true gold is in using automation as a starting point to deeper analysis, to find the bugs others miss — that’s where real quality lives. As an example, consider an automated test that passes but takes increasingly longer to run each time. That's a signal of something going wrong!

We also spoke about the critical role of people: leveraging individual strengths, mentoring next-gen testers, and recognizing that automation is a tool, not a magic wand.

The human insight, intuition, and judgment are irreplaceable — especially in times of rapid change and AI influence.

AI and automation can mimic, but never replicate, our innate reasoning, intuition, and the human spark that drives innovation.

We must remember: technology is an extension of our skills, not a replacement for them.

Maybe that’s the point — the true craft of testing is marrying the right tools with sharp human insight. It’s a balance worth protecting and worth thinking about.

Listen to the full conversation here:

Unlocking the True Value of Test Automation (With Greg Paskal)

04/15/2026

Last evening I was watching some YouTube videos to wind down. One video in particular looked interesting, so I started watching.

At first, I didn't notice anything off. But the longer I watched, the more I got the feeling that this wasn't real. It looked close. It was a gentleman talking to the camera. Highly scripted, but there were subtle signs of AI.

The longer I watched, the more I wanted to leave. That's unusual for me because I tend to deep-dive on most videos and watch to the end. I've even watched some car restoration videos over 2 hours long!

It just became too boring. The information was overshadowed by the vanilla perfect video format.

All of this causes me to think about when (or if) people will accept the artificial over the authentic. We are at the point right now where the choices are being made.

I expect that the person creating this video felt good that they had created a "perfect" video. But when is "perfect" too perfect? And, what good is a perfect AI video if people tune out?

I promise you, in my training courses, podcasts, and other content, you will hear some "uhs", a chuckle here and there, and other imperfections. Yes, I do some editing, but I also leave in some character.

I remember many years ago, as a child, anything artificial was something to be hidden to some degree and sometimes even ashamed of. Take artificial flowers and fruit, for example. Granted, artificial flowers are much easier to care for, and they look close to the real thing, but they lack the full beauty of real flowers. It takes skill to care for real flowers.

If you are a creator, I beg you to keep creating real stuff. Write the real article or song. Paint or draw the real picture. Write and sing the real song.

Yes, I do use AI as a tool for basic tasks like generating video thumbnails, getting topic ideas, summarizing videos, etc. But I try to stop short of relying on AI to pump out content that might look good, but is inauthentic.

My goal is to stay authentic. You get what you see with me - warts and all.

Please join me!

04/10/2026

I’ve released a number of podcast episodes over time…but one stands out.

The recent conversation with Ruslan Desyatnikov, CEO of QA Mentor, on The Human Edge in Testing just crossed 5,000 views—more than any other of my podcasts so far.

That got my attention.

After all the discussion around AI, automation, and speed…this is the topic that resonated the most.

Not tools.
Not frameworks.
But people.

There’s something important in that.

Maybe now more than ever, the real question isn’t what AI can do—but what only humans can do.

What do you think is the most important “human edge” in testing right now?

BTW, in case you haven't seen this podcast yet, you can see it at

04/07/2026

Ready to strengthen your testing fundamentals and earn your ISTQB Foundation Level certification?

I’ll be leading a live, virtual ISTQB Foundation Level class on April 20–24—and I’d love to have you join us.

This is practical, real-world training built on decades of experience—not just theory.

Reserve your spot today! https://mysoftwaretesting.com/ISTQB-Foundation-Software-Testing--Live-Virtual-Course_p_46.html

ISTQB Foundation Level Course in Software Testing - Public Course