Akatama's Slice of the Internet

AI, Testing, and DevOps: Practical Insights from StarWest 2025

By Jimmy Lindsey

When I first heard about StarWest, I wasn’t sure it was the right conference for me. As someone working as an SDET today but aiming for a future in DevOps, I wondered if a QA-focused event would really be valuable. What I found was a mix of vendor sales pitches, insightful talks, and great networking opportunities. In the end, I walked away with some surprising lessons, many of which extend well beyond QA into engineering and DevOps.

Overall Feel

StarWest has a corporate feel, but that’s not necessarily a bad thing. Most attendees wore business casual or formal attire. Conversations tended to focus on your role and the hard problems you’ve solved.

At StarWest, everyone is here to learn, to improve themselves, find solutions to problems they face at work, to network and to possibly find a new role. All of those aspects mean that the people at this event are in business mode. As long as you aren't expecting it to be something it is not, you can definitely use this to your advantage if you go with the right mindset.

Vendors

The exhibition hall at StarWest is entirely company booths, most of them selling tools and platforms. That’s not unusual, but it does mean you’ll hear a lot of pitches. The common theme this year was an “agentic low-code/no-code” solution that also requires handing over all your data, something I grew tired of hearing about after the fifth time.

It’s not StarWest’s fault, but it does make the $2,145 ticket feel steep if you’re paying out of pocket. Since my company covered the cost, I didn’t mind as much, but if you’re self-funding, the heavy vendor presence is worth keeping in mind.

Networking Events

Throughout the day, they had several networking opportunities. These events were generally a meal (breakfast and lunch), with two other breaks throughout the day to meet people. The breaks were short, but still enough for a few quality conversations with interesting people.

The evening event was the highlight: food, drinks, and a raffle for those who completed the vendor “passport”. The atmosphere was relaxed, people were in a good mood, and I even walked away with a $100 Visa gift card. More than anything, it was the best time to connect with others.

Talks and Keynotes

Overall, I think the talks and keynotes were amazing. If I had to have any criticism for them, it would've been nice to have a few more that weren't focused on AI or LLMs. LLMs are the hot topic right now, so it was no surprise to see many talks focused on them. In fact, it was interesting to see different people evaluating this technology's usefulness, but it would've been nice to see talks that focused on automation testing or programming fundamentals as well. These fundamentals are crucial to using AI effectively.

I only attended one "industry insight" talk, which are put on by the vendors of StarWest. I have to say in the future I will likely avoid them. It was about 20 minutes of useful information, and 40 minutes of a sales pitch. Unless you think you will actually want to purchase what the vendor is selling, I would recommend the same for you.

What I Missed

I never made my way to The Test Lab and Automation Zone. I did meet the two excellent leaders who run it, Julie Gardiner and Christ Loder. I felt like I had to go to as many talks as possible, but when I go next year I will make sure to stop by and check it out!

What I learned

Below I am going to distill some things I learned. I still have to go through some of the talks I missed, but here's what I have for now.

Put Our Service to the Test: A Beauty and the Beast Tale of Quality Engineering by Janna Loeffler

This talk focused more on the communication and collaboration aspects of testing. The tests themselves aren’t the point. Their value is in guaranteeing quality.

• Empower and uplift the teams you support
• Join forces with security and SRE to improve quality
• Provide quality expertise to teams in your organization

OK, AI I am Ready to Meet you Halfway — Help Me Write My API Test Automation by Tim Heck

To this point, I have mostly used GitHub Copilot to help me write tests. However, this talk took a very grounded approach in showing the best ways to use LLMs to help write API tests. In particular, I liked how Tim didn't suggest that he had all of the solutions, and instead just showed what he had tried so far, and how well it worked.

• Pure ChatGPT can get you there, but you can achieve better results with structured prompt chaining
• If your tool has an MCP server, you can connect to it with GitHub Copilot and achieve similar results
• Do not use LLMs to directly test the API, instead use them to generate the code or scripts that test the API
• This is because testing is transparent and repeatable, but LLMs aren't

Are API Tests Overrated? Let's Mitigate Risks by Andrew Knight

I definitely do not agree with everything Andrew said in his short talk. However, I think he made some great points, and if nothing else he made me think about the most valuable way to test software.

• If you have software engineers that write their own unit tests, then API tests become less significant
• Unit tests test your business logic, like API tests, but they don't have to worry about the network connection
• Consumer-driven Contract testing can replace the part of API testing that focuses on the connection and API framework

Agents All The Way Down by Dionny Santiago

Dionny suggests that we use AI to test AI, and suggests a way we can do this. Unfortunately, the easiest way to see what he means is through the diagrams he provided during his talk. I could reproduce them here, but I didn’t include them due to copyright concerns.

• Hard or impossible to test AI with traditional testing
• Not Always a single "right" string
• Often multiple valid strategies
• Domains mix task + safety + cost
• Ground truth is rare or moves
• Test an AI with another AI
• Can run thousands of test variations 24/7
• Can verify very quickly
• AI can generate bizarre edge cases
• Testing agents can learn and improve themselves over time
• Eventually you will need to escape the infinite recursion of testing
• Gold standard tests
• Human expert review
• Statistical confidence bound
• Depending on what you are using the AI for, you may be able to short circuit the process earlier

How to Steal AI's Testing Job by Filip Hric

Filip talks about a time he lost his job when his company made a pivot to be an AI company.

• Demystify AI for yourself
• Turn fear into hope
• Confusion into clarity
• Pessimism into activation
• AI is not the only reason there are a lot of jobs being lost:
• Interest rates
• AI made a lot of companies uninvestable, so they had to pivot
• Work transformation happened as well
• It's okay to be late on AI trends
• Look at edge cases
• Look into security
• Become good at context engineering for LLMs

Optimizing Delivery Life Cycle: Harnessing DevSecAIOps for Secure and Efficient Solutions by Sudipta Debnath

So the title of this talk is a bit heavy on buzzwords, but the content of it was great.

• Shift left on security and integrate it with developer workflows
• Security isn't a checkpoint, but a continuous journey
• Integrate AI and Operations for operational efficiency
• Enhance security with predictive insights

Takeaways

Here are the main takeaways:

• Focus not on the tests you write, but the quality you ensure and provide
• this is something I already knew, but it was great to hear it framed so succinctly
• Context engineering and configuration of LLMs is important
• One useful trick: export your LLM session as a Markdown summary, then use that summary to restart and refine the dialogue
• AI can further enhance DevOps / DevSecOps workflows

Conclusion

StarWest isn’t cheap, and it’s not perfect. You’ll sit through vendor pitches and pay a premium for the experience. If your company is willing to cover the cost, the fresh perspectives on testing, AI, and DevOps make it worthwhile. For me, the biggest value came from the conversations, the ideas I brought back to my work, and the connections that will last long after the conference. If you go with the right mindset, you’ll get more than your money’s worth.