It’s a great time to be a builder. The path into technology and product development is rarely direct, and it takes a wide variety of people and skills to create amazing things. Many of us find our way into this field through a love of technology and a passion for making things that solve real-world problems.
This journey often leads to product management—a fascinating discipline that sits at the intersection of user needs, business goals, and technical feasibility. It’s about figuring out what to build, for whom, and why. You work with numerous talented people to bring a concept to life. Whether in a small startup or a massive corporation, the core challenge remains the same: making smart choices with constrained resources.
A product manager’s job is to champion the user and the business. This requires a deep understanding of the problems you’re trying to solve. A common question is how deep into the technical details a product manager should go.
Personally, I believe in knowing just about everything—perhaps not down to the specific library, but certainly understanding the medium you're building in. Computing is a creative medium, and you can't make effective decisions without understanding what it can do and what it takes to do it. If Plan A takes six years to deliver a feature, but Plan B can deliver 10% of the value in two weeks, is that 10% worth it? You can't answer that without understanding the technical trade-offs.
This philosophy is about being micro-informed, not micromanaging. It’s about having all the information to guide the team, trust their expertise, and make the best possible decisions. This is especially true in the rapidly evolving world of AI. As a product manager, you need a solid set of tools at your command to help guide engineers in exploring the right solutions.
A healthy dynamic often involves the product manager defining the "what" and "why," while the engineering team owns the "how." The product manager’s role is to define the problem, motivate the team, and provide constraints. The engineers then explore multiple paths and present the trade-offs. However, these roles are a spectrum. The best results come from a collaborative conversation where everyone contributes ideas.
With the explosion of AI, a common question we face is: which model is the best? You see a Super Bowl commercial for one AI tool, and that’s all some people know. But for those deep in the industry, there are hundreds of thousands of models, and choosing the right one is not a trivial question.
This article was inspired by the idea that there is no single answer. You need to figure out the answer for your specific context. It’s about using the right tool for the job, not just the tool that’s most popular or easiest to access. Every AI company tells you to use their tool, but they rarely help you understand the why.
AI models can seem like magic. They produce captivating, human-sounding output that can lure you in. But that’s not what they are. We have to systematically test for a broad range of inputs and outputs, which is a new challenge. Quality assurance for deterministic systems is well-understood (2 + 2 always equals 4), but testing non-deterministic AI systems is a much tougher problem.
In this new era of AI, evaluation should be a conversation between product, engineering, and the business. Here are several tools and platforms that can help facilitate that conversation.
Hugging Face is an amazingly deep marketplace for finding the latest AI models. It’s not just for big-name brands; anyone can upload a model.
What makes it powerful is its public evaluation leaderboards. An evaluation is a set of technical measurements of how well a model performs on specific, standardized tests. These tests are structured so you can ideally run the same test on two different models and compare the results.
On Hugging Face, you'll find a raw, data-heavy view of model performance across various benchmarks like GPQA or math proficiency tests. While highly technical, this allows you to get up-to-the-minute, broad comparisons. If your product’s success depends on a specific capability, like language translation or mathematical reasoning, you can sort the leaderboards by that metric to see which models excel. This is not a quick, five-minute exercise but an iterative, exploratory process for deep technical analysis.
On the other end of the spectrum is Chatbot Arena, which is all about human interpretation. It provides a simple interface where you can enter a prompt and get responses from two randomly chosen, anonymous models (Model A and Model B). As a human, you read both and vote for the one you think is better.
For example, asking, "What's the best pizza in New York City?" will yield two different answers. Your vote for "better" could be based on accuracy, tone, formatting, or any other subjective factor. After you vote, the platform reveals which models you were comparing. The aggregate of thousands of these head-to-head battles creates a leaderboard based on human preference.
While useful for getting a general sense of which models people "like" more, this approach has limitations. It primarily focuses on single-turn, question-answering tasks and often conflates persuasiveness with truthfulness. Large language models are not sources of truth, yet we often unconsciously evaluate them as such. They are good at many other things, like natural language understanding and intent analysis, which this format doesn't capture.
Chatbot Arena also has leaderboards for more specific tasks, like coding. For instance, a prompt like "create a pizza rating website" challenges models to generate functional code.
This is where things get interesting. One model might produce a simple HTML structure, while another generates a more interactive application using a library like React.
Model A Output: Basic Structure ```javascript // Model A might produce a simple, static output. import React from 'react';
const PizzaWebsite = () => { return (
Time to rate the pizza.
{/* Further implementation would be needed here */}export default PizzaWebsite; ```
Model B Output: Interactive with State ```javascript // Model B might interpret the request more dynamically, using state and better styling. import React, { useState } from 'react';
const PizzaWebsite = () => { const [ratings, setRatings] = useState([]);
return (
Welcome! Add your ratings below.
{/* Logic to display and manage ratings would go here */}export default PizzaWebsite; ``` In this scenario, you can inspect the code, see the rendered output, and vote. This reveals how different models interpret the same prompt and their proficiency with different libraries and coding patterns.
Ultimately, choosing a model is about trade-offs. This is where a tool like Artificial Analysis shines. It aggregates data from multiple sources and presents it in a way that’s incredibly useful for product people and engineers.
It moves beyond simple leaderboards to multi-dimensional charts, such as: * Quality vs. Price: This helps you see which models offer the best performance for their cost. * Quality vs. Output Speed: This is critical for user-facing applications where latency matters.
These visualizations allow you to quickly identify models that fall into your desired "sweet spot." For example, you can instantly see which models are in the top-right quadrant for high quality and high speed.
More importantly, this framework forces you to consider factors beyond raw performance. For a project at a major healthcare institution, the best-performing open-source model was unusable because its terms and conditions prohibited use in medical applications. That kind of critical business or legal constraint won't show up on a technical leaderboard. Artificial Analysis helps you narrow down the options so you can perform that final, crucial due diligence.
Finally, look for in-depth, rigorous studies from organizations that specialize in AI quality. Galileo, for example, produced a detailed report on model hallucination—one of the biggest concerns for business and product leaders. If you can't trust the results, the AI is just a cool party trick.
The strength of such reports is their depth and thoughtful analysis. They often evaluate models on more nuanced tasks, like their effectiveness within a Retrieval-Augmented Generation (RAG) system, which is a much better use case for LLMs than treating them as standalone sources of truth.
The downside is that these reports are static and can become outdated quickly in this fast-moving industry. However, they provide an invaluable snapshot and a methodological blueprint for how to conduct your own deep evaluations.
This brings us to the critical topic of Quality Assurance (QA). How do we truly test these things? The state of the art for testing LLMs is still a work in progress, even within the QA community. The non-deterministic nature of these models throws a wrench in traditional testing paradigms.
The seductive nature of a smooth-talking AI is the biggest hurdle. People type in one or two prompts, get what looks like a compelling answer, and assume it works. But that answer could be built on completely fabricated points. Getting past this superficial validation is the frontier.
One effective strategy is to limit the LLM's role. Instead of using the LLM to find the answer, use it to understand the user's question and then to format the final response. The core logic—finding the actual answer from a reliable data source—should be a deterministic, testable process. This RAG-like pattern gives you a middle chunk that you can easily test, reducing the risk associated with the less predictable parts of the system.
Ultimately, evaluating and choosing an AI model is not a one-time decision. It's a continuous process. The tools and models will change daily, but the principles of diving deep, understanding trade-offs, and never trusting a headline will always apply. You must get into the details to find the right answer for your product and your users.