I had a problem: as an AI implementation consultant, I had no automated pipeline to discover and qualify leads and insert them into my self-hosted Twenty CRM instance. I thought I could solve this whole problem using LLMs.
The first version of my prospect discovery system gave one LLM agent the full scouting pipeline: web search, deduplication, validation, and database insertion. About 10–20% of the prospects found were irrelevant, duplicated, or improperly inserted into the CRM. To make matters worse, the system was extremely difficult to debug, as each instance of the agent took a different approach and ran into its own unique process and tooling speed bumps. I ended up having to search through the entire list of 800+ prospects in the CRM manually myself, checking for duplicates and validating if the prospect was relevant. Clearly this solution wasn’t going to work as-is.
Executives seem obsessed with implementing AI in any form possible. I often hear that AI adoption progress is measured using arbitrary metrics such as token usage or lines of code written, which don’t measure if AI is being used in a helpful or a harmful way. Large language models (LLMs) are fundamentally just a tool. If your CEO thinks AI is a hammer, and she wants to use that hammer for everything, then everything ends up looking like a nail. This is the wrong approach to AI solution architecture.
The goal should not be to say “we’re an AI-first company” or “our product uses AI”. The goal should be to solve the most pressing and relevant problems using the appropriate tools. LLMs are one of those tools. The first question asked should be “what problem are we solving?”, not “where can we apply AI?” When the problem to be solved is decided, only then can the solution be specified. What capabilities does the solution require? Where does the current solution, if any, fail? And is integrating an LLM into that solution actually necessary?
Every tool in the software development tool belt has strengths and weaknesses, and LLMs are no different. The question is not whether LLMs are useful, but whether the capabilities justify the trade offs they bring with them. LLMs provide flexibility by sacrificing determinism. They are useful when the work involved requires interpretation of natural language, synthesis across abundant or varied information, and step-by-step reasoning where the rules of a process can’t be specified before it begins.
These gains in flexibility result in losses in repeatability and determinism. LLMs produce variable output — that is, the same prompt given to the same model twice will produce tangibly different results. This makes testing and failure analysis much more difficult than for traditional code because success criteria are often subjective. Additionally, a process executed by an LLM will typically take much longer and cost more than the same workflow executed using plain code. These limitations mean that we need to evaluate the need for AI in a solution thoroughly before we blindly assume it will be helpful.
When evaluating if an LLM will be useful, it’s helpful to think through the decision using concrete questions.
In my prospecting system, I originally had the LLM execute the entire prospecting pipeline, searching, deduplicating, validating, and DB insertion. The model would generate the search queries, execute the queries with an MCP web search tool, deduplicate the returned prospects against the ones currently in the DB, validate the prospects against my ideal client description, and then write the surviving list into the CRM. Because there were so many steps, it messed up often.
Let’s use this example to think through the questions I outlined in the previous section.
The decision of whether or not to use an LLM is not straightforward. Trade offs have to be made: flexibility versus determinism, adaptability versus accuracy, cost versus performance. Even in my relatively simple prospecting system, the answers to the validation questions are not always a clear yes or no. I’ve found this often to be the case: usually a solution that contains an LLM also contains deterministic validation and process-automation code. The best solutions combine the strengths of LLMs with the strengths of deterministic code — it’s not a one-or-the-other decision.
Though many problems can be solved using deterministic solutions and would not benefit from the addition of a large language model, we live in a time where many problems that previously could not be solved with a computer now can be. These solutions don’t require your engineers to spend hundreds of dollars in tokens every day — they require your team to evaluate the problem at hand and decide to use the right tool for the right job.
Where did replacing an LLM with deterministic code improve your system, and what new trade offs did that introduce?
Source: Hacker News — This article was automatically imported from the source. Read full article at original source →
