Managing costs when working with large language models (LLMs) can be tricky. This guide shows how to create a smart routing system that directs prompts to the most suitable model based on complexity. Using local prompt classification and model switching, developers can save money while maintaining performance.

Setting Up the Environment and Tools

The first step involves installing the necessary Python packages, including NadirClaw, OpenAI, SentenceTransformers, and others for data handling and visualization. The process also includes capturing an API key for Gemini, a service used for model routing. If no key is provided, the system defaults to local classification, skipping live API calls.

This setup ensures that local prompt classification works independently, making it accessible even without API access. Once installed and configured, the system is ready to classify prompts and compare different models’ behavior.

Local Prompt Classification and Routing Logic

The core of the system is a classifier that sends prompts to the NadirClaw CLI, which returns a JSON with details like the estimated complexity tier, confidence score, and suggested model. Prompts are categorized as simple or complex, and this classification helps decide which model to use for each task.

Developers can test this locally by providing various prompts—ranging from simple questions to complex technical requests—and inspecting the routing results. The classifier uses centroid vectors representing simple and complex tasks. By measuring cosine similarity between prompt embeddings and these centroids, the system determines the task’s nature.

Visualizing and Comparing Model Similarities

To understand how prompts relate to the centroids, embeddings are generated using a sentence transformer encoder. These embeddings are then compared to the centroid vectors through cosine similarity, allowing visualization of how prompts cluster based on complexity.

This step helps verify that the classification logic aligns well with the actual prompt content. Visual plots show the distribution of prompts, highlighting those deemed simple or complex. Adjusting confidence thresholds can fine-tune the classification accuracy, balancing cost savings and response quality.

Switching to Live Routing with Model Proxy

Once the local classification is reliable, the system can be extended to live routing. This involves launching a NadirClaw proxy server that intercepts requests and routes them to the appropriate models, such as OpenAI’s GPT variants. The proxy handles prompt forwarding, model switching, and cost tracking seamlessly.

Developers can send prompts through this proxy, which automatically determines whether to use a cheaper, simpler model or a more powerful one based on the earlier classification. This setup reduces costs by avoiding unnecessary use of expensive models for simple tasks.

Comparing the behavior and output quality of routed models helps ensure that cost savings do not significantly impact performance. The system can estimate potential savings by comparing the always-on premium model baseline against the cost-aware routing approach.

Overall, this approach provides a flexible, efficient way to manage LLM usage, balancing performance and expenses effectively. It can be integrated into existing workflows or scaled for larger applications, offering a practical solution for organizations working with large language models.

Inspired by

• https://www.marktechpost.com/2026/05/10/how-to-build-a-cost-aware-llm-routing-system-with-nadirclaw-using-local-prompt-classification-and-gemini-model-switching/

Sources

• github.com
• aidevsignals.com