Experiment Objective
To determine if a small, extremely fast embedding model (MiniLM) can accurately classify user prompts and route them to fine-tuned specialized models without introducing unacceptable latency (>50ms) into the critical path.
Hypothesis & Problem
Problem: Loading a single giant model (e.g., 70B parameters) to handle all queries is memory-prohibitive for edge nodes. Loading multiple smaller, specialized 8B models is feasible, but requires a routing mechanism to send the query to the correct expert.
Hypothesis: Computing cosine similarity between the incoming query embedding and pre-computed cluster centroids will take less than 20ms in Rust and yield >95% routing accuracy, effectively creating a poor-man's Mixture of Experts (MoE).
Dataset
I synthesized a dataset of 10,000 prompts across 4 distinct categories:
- Code Generation (Rust, Python)
- Creative Writing
- Data Extraction / Structuring (JSON outputs)
- General Chit-chat
Methodology & Models
The routing layer was written in Rust using the candle ML framework. The embedding model used was all-MiniLM-L6-v2. Upon initialization, the router computes the centroid embeddings for each of the 4 categories based on 100 sample prompts each.
When a new prompt arrives:
- Tokenize and embed the prompt using MiniLM.
- Calculate cosine similarity against the 4 centroids.
- Route to the model assigned to the closest centroid.
Experiments Run
I ran the 10,000 test prompts through the router under varying load conditions (1, 10, 50 concurrent requests) to measure embedding latency, routing latency, and overall accuracy against the ground truth labels.
Results & Visualizations
The results largely validated the hypothesis, though concurrency exposed some bottlenecks in the tokenization step.
Conclusion
Semantic routing is a highly viable strategy for edge deployments. The <20ms overhead is negligible compared to the generation time of the LLM. However, a dedicated batching tokenizer is required if concurrency exceeds ~20 requests/sec on standard hardware.