Hypothesis
I hypothesized that a heavily quantized, sub-billion parameter model (Qwen 0.5B) could reliably act as an offline automation brain if the prompt strictly enforced JSON formatting. The goal was low-latency execution without eating up developer RAM.
Setup & Variables
- Model: Qwen 0.5B via Ollama.
- Input: System prompt enforcing JSON format:
{"action": "string", "parameters": {}}. User prompt asking it to map a natural language command to a specific action. - Execution Environment: A Python script capturing stdout and attempting to parse the result via
json.loads().
Observed Result
While the model's latency was astonishingly fast on standard CPU hardware (avg 200ms to first token), its deterministic reasoning was severely flawed.
- Hallucinated Syntax: The model frequently forgot closing braces
}or added Markdown code block ticks ````json` inconsistently. - Context Loss: In multi-turn conversations, it rapidly lost the strict structural requirement and began responding conversationally ("Sure! Here is your JSON:").
- Parameter Bleed: It struggled to map variables to the correct nested keys, often inventing new top-level keys.
Interpretation & Conclusion
The experiment failed its primary objective. Qwen 0.5B is not currently viable for zero-shot strict JSON generation in a production-like automation pipeline.
The memory footprint is incredible, making it suitable for simple summarization or semantic search embeddings, but it cannot function as the "brain" of a deterministic API without extensive middleware to heal broken JSON strings. This observation directly drove the architectural pivot in the StudyOS project towards a hybrid cloud-reasoning approach.