Tool-using agents often break on private, idiosyncratic APIs. RIMRULE turns a failed execution trace into a compact if-then rule, consolidates rules with Minimum Description Length, stores each rule in natural-language and symbolic form, and retrieves relevant rules at inference time. The method adapts the agent without changing model weights. 1
What changes
The learned artifact sits outside the model. Its symbolic schema uses five fields: domain, qualifier, action, strength, and tool category. That gives a product team something it can inspect, prune, scope, and reuse across different LLMs. 1
The product problem
A retry can expose the same tool error without fixing the reasoning pattern that caused it. RIMRULE targets those repeatable failures in decomposition, tool selection, and argument construction. The paper evaluates the approach on ToolHop and BFCL. For Llama 3.2 on held-out tools, accuracy rises from 35.1% to 43.1% on ToolHop and from 45.0% to 48.5% on BFCL after rule generation and consolidation. These are benchmark results, not a deployment guarantee. 1
First build
- Capture
query + available tools + wrong call + error + correct trace. - Ask a model to propose one short if-then correction, then test it against the originating failure.
- Store only rules that improve replay performance and whose scope is explicit.
- At runtime, match the current query and tool set to the rule library, inject the top few rules, and log the decision.
Start with the public ToolHop dataset or Berkeley Function Calling Leaderboard dataset, then replay failures from your own tools. Keep the rules external so you can compare models without retraining.
The paper assumes useful failure signals during learning, uses a greedy consolidation procedure, and relies on a hand-designed symbolic schema. Those are the first production checks to reproduce before treating the rule library as a reliable memory. 1
Read the RIMRULE paper.
Comments
Sign in to comment.