Massive Language Fashions (LLMs) are more and more deployed to autonomously remedy real-world duties. A key ingredient for that is the LLM Operate-Calling paradigm, a extensively used method for equipping LLMs with tool-use capabilities. Nevertheless, an LLM calling features incorrectly can have extreme implications, particularly when their results are irreversible, e.g., transferring cash or deleting information. Therefore, it’s of paramount significance to think about the LLM’s confidence {that a} perform name solves the duty appropriately previous to executing it. Uncertainty Quantification (UQ) strategies can be utilized to quantify this confidence and stop probably incorrect perform calls. On this work, we current what’s, to our data, the primary analysis of UQ strategies for LLM Operate-Calling (FC). Whereas multi-sample UQ strategies, resembling Semantic Entropy, present robust efficiency for pure language Q&A duties, we discover that within the FC setting, it provides no clear benefit over easy single-sample UQ strategies. Moreover, we discover that the particularities of FC outputs could be leveraged to enhance the efficiency of current UQ strategies on this setting. Particularly, multi-sample UQ strategies profit from clustering FC outputs primarily based on their summary syntax tree parsing, whereas single-sample UQ strategies could be improved by choosing solely semantically significant tokens when calculating logit-based uncertainty scores.
- † College of Oxford
- * Equal contribution
- ‡ Joint senior authorship
