WESE: weak exploration to strong exploitation for LLM agents

Recently, large language models (LLMs) have demonstrated remarkable potential as autonomous agents. However, existing studies mainly focus on enhancing the reasoning or decision-making abilities of the agent through well-designed prompt engineering or task-specific fine-tuning, while neglecting the procedure of exploration and exploitation. When addressing complex tasks within open-world interactive environments, these methods exhibit limitations. First, the lack of global environmental information leads to greedy decisions, resulting in suboptimal solutions. By contrast, irrelevant information acquired from the environment not only adversely introduces noise but also incurs additional cost. To address these limitations, this paper proposes a novel approach, namely, weak exploration to strong exploitation (WESE), to enhance LLM agents for solving open-world interactive tasks. Specifically, WESE involves the decoupling of the exploration and exploitation process, employing a cost-effective weak agent to perform exploration tasks for global knowledge. A knowledge graph-based strategy is then introduced to store the acquired knowledge and retrieve task-relevant knowledge, enhancing the stronger agent in terms of success rate and efficiency for the exploitation task. Our approach is flexible enough to incorporate various methods in solving diverse tasks, thereby achieving remarkable improvements in effectiveness (success rates), efficiency (number of steps), and cost (expenses for API tokens) across four interactive benchmarks.