ScreenTK: Seamless Detection of Time-Killing Moments Using Continuous Mobile Screen Text and On-Device LLMs

Smartphones have become essential to people's digital lives, providing a continuous stream of information and connectivity. However, this constant flow often depletes users' limited attentional resources and time, leading to decreased productivity and increased stress levels. This issue underscores the need for tools that empowers users to maximize their potential for achieving personal objectives. One effective approach is to identify "time-killing" moments--a specific type of attention surplus--during which users seek to fill perceived free time without a specific purpose. Recent work has utilized screenshots taken every 5 seconds to detect time-killing activities on smartphones. However, this method often misses to capture phone usage between intervals. We demonstrate that up to 50% of time-killing instances go undetected using screenshots, leading to substantial gaps in understanding user behavior. To address this limitation, we propose a method called ScreenTK that detects time-killing moments by leveraging continuous screen text monitoring and on-device large language models (LLMs). Screen text contains more comprehensive information than screenshots and allows LLMs to summarize detailed phone usage. To verify our framework, we conducted experiments with six participants, capturing 1,034 records of different time-killing moments. Initial results show that our framework outperforms state-of-the-art solutions by 38% in our case study.