Aqueduct: Task-based entry points in Android apps

Abstract

Modern smartphones offer voice assistants to ease a variety of tasks. However, the actions that can be performed by current voice assistants are limited – a predefined set of built in actions like checking the weather, and a few hooks that can be built into third-party applications. To extend assistant actions to third-party applications, the onus is on the application developers to manually add support for voice assistant integration. To improve the link between voice assistants and third-party apps, we built Aqueduct, a data driven task-based app search and task entry point discovery system for Android. We search over app UI data augmented with semantic annotations to find applications and screens within those applications that can accomplish a given task. Furthermore, Aqueduct can leverage the package name and the activity name of the discovered screen to automatically navigate users to that screen. A user study was conducted to compile a set of common smartphone tasks and evaluate the effectiveness of Aqueduct, which showed that it is effective at finding task-based entry points for a wide range of tasks. Aqueduct is also useful for augmenting search in application repositories, finding starting points for execution for task-automation systems, and even generating deep link suggestions for applications.