Interaction

Ideally, user interaction with smart products resembles the natural interaction with products in an environment. To achieve this goal, smart products pro-actively manage a dialogue with the user. Unlike traditional human-computer dialogues, this dialogue does not only comprise acts of explicit interaction (see interaction types: D.5.1.2), but also acts of implicit interaction, i.e., the dialogue is context aware, e.g., it may react to physical actions of the user. The SmartProducts platform enables the developer to manage such complex user-product dialogues by using the well known concept of workflows.

For more information on the detailed design of the interaction subsystem, see deliverable D.5.4.1.

Developers specify a workflow for each procedure they want the smart product to support. The activities in a workflow can link to context events or services on other smart products. However, these links are not hard-wired. The SmartProducts platform dynamically embeds the workflow in the runtime environment of the smart product, connecting to the right context and service providers. If no suitable service is available for executing an activity, the SmartProducts platform will automatically switch to explicit interaction an approach the user.

For more information on how to use workflows in the SmartProducts platform, see Interaction Manager.

The user interface for explicit interaction can be provided by the developer, or it can be automatically generated by the platform, based on the description of the activity. The interaction can take place using on-board interaction devices, like a built-in screen or interaction devices discovered in the environment, e.g., a text-to-speech service. The actual rendering to the different devices is performed by the SmartProducts platform.

Smart products must tailor their behaviour to the individual user to be easy to use. This adaptation is automatically performed during runtime by the UI Adapter component. The UI adapter requires the specification of the user’s preferences in a user model. We want to free the developer from manually specifying all this information by automatically learning the user model from observation. This should be done at the level of product functionality, e.g., by learning which type of coffee a certain user prefers, and also at the level of interaction, e.g., by learning whether a user wants to use or does not want to use voice based interaction.

Relevant Platform Components

• UI Adapter The UI Adapter changes the UI taking into account the preferences of the user (e.g., native language) and available interaction resources.
• Interaction Manager The Interaction Manager controls the overall flow of the user interaction, e.g., by following predefined workflows and generating the right UI at the right time.
• Multimodality Manager The Multimodality Manager keeps track of available interaction devices (local or remote) and renders a UI produced by the UI Adapter using these devices.
• User Model Learning The User Model Learning component extends the products knowledge by improving the products’ model of the user, e.g., by learning her preferences.