• Home
• Research
• Courses
• Publications
• People
• Affiliations
• Software

Large Display Framework Overview

Introduction

The Large Display Framework is the current C++ version of the buffer framework developed in the Interactions Lab at the University of Calgary [1, 2]. The goal of this software framework is to provide a reusable code base for creating large display applications with responsive interaction for a large number of visual components.

This goal is achieved by the use of sets of buffers as intermediate data structures for communication among objects. These buffers are called "Interaction Buffers" or I-Buffers. They are organized in stacks and keep positional informational that can be quickly accessed by components. They can also be used to steer components and store their properties. For more information on interaction buffers, please refer to [1] and [2].

Overview

This document provides an overview of the current version of the framework, which has substantial differences from the original prototype implementation [1, 2]. We start by providing some important definitions used in the description of the framework architecture. Next, we present the layered structure where a large display application that uses the framework is inserted, followed by a more detailed description of each layer and its classes. Finally, we conclude and invite the reader to learn more in the code tutorials.

Definitions

In the conception of the framework [1, 2], two different types of entities in interactive systems are identified:

1. Visualization Objects: entities that carry the information to be communicated; and
2. Control Structures or Interface Components: entities that guide the behavior of the visualization objects and are manipulated by user interactions.

In the context of this description of the Large Display Framework, the term Visualization Component (or just Component) is used to refer to entities of both types. When a visualization component can contain other visualization components inside its boundaries, it is referred as a Container.

I-Buffers are data structures that hold positional information relative to some visualization object's property or steering directive. They can be classified into three categories:

1. Active Buffers: buffers owned by an interface component that influence other interface components and visualization objects [2];
2. Passive Buffers: buffers that a visualization object or interface component looks into for specific information (usually, a passive buffer of a component is the active buffer of another) [2];
3. Private Buffers: buffers that hold information that only their owner uses and, therefore, should not be made visible to other components.

Structure

The framework is organized in a layered structure where each layer builds on top of the functionalities provided by the layers below. Each layer, with the exception of the Application layer, corresponds to a separated software library (e.g., a Windows DLL or a Mac OS Framework). A scheme of these layers is presented in the following figure.

The I-Buffer layer provides the interaction buffer data structure based on which the entire framework is proposed. The Large Display Framework layer is where the core of the framework is found. It is in this layer that the visual components are organized and that drawing and input handling are assigned to individual components. The Toolkit layer is a collection of classes that provides common functionalities that applications using the framework can employ. The iLab Toolkit layer holds classes shared by researchers in the Interactions Lab at the University of Calgary and that might eventually make their way into the Toolkit layer. The Application layer is where the user creates an application that uses the framework. It takes care of window management and input collection. Any additional classes written by the user are also in this layer.

In the following sections, we will describe each of these layers in more detail.

I-Buffer Layer

This layer is composed of a single class, called IBuffer. This is a template class so that it can be used with any data type. It defines the buffer data and the operations that can applied to this data.

Some of the important data and operations found in this class are:

• The number of channels in a buffer;
• The actual dimensions of the buffer, i.e., the width and height of the buffer data structure;
• The virtual dimensions of the buffer. The buffer may be accessed as it had different dimensions than it actually does. These different dimensions are called virtual dimensions. They can be useful for application with space constraints and low-resolution data;
• Resizing methods that can interpolate, copy, and erase the values in the resized buffer as well as fill it with a default value;
• Filling methods that can work on the entire buffer, on a quadrilateral region, or on an arbitrary polygonal region;
• Validation of buffer coordinates, i.e., checking if the coordinate pair is validly inside the buffer;
• Output the buffer contents as a PPM image, which can be very useful for debugging.

Framework Layer

The Large Display Framework layer encapsulates the mechanisms for integrating the visual components with the buffer paradigm. It manages the structure that organizes the visual components and distributes input events received from the application. It also controls the rendering loop. This layer represents the core of the framework. Its C++ source code is independent of any auxiliary library (besides the one provided by the I-Buffer layer, of course).

A scheme illustrating this layer is given below.

Let's look at each of the classes that compose this layer.

VisComponent

A central class in this layer is the VisComponent class. It represents the visual components (i.e., visualization objects and interface components) in an application. It has a state (position, width, height, rotation angle, color, and scale factors), a state stack, a collection of active buffers, a collection of passive buffers, and a chain of strategies (see the VisComponentStrategy class for more details). It inherits from the CompositeNode class.

CompositeNode

The CompositeNode class provides an implementation of the Composite Design Pattern [3]. This design pattern "composes objects into tree structures to represent part-whole hierarchies. Composites let clients treat individual objects and composition of objects uniformly" [3]. This is the kind of behavior we want the visual components to have. Therefore, the VisComponent class is made a subclass of this one. Objects of this class have references to their parent and child nodes. The order in the list of children of a node is important when drawing the composition of components, since children components are rendered in reverse order (i.e., the first component is drawn on top of - or after - the following ones). The methods in this class allow moving a node (and its underlying branch) from one parent to another, to bring a child branch to the front of the list of children of a node, and to add and remove children. When a leaf node (a node with no children) is necessary, it can be indicated by a flag.

VisComponentState

The VisComponentState class holds the transitory data of a component (i.e., a VisComponent object). This data is composed of:

• the 3D coordinates specifying the position of the center of the component;
• the rotation angle (about the z-axis) in radians applied to the component;
• the dimensions of the component, namely, its width and height;
• the scale factors along the x-, y-, and z-axis;
• the RGBA color associated with the component.

This class also provides a method for interpolating these data between two VisComponentState objects. The idea is to provide a simple way to have state change animations for a component. This is also one of the motivations for providing the VisComponent class with a current state and an auxiliary stack of states (so that previous or alternate states can be easily saved and retrieved).

IBufferProxy

Since the IBuffer class is a template, the IBufferProxy class provides a safe and abstract reference to it without knowing its actual data type. It holds a reference (a void pointer) to an IBuffer object, the identifier of the type of the buffer, and a reference to the component that owns this buffer (needed when calculating the buffer coordinates of a point). The elements of a component's list of active and passive buffers are objects of this class.

LargeDisplayEvent

The LargeDisplayEvent class provides a base class for events in the framework. The framework itself does not handle the events. Rather, they are forwarded to the components, which let their strategies handle them (see the VisComponentStrategy section). An event object should be created at the application level (i.e., by the user) when mapping a specific input to a type of event. The strategies in a component should be listening to events of that same type in order to be able to handle them.

Objects of the LargeDisplayEvent class can carry information about the event they represent. In the base case, this information consists of the identifier of the type of the event, the identifier of the user who generated the event, a reference to the VisComponent object that sent the event (in the case when the event is generated by a component instead of a user), and the 3D coordinates of a position associated with the event. More specific events (such as a KeyEvent for keyboard input) can be defined by inheriting from the LargeDisplayEvent class as needed.

VisComponentStrategy

The VisComponentStrategy class determines how a component should behave and appear. It is a combination of an adaptation of the Strategy and the Decorator Design Patterns [3]. The Strategy Pattern "defines a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it" [3]. It is adapted in the sense that, in this class, instead of defining a single family of algorithms, we define a set of families, covering drawing, event handling, and i-buffer management for an associated component. This adaptation is useful to allow data to be easily shared among different families of algorithms. The Decorator Pattern "attaches additional responsibilities to an object dynamically. Decorators provide a flexible alternative to subclassing for extending functionality" [3]. This design pattern allows having a complex behavior as the result of the combination of several simpler ones. In this way, a component can be specified as the combination of a "stack" of simple strategies, which can be reused and recombined in different ways to provide different components.

This approach replaces the hierarchy of component classes provided in the original framework. It follows the Object-Oriented Design principle that states: "Favor object composition over class inheritance" [3]. It allows for better encapsulation by having classes more focused on a specific functionality and facilitates reuse by the dynamic combination of functional modules. The absence of a long class hierarchy also contributes to more flexibility when assigning different independent behaviors to a single component.

The VisComponentStrategy class is the base class for the specific strategies created by the user. The specific classes should implement one or more of the virtual methods given in the VisComponentStrategy class depending on their needs and goals. The list of these methods is given below (for more information, please see Part II of the Large Display Framework Tutorial):

void initProperties()
void process()
void draw(const std::vector<unsigned long>& selectedIds)
void drawForPicking()
void onEvent(LargeDisplayEvent* evt)
void resize(unsigned int width, unsigned int height)
bool drop(bool parentChanged)
void readPassiveBuffers(const std::vector<unsigned int>& types)
void readAllPassiveBuffers()
void readAllPickingPassiveBuffers()

As per the Decorator Design Pattern, instances of VisComponentStrategy subclasses can be linked together and add functionality to a component. When a component should perform a given action (for example, draw itself or handle an event), it calls a corresponding method in the strategy object that is on the top of its stack of strategies. This object will recursively call this same method in the following strategy object in the stack (if any), before executing its designated action. This other object will do the same until it reaches the strategy object on the bottom of the stack. In this way, the first action to be processed is the one associated with the bottom strategy object and the effects of the actions of the remaining strategies are accumulated from the bottom up. The diagram below illustrates an example of this execution flow for a component associated with three different strategy objects:

LargeDisplayManager

The LargeDisplayManager class is responsible for controlling all the components in an application. It holds a collection of all the current components (a hash map with key given by the component's ID) and keeps track of the current dimensions (width and height) of the display. It has a list of components that might have had their composition tree structure changed (as when "dropping" a component into a container) and that should be updated. This list is processed (i.e., the components in the list are updated) at the end of every frame just after drawing the components. The LargeDisplayManager class is also responsible for distributing unique identifiers to the components that are added to its domain. A stack of freed identifiers is kept so as to optimize the use of identifier values.

As mentioned before (see the CompositeNode section), the visual components are organized in a tree structure. The LargeDisplayManager class keeps a reference to the root of this tree. From this root, the tree is recursively traversed for operations such as rendering and picking.

Events produced by the user are used to transmit asynchronous messages to framework. These messages can transmit information such as movements of a pointing device or the pressing of a key. The events that carry these messages are not handled by the framework. Instead, the manager forwards the received events to specific components, and their respective strategy objects are responsible for handling these events. The manager allows for four different types of event forwarding:

• forwarding the event to the component that is currently selected by the user who generated the event;
• forwarding the event to a specific component;
• forwarding the event to a subset of the components (multi-cast);
• forwarding the event to all the components (broadcast).

A special kind of component is used by the LargeDisplayFramework to indicate the position of a touch. This component is referred to as a touch indicator. The manager class has a list of touch indicators, one for each possible user of the application. Touch indicators are created exactly in the same way as other components (an instance of a component is created and initialized, and strategy objects are pushed into it). However, instead of adding the touch indicator to the list of components, it is added to the list of touch indicators. The basic difference in treatment given to the touch indicators by the framework is that every event produced by a specific user is forwarded to its corresponding touch indicator (in addition to the proper target of the event). Naturally, the touch indicator component should be associated to a strategy that will update its position based on the event coordinates (see the TouchIndicatorStrategy in the Toolkit layer).

Another responsibility of the LargeDisplayManager class is to keep track of the display frame rate. An object of the FrameTimer class is used with this purpose. This object is passed in the constructor of the LargeDisplayManager. The FrameTimer class is discussed in the next section.

FrameTimer

The FrameTimer class is used by the LargeDisplayFramework to measure the time passed between rendering intervals and, therefore, to calculate the application frame rate. The interval values are placed in a queue and their average is then used for the frame rate calculation. The size of this queue, or the maximum number of interval values in it, is defined in the class constructor. To measure time, this class needs to make platform dependent system calls. Therefore, the FrameTimer class must be extended in order to provide proper functionality. The subclass should implement the virtual methods getCurrentTime()(), which queries the system for the current time, and calculateFrameRate()@@, which calculates the frame rate based on the time unit provided by the system.

Utils

The Large Display Framework makes use of a set of auxiliary classes that are grouped under an 'Utils' directory. These classes consist in a Point3f class, which holds data for a 3D point and the basic operations over it, and a VectorTemplate class, which is a generic class for simple vector math. Common vector types are already defined, such as Vector2f, Vector3f, Vector2i, Vector3i, etc.

Toolkit Layer

The Toolkit layer sits on top of the Large Display Framework layer and provides a set of classes that can be used when creating an application for a large display. These classes make available some common strategies as well as utilities for loading textures and managing constants, for example. The GLUT (http://www.opengl.org/resources/libraries/glut/) and DevIL (http://openil.sourceforge.net/) libraries are used by a few classes in this layer.

The goal of this layer is to share some well established elements that were developed for previous applications. By reusing these elements, creation of new applications can be made easier and faster. Notice that this layer is not part of the "core" framework and that the framework can be used without the Toolkit.

Below is a list of the classes in this layer together with a short description of their responsibilities. For more information, refer to the documentation of each individual class.

ConstantProvider: provides dictionaries for mapping descriptions into numeric values to be used as constant identifiers for buffers and events.

DefaultManager: provides a default implementation of the LargeDisplayManager class, defining touch indicators and picking methods.

VisComponentGL: specializes the VisComponent class by applying the linear transformations relative to the component state before calling its drawer strategies.

FrameTimers/WindowsFrameTimer: provides a frame timer for the MS Windows operating system.

Utils/BufferFillingHelper: provides methods to help filling a buffer using multiple threads.

Utils/ImageLoader: loads an image from a file.

Utils/Spline: provides a spline curve obtained using subdivision.

Utils/TextureManager: manages a set of textures loaded from image files, providing the OpenGL texture names for them.

Strategies/Behaviors/DestructionStrategy: provides a base class for different types of destruction actions/animations performed by the component when it is deleted by a container/widget that has a DestroyerStrategy.

Strategies/Behaviors/DraggingResizerStrategy: provides resizing behavior from a dragging gesture.

Strategies/Behaviors/ResizerButtonStrategy: extends the DraggingResizerStrategy by providing an invisible button on lower right corner of component for resizing.

Strategies/Behaviors/RNTStrategy: provides RNT (Rotation 'N Translation) behavior.

Strategies/Behaviors/TossableRNTStrategy: combines tossing and RNT into a single behavior.

Strategies/Behaviors/TossableStrategy: allows tossing components.

Strategies/Behaviors/TouchIndicatorStrategy: provides a touch indicator (see LargeDisplayManager section) behavior.

Strategies/Behaviors/TranslationStrategy: provides translation behavior.

Strategies/Containers/BeltContainerStrategy: provides a container that has a belt shape defined by two closed spline curves.

Strategies/Containers/BoundedContainerStrategy: provides a container with boundaries given by a closed spline curve.

Strategies/Containers/CurrentBeltContainerStrategy: provides a belt container as an interface current.

Strategies/Containers/FrictionSurfaceStrategy: provides a surface with friction for the associated component.

Strategies/Drawers/BorderStrategy: draws a quadrilateral border based on the dimensions of the associated component.

Strategies/Drawers/GradientBorderStrategy: draws a border with a fading gradient around the dimensions of the associated component.

Strategies/Drawers/GradientCircleStrategy: draws a circle with a fading gradient from the center of the associated component.

Strategies/Drawers/GradientRoundBorderStrategy: draws a circular border around a component with radius equal to its shortest dimension and with a fading gradient.

Strategies/Drawers/ImageStrategy: draws a quad with a texture applied to it.

Strategies/Drawers/ResizerHandleStrategy: draws a handle on the lower right corner of the associated component that allows changing the component's dimensions.

Strategies/Drawers/RoundBorderStrategy: draws a circular border around a component with radius equal to its shortest dimension.

Strategies/GUI/CreatorStrategy: provides a base class for buttons that create visual components.

Strategies/GUI/DestroyerStrategy: provides a widget that deletes the components that are dropped on it.

Strategies/Observers/CurrentObserverStrategy: provides the associated component with the ability to respond to interface currents.

iLab Toolkit Layer

The iLab Toolkit layer contains a set of classes that are part of current research in the Interactions Lab at the University of Calgary. This layer is not part of the general release, but some of its classes might make their way into the Toolkit layer after publication. Some of the classes in this layer might be for testing purposes only, not providing very attractive or useful functionalities, and some might be still incomplete.

A list of the current classes in this layer together with short descriptions is given below.

Events/KeyEvent: extends the LargeDisplayEvent class to deal with simple keyboard events.

Strategies/Behaviors/FadeDestructionStrategy: provides a fade out animation when the associated component is deleted by a DestroyerStrategy.

Strategies/Behaviors/GaussianLensStrategy: provides a lens behavior with magnifier factor given by a Gaussian distribution.

Strategies/Behaviors/MagnifierLensStrategy: provides a magnifier lens behavior with a constant magnifier factor.

Strategies/Behaviors/QuadMagnifierLensStrategy: extends the MagnifierLensStrategy by given the lens a rectangular shape.

Strategies/Behaviors/RoundMagnifierLensStrategy: extends the MagnifierLensStrategy by given the lens a round shape.

Strategies/Behaviors/ThickeningBorderStrategy: thickens the border of the associated component when a state change animation is running.

Strategies/Behaviors/VortexDestructionStrategy: provides a vortex-like animation (rotation and scale down) when the associated component is deleted by a DestroyerStrategy.

Strategies/Containers/ColorMixerStrategy: adds its color to the components that dropped inside a container with this strategy.

Strategies/Containers/ResizerContainerStrategy: resizes the components that are dropped inside a container with this strategy.

Strategies/Containers/RotationContainerStrategy: applies a rotation to the components that are dropped inside a container with this strategy.

Strategies/Containers/UniformResizerStrategy: similar functionality as the ResizerContainerStrategy.

Strategies/Drawers/QuadStrategy: draws a rectangle with a plain color.

Strategies/Drawers/TextStrategy: draws text strings inside the associated component.

Strategies/GUI/GaussianLensCreatorStrategy: provides a button that creates a component with Gaussian Lens behavior.

Strategies/Observers/ColorMixerObserverStrategy: enables its associated component to respond to a container with a ColorMixerStrategy.

Strategies/Observers/RotationObserverStrategy: enables its associated component to respond to a container with a RotationContainerStrategy.

Strategies/Observers/ScaleObserverStrategy: enables its associated component to respond to a container with a ResizerContainerStrategy or a UniformResizerStrategy.

Application Layer

The Application layer is where the code for user-defined applications sits. In this layer, the user provides code for window management (e.g., Qt), definition of the graphics context (e.g., OpenGL), and input handling (e.g., SBSDK). Additionally, all the new functionalities and behaviors the user creates as Strategy classes are in this layer. The code provided in the tutorial for the framework is all in this layer.

Conclusion

This concludes the overview of the Large Display Framework. This framework provides a way to create large display applications that focus on modularity and reuse of components' features. The buffer approach, which consists in the base of the framework, allows for more responsive interactions when many dynamic components are part of an application. The main interface of the framework with its users is via the LargeDisplayManager and VisComponentStrategy classes.

To learn more on how to create applications using the Large Display Framework, please see the Large Display Framework Tutorial.

Fabricio Anastacio

Calgary, November 23, 2007
''Last Review: January 16, 2007\\ by Julie Stromer

References

[1] Tobias Isenberg, André Miede, and Sheelagh Carpendale (2006). A Buffer Framework for Supporting Responsive Interaction in Information Visualization Interfaces. In Proceedings of the Fourth International Conference on Creating, Connecting and Collaborating through Computing (C5 2006), pages 262-269, Berkeley, CA, USA, January, 2006.

[2] André Miede (2006). Realizing Responsive Interaction for Tabletop Interaction Metaphors. Master's thesis, Otto-von-Guericke-University of Magdeburg, 2006.

[3] Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides (1995). Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, Boston, MA, USA, 1995.