In addition to providing pleasant and stimulating experiences, complex cultural collections can require significant amounts of cognitive work on the part of visitors. Whether collections are situated in physical spaces or presented via web-based interfaces, the sheer richness and diversity of artefacts and their associated information can frequently lead to cognitive overload and fatigue. In this article we explore visualization methods that can be used to fend off fatigue and to support cognitive tasks such as collection exploration and conceptual comprehension. We discuss a variety of options to generate collection representations with multiple views and focus on the rarely heeded challenge of how to integrate information from these views into a bigger picture. By utilizing multiple space-time cube representations (through the PolyCube framework), we discuss an effective approach to integrating and mediating multiple perspectives on cultural collection data. We illustrate its potential by the means of a case study on the work of Charles W. Cushman and outline first insights drawn from a heuristic evaluation. Finally, we situate our approach within the larger epistemic and methodological environment of humanities approaches to visualization design.
Can we address the elephant in the room? Spaces containing complex cultural collections (CCC) pose thorough challenges to the cognitive systems of visitors. Encounters with galleries, libraries, archives, or museums require sense-making activities with a vast number of mostly unknown objects. These are frequently of high perceptual diversity and rich in detail, each one connected to many threads of further information; and are commonly arranged in physical architectures based on unfamiliar principles. Even if visitors intend only to experience leisurely pleasure, such encounters require significant amounts of perception, interpretation, and learning. In short, considerable
So aside from their well-known marvelous and inspiring aspects, it is rarely made explicit that CCCs require considerable support from a perception and cognition perspective. Learning about collections–i.e. building up a mental model (
We are reminded of the parable on the elephant and the blind men.
Following decades of digitization, CCCs often exist both as traditional object collections in physical spaces, and as digital collections in data and information spaces.
Looking around, we find numerous design strategies which help visitors to grasp the elephant while shunning, minimizing, or ameliorating fatigue. Many of them have been applied both in physical museum spaces, as well as in digital information spaces. Prominent methods include storytelling (
In the following section, we zoom in on approaches which utilize methods of visualization to support the understanding of complex cultural collections. A synoptic approach is outlined by section three, its exemplary implementation in the fourth section, and its evaluation in the context of a case study in section five.
Visualization creates graphical representations from complex data allowing visitors to explore them interactively, and to acquire insights that unaided perception would not allow for (
Digital collections commonly integrate digitized object representations of artefacts (such as images, text, audio, videos, or 3D models) and associated metadata entries, such as place of origin, date of origin, creator, style, or inter-object relations (see Figure
Cultural collections comprise a diversity of object types (left). As digitized collections, they are commonly translated into a digital object (image, text, audio, video, or 3D model), and enriched with multiple dimensions of metadata (right).
In some cases the databases of GLAM institutions already mirror the complexity of their physical collections, constituting a prototypical example of massive, heterogeneous, abstract, and often time-oriented data. Such digital databases are often even less amenable to human sense-making than their physical counterparts, a problem exacerbated by the fact that visitors to digital collections are often treated as if common information seekers on the web, and so are provided with only the most basic (search-centered) access technologies. Such search-based interfaces require a thorough understanding of the collection’s structure and available metadata to retrieve meaningful results (
Let’s imagine a visitor arriving at the landing page of an art gallery, an archive or a museum, with a collection he doesn’t know well (cf.
As a powerful paradigm for interaction with abundant information, the ‘search box’ approach to information retrieval has dominated interface and interaction design since the emergence of the web. Search boxes still are often chosen as the main method of access and are used even by the largest cultural collections such as Europeana, which, as of June 2018, contained more than 50 million artworks (
Dörk et al. (
This ‘generous design’ avoids starting with questions but prefers to directly show: it aims to offer rich overviews and context, as well as high quality primary content and detail on demand (Butler, 2013). Because it has the privilege to deal with data that does not have to be hidden it can throw the doors of collections wide open and so transform databases into giving and sharing visual repositories, which represent scale and richness; but also allow multiple ways to focus on specific details. To honor the complexity and diversity of a collection, generous design offers multiple access or vantage points, and encourages multiple perspectives on the assembled riches. Understanding that any given visualization method can capture only certain aspects of a collection’s composition or structure, it calls for multiple views to be used in the presentation of objects, combining the strengths of different methods and forming complementary composites to reveal different aspects of a collection. Such a multi-perspective interface enables the ‘open-ended proliferation of partial views, rather than a single total or definitive representation’ (
Another key facet of human information acquisition that visitors can utilize in such interfaces is ‘serendipitous’ engagement. In museums, libraries and other open object collections, visitors frequently find interesting and inspiring information by chance. Several studies on everyday information practices show that serendipitous encounters constitute a key component of information acquisition (
As a standard technique for fostering multiple entry points and a plurality of perspectives and interpretations, the method of ‘multiple views’, or ‘coordinated multiple views’ (
Multiple spatial or structural visualization methods (left) and multiple methods to visualize time (right).
However, offering multiple views can also be a way to cover a specifically interesting data dimension in a more diverse or in-depth fashion. For the cultural heritage domain,
While we consider this generous provision of multiple (spatial, structural, and temporal) perspectives as a strength of novel interfaces, their implementation also comes with a notable downside, which has been barely mentioned or problematized up to now: multiple perspectives recreate perceptual complexity and diversity on the overview level on our screens. The resulting challenge has various consequences for macrocognitive reasoning operations (
Split-attention challenges arise when observers of multiple views start to wonder about the bigger picture of a collection—or what the whole elephant looks like—yet their diverse information sources appear spatially or perceptually separated, and do not easily merge.
Screenshots taken from Palladio (cf.
For a synoptic integration of the displayed data, users have to combine information from both views (i.e. from the spatial and temporal perspective at the same time) and build up a mental model that integrates both data dimensions. Cognitive science researchers have called attention to the fact that such synthetic operations are cognitively demanding in general, but require even higher cognitive effort when the aim is to construct a coherent and consistent mental model rather than a sketchy ‘cognitive collage’ (
With regard to the synthesis of bigger pictures, we distinguish between possible results along a quality gradient of construction. According to Tversky’s distinction (
The visual-synthetical continuum from microcognitive representations (single perceptions) to macrocognitive representations, that are either cognitive collages or mental models.
While it is relatively easy to synthesize jumbled and fragmented collages from multiple views, their coherent assembly requires either more mental effort by the user—or the development of more effective techniques of visual-synthetical design on the visualization side.
With regard to ‘coherence techniques’, which support cognition by connecting insights from different views to larger units of sense-making, we find two basic approaches: the use of consistent visual variables or design choices across multiple views (
As Funtowicz and Ravetz (
The research project
The PolyCube emerges from the space-time cube representation (STC), first developed and utilized in human geography to support the visual analysis of human movement patterns and the spatial diffusion of innovation (
By the means of a space-time cube representation, the PolyCube scaffold can arrange CCC objects as point clouds according to multiple spatio-temporal arrangement principles. On the bottom, a data plane initially features a geographic map, and each object’s place-of-origin determines its horizontal position. The vertical axis of the cube represents time–and thus date-of-origin assigns an individual altitude to each cultural object above the ground (Figure
The space-time cube as an integrated visualization method for collection data.
Contemplated from a distance, this framework rearranges every corpus as a characteristically shaped ‘hyperobject’, which invites on-demand probing, zooming and close-up display. Further visual structures are
Different spatiotemporal patterns and expressions within the space-time cube, which can be generated from rich CCC data to show distributional, relational and categorial (set-like) shapes of a CCC in parallel.
Basic flow patterns to visually parse and analyze the evolution of CCC developments.
Figure
For categories of objects—accumulated and delineated by sets—the framework generates expressive flow patterns (second from right and far right), which exemplarily can disclose the parallel evolution of cultural styles or schools, or their mutual genealogical influences. For these accumulating perspectives—which can also indirectly visualize the associated development of cultural organizations, art schools, religions, fashions, disciplines, or any other collective entities—a simple pattern language helps users visually parse complex developments as composites of basic temporal patterns (Figure
When utilizing the third dimension in InfoVis, one should prepare for some additional explaining. As Munzner (
Firstly, the STC achieves the integration of spatio-and-temporal in a fair and balanced manner by distributing the strongest and most effective visual variable (i.e. position, cf.
Secondly, this unfolds a whole new visual-analytical morphology as an expressive and technically open-ended, time-oriented pattern language, that could be parsed and read by highly trained faculties of 3D gestalt perception (see below), and which synoptically encodes time like no other method we know (see Figures
Thirdly, as Bach et al. (
Seamlessly traceable transitions on the space-time cube allow to derive various alternative visualization perspectives on a collection’s spacetime (from
Fourthly, empirical studies on casual users (
Fifthly, studies show that STC representations are liked because they are ‘cool’ (
Sixthly, Sorger et al. (
Drawing these arguments together, we consider STC representations to provide a powerful and largely untapped potential for visual-synthetical mediation—not in spite of but due to their use of an additional display dimension. While this also increases visual clutter and interaction costs (e.g., due to additional rotating, zooming and panning, cf.
To consolidate the outlined design principles, we present first patterns and insights from a digital collection case study, reshaped by the first implementation of the PolyCube framework as a visual-analytical research prototype.
Three main considerations guided the technical implementation of the PolyCube concept: reusability; modularity for ease of reading and extension; and compatibility with DOM selection in order to accommodate various document object model (DOM)-related libraries such as data driven documents (D3.js). We aimed to build the PolyCube 3D rendering environment on CSS3D, doing without the WebGL engine as much as possible, as this is still not supported by browsers and older devices with limited exposure and instability of the current HTML5 canvas.
To ensure modularity, the code was built in a modular fashion using the popular JavaScript framework to support easy integration and creation components by creating a PolyCube object as a function with properties such as
As for the data, we make use of the Cushman Collection (
For the case study, a geographic map and a set-diagrammatic visualization were implemented as cross-sectional visualization methods. These views have been transferred to an STC, which also offers a juxtaposition and a superimposition perspective on demand. Figure
Space-time cube representation of a sub-selection of Charles W. Cushman’s work, comprising 800 images between the years 1938 and 1955.
The representation allows for rotation, zooming and panning, and access to previews of photographs (see Figure
Animated canvas transitions, seamlessly translating an STC representation into a juxtaposition perspective (top row) and from a juxtaposition into a layer superimposition perspective (bottom row).
The ‘time flattened’ superimposition perspective on a subselection of the Cushman Collection, with a highlighted picture from Annapolis, MD, 1940, on the left hand side.
From a model-based reasoning perspective, these transitions strengthen the visual momentum of the visualization system (
The flat superimposition layout allows for inspection of the overall spatial distribution of objects, and the precise reading of spatial positions from an orthogonal point of view. As the time-axis has been shortened, it is possible to encode time into another retinal variable like the color of the data points to allow for a balanced comparison of different spatiotemporally integrated perspectives.
Figure
The layer juxtaposition (or small multiple) perspective on the Cushman Collection, visualizing photographs taken from 1938–40, 1940–45, 1945–50, 1950–55 (from left to right).
Figure
The set-diagrammatic STC perspective on the Cushman collection.
If such set-diagrammatic cuts through the longitudinal development of a collection are further enriched (for instance by differentiating subsets), the flow-patterns of Figure
Whether for online or offline collections, such data sculptures can serve as prime exhibits among others, featuring as a bigger picture of the whole elephant, and as a novel interpretation of the advance organizer concept (
We conducted a qualitative evaluation of the PolyCube prototype with three casual users: two female and one male. None of them had prior knowledge about the Cushman collection, nor any expertise in the field of information visualization. They participated voluntarily in this study without any remuneration besides some complimentary chocolates.
Following a short introduction to the Cushman Collection and the interaction techniques offered within the prototype (rotate, zoom, pan, select), participants were left to freely explore the prototype on a 24’’ screen while thinking aloud. The visual structure of the STC was not further explained as we sought to understand how casual users make sense out of the unfamiliar PolyCube system (similar to the procedure in
During the free exploration, Participant 1 started with an extensive phase of close reading—viewing and evaluating the different photographs—before she was encouraged to explore the arrangement of the data points (7’) and slowly gained an understanding of the visual structure (15’). The other two participants explored and understood the visual structure right from the beginning. Participant 3 rightly observed that significantly fewer than 800 data points became visible in the various perspectives, which was caused by the merging of spatiotemporally adjacent data points on the chosen scale.
Participants reported no significant problems while answering our questions. They could identify spatiotemporal patterns efficiently with the STC. All three participants were able to describe where Cushman lived or travelled during each period. Participant 1 was the only one to show initial difficulties in relating the data points to the correct geographic regions, but came to grips with the task after rotating the STC. Confronted with the task to identify the time periods when Cushman was the most active or inactive, all three participants could instantaneously point out the corresponding time periods. When asked to describe the collection to someone else, they focused on their (mostly emotional) evaluation of the explored photographs rather than on the collection’s spatiotemporal characteristics. As Participant 2 phrased it: ‘a number of uninteresting photographs, but in a nice toy to play with’.
During the final interview, all three participants preferred the STC over the juxtaposition and superimposition visualizations. As Participant 1 stated: ‘you can’t feel the logic at once, but then it is becoming clear … You can compare period, territory, the main objects. This is nice’. All participants highlighted the STC’s potential to support an integrated understanding of the geographic and temporal distribution and interdependencies of the data, which cannot be as easily derived from the other views. They also highlighted the attraction and user experience of the STC. As Participant 2 put it: ‘if I have something boring [the photographs] and fun [the STC]—and something boring and no fun—I’ll take the former, obviously’.
The participants suggested numerous improvements. With respect to the visual design, Participant 2 would stated they would have found the STC more logical or natural if the time-axis were inverted. For the juxtaposition perspective, all participants missed labels specifying the temporal periods. Participant 3 suggested improving the labeling on the time axis so that it can be easily read regardless of rotation. We also collected some design suggestions, such as the ability to enlarge selected photographs on demand and the addition of data layers of related (historical or political) events, so that the artworks of the collection could be contextualized in a broader space-time context.
As for the set-diagrammatic visualization (Figure
In this article, we have reflected on both the curiosity and openness that drives people to explore cultural collections and on the well-known limitations of their cognitive resources. Information visualization offers a powerful spectrum of methods to provide visitors to complex collections with facets of a bigger picture. Interaction with such representations can add to the visitor’s sense of overview and orientation – and thus facilitate conceptual understanding. Following our discussion of recent achievements of generous interface design, we focused on a second-order problem that arises from one of its central design strategies: multiple views allow visitors to inspect CCC data from diverse perspectives and support the investigation of spatial, structural, and temporal data aspects. Yet, most of these interfaces leave users to themselves when it comes to the integration or mediation of these perspectives.
We introduced the PolyCube framework as a method to mediate and integrate a diversity of local views on a global level of representation. Analogous to the provision of overviews on a local level, this enhances the ease of global cognitive syntheses and reduces cognitive efforts to integrate various perspectives without sacrificing any of the benefits offered by plain local ‘standard’ views. In particular, the options provided by seamless transitions appear as a promising technique to support visual macrocognition and as a noteworthy strategy for strengthening the visual momentum of advanced interfaces for use with and by cultural collections (
Revolving around an organismic metaphor of complexity, we have discussed a specific combination of techniques to reassemble elephants as a whole. Towards the end of this endeavor it seems necessary to look into one of the most obvious limitations of this metaphor: cultural collections—like so many other complex phenomena—have no original (spatial or visual) superstructure that can be visually reconstructed in an isomorphic fashion. Diagrams and information visualizations are indispensable techniques because they successfully create
If we aligned a good part of this article’s argumentation with the blind men’s quest for information integration, we know that other observers of the scenery can see things differently. To also bring in their perspectives, we close with a reflection on expected reservations about our holistic approach. Regarding various humanities approaches to interface design (
If interpretation is a central operation underlying the thinking and working of the humanities, then interface design has to support this activity, conceived as an open-ended, critical and constantly self-challenging endeavor. Related approaches thus sometimes question traditional HCI objectives like ergonomic efficiency, but strive to foster elaborate evaluation and reframing activities like critical reflection, intellectual argument and rhetorical engagement. To this end flow, or pleasure-driven engagement with data is also deemed essential; as well as the acknowledgement of the subjective, situated, and partial character of every emerging result (
As with many controversies, it is possible to tap into such lines of contrarian argumentation by mapping them within a ‘square of opposition’ (Figure
Square of opposition, showing lines of contrarian argumentation (orange) between humanities design positions together with possible lines of development (blue).
In a reliable fashion, contrarian arguments emerge from a diagonal polarization, where charges (arrows in orange) are directed from the upper corners of a position (A1 and B1) to the opposite corners at the bottom (B2 and A2). Corresponding controversies thrive on the common self-idealization of a position in combination with the devaluation of the opposite value. Yet the square can also show ways for mediating tensions by developing dynamically balanced or hybrid positions in between (arrows in blue). While not being especially popular in the academic context, pragmatic approaches to the mediation of controversies can move both sides forward.
Connectivity is key. While it is possible to enjoy many humanities controversies as explication of competing and contrarian positions, advanced interface design is well-advised to read them in a complementary fashion and to bring their best arguments into a dynamic balance. This will also allow us to take care of a more informed development of bigger pictures in the realm of the humanities, despite the damage that approaches concerned with these big pictures suffered from poststructuralist decrees. As has been stated with regard to ambitious accounts of culture and history in general: ‘[i]f the grand narratives known so far … have been seen through as unsuitable attempts to seize power over the world’s complexity, this critical realization neither delegitimizes the narration of things past nor exempts thought from striving to cast an intense light on the comprehensible details of the elusive whole’ (
To remake and refine visual representations of cultural collections and other complex humanities topics, we advocate synoptic visualization approaches which coordinate the best knowledge representation strategies of multiple communities. Such hybrid endeavors will generate more effective approaches to the support of macrocognition in face of data diversity, and the facilitation of switching between multiple perspectives and sense-making frames. This seems to us to be not only a design task worth strengthening, but also a cognition technique which comes close to a civic meta-competence for these times, arguably not only needed in digital humanities’ and cultural sciences’ research domains.
The fable, which has been traced back to Buddhist, Hindu and Jain texts around the 1st millennium BCE tells the story of a group of blind men, who learn and conceptualize what an elephant is by touching it. Each blind man feels a different part of the elephant body, such as the side, the tusk or the trunk. They then describe the elephant based on their partial experience. Their descriptions lead to disagreement on what the object essentially is.
Concerning the scope of contemporary ‘crowd-curated’ CCCs consisting of native digital objects, estimates approach hilarious numbers: 70 million Instagram uploads a day (
For an investigation of the state of the art and future challenges for information visualization approaches to cultural heritage collections, see Windhager, Federico, Schreder et al. (
‘Split attention effect’ is the name for a phenomenon where learners are offered multiple descriptions or depictions of the same topic, and thus have to integrate these representations mentally. This forced integration process stresses the learner’s working memory and can negatively impact learning if the mutually dependent or complementary sources are designed poorly or cannot easily be synthesized. To create effective learning environments, it is recommended that designers avoid split-attention by externally integrating the different sources of information together into a single integrated source of information (
Due to their unique visual syntax and data spatialization principles, information visualizations are specifically challenging components for top level-integration. Yet also with each visualization type itself, the challenge to mentally merge cross-sectional and temporal perspectives ranks high.
See the etymological origins from the Greek
Cognitive science research points to the significant payoffs that coherent representations can have on local visual sense-making (cf. Figure
As Dörk et al. (
As for the visitor group of visualization experts, who are expected to rather feel provoked by the use of 3D representations, we agree with Bitgood (
The cultural collection visualized with the Palladio toolkit in Figure
Known issues for poor holistic elephant paintings or system designs commonly include a lack of interpretive plurality; a lack of declared positionality; a lack of openness for critique, modification, or revision; a lack of transparency on data and visualization methods’ provenance; a lack of system performativity or procedurality; and last but not least, claims for impartiality and objectivity. We consider all these problematizations of bigger pictures to be frequently justified and want to foster their systematic discussion for future visualization system developments (e.g. Dörk et al., 2013;
To a certain extent, we can even consider the visual information-seeking mantra (‘overview first, then zoom and filter, details on demand’,
Furthermore, pro-plurality approaches rarely want to be read as ‘argument in favor of bad, inefficient, or obstructive design’, nor provide a ‘perverse justification for the ways in which under-resourced projects create confusion, as if that were a value for humanists’ (
This research was supported by a grant from the Austrian Science Fund (FWF), project number P28363-24.
The authors have no competing interests to declare.