Report from workshop 3 – Evaluating Use and Impact

The workshop brought together speakers and participants from a range of cultural heritage areas and academic disciplines in a convivial atmosphere for fruitful exchange of experiences.

ColourPostIts manyIan Ruthven, Professor of Information Retrieval at the University of Strathclyde, opened the day, welcoming all to the University of Strathclyde hosting the event. He explained how the workshop themes relate to the research undertaken at the Department of Computer and Information Sciences which he chairs, but how they also reflect important concerns in the field.

Maria Economou (HATII & The Hunterian, University of Glasgow) who co-0rdinates the network, outlined how the workshop’s topic on evaluation methodologies and the “how” of evaluation work followed from the previous two workshops which looked at models of digital access in Scotland and then at user needs with particular emphasis on Crowd-sourcing, Co-creation, C0-creation.

Jen Ross (Lecturer in Education, Community and Society and  Co-director of the Centre for Research in Digital Education, University of Edinburgh) talked about Artcasting, an AHRC-funded project which is developing, testing and assessing a new digital mobile platform for evaluating arts-based engagement, using as a case-study the ARTIST ROOMS On Tour. One of the project’s research questions examines how can a mobilities approach which asks visitors to make connections between art and place constitute meaningful evaluation practice. Artcasting is using qualitative methods and a design-based approach to generate, pilot and evaluate artcasting prototypes. These methods include interviews, workshops with young visitors, iterative design of the artcasting application, in-gallery observations, and analysis of usage data and user-generated content shared by artcasting users. Jen showed aspects of the rich material collected from users through the Artcasting app that is still being analysed and raised interesting questions about the use of the digital platform itself as a method of evaluation for understanding engagement with art.

Staying with the theme of mobile interpretation and engagement, Areti Galani (Lecturer in


Dr Areti Galani, University of Newcastle, talking about experience-drive evaluation of heritage/art mobile apps

Museum and Heritage Studies in Media, Culture, Heritage at Newcastle University) talked about experience-driven evaluation methods for mobile cultural applications. She referred to the situated, personal/private, and dynamically constructed nature of the ‘felt experience’ of heritage and art apps and the importance of understanding the process of sense making that users go through when using them. She presented two quite different case studies which they evaluated, the Rock Art on Mobile Phones (RAMP) web app and the Second Moon art app. Unlike Jen’s Artcasting example which focused on engagement in or stemming from the art gallery space, both of Areti’s examples focused on the every day experience of heritage and art in an outdoor setting.


Mind map used in the evaluation of the Rock Art Mobile app – from Areti Galani’s presentation

The first one studied rock art web apps in rural Northumberland to evaluate whether the app supported the discovery of rock art, speculation about is meaning and a sense of place. It used ten self-selected participants, which the researchers shadowed on site while they were using the app, debriefing qualitative interviews and mind maps based on prompts. This qualitative in-depth approach provide a rich and deep understanding of the nature of the users’ experience and how they constructed meaning and used it in their exploration. The second study that Areti presented on the Second Moon was an art project by British artist Katie Paterson that was commissioned as part of the British Science Festival in 2013. The evaluation used again a small sample (7) of self-selected app users who were asked to keep an online diary using Google docs. As with the previous case, it was impressive to see the depth and richness of the self-reflective information users volunteered and how they allowed the researchers to see the ‘felt experience’ of the app in action and they ways they appropriated the technology. Areti highlighted the lessons to be learnt from these studies, including the observation that good usability does not necessarily equate with a meaningful experience but meaningful experiences require good usability.

Moving from qualitative approaches to evaluation methods that can capture large quantitative of data, the next presentation focused on eye tracking and gaze analysis. Jan Hendrik Hammer (Fraunhofer Karlsruhe Institute of Technology) explained how eye tracking and gaze analysis can help us understand gaze-based interaction and highlighted its potential and limitations in cultural heritage evaluation. He outlined the type of investment in hardware and software, as well as the skills involved in analysing the data, which might pose limitations for smaller cultural heritage organisations. Jan used the case study of the ArtSense project and the Valencian kitchen display of the National Museum of Decorative Arts in Madrid where eyetracking was used to compare where visitors’ attention focused and points of interest when the explored on their own and with an audio-guide. He also referred to the use of eye tracking via webcams where no eye tracker is needed and can involve online users from around the world, using the example of Glasgow’s well known landmark of the Duke of Wellington statue in from of the Gallery of Modern Art, where most it is obvious that the users focus on the cone on his head. This webcam use of eye tracking has considerable potential for user interface analysis for online digital libraries and websites.


Jan Hendrik Hammer, Fraunhofer Karlsruhe Institute of Technology on use of eye tracking with webcam, using the Duke of Wellington statue outside the Gallery of Modern Art in Glasgow


His talk raised an interesting discussion about how eye tracking and gaze analysis helping you record where the users and visitors are actually looking but not necessarily understanding the reasons for doing so and their type of engagement.

The last speaker, Professor Paul Clough from Sheffield Univesity School of Information, talked about the evaluation of information searching in digital cultural heritage. He looked at the question of what makes a search system successful and the different criteria used to evaluate this (like the retrieval of relevant documents, user interaction support, user satisfaction, impact on the wider user environment) and remarked that the answer actually depends on who you ask, the users and the context. The is even more important in cultural heritage settings. Most typical in information retrieval evaluation, the focus is on the quality of the search results but as many search-based applications are typically rich in features, it is necessary for evaluation procedures to move beyond studying just the search box. He presented the lessons


Professor Paul Clough, University of Sheffield, presenting the PATHS project at the workshop

learned from the study they carried out of the European PATHS (Personalised Access To cultural Heritage Spaces) project. This developed techniques to support expert and non-expert users navigating and using cultural heritage materials from Europeana. It also investigated the use of trails/paths to facilitate narrative-like structures through digital collections for use as guides and learning aids (like exhibitions/guides in physical space). The evaluation activities in PATHS covered a range of approaches and perspectives, ranging from evaluations carried out by researchers to select best algorithms, evaluations carried out by user interface designers and evaluations of the integrated prototype by end users both in controlled lab-based user testing and field trials. He identified the challenges they faced when ‘thinking outside the search box’, including sharing evaluation practices between domains and disciplines. This was a useful point to remember when working with the diversity of material and interdisciplinary nature of digital cultural heritage.



The groups working hard

As with the previous workshops, the afternoon session included group work with five different groups exploring the participants’ own experiences of evaluation in this area and the advantages and limitations these brought. Some of the issues highlighted at the plenary at the end was the need for a balance between quantitative and qualitative methodologies, the lack of a single golden evaluation method, the importance of integrating evaluation in a life-cycle from the beginning of a project and not bring in as an afterthought, but also moving from projects to integrate evaluation work in everyday practices of organisations and getting institutional support at all levels.IMG_2761


Thanks are due to all speakers, participants and moderators for their hard work: Seamus Ross (Dean of Information School, University of Toronto & Visiting Professor, University of Glagsow); Ian Ruthven, Professor of Information Retrieval, University of Strathclyde; Areti Damala, Research Fellow, University of Strathclyde (not only for moderating but also the hard work in the organisation and planning); Monica Callaghan, Head of Education, The Hunterian; David Scott, Digital Media Curator, Glasgow Museums.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s