Author: Mike Leggett

ABSTRACT : The contemporary burgeoning usage of digital movies, photos, audio and text, their distribution through networks both electronic and physical will be considered in the context of a convergence of these media with a popular interest in personal and community history and identity.
The paper introduces interdisciplinary research into human memory as a context for understanding its relation to machine memory and methods of storing and retrieval. It proposes an approach to indexing audio-visual media utilising a time-space representational system, drawing upon a real-world time-space representation as the taxonomy of the indexing procedure.
An interactive experimental prototype, PathScape, will be described and evaluated and further practice-based research approaches to author-defined storage and retrieval systems will be outlined.
Author Keywords
Interactive, digital media, taxonomy, index.
ACM Classification Keywords
H5.2 User interfaces: user-centred design.

INTRODUCTION
Storage of artefacts is far easier than finding them again, as any dog will tell you. (Anon)
This paper outlines some research that seeks to develop tools for storing and retrieving audio-visual digital media. The design of the system will need to accommodate the needs of the ‘memory worker’, whether as an individual, or part of a closed or open working group.
The contemporary burgeoning usage of digital movies, photos, audio and text, their distribution through networks both electronic and physical will be considered in the context of a convergence of these media with a popular engagement with personal and community history and identity.
Interdisciplinary research into mind and memory, perception and cognition, presence and embodiment, media representation, creativeness and meaning, will provide a context for understanding this approach to investigating machine memory. A short survey of methods of storage and retrieval of audio-visual digital media will provide the background for the further development of an existing experimental prototype.
DESPERATELY SEEKING….
“Memory is a label for a diverse set of cognitive capacities by which humans and perhaps other animals retain information and reconstruct past experiences, usually for present purposes.”(30)
Lansdale and Edmonds in a 1992 study investigated the design of document filing systems by developing a prototype, MEMOIRS, that treated “…documents as a particular form of event memory”, referring to it as episodic memory. (20) Sutton describes episodic memory as “personal memory for past events and experiences accompanied …by a feeling of familiarity and a reflective awareness of having had the experiences in the personal past.” (30) Semantic memory delivers to us facts derived of the world – Freud died in London – knowledge by association. As the MEMOIRS project observed: “It is enough that the distinction between episodic and semantic memory throws into perspective an approach to the design of filing systems based upon event memory as opposed to the associative relations between items.” (21)
With interest in and the relevance of the field increasing, interdisciplinary memory research is becoming increasingly recognised and valued (17). As Sutton points out, “It’s no accident that memory is at the heart of recent work on dynamical cognition and the embodied, embedded and extended mind…” and that the “…brain and world are often engaged in an ongoing interactive dance through which adaptive action results.” (30)
Interacting with external memory machines such as collections and libraries of knowledge located on computer servers around the globe are central to academic pursuit and increasingly, the education and edutainment of the population. The machine-based memory industries that specialise in servicing this demand by storing data and knowing how to retrieve it again, are moving away from notions of information retrieval and database management towards information gathering, seeking, filtering and visualisation. (29)
DIGITAL SHOEBOXES
Another computer-based industry, growing annually, is digital video. (Notes 1). Disseminated by cable, broadcast, the internet and more recently the mobile phone into the home and the workplace, audio-visual media is ubiquitous (Notes 2) and will increasingly become the format of document that will need an advanced design of filing system. Digital media can be used simply to document an object or the appearances of an occasion, but it is also expressive. In the hands of a trusted author, (or authors), visual media can inform us and reflect us in ways of which we are often unaware. Many of us have the option to gather these images, as photos, as video, as sound. In making images as records of the passing moment, we are able to display our appearance, our presence, often instantaneously, in a place, of a time. But having made the record and following its initial consumption, what then happens to the artefact? “..there has been very little research attention given to how people organize and browse their photo collections, whether digital or non-digital.” (Notes 3) (28)
As collective or personal memory decays, whether a corporate memory or a family memory, the connectedness of events to the media artefact fade and the narrative thread is disrupted. The significance of the memory, the meaning of the image even, can be lost.
‘Episodic memory’ or personal memory is discussed by philosophers at length. Like semantic memory, episodic memory is declarative memory which sets out to represent the world, usually with the aim of truthfulness (30). Epistemologies of representational systems are debated between interdisciplinary researchers working in the fields of philosophy, cognition, perception, cultural theory and semiotics:
“Signs represent the present in its absence; they take the place of the present … when the present does not present itself, then we signify, we go through the detour of signs.“(11)
The notion of ‘memory traces’ and representations for and of recall, while remaining contested ground, form the basis of memory storage and retrieval devices, from the dictionary to the encyclopedia, from the diary to the snapshot. Autobiographical and personal memory can be prompted by what Tulving terms “synergistic ecphory” (32) whereby the emotion or the memory is evoked or revived by means of a stimulus (27). Often aided by the context of the recall, a writer for instance, through placement of artefacts or words in spatial relationship can create the circumstances which connect with the narrative (of a memory trace, event, object etc). We are not unfamiliar with the use of postcards and palm cards or scraps of paper placed around the room as a way of organising complex sources in the process of synthesising thoughts and events into fresh formulations. (Notes 4)  
Within the repositories of collected memory, in large public collections for instance, the stimulus relies on a common rather than private language of signs, most often expressed in a word index form.
INDEXING OPTIONS
“Indexing is a way to increase retrieval precision and accuracy by consistent application of subject terms in their preferred forms. … A taxonomy is a controlled vocabulary presented in an outline view, also called a classified view or hierarchy. Terms are organized in categories reflecting general concepts (Top Terms), major groups (Broader Terms), and more specific concepts (Narrower Terms). The final terms at the end of a branch, often called nodes, can represent any specific instance of a Broader Term, including terms from an authority file of people, organizations, places, or things.” (7)
A taxonomy of indexing enables an overview of the topography of the system, by reducing scale and quantity to proportions that can be comprehended, particularly by new or inexperienced users. In many ways ideal for text-based data such as large ICT parallel database systems (31), such to approach audio-visual data based upon word interpretation is constraining, useful only when words in documents need to be illustrated. On-line picture libraries use keywords associated with location, subject, artist, colour, date, owner etc. – the AHDS Data Service Visual Art (2) image resources site is an example of this tradition as are many photographic archives and stock-shot libraries. Whilst a word index is admirable for locating traces within written language sources, “..keyword searching is a crude and unsatisfactory method for sampling the information content of complex sources….” such as media collections. (8) Likewise, seeking images on the web with a search engine is similarly hit and miss, having to double guess a file name or location descriptor or other aspects of the meta-data, if present.
Glorianna Davenport is one of a group of researchers who have developed approaches to storing and retrieving the complex nuances of the audio-visual artefact within machine-memory database systems. One of these was developed by a research team in the Media Lab at MIT during the mid-90s, ‘Jerome B. Wiesner, 1915-1994: A Random Walk through the 20th Century’ (9) By monitoring the users initial selection subsequent options are reorganised to cluster related topics, using a combination of image and words and re-shuffling their relative positioning on the screen. Each thumbnail image is able to operate as an iconographic link to play the archival media material.
At about the same time, a British artist, Chris Hales made ‘Twelve of My Favourite Things’, an interactive diaristic installation, accessed using a touch screen. A composite of three Quicktime movies, through interaction with ‘hot spots’ based on visible colour zones, movies narrating the world of some young children recorded talking about their favourite colours, places and people, replace one another within the composite on the screen. Contained in scope and size by the technology of the time, the work was an early model of how it could be possible to navigate a series of recollections using wholly visual means. (15) Hale’s overall project to develop an interactive cinema based on these indexing principles has currently reached fourteen iterations of the touch screen-based model.
Research projects seeking industrial objectives, visual indexing systems for the television and cable industries, have included the IBM CueVideo research project. The project measured the productiveness of automated indexing, browsing and retrieval based on different means of summarising digital video using keyframe storage, and accelerated sound reproduction employing audio processing TSM technology. (1) Whilst the taxonomy is text-based, the final indexing stage which locates sequence or shot, is a audio and/or visual abbreviation of content, of relevance to our current concerns. (Notes 5)
Well established software tools, such as ArcView, are related to topography, recorded time and place, and are widely used in industries related to environmental planning, water and land management, urban layout, national parks, mining and agriculture, etc. These are specialised tool sets based on data derived from various methods of measurement. GIS satellite data and a range of plug-ins to the system enable digital images, sound and text files to be attached to specific coordinates. This provides extensive profiles to be constructed and navigated in real-time from numerical data using graphical and map visualisations. Such tools have been adapted by archaeologists and social scientists. In the west of Sydney, the NSW Migrant Heritage Centre has commissioned a website [13] using a application called TimeMap that links a combination of text and map metaphors with personal oral histories and localities around the City of Fairfield in western Sydney.
These tools offer a plethora of styles and codes that incorporate maps, diagrams, graphical and typographic devices, each inflected with current tools and fashions in interface design. The Fairfield project takes an approach closely related to the archaeologist’s inventory, making it possible to store and retrieve data about the past, but making the oral and written evidence useful for archaeologists and educationalists but uninvolving and distant as an experience for individuals in the community.
LOCI SYSTEMS
The Greeks oracists and rhetoricians, who before the alphabet had been handed down, developed an elaborate form of artificial memory, described so fully in Yates’ Art of Memory. Ars memoria, “…a series of loci or places. The commonest, though not the only type of mnemonic place system was the architectural type ….. We have to think of the ancient orator as moving in imagination through his memory building whilst he is making his speech, drawing from the memorised places the images he has placed on them.” (34) It could be claimed the first movies were a conceptual model made by the Greek rhetoricians, complete with wide shots, tracking shots, panning, tilts, close-ups and flashbacks, all played in the cinema of the mind’s eye, the first ‘classic film narrative’.
In a 2002 edition of Nature Neuroscience, a study included a range of tests carried out on people who were highly ranked in the World Memory Championships. Whilst their brain capacity and structure was determined to be average it, it was found with functional magnetic resonance scanning (fMRI) that the regions associated with navigation and memory were more active than in a control group attempting the same memory tasks. The contestants confirmed that they used a strategy called the ‘method of loci’ in which the objects to be remembered were placed along an imaginary pathway that could be retraced when recalling the items in order. “The longevity and success of the method of loci in particular may point to a natural human proclivity to use spatial context – and its instantiation in the right hippocampus – as one of the most effective means to learn and recall information” (23)
In this, the age of the rhizome (10), linearity need not structure thought within the confines of logic and rhetoric. In the same way as the walk from home to the station may allow interventions of the everyday to structure the day itself, even enhanced by the imprecision of the visual cues that guide us during the walk, then too the invention or re-invention of a visual literacy based on digital video and ‘machine memory’ technologies, would enable us (with the happenstance of chance encounter), to employ indexing and classification appropriate to the task in hand.
An experiment in the late 1970s by the Architectural Machine group at MIT, ‘Aspen Walk’, linked two video disc players with a computer system. By interacting with a touch screen display, the viewer could navigate the image of a drive around the town of Aspen, determining as each crossroad approached on the video screen whether to turn left or right or to proceed forward. With an appropriate touch, the video would be cued to change the image correspondingly. (25) Our familiarity with the visual cues of the urban landscape and of the principles of physical movement through linking streets, enable us in the machine version to navigate, cognitively, the visual system representing the physical layout of the town.
Criss-crossing the virtual town would enable us to gradually install in memory at first the main features of place and their relation to other features and the grid of the streets. Later as our familiarity increases, then the ‘bird’s eye view’ could be constructed in the mind at the moment it becomes necessary to reckon the most direct route between two points in the town. Such a process of conceptualizing would be similar whether in front of the representational system or within the town itself.
This begins to illustrate the complex way in which physiology, mind, agency and artefacts can interact to inform action, the outcomes of which can cause physical passage through a space as well as further updates from the system of representation.
“Clark (5,6) and Hutchins (18)… and others, have argued that just as basic forms of real-world success turn on the interplay between neural, bodily and environmental factors, so advanced cognition turns – in crucial respects – upon the complex interplay between individual reason, artifact and culture. …The external environment, actively structured by us, becomes a source of  cognition-enhancing ‘wideware’: external items (devices, media, notations) that scaffold and complement (but typically do not replicate) biological modes of computation and processing, creating extended cognitive systems whose computational profiles are quite different from those of the naked brain. Hutchins for example, gives a wonderful  and detailed account of the way multiple biological brains, tools (such as sextants and  alidades), and media (such as maps and charts) combine to make possible the act of ship navigation.” (5)
A final example of memory systems based on loci is the Exeter Cathedral ceiling website. Here the narrative of a learnéd treatise, the index of a catalogue and a graphical map of the ceiling are each linked to pictorial details of the magnificently restored ceiling of the structure. However, the authors are quite upfront: “There are two main routes into the material, Visual and Verbal. ….. The Verbal route is for those who are more at ease with text than images.” There is an elegance and appropriateness in the visual component of the site in associating a contemporary on-line database design with a medieval equivalent – the vaulting and keystones in a 700 year old cathedral. These are pathways and nodes that actual store 15th Century arcane and local knowledge using, like their modern counterpart, visual coding and systematic method. (16)
PATHSCAPE
An interactive multimedia prototype of PathScape was developed in 1999/2000 with a small team of which I was project leader, in association with the Australian Film Commission. The prototype has an interface and navigation system giving access to ‘narratives’ by their association with a specific place or location or series of locations.
The taxonomy is represented with images of contiguous cinematic space – individual photo images are pixilated to produce apparent motion in a forward direction, perceived as a movement ‘into’ the space recorded, a landscape. The movement is achieved by gesture, using a mouse in the prototype. (Figure 1)

Figure 1: Screen Cursor Areas and Gesture Outcomes
The taxonomy of the Path is ordered sequentially by three indexical devices. These are located in the border area that surrounds the central image of movement along the Path. Within this border are seen at various points, fragments of images, visible for short durations. These indicate a nodal junction which, when ‘captured’ by halting all apparent forward movement, enable with a click the launch of a movie to replace the image and sound of the Path. Thus along an X-Y axis are the 1, 2, 3, …. 8, 9 etc options, or loci ‘in’ which are stored the ‘narratives’. (Figure 2)

Figure 2: Schematic for accessing image/sound database
The second device is changes in background colour to the border and background sound, signifying changes of zones. (Differences in ecology along the Path in this prototype). In Figure 2, along the X-Y axis are the AA, BB, CC …. FF etc axes. By gesturing to the left of the screen (or to the right) will launch a 360˚ panning movement, a movie representation of the zone through which the user is currently ‘passing’ – to the right will pan right, to the left will pan left. Within the pan will be ‘found’ further nodes to launch movies storing more narratives.
At the completion of a narrative, the third indexical device appears as a series of circle shapes that appear over the final frame of the movie. (Figure 3) Blue, yellow and brown and green circles function as ‘buttons’ to linked topics colour coded to symbolically represent a broad sort (in this prototype) under the descriptors: Anecdotes, Historical Context, Commentary and Analysis. Each option extends and develops the background of what has gone before, in effect narrowing the index path to the specific, reducing from the broad.
FURTHER DEVELOPMENT
Following demonstrations to several groups of the initial prototype and receiving anecdotal responses and with the limited resources left to the development project at that stage, it was decided to implement a text-based component to PathScape. This would not compromise the initial intention of devising a visually-based indexing system as the choice to use text would be clearly indicated and separated from the visual path.

Figure 3: Screen grab within a narrative branch, with colour-coded circles.
The grey/black circles on the screen that sit behind each of the coloured circles are the route through to the traditional text-based index – the text information sits in the shadow, as it appeared, of its iteration as a movie. The text is organised sequentially as a series of ‘browser pages’ gathered, utilising XML protocols, from the Sources database of content, specifying:
For each narrative: Sound; Picture; Transcript; Keywords; Web Search;
For the whole prototype: More Stories (as a Table of Contents – the narratives – with the frame numbers of the Path movie listed against each item, from which the narratives could be launched); Keyword Index
The user in the prototype therefore has a choice – to navigate the index by using images and sounds, or by using words, or a mixture of both. The probable usefulness of the feature in an educational context was also noted.
FURTHER RESEARCH
PathScape is a project progressing through several stages and adopting several iterative forms. It could be delivered on disc (CD or DVD) or via the internet or broadband cable or conceivably, as it uses XML protocols, via a PDA or mobile phone. The software framework is dynamic, rebuilding the database interface at each launch. With the further research into the development of appropriate interfaces that help the author(s) define the ontology and epistemology of personal and collective memory, the PathScape paradigm will examine models for placing and retrieving audio-visual digital media artefacts.
At a later stage it may be appropriate to consider meta-design as an approach to developing the tool further. Fischer describes “…a fundamental objective of meta-design is to create socio-technical environment that empower users to engage in informed participation rather than being restricted to the use of existing systems.”(14) In such an event, this representational system will be open to invention by its author(s) through the placement of appropriate media into the chosen taxonomic indexing system. Different modes of taxonomic representation could be suggested in such a scenario to provide ways of thinking about the representation of memory.
RESEARCH BACKGROUND
Lansdale, Scrivener and Woodcock have shown that “useful theories of spatial memory can be developed of general utility in the design of pictorial databases” but that “…the specificity of task domain and visual material is more likely to dictate issues of design than is any generic theory of visual cognition.”(21) The prototype of PathScape is a specific model using the familiar figure of a landscape into which we walk and from which we can return as a paradigm with which to address this conclusion. Like many of the aspects of contemporary interface design, the various devices and indexing systems could become options at application launch, easily switched on or off by the user, helping the user to define for themselves, the interface with which they felt most comfortable and productive.
Though setting out to be a storage system for movies and narratives rather than just pictures, the direction indicated by Lansdale, Scrivener and Woodcock’s research into designing a system is in the same area as more recent thoughts by Clark about “…the challenge of tractable search and recall given an extremely large database.” (4) Though an interactive system may ameliorate the apparent size of a digital media database, at some point the ‘visitor’ to such a system will want tools to enable a meaningful encounter with it.
In addressing the problems associated with other ‘unknowable’ database resources like the web, Clark describes Kleinberg’s procedure, “..which exploits information implicit in the links between pages so as to identify patterns of connectivity indicative of ‘authoritative sources.’” Recent work on this approach to “…information-about-information (or second-order information) implicit in the link structures…” may be of value in creating “…a useful, low dimensional reflection of the high dimensional knowledge-space.”(4) A taxonomy based on making visible connections between locations of knowledge or evidence, whether on the unordered space of the internet or the more ordered (but possibly idiosyncratic) space of an artificial topography, provides the visitor to the system with some shapes, some vectors to move within at the outset.
The appeal is to the users knowledge and experience of moving through three-dimensional space, in the urban or rural setting and its remediation as an artificial topography. Encountering a range of spaces in representational form (loci) that engender in the user a sense of a favoured space raises the issue of motivation, particularly for the visitor to the system, or one who is not familiar with it. A ‘low dimensional reflection’ of this kind will at very least be a means by which the scale of the database and its contents can be comprehended. But the registering of presence of the user both in the space of the system and the images and sounds it can retrieve, and within the physical space the system stands, together will provide reassurance and encouragement to interact, to explore and to respond to and move through what is retrieved.
Mantovani and Riva, building on the work of Zahoric and Jenison (1998) through Heidegger and J. Gibson, proposed an ‘ecological approach’ to establishing a relational presence. Like Kleinberg’s ‘second-order information’, this is based on resources not being the ‘properties of either object or subject, but of their relation’(23). Gibson’s image of a tree in the middle of a field on a summer’s day being only an ‘affordance’ to those who seek its cool shade being an illustration of ‘resources, which are only revealed to those who seek them’. Mantovani & Riva go on to amplify this distinction with the argument that presence is a social construction “mediated by both physical and conceptual tools which belong to a given culture” in which there is “the emphasis of ecological approach on the primacy of action on mere perception” and that “action is not undertaken by isolated individuals but by members of a community. …. Ultimately, there are only two elements which guarantee presence: a cultural framework and the possibility of negotiation of both actions and their meaning”. (23)
This tends to support work developed a decade previously by R.S.Lazarus under the heading Cognitive-Relational Emotion Theory which set out to propose
“..that emotions work through a set of interdependent systems including processes for cognitive appraisal, physical interaction between person and environment, coping, and emotional response itself.” (19).
Discourse around the term embodiment has ventilated many of these concerns about presence. Dourish giving central place to Merleau-Ponty captures “…a sense of ‘phenomenological presence’, the way that a variety of interactive phenomena arise from a direct and engaged participation in the world [which] includes both physically realized and socially situated phenomena…”  Meaning and meaningfulness “…is to be found in the way in which it reveals itself to us as being available for our actions. It is only through those actions, and the possibility for actions that the world affords us, that we can come to find the world, in both its physical and social manifestations, meaningful.” (12) (Author’s emphasis)
CONCLUSION
“One of the most basic principles of plot construction is that the remembered ‘I’ traces a continuous spatio-temporal route through all the narratives of memory, a route continuous with the present and future location of the remembering subject. … This principle imposes a kind of unity on all the narratives; …” (3).
The narrative that I conclude here has briefly discussed, if not imposed unity upon, the interdisciplinary nature of the Pathscape project. Thinking about ways in which a system may be further developed has unavoidably caused me to consider the often separated disciplines that are the study of mind and memory, perception and cognition, presence and embodiment, media representation, creativeness and meaning. I am not forgetting the Machine and the interdisciplinarity of connecting with another or others through computational complexity and its magnetic appeal.
Increasingly in the contemporary context of tools like the Macintosh lifestyle suite iLife, we can anticipate if not fewer words, then a lot more images to be digitally authored and then consigned to data media, before being finally consigned to the bottoms of drawers for a want of a means of retrieving their autobiographical or historical significance. PathScape and similar projects set out to extend the potential of these cultural resources and the authors who will provide a signifying unity for the benefit of others to make meaningful enjoyment. Enjoyment should be the key because, after all is done, and as Andy Clark has recently commented, “Memory is but constrained confabulation”.
NOTES
1. Communication of digital video signals has many aspects: content; creation; formatting; encoding for data compression and channel error control; modulation; satellite, terrestrial, cable, and networked transmission; and reception – demodulation, decoding and digital signal processing. Accompanying every signal operation is a piece of hardware to perform the task. Cameras, displays, switching arrays, servers, mass storage devices, and computers are examples of the kinds of hardware required for the generation and distribution of digital video and which will be affected by technological advances in the state of the art. (17)
2. Broadband services enable, if not video quality, access to audio-visual digital media. Broadband subscribers have increased six fold from a base of 6.6 mil in 2000 to 35.8 mil in 2004. Source: US Bancorp Piper Jaffery. ‘Streaming Media Guide’ Viewcast (32)
3. Rodden and Wood’s research came up with several interesting proposals for further research. In the conclusion they went on to cast doubt on the usefulness of text-based indexing and retrieval providing the subject group with “enough extra motivation to invest the effort in annotating their photographs.” (27)
4. The author witnessed two professional script-writers working method, which involved them laying out palm cards and images around a studio, whilst working with a computer in the centre of the room to synthesis their content. Russell Crowe portrayal of the schitzophrenic John Nash in the movie ‘A Beautiful Mind’ provides an image of this process in its pathological state. (19)
5. “The on-line video server is composed of our speech-based search and retrieval system, a multimedia streaming server (Real Networks, IBM’s VideoCharger and/or Apple’s Quicktime), and query processing and a process that compose and deliver the retrieved results back to the user. The search and browse system includes an Internet-based Graphical User Interface (GUI) that can be run by any browser, on different platforms, using standard plug-ins. The GUI includes a text query box and associated advanced searching options, and allows easy navigation between the different views which blend together into an advanced video browser. … The results further show that there is no difference between speed assessment of video, MSB and audio only. This means that in many cases of remote education we can replace the video with a moving storyboard, which is much smaller in size and can be streamed across low bandwidth networks. … The results also vary between people. Among the 24 subjects we have some prefer to watch the full video, some prefer to watch the MSB, and others prefer audio only. The main lesson from this diversity in preferences is not to “optimize” the system for an “average” user, but to leave him/her to decide which media and what speed to use for a given task.” (1)
6. Pixilation is defined as “A technique used in theatrical and cinematographic productions, whereby human characters move or appear to move as if artificially animated. (26) This should not be confused with the pixel, a compression of the term, ‘picture element’ being “The smallest resolvable rectangular area of an image” (18)
REFERENCES
1. Amir, A. Ponceleon, D. Blanchard, B. Petkovic, D. Srinivasan, S. Cohen G.  Using Audio Time Scale Modification for Video Browsing, Proceedings of the 33rd Hawaii International Conference on System Sciences  2000.
2. AHDS Arts and Humanities Data Service Visual Arts. http://vads.ahds.ac.uk/about/index.html. Accessed 1.9.04.
3. Campbell, J. The Structure of Time in Autobiographical Memory, European Journal of Philosophy 5:2, Blackwall Publishers, Oxford 1997.
4. Clark, A Global Abductive Inference and Authoritative Sources, or How Search Engines can Save Cognitive Science. Cognitive Science Quarterly 115-140 2:2:2002
5. Clark, A.  Being There: Putting Brain, Body and World Together Again, MIT  Press, 1997.
6. Clark, A. Where Brain, Body and World Collide Daedalus 127, 257-80, 1998.
7. Data Harmony Inc http://www.dataharmony.com/ faq.htm#b1 Accessed 1.9.04.
8. Davenport, G.Indexes Are Out Visions & Views, MIT Media Lab Fall 1996.
9. Davenport, G. et al  ‘Jerome B. Wiesner, 1915-1994: A Random Walk through the 20th Century’ (1994) http://ic.media.mit.edu/projects/JBW/ Accessed 1.9.04.
10. Deleuze, G & F Guattari ‘A Thousand Plateaus: Capitalism and Schizophrenia'(1994), trans B Massumi, Minneapolis: University of Minnesota Press.
11. Derrida, J. Difference. Speech and Phenomena, NW University Press 1973.
12. Dourish, P  Where the Action Is – the foundations of embodied interaction, MIT Press. 2001
13. Fairfield, City of ‘Peopling Fairfield’ website http://acl.arts.usyd.edu.au/projects/consulting/fairfield/index.html Accessed 1.9.2004.
14. Fischer, G Meta-Design: Beyond User-Centered and Participatory Design, Proceedings  of HCI International 2003, Julie Jacko and Constantine Stephanidis  (eds.), Crete, Greece, June 2003, pp. 88-92.,
15. Hales, Christopher. http://www.smartlabcentre.com/ 4people/coreres/chales.htm Accessed 1.9.04.
16. Henry, A. and Hulbert, A. Exeter Cathedral Keystones and Carvings. http://hds.essex.ac.uk/exetercath/ Accessed 1.9.04.
17. Hoerl, C. and McCormack, T. (eds) Time and Memory: philosophical and psychological perspectives. OUP 2001.
18. Hutchins, E.  Cognition in the Wild, MIT Press,1995
19. Huang, M  Presence as an Emotional Experience, Medicine Meets Virtual Reality ed Westwood et al, IOS Press, Amsterdam. 1999
20. IMDB, International Movie Database. http://www.imdb.com/title/tt0268978/ Accessed 1.9.04.
21. Lansdale, M Edmonds, E  Using Memory for events in the design of personal filing systems, International Journal Man-Machine Studies, 36 97-126. 1992
22. Lansdale, M Scrivener, S Woodcock, A Developing Practice with Theory in HCI: applying models of spatial cognition for the design of pictorial databases, International Journal of Human-Computer Studies, 44, 777-799, 1996.
23. Maguire, Dr Eleanor, et al  ‘Routes to Remembering: the brains behind superior memory’ in Nature Neuroscience V6 N1. 2002
24. Mantovani, G Riva, G  “Real” presence: how different ontologies generate different criteria for presence, telepresence and virtual presence’, Presence: Journal Teleoperators and Virtual Environments, 8 (5) 538-548. 1999
25. Naimark, M. Place Runs Deep: Virtuality, Place and Indigenousness. Virtual Museums Symposium, ARCH Foundation, Salzburg, Austria 1998.
26. NSF Industry/University Co-Operative, Research Center for Digital Video at Rensselaer Polytechnic Institute Troy, New York.  http://www.ecse.rpi.edu/CNGV/proposal.html#intro. Accessed 1.9.04.
27. OED, Oxford English Dictionary http://dictionary.oed.com/ Accessed 1.9.04.
28. Rodden, K Wood, K How Do People Manage Their Digital Photographs? Proceedings of CHI 2003, Ft Lauderdale, Texas, 2003.
29. Schneiderman, B. Designing for User Interface, Addison-Wesley, 3rd edition 1998 pp 510-511.
30. Sutton, J. Memory, The Stanford Encyclopedia of Philosophy (Summer 2004 Edition), Edward N. Zalta  (ed.),http://plato.stanford.edu/archives/sum2004/ entries/ memory/.2004
31. Taniar, D Rahayu, W. A Taxonomy of Indexing Schemes for Parallel Database Systems, Distributed and Parallel Databases, 12, 73–106, Kluwer Academic Publishers. 2002.
32. Tulving, E. Elements of Episodic memory, OUP 1983
33. US Bancorp Piper Jaffery. ‘Streaming Media Guide’ Viewcast http://www.viewcast.com/whitepaperdownload.asp# Accessed 1.9.04.
34. Yates, F A. ‘The Art of Memory’, Pimlico, London 1966 (1992 ed)
 

ABSTRACT
A series of Models were built to explore and test the precept of navigating movies using gesture to control both forward and backward movement, and to launch movie files linked using visual elements associatively and semantically related to the knowledge domain represented within a movie collection.

Author Keywords
Video, indexing, hypermedia, mnemonics, meta-design
ACM Classification Keywords
H.5.4 Information interfaces and presentation: hypermedia.

INTRODUCTION
The browse searching of digital video files using proprietary software and commercial applications relies on alphanumeric indexing and keyword selection. This is appropriate for ontologies with established taxonomies and structures for maintaining Knowledge Management Systems (KMS) including information contained in movie files. But as movie files become ubiquitous in our everyday lives as a means of conveying information for a range of purposes, the design of computer systems for storage and retrieval employing other forms of visual mnemonics could add efficiencies of speed with ease of accurate access.
The research approach taken utilises the mnemonics contained in the motion-picture images of a movie collection and offers possibilities for non text-based interaction with a KMS. A series of Models were built to explore and test the behaviours of subjects navigating movie files, encountered as full screen motion-picture images, using either arrow keys on the keyboard, or mouse, to effect 4-way control: the playback of the movie – up for forward, down for backward; and launch movie files linked to mnemonics in the movie being viewed – left and right to link.
Linking to left and right is according to a schema, (from the Greek skhema, meaning shape), designed for each Model, that aids in the retrieval of movies in the collection.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Using visual elements associatively and semantically related to the knowledge domain represented within the movie collection, evaluation will compare Models operated by novice and expert groups.

METHODS
The software tool, Mnemovie, was developed by the researcher, in collaboration with a professional multimedia developer. The approach to this initial task has been described by Fischer, Giaccardi et al, using the seeding, evolutionary growth, and reseeding (SER) process model [1] p 492. The practice of building the software tool by Leggett and Hinshaw was guided by the fourth model described by Fischer et al for collaboration paths in software development, where both the domain practitioner and the software professional has some knowledge of the others practice (p 487).
Building on related hypermedia research by Girgensohn et al [2] the Mnemovie tool facilitated the rapid building of Models to create linkages between movie files as a means of realising proof of concept. Following analysis and reflection on the semantic domain of each of six movie collections, navigational schemas were designed before links were created between movie items. The tool enabled the researcher to define parameters within the code to link movie files and iteratively develop and refine the schema for each Model.
Retrieval Schemas
Four of the six schemas were advanced further in preparation for evaluation using a specific movie collection of people talking about their research activity.
1. The segmented Loop schema, where each segment is a compression of each movie item in the collection. Linking thereby has a direct indexical relationship between a Loop segment and the item, and vice versa to return to the Loop. (Figure 1)
2. The Pathway line schema, from A to B, linking to the researcher collection has an indirect connection, where a particular location along the path is the mnemonic for a particular movie item.
3. A development of the Pathway line to the horizontal Grid is applied to a collection of movies shot and linked by the system through the intersecting streets of an inner city block. Linking to the collection of movies about the work of the researchers has an indirect connection with the schema, a particular location in the grid of streets being the mnemonic for a particular movie item.
4. The Clock face schema, dividing by convention the passing of time and indexically, direct links to the proportional durations of a movie; or indirectly to different movies in the researcher collection.

Fig 1. Sample movie collection using Loop schema.
Mnemovie Tool
Arising from an earlier prototype that used Macromedia Director to construct the system framework, conceptual and technical elements used for the PathScape project were extended into the requirements for the current investigation. Specifically this moved away from hard-coded ‘content’ to a modular and externalised framework, (.mov, .swf, .dcr files), subject to an “external importation routine” incorporating an XML file, “..more extensible to handle growth of later versions.” This approach has more recently adopted by other developers whereby a:
“…presentation engine allows content authors to describe … content through associated XML files. Interpretation of those files, content layout, and all … communication is automatically handled by the presentation engine. The content is described external to the application, creating a natural separation from the … interface.” [3]
In Table 1, the conceptual data model for building the Mnemovie tools includes the software framework, the presentation engine and the media directory.
Root Directory of Model
Presentation Engine
Media
Mnemovie b3
MNEMOVIE (application) •Mnemovie.dcr •movie_data.xml
MOVIES (video files directory)
Table 1: Mnemovie data model
Media files are prepared using a digital video editing application and saved with consistent resolution and frame size into the directory. The .dcr file compresses
specification data for the Director application and is prepared by the software professional. The file movie_data.xml contains a description of the tags and the layout of the program source code specific to the manipulation of the movie files contained in the adjacent Movies directory. The modular construction of the source code enables the researcher to expand the scale of the instruction set according to the requirements of the interactive Model.
The XML-file structure throughout was based on each <track> having a <movie id> for the forward motion movie and a different <move id> for a reverse motion movie. From each <track> links to other movies could be created. The Beta 1.0 used the following structure (sample):
<track id=”PD”
 <movie id=”PZF” file=”movies/PulledZfore.mov” dir=”F” ><!–F–>
<link side=”L” start_time=”00:00:13″ end_time=”00:00:14″
movie_id=”11″ link_start_time=”00:00:00″ />
    </movie>
<movie id=”PZB” file=”movies/PulledZback.mov” dir=”B” ><!– B –>
    <link side=”R” start_time=”00:00:12″ end_time=”00:00:13″
movie_id=”R11″ link_start_time=”00:00:00″ />
    </movie>
</track>
As experimentation progressed iteratively, the <link_> group of tags operating within the system were increased and thus offered additional linking possibilities for schema design.
Evaluation
Each Model is in the process of predictive and operator evaluation with a movie collection to assess interaction efficiencies between Models, the quality of the experience and the interacting subject’s ability to:
navigate a movie collection using the schema approach;
retrieve information contained within movie files;
retain memory traces from the navigational process, such that subsequent interactions with the Models can demonstrate accumulated learning behaviour.
REFERENCES
1.    Fischer, G., Giaccardi, E., Eden, H., Sugimoto, M. and Yunwen, Y. Beyond binary choices: Integrating individual and social creativity. International Journal of Human-Computer Studies, 63. 482-512. 2005
2.    Girgensohn, A., et al. Designing Affordances for the Navigation of Detail-on-Demand Hypervideo. in ACM Advanced Visual Interfaces. 2004.
3.    Mentor, K. Director and SCORM 1.3 SCORM SCO Presentation Engine (S2PE) Director Developer Center, Adobe Inc, 2006.