Digital Media Collections Are an IT Problem But Not an IT Solution

By October 31, 2011Blog

The power and the flexibility of content use and distribution in the digital realm is enabled by the ability to break everything down into the same essential components, into the 1’s and 0’s that form the atomic structure of data. In its idealized form, that content and the persistent structural wholeness of digital files do not matter in the same way they do with analog materials. One would not tear pages out of a book to ship it separately in smaller envelopes, nor would one store half-second fragments of a film on separate shelves in a room. The works could be reformed, but no easily nor cleanly. Data, however, those 1’s and 0’s, is sent, received, and shuttled around in packets, the fragmentation and compressibility of the whole, unlike with analog works, supporting efficiency, portability and far reaching usability for research and creativity.

The shift to digital workflows has necessitated a major shift in how we conceptualize the use and storage of assets. Creators, owners, records managers, and archivists are no longer the sole stakeholders in how documents and materials are taken care of long term. There is now a greater need to understand data management, technological infrastructure, and the particulars of software, hardware, files types, codecs and more. Likewise, the ease of creating and versioning digital works has led to an explosion in the number of files (as well as the number of network, local, and detachable drives to squirrel them away on), resulting in an overwhelming bevy of content to track and maintain. In a corporate or institutional environment, a creator or overseer of digital assets must either educate oneself on these topics or rely to a greater degree on IT departments to help manage their materials.

Integration and collaboration between departments is an essential component of organizational success today -– sharing resources, eliminating redundancy, and open communication help prevent the waste and lack of innovation that can doom an organization to irrelevancy and worse. However, the people who should be in control of setting policies for file management and for selection and implementation of asset management tools — the archivists and records managers out there — have ceded too much ground to a pure IT mindset.

As I see it, providing solutions to problems means applying one’s areas of expertise to derive something that attempts to approach a balanced mix of functionality, efficiency, usability, and elegance. In the world of archives and media collections, this means, among other things, making decisions about metadata, file types, storage systems, and distribution systems that support findability, longevity, and flexibility for current and future use. Under an IT mindset, solutions hinge, among other things, more on processing speed, maximizing storage capacity, decreasing time to market or implementation, and monitoring data flows. Of course these things matter to people using or providing access to digital assets, but the paths to the end solution — compression or low resolution, out-of-the-box asset management, decentralized or uncontrolled metadata creation, etc. — are fraught with hazards for media. By not taking a more active role in the policy and decision making process, caretakers for media collections put the safety and usability of their assets at risk as well as their own ability to perform their responsibilities to the collection and to the organization.

At their core files are just data, but the ways we manage, use, interact, and create with them rely on intellectual, humanistic, or organizational structures that step away from data and back into nuance, language, and user experience. When we bandy about terms such as digital archive and digital asset management, we are actually using broad categorizations to simplify references to a host of complex and distinct solutions for working with file-based collections, solutions that vary greatly depending on the avenues of access and the functional needs of the organization.

This is especially true with audiovisual content, which presents much different needs and distribution methods than straight text files, including considerations for time-based presentation, aesthetic quality, and the management of very large files. For example, distributing assets publicly over the Internet may utilize lower-quality, “access copy” versions of content in a system designed to promote simple search and playback through streaming. Distributing assets internally to a marketing or development department may instead utilize high-resolution copies of content that can be downloaded and edited into new assets, retrieved through a system that promotes advanced search and integration with editing software. But both of these solutions only support findability and usability for media collections; they do not represent the needs of preservation for the highest resolution originals or preservation masters. These versions are infrequently accessed and, for audiovisual content, may range in the hundreds or thousands of gigabytes per file, thus solutions may include offline storage and ought to include redundancy and geographical separation of backups.

This is one area where the interpretation of what an archive is and what an archive does come into conflict. In environments such as email programs, “archiving” has traditionally been used to mean moving data off into deep storage so it is not eating up active space needed for incoming information. This is considered to be data maintained primarily under retention policies and is not meant to be quickly searched for and called up. Deep storage has its place as a strategy, but it should not be confused with the true sense or value of an archive or collection. An archive is a living resource within an organization, maintaining legacy assets but also bringing in new creations, and providing accessibility to both…If the proper resources and support are allocated to the archive itself. Archives are long-term investments, paying off over time by extending the usability of short-term investments, i.e., acquisition and creation of assets. Shortchanging the archive’s ability to do its work now devalues past and current efforts by denying them a future.

Archivists have centuries of tradition, learning, and research which have informed the development of current practices, with an increasing focus on managing digital collections. IT professionals have their own areas of expertise, but these do not expand to all aspects of dealing with file-based materials. Tracking complex relationships among related or derivative assets… Providing accessibility at the intellectual rather than just the physical level… Selecting file formats and codecs based on potential longevity and fidelity to analog source originals… Developing metadata models that adhere to professional standards and that support the activities of collection management… These and more are areas of digital archiving that rely on data practices but that include considerations well beyond those of ground level data management. Today’s archival professional needs to collaborate with IT — as well as many other departments — but we also need to step up and take back control of those aspects of our collections that rightfully belong in our care.

Joshua Ranger

Join the discussion 6 Comments

  • I agree with your concerns about an IT ethic that disassociates media processing from media context, and the resulting hazards at play as the media circulates over time unprepared for its interactions with research, academic, and production communities. But I also think it is naive, especially in the States, to think that archivists have the muscle or institutional credibility to go head to head with the massive cost centers that are becoming stronger, not weaker, offices of organizational decision-making. (And why does this rightfully belong to archivists? Did we fund or create the content in the archives? I don’t understand this “right,” although I agree that archivists’ instincts, mission, and training are a better friend of a media’s life cycle in digital environments). Archivists were able to control their systems in the past because nobody paid attention to them. With few exceptions, the advocates for massive investments in digital systems and work-flows are not archivists but business development groups, marketers with new business models, and corporate CTOs who are advancing various technological solutions to address inefficiencies (of which archive departments are often flouted as the poster child). I have found myself deep in the trenches at odds with IT departments at global media companies. I was able to win the case from time to time: not because of the archivists’ superior professionalism, but because I offered an alternative value that could only be realized with strong contextual resources: short form programming for distribution on mobile devices globally; news packages built out of archival materials to connect history and memory with current events; television series created out of archival materials that cost less and rated higher than the blue chip pipeline (and occasionally won awards). If they handed me the keys, it’s because they believed in the viability of the project and our ability to manage it effectively. In the end, the most valuable attribute of a media archivist is his/her understanding of the content, not the technology: someone who has a visceral and creative understanding of the content’s meaning, can engage the core mission of a media company (which is to make media!) and who can argue that to succeed an IT department should support your initiatives, not challenge them. So it can be done: not on the battlefield of technological functionality, not on a preservation argument (which is at best cocktail conversation for media execs), but on the creative values represented in the archives. The use case. If you get that right, and you deliver as promised, you’ll have plenty of room to build out the preservation and contextual resources that meet the unforgiving demands of the future. It’s hard work, but it’s the only model I’ve seen that gets archivists to the table.

    • Josh says:

      Matt, I feel like your comment, which is full of spot on points and along the line of some ideas I was struggling to communicate, deserves to be a post in itself and not just an appendage to my ramblings. Based on the budgets, human resources, internal politics, broader organizational concerns, and other machinations involved, I in no way think that archives can go head to head, but I do think that as a profession we have the responsibility to step up our game and have a place at the table. Not to make demands on uncompromising positions, but to be a part of the discussion and understand the content, technology, and potential uses enough in order to state the pros and cons of various decisions under consideration. And I think this leads to a strain of my argument that I edited out of my post — as with most professions, archiving has fundamentally changed and has bled over to (or been bled over into) a number of different roles and scenarios. I was trying to avoid using the word archivists because, really, this work encompasses all kinds of records managers, collection managers, media producers, creators, etc. at the corporate and individual level, and the concept of academic or historical archives is much different than a media company’s “archive”. But how does one write a comprehensible sentence when having to include a cornucopia of titles and exceptions. My problems aside, the connotations of and weighted associations with archive/archivist/archiving do not match up to the work that needs to be done with media today. We need to diversify and extend our knowledge and skills to live up to the responsibilities we are hired to execute.

  • Maura White says:

    Well said.
    Thanks for putting my feelings into eloquent words.

  • Josh, as long as the archival community (including all the stragglers you identified above) approaches the problem as professionals, the seat will remain empty. I understand that professional archivists have the right skills and mindset to do the job, will always work in the best interest of the media, and have answers to questions that IT groups are unprepared to ask, let alone solve. But the profession was marginalized long ago, as evident in the going salaries for archive positions these days, and it will take more time than we have to elevate its status. Meanwhile, most of the important decisions are being made right now as corporations and media companies and Universities and digital media superpowers invest in a future that depends, in dramatic ways, on the manipulation and distribution of moving images. The IT groups and engineers have been enormously effective at staking their turf, aligning themselves with the agenda of senior management, and being party to land grabs that have created the digital superpowers (Google is putting $100 million into original YouTube content, CISCO is building their new five year plan around an all-video world, the Department of Education is launching a massive infrastructure to support their Learning Registry (tag-line: social networking for metadata). With a few very notable exceptions, these entities are resistant and at times hostile to the archivist/librarian mission: it slows them down; archivists don’t appreciate automated systems and semantic web solutions to moving image discovery; the profession gives too much privilege to the content owners, who may rig the contextual data in their favor, etc. So how does the archival community get a seat at the table? As long as the archival community is defined as a support system (a cost center), it creates a competitive dynamic with IT groups, which are also support systems, but highly valued support systems that have already established credibility as effective managers of large-scale projects dependent on technology solutions. The archival community has to make their case another way: by introducing new projects to senior management that have dependencies on the archives and good archival management. There are the program-development methods I’ve used and described above, there are also escalating opportunities in the educational community to tap into moving image archives, and there are strategic cases to be made concerning the use of archival materials in the future (and some very nice European models that can be introduced to senior management). Whatever, the objective is to transition the archival community from a professional with standards to visionaries who are uniquely positioned to contribute to the identity of the institution, be it commercial, corporate, governmental, academic, or mission-based. In today’s chaotic environment, everyone is hungry for new concepts and leadership. If the archival community steps to that plate, they will drive the decision-making, and IT groups will be asked to support their vision. Lots of work, but I don’t see other credible ways of getting there.

    • Josh says:

      Matt, I think we’re in agreement on a lot of core concepts you bring up, especially along the lines of making new business cases and adopting new methodologies. We have a number of colleagues working along those lines and are advocating or developing in those areas with many of our clients. I would say too, however, that we’ve seen a number of projects turned down or not receive funding explicitly due to a reliance on novel approaches that utilize new technologies, automated processes, and a degree of self-reliance on the archive’s part. I came to the realization a few months back that most people don’t really like archives, because they are seen as slow, or stodgy, or too strict, or won’t put everything online right now, etc. It’s a major image problem we’re facing. At the same time I would also posit that the repeated failure of DAMS or digital archives projects have often been caused by decisions being made outside the archive that do not take a broad enough collection of functional requirements for a usable system into account or consider the future of those assets. I feel like it’s an iPhone/AT&T situation where perception of the players drives the interpretation of the situation. I don’t think standards are a bad thing here — the problem is that in lack of strong enough comprehension of the technological issues, standards are leaned too heavily upon rather than used as guidelines to inform decisions and manage risk, based on having enough knowledge to decide how those standards are interpreted or how closely they should be adhered to in specific situations.

  • What I’m Reading 11-3-11 | says:

    […] Digital Collections are an IT problem, not an IT solution. […]

Leave a Reply