search

FAQ & User Guide
Version 1
2013-08-27
PDF Version

Contact information:
AudioVisual Preservation Solutions
www.avpreserve.com
[email protected]

Github repository:
https://github.com/avpreserve/interstitial

Download pages for application:
Windows:
http://www.avpreserve.com/wp-content/uploads/2013/07/interstitial-win.zip

Mac:
http://www.avpreserve.com/wp-content/uploads/2013/07/interstitial-osx.zip

FAQ:
1) What are interstitial errors?
2) What do interstitial errors look like?
3) What do interstitial errors sound Like?
4) What is Interstitial and why is it needed?
5) What recording setup does Interstitial require?
6) How do I use Interstitial?
7) What is the copyright and license for Interstitial?
8) Further reading

1) What are interstitial errors?

As described on the Federal Agencies Digitization Guidelines Initiative (FADGI) website : “Interstitial errors consist of lost or altered samples within the recorded file, resulting in the loss of content and integrity. These errors — often very momentary –result from a failure in the chain of digital data, i.e., in the handoff from the analog-to-digital converter (ADC) to the digital audio workstation (DAW), and in the DAW’s writing of the file to a storage medium.”

These errors occur within the chain between the ADC and the writing of data to disk. Due to the complexity of the hardware/software stack between the ADC and storage media these errors are fleeting and intermittent, and the precise causes are extremely variable although they typically result from resource allocation and buffer issues.

2) What do interstitial errors look like?

Interstitial errors can take many forms, manifesting as missing samples, corrupted samples, or misplaced samples. Below are several images of interstitial errors.

Figures 1, 2 and 3 below provide a side-by-side comparison of a DAW file containing an interstitial error and the Reference file that does not have an error. They are presented as waveforms in a timeline. For clarity in these illustrations, the polarity of the DDR file has not been reversed, as it is when the null test is performed. You will notice that the waveforms are the same from the beginning up until a certain point after which they differ. That point is where the interstitial error occurs; as will be shown in the next section, this error point indicates that a series of samples are missing from the DAW file.
Figure 4 illustrates the alignment at the beginning of a file in contrast to the end where a shift forward can be seen in the DAW file caused by the loss of samples.

Screen Shot 2013-09-03 at 12.24.48 PM
Figure 1: DAW file with interstitial error (Top Waveform) compared to the intact Reference file (Bottom Waveform)

Figure2
Figure 2: DAW file with interstitial error (Top Waveform) compared to the intact Reference file (Bottom Waveform)

Figure3
Figure 3: DAW file with interstitial error (Top Waveform) compared to the intact Reference file (Bottom Waveform)

Figure4
Figure 4: Demonstrating the beginning and end of the DAW and Reference files. Note that they are synchronized in the beginning and that by the end they are out of sync due to missing samples in the DAW file.

For Figures 5 and 6 the two waveforms at the top of each figure are similar to what is shown in figures 1, 2, and 3, except in these figures the polarity of the second (Reference) waveform has been reversed. The third waveform in these sets shows the DAW file but aligns all of its content with the Reference file. Thus, the gap represents the missing samples. The last waveform is the Delta file, comparing the Reference file with the DAW file; the “null” results are the flat line while the samples missing from the DAW file are represented by a waveform contour.

Figure5
Figure 5: Demonstrating the original DAW file, the Reference file and the Delta file along with a DAW file showing a gap where the missing samples are.

Figure6
Figure 6: Demonstrating the original DAW file, the Reference file and the Delta file along with a DAW file showing a gap where the missing samples are.

Figure7
Figure 7: Demonstrating an interstitial error in the form of silence, or digital black where there should be waveform information.

Figure8
Figure 8: Demonstrating an interstitial error taking place in 2 channels at the same time. The left channel has highlights showing the effected area. The original signal is a 50Hz tone. The error resulted in reversing the samples in the highlighted area. The samples covered by the orange highlight should be in the area that is highlighted in green and vice versa.

3) What do interstitial errors sound like?

As shown above, Interstitial errors manifest in several ways. The audible impact can range from sounding like a small click, to momentary dropouts, to missing whole words. In a study performed by FADGI the interstitial errors that occurred offer a couple of examples of interstitial errors resulting in missing samples. These can be heard here:

Sample Set 1:
Good File (http://www.avpreserve.com/interstitialerrorsamples/201206081082_DDR.wav)

Bad File
(http://www.avpreserve.com/interstitialerrorsamples/201206081082_DAW.wav)

Sample Set 2:
Good File
(http://www.avpreserve.com/interstitialerrorsamples/201206121086_DDR.wav)

Bad File
(http://www.avpreserve.com/interstitialerrorsamples/201206121086_DAW.wav)

4) What is Interstitial and why is it needed?

A variety of vendors offer real-time (i.e. during digitization) and non-real-time (i.e. post-digitization) analysis systems. These systems monitor a variety of important issues and they successfully identify many problems, including certain types of digital errors. These systems, however, generally do not detect interstitial errors. Interstitial errors happen during the writing of the bitstream to disk, while real-time systems monitor the bitstream prior to being written to disk. Additionally, both real-time and non-real-time analysis systems use algorithms to detect errors that are not accurate for detecting interstitial errors. They under-report and over-report making their output of little use.

Interstitial is based on a model of comparative analysis, utilizing a setup that generates two files from the output of a single analog-to-digital converter (described in further detail below). Interstitial is a Python based tool that automatically detects matching sets of files regardless of naming conventions. It then aligns the beginnings of the files and performs a sample-by-sample comparison. In sets of files with no errors there will be no differences. Differences between the two files represent interstitial errors and are reported by the application.

5) What recording setup does Interstitial require?
There are two important things to understand about the use and setup surrounding Interstitial. First, because Interstitial utilizes comparative analysis it requires the generation of two files, and these two files should be created on two separate systems. This is achieved through implementing a parallel recording chain, taking the AES output of the ADC and sending it to your DAW and a separate recording device. Two illustrations demonstrating possible basic signal paths are below.

Figure9
Figure 9: One possible setup for distribution of the ADC AES output, showing use of an AES distribution amplifier

Figure10
Figure 10: One possible setup for distribution of the ADC AES output, showing use of dual AES outputs from the ADC

Note that the AES signal must be distributed properly. Simply splitting the signal using a Y-Cable is not sufficient and will cause errors. The distribution of the AES signal must either be achieved through use of outputs on the ADC or through a distribution amplifier as illustrated above. Another possible setup is to use a device in pass through mode. Testing has found that not all devices will work properly in this way, despite nominally supporting it – one device, for example, was found to introduce one-sample pauses upon exit from the device.

The Reference Device can be any device with an AES input that captures at the same sample rate and bit-depth as being used by the DAW. The device may write to any type of storage medium, internal or external. Before using a new device and signal path in production, we highly recommend thorough testing to ensure that no errors are introduced by the device or signal path.

The second important thing to understand about Interstitial is that the expectation is that testing will be performed on all files generated through digitization. The workflow that has been used most commonly is to record to both the DAW and Reference Device throughout the day. At the end of the day, place the Reference Device files on a portable drive (unless the device writes directly to a portable drive already) and connect that drive to the DAW. Open Interstitial and point it to the appropriate DAW and Reference Device directories/files. Begin the process and let it run automatically, leaving for the day. Interstitial will automatically match files from the two devices and perform analysis. In the morning, arrive to find and review the report output by Interstitial showing the results. If there are no errors you may delete the files on the Reference Device drive. If there are errors reported these are inspected to confirm their existence on the DAW file and the DAW file may be replaced by the file on the Reference Device drive.

6) How do I use Interstitial?
Interstitial can be used in a number of ways. All that is required is the Interstitial application and access to the two sets of files to be analyzed. Analysis can be performed in any configuration that meets these criteria. The following workflow description is one possibility:

After recording audio to the reference device and DAW, attach the reference storage to the DAW (or another workstation with access to the DAW files). Run Interstitial, which brings up a window asking for three directory paths. Provide the paths to the directory of DAW files to test, the reference device directory, and a location to write the final report to. Interstitial will then compare the files in these directories, match and align them, and scan them for any sample-level differences between the reference and DAW recordings. Following this, it will write a manifest of files compared and report the sample location of the first error detected if applicable.

Note that if Interstitial fails to match two files that should be matched this likely indicates that there is an issue in the files that demands analysis (such as clocking errors and pauses being introduced).

7) What is the copyright and license for Interstitial?
Copyright 2013 AudioVisual Preservation Solutions, Inc.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

8) Further reading
http://www.avpreserve.com/wp-content/uploads/2010/01/Digital_Audio_Interstitial_Errors.pdf

http://www.digitizationguidelines.gov/audio-visual/documents/Interstitial_Error_Report_2013-04-08.pdf