The Future of Data Integrity


September 10, 2019

The New Age of Archiving

In July 2019 the Marriott International Inc was fined over 120,000,000 Dollars for violating data protection laws in the EU. During the same month British Airways was fined 225,000,000 Dollars for having insufficient cybersecurity measure on their website which got exploited compromising the data of millions of EU citizens.  

In less than a month almost 450 million Dollars in fines were paid to the European Union due to poor cybersecurity measures regarding the repository of critical customer information. 

Through this immense contextual change, the pressure on companies to have their archives comply with regulations while not losing the critical business value of data points and their value for analytics is increasing drastically 

To better understand this intense change a short overview of the evolution of archiving is necessary. 

In the 19th and 20th centuries, businesses and government agencies maintained large staffs of archivists –called file clerks and librarians. Archives, also called records or repository for an organized body of records produced or received by a public, semi public, institutional, or business entity in the transaction of its affairs and preserved by it or its successors. 

With introduction of computers typewrites were replaced and that organization withered. They were presumably replaced but with what? Documents created and stored on someone’s hard drive that disappeared when that hard drive eventually died? 

The first generation of electronic archiving grew out of a turn-of-the-century legal panic driven in part by high profile business torts in which large companies had to pay huge fines for failure to produce required documents promptly. A very similar reason why it is happening today as well. To solve this issue businesses have implemented document search capabilities to accelerate the finding process and enable their legal departments to provide the necessary documents. The process developed was a tape-based backup system that supposedly preserved everything. However, finding and getting at them was hugely difficult and time-consuming: No one knew what was on any given tape, often vital tapes were lost and those that were found might or might not be readable. It was easier and less expensive trying to recreate the document manually by repeating the research than trying to recover it from backup. This became an acute issue in the late 1990s, where the value of electronic records was discovered by the legal system. Electronic documents were requested by courts through subpoenas while governments passed new regulations simultaneously. Businesses suddenly realized that they needed to know what the thousands of formal and informal electronic documents they had contained, where they are, and how they can be re-produced when needed. 

In the 20th century, archivists were faced with handling new kinds of records, such as photographic records, motion pictures, sound recordings, and computer-kept records. With the dawn of new industries such as the Internet of Things and companies trending towards digitization the amount of records that need to be archived has increase. The increasing level of digitization means an increase in device being used an increase of data created. To be more specific DBS Asian Insights is predicting that the IoT installed base will grow from 6.3M units in 2016 to 1.25B in 20303 and around 29 billion connected devices are forecast by 2022, of which around 18 billion will be related to IoT4. 

Read the whole white paper about “Compliant Information Archiving – Digitalization and Regulation” here.