7 Reasons to Manage Application Data via Filesystem vs. DBMS

Introduction

Database management systems (DBMS) have become an increasingly popular data storage solution with the proliferation of mobile and web development, where filesystems are either not available at all or have restricted access. This trend has caused a certain bias towards using DBMS (and specifically SQLite, which often comes a built-in on mobile platforms) as a universal solution to data storage needs.

However, filesystems are not a thing of the past. The advancement of mobile software brings back the necessity of design practices similar to those seen on desktop systems. In particular, the need arises to use files and file-based approaches to manage heterogeneous information. Given that files are more suitable for the classes of data management tasks discussed below, file-based data storage solutions will continue to be in demand, and CBFS Storage is an all-in-one embedded filesystem solution that you can use anywhere.

Contents

• Encryption
• Compression
• Versioning
• Large and Heterogeneous Files
• Remote and In-Memory Storage
• Easy Backup
• Exposure to Other Applications

Encryption

Strong data encryption is a goal that is, to some extent, contradictory with the nature of DBMS. The database needs to know the size and in many cases the content of the data fields in order to operate efficiently, and encryption can limit flexibility in terms of data management; and while some modern DBMS solutions (mostly client-server ones) provide means to encrypt table fields, such capabilities are often absent on mobile devices. Additionally, field encryption doesn't hide metadata, disguise the structure of the database, or hide the number of entries. This can lead to each developer needing to invent their own encryption schemes, and the proper implementation of encryption can be a complex task.

Generally speaking, encrypting files and the filesystem is relatively simple nowadays. CBFS Storage implements on-the-fly encryption and decryption of files in streaming mode, so developers don't need to waste any time on it. Moreover, certain filesystems—such as CBFS Storage—support both whole-filesystem and per-file encryption simultaneously, and allow the use of different keys and encryption mechanisms for each file as well as for the filesystem itself. CBFS Storage also offers event-based APIs that developers can use to supply their own encryption mechanisms (e.g., certificate-based).

Compression

Compression is very important on mobile devices, where the user pays for traffic and where the storage space is severely limited. Databases are space-hungry and rarely include mechanisms for data compression, which leads to excessive complication in applications that need to deliver data in compressed form and then import it into a database (and even then, uncompressed data takes precious space in device memory).

Modern filesystems have built-in data compression implemented at the file level; i.e., you can compress all or some files as needed. CBFS Storage is no exception, and even allows developers to supply their own compression mechanisms if desired.

Versioning

A common requirement is the ability maintain a history of past versions of a named entity (be it a calendar record, a contact, a text note, or anything more complicated). Each version of your data must be stored somewhere, regardless of whether it's used for something as simple as an undo mechanism or something as complex as audit compliance. Alternate Data Streams support is typically the best solution since it allows you to store all variants in one file, which in turn means that simple file operations like copying and moving perform operations on all variants of the file at once.

To have versioning in a DBMS, you need to maintain a separate table or two, as there are no built-in mechanisms for versioning available. With filesystems, version management can be almost transparent—to create a version, you don't open the file itself, but its alternate stream. Just add a suffix to the filename and you have a new version.

Large and Heterogeneous Files

Relational databases were designed for storing uniform data that has a predefined number of typed fields per record. Such a design is effective when processing large amounts of such uniform data, but doesn't work well with objects that have different sets of properties. The necessity to manage such object sets lead to the creation of OODBMS (object-oriented DBMS) and later NoSQL DBMS. Still, SQL-based DBMS are mainstream, especially on mobile devices.

SQL-based relational DBMS don't handle large fields of undefined length (BLOBs) well. Such fields break the sleek structure of the database table. Various DBMS offer different ways to manage BLOBs. Frequently, BLOBs are kept in separate files on the disk with references from the DB table. Another solution is for the DBMS to have a mini-filesystem built into its engine to manage BLOBs and keep them in one file. However, both of these approaches require an extra layer of code that slows down operations with large data and reduces stability and reliability.

With a filesystem, you can have files as large as you want, and some filesystems let you add custom properties to files (in CBFS Storage, they're called "tags"). Such properties let you organize object storage easily.

Remote and In-Memory Storage

The database in a DBMS is almost always a set of files in local persistent storage, like a filesystem. Some DBMS allow the creation of in-memory tables for temporary operations, but migrating data to and from such in-memory solutions can be cumbersome.

Virtual filesystems, such as CBFS Storage, allow you to keep data anywhere—in memory, on a network-attached storage (NAS), in the cloud, or even in a database. Using event-based mechanisms, CBFS Storage offers unprecedented flexibility in the placement and maintenance of your data. Moreover, you can have on-the-fly mirroring of your data, (i.e., updating a remote backup copy as soon as the data is changed locally).

Easy Backup

Most DBMS are not designed for easy backups. Rarely can you simply copy the files that the DBMS uses to store its data; and even if you can, the restoration of these files is usually a tricky operation. Typically you have to export the data to a large—and not easily maintainable—SQL file, which is then packed and stored somewhere. If you need to recover the data, you will need to execute this SQL file and potentially need to edit it.

In the case of filesystems, backup and restore operations are trivial—you can just copy the files. In the case of CBFS Storage, which stores a complete filesystem in a single file (or other medium of your choosing), you can copy the entire filesystem; and when you restore that copy, it already has everything that's needed to continue operations. And if you use mirror, as discussed above, then your backups can be created on-the-fly.

Exposure to Other Applications

A final point of inefficiency with a DBMS is that you often need to expose the data contained within it as files for processing by other applications, and there's no way to accomplish this task without external assistance. On desktops you can use third-party solutions (e.g., CBFS Connect on Windows, or FUSE on Linux) to expose custom data as files, but on mobile platforms this is not possible at all.

With a filesystem, such a task is obviously of no consequence since the data is already available in the files-and-folders format. With CBFS Storage you can mount your embedded filesystem as a virtual drive on Windows, macOS, and Linux.