Implementing Virtual Files with CBFS Filter

The CBFilter component of CBFS Filter provides mechanisms for redirecting file access to other files and directories and for creation of files that don’t exist in the filesystem at all. Such files are called virtual files. This article describes what virtual files are, how they may be used, and how one can implement them.

The Basics

Filesystems, as containers for data, lack mechanisms for dynamically generating content. When an application needs to present external data as a file, it has two primary options: create a complete virtual filesystem (e.g., using our CBFS Connect product) or create a virtual file within an existing filesystem.

Creating a complete virtual filesystem requires a significant investment of resources, which may not be justified if you only need to create one or a few virtual files. This is where CBFilter comes to the rescue. With this component, applications can leverage the same mechanisms used for file monitoring and control to inject virtual files directly into an existing filesystem.

Some uses of virtual files include

• Dynamic Configuration Management: Virtual files can be employed to present configuration files to applications that are hardcoded to look for them in specific locations. This allows for centralized configuration management without the need to change the application’s code or its deployment structure.
• Legacy Application Support: In environments where legacy applications require specific files that are no longer needed or have been deprecated, virtual files can provide these files on-the-fly without having to retain outdated or unnecessary physical files on the system.
• Testing and Debugging: Developers can use virtual files to simulate various file conditions or data scenarios when testing applications. For instance, a developer can create a virtual file that mimics a corrupted file to test how the application handles errors.
• File Redirection: Virtual files can be used to redirect file access requests to alternative locations. This can be beneficial in scenarios where an application is hardcoded to a particular file path, but the actual file has been moved to a new location, allowing seamless access without modifying the application.
• Data Virtualization for Enhanced Security: By using virtual files, sensitive data can be stored securely in a central location, while still allowing applications to access this data as if it were stored locally. This can help reduce data leakage and enhance security protocols.
• Multi-Version File Handling: In environments where different versions of files or libraries are required for different applications, virtual files can allow the same application to operate with different versions without conflict. This is particularly useful in scenarios like software development and testing environments.

CBFilter implements virtual file support through a filesystem filter driver. This driver intercepts operating system requests sent to the filesystem and handles requests related to virtual files internally, without passing them further down the filter stack.

Types of Virtual Files

CBFilter recognizes two types of virtual files: static and dynamic.

• Static Virtual Files: An application adds a static virtual file by calling the AddVirtualFile method of the CBFilter component. Later, the application can use UpdateVirtualFile to modify file information and RemoveVirtualFile to delete the virtual file's information. Static virtual files are listed in a directory because the driver is aware of their presence. The parameters for these files—such as size, timestamps, and attributes—are derived from the information provided during the AddVirtualFile call.
• Dynamic Virtual Files: To create a dynamic virtual file, an application handles the BeforeCreateFile and BeforeOpenFile events and instructs the driver to open the file with isolation (setting the Isolate parameter to true) and without redirection (leaving the BackendFileName parameter empty). Dynamic virtual files are not visible in the listing by default. However, if a real file exists on the disk, it will be listed with its actual size and attributes. When a dynamic file is opened, it overlays the virtual file, masking it from the process that opened it. If no real file exists on the disk, nothing is listed; nonetheless, a process can still open the virtual file by specifying its path.

The dynamic virtual file mechanism is particularly useful when the exact file names are unknown in advance, allowing the application to provide virtual files based on the needs of the process attempting to open them. Additionally, dynamic virtual files can conditionally appear or become virtual based on certain properties of the process. This means a file might exist for one process but not for others, or the application can provide different content for different processes.

Using Virtual Files

In the previous section, we discussed two ways to create virtual files, depending on whether and how they should be listed in a directory. An application can either add the file to the list by calling AddVirtualFile, or it can handle the BeforeCreateFile and BeforeOpenFile events, specifying that the file should be isolated without any backend file to redirect requests.

When a virtual file is added using AddVirtualFile, the BeforeCreateFile and BeforeOpenFile events are still triggered. In this case, the Isolate and BackendFileName parameters are already set to the values required for the file to be virtualized.

The contents and size of a virtual file can differ for each application that accesses it. When two processes, A and B, open the same file concurrently, file isolation ensures that their operations remain completely independent. Process A may receive a different set of information than Process B, and both processes can modify the data simultaneously. This means the application managing the virtual file can handle all requests from Process A using an on-disk file X, while Process B interacts with and updates data in a different on-disk file Y.

However, this separation has a downside: if the data must remain consistent between Processes A and B, the application must manage the DesiredAccess, ShareMode, and CreationDisposition parameters during the BeforeCreateFile and BeforeOpenFile events. The application is responsible for determining whether a second opening of the same file should be permitted or denied, as well as what rights each operation would have. Normally, these checks are performed by the filesystem, but with virtual files, this responsibility shifts to the application.

For static virtual files, the driver retains basic information about the file, while for dynamic virtual files, it has none. Nevertheless, the application must deal with a variety of OS requests, including file data manipulation.

To process requests related to virtual files, the application should handle the following events:

• BeforeQueryFileInfo
• BeforeSetFileInfo (handling of the FileBasicInformation, FileRenameInformation(Ex), FileDispositionInformation(Ex), FileEndOfFileInformation, FileAllocationInformation classes is required; other classes are optional)
• BeforeReadFile
• BeforeWriteFile
• BeforeGetFileSecurity
• BeforeSetFileSecurity
• BeforeCloseFile

The corresponding rule can be added like this:

filter.AddFilterRule("C:\\path\\to\\virtual.file", Constants.ACCESS_NONE, Constants.FS_CE_BEFORE_CREATE | Constants.FS_CE_BEFORE_OPEN | Constants.FS_CE_BEFORE_QUERY_FILE_INFO | Constants.FS_CE_BEFORE_SET_FILE_INFO | Constants.FS_CE_BEFORE_READ | Constants.FS_CE_BEFORE_WRITE | Constants.FS_CE_BEFORE_GET_SECURITY | Constants.FS_CE_BEFORE_SET_SECURITY | Constants.FS_CE_BEFORE_CLOSE, Constants.FS_NE_NONE); // reference your handler methods here filter.OnBeforeCreateFile += HandleBeforeCreateFile; filter.OnBeforeOpenFile += HandleBeforeOpenFile; filter.OnBeforeQueryFileInfo += HandleBeforeQueryFileInfo; filter.OnBeforeSetFileInfo += HandleBeforeSetFileInfo; filter.OnBeforeReadFile += HandleBeforeReadFile; filter.OnBeforeWriteFile += HandleBeforeWriteFile; filter.OnBeforeGetFileSecurity += HandleBeforeGetFileSecurity; filter.OnBeforeSetFileSecurity += HandleBeforeSetFileSecurity; filter.OnBeforeCloseFile += HandleBeforeCloseFile;

Each event must be handled in full; if it is not addressed by the application, there is no filesystem to process the request. Each event handler should set the ProcessRequest parameter to false, signaling the driver that no further processing is required.

If this parameter remains true (its default state), the driver will perform basic processing of the unhandled request, which typically results in returning STATUS_NOT_SUPPORTED or another appropriate status to the system. This is undesirable, as it will likely disrupt the operation of the caller application.

Note that file size and allocation size are communicated through different information classes within the BeforeQueryFileInfo and BeforeSetFileInfo events.

If the application needs to deal with other requests related to virtual files, such as extended attribute or reparse point manipulations, it can do so by handling the corresponding Before* events similarly to the main events listed above.

Renaming and hard links

These two types of file operations—renaming and creating hard links—must be explicitly handled by the application if support for them is required. The relevant events to handle are BeforeRenameOrMoveFile for file renaming and moving, and BeforeCreateHardLink for creating a hard link to a virtual file.

When a virtual file is renamed, only the application knows how to manage this situation appropriately. If no corresponding file exists on the disk, the application can update its internal information to manage the virtual file under its new name. However, it is important to ensure that the filter rules for handling events related to the virtual file also cover the new name. If the filesystem contains a real file with the same name as the virtual file, the application must decide whether to rename that real file or leave it unchanged.

Hard links are directory entries that point to the same file data. The first file name is itself a hard link, meaning that when another hard link is created, both names are functionally equivalent. Hard links can exist in different directories within the filesystem. However, because hard links are directory entries, they can only be created on a real filesystem where an existing file must already be present. The application can choose to allow the creation of a hard link to the existing real file or deny this operation if no real file exists or for other reasons. If the appropriate events are not handled, the driver will respond to these requests with an error.

Deletion of virtual files

For both static and dynamic virtual files, the application is responsible for managing requests related to file deletion. Simply put, if a user deletes a static virtual file through the file manager, CBFilter will not automatically remove this file from the list of virtual files. The application can utilize the BeforeCanFileBeDeleted and BeforeDeleteFile events to facilitate file deletion or to prevent deletion in a way that is visible to the user.

If the BeforeDeleteFile event is handled for a static virtual file and the application allows deletion, it must explicitly call the RemoveVirtualFile method to remove the virtual file from the list. For dynamic virtual files, no specific operation is required when it comes to deletion.

Conditional provision of data

Within the event handlers, an application can collect additional information about the request, based on which it can make decisions about file availability, size, content, and available operations. The information available to an event handler includes the parameters received in the event (such as the path to the file or directory and operation-specific parameters), as well as the following supplementary details that can be obtained by calling the corresponding methods of CBFilter:

• the name of the process that opened the file handle related to the request
• the PID (Process ID) of the process that opened the file handle related to the request
• the ID of the thread that opened the file handle associated with the request
• the security token of the process that opened the file handle; this token can be used to obtain the SID of the user account, along with the user name and other related information
• the name of the process that initiated the request (which may differ from the process which opened the handle)
• the PID (process Id) of the process that initiated the request (which may differ from the process that opened the handle)
• the Id of the thread that initiated the request (which may differ from the thread which opened the handle)
• the security token of the process that initiated the request; this token can be used to obtain the SID of the user account, along with the user name and other related information
• the time when the request reached the filter
• the information about remote access if a local file was accessed over the network.

Getting Started with CBFilter

You can find an evaluation version of the SDK for your platform and programming language in the Download Center. After downloading and installing the SDK, you will receive a library, sample code, and comprehensive documentation on your system. The documentation includes a "Getting Started" section with programming instructions. Additionally, free technical support is available during the evaluation phase.