Creating a Replacement for FileSystemWatcher

Since the early days of the .NET Framework, the .NET library has offered a convenient wrapper around the FindFirstChangeNotification system API function, known as the FileSystemWatcher class. However, this class and its underlying technology have notable limitations, such as providing insufficient information about changes and introducing an unpredictable delay between the change and the triggering of the event handler. For instance, a file may be created and renamed, but when the event fires, the file for which you receive a creation notification may already be gone. These complications lead to non-deterministic application behavior, making it difficult to track file changes reliably.

The CBMonitor and CBFilter components of CBFS Filter enable comprehensive monitoring and control of all types of filesystem operations within Windows applications. Separate articles address the monitoring capabilities in both CBMonitor and CBFilter, as well as the control functions in CBFilter, covering the theoretical aspects of using CBFS Filter for change tracking.

This article provides practical guidance for creating a more reliable alternative to FileSystemWatcher using either CBMonitor or CBFilter.

The .NET edition of CBFS Filter includes a drop-in replacement for FileSystemWatcher as a sample project. This article aims to explain the implementation to users of other editions, where this sample is not available (although its creation is planned). Users of the .NET edition will also learn about omitted details from the sample, potential extensions, and the rationale behind certain design decisions.

Tracked operations

CBMonitor and CBFilter can track and pass a wide range of filesystem operations to applications, including file creation, opening, deletion (which is not even an atomic operation in Windows), metadata changes, operations with extended attributes, reparse points, and alternate data streams. This capability also extends to directories.

In contrast, the scope of operations for which FileSystemWatcher can send notifications is more limited. While you can extend its implementation to cover additional operations, the sample provided closely follows FileSystemWatcher and includes events for the following types of operations:

• A file or directory has been created.
• A file or directory has been deleted.
• A file or directory has been renamed.
• A file has been modified.

The types of detected file changes include:

• Changes to attributes
• Changes to timestamps
• Changes to security attributes
• Changes to file size (for files)
• Changes to file allocation size (for files)

While the sample does not track changes in file extended attributes or reparse points, it can be easily extended to include these operations.

Internals of Tracking

If an application needs to respond to operations after they occur, it can utilize After* events or Notify* events. An After* event is triggered after the operating system has processed the request through the filesystem, but before it returns control to the OS and the originating process. The driver waits for the After* event handler to complete before passing the request further. In contrast, Notify* events are scheduled for firing after the corresponding After* event completes but do not require the driver to wait for the Notify* event to finish. This means that Notify* events are fired asynchronously, while After* events are not.

After* events are exclusive to CBFilter, whereas Notify* events are available in both CBMonitor and CBFilter.

Events are triggered when the corresponding rule is added, with examples of such rules provided later in this article.

The sample implementation uses both After* and Notify* events, with each filesystem operation covered by either an After* or Notify* event (but not both). The choice between event types is based on the frequency of operations and the volume of data involved.

For instance, writing operations occur frequently, and the AfterWriteFile event carries the actual written data in its parameters. Additionally, the driver waits for the event handler to return. Therefore, using AfterWriteFile for tracking file changes can introduce delays in processing write operations. Since the component does not need the written data, it relies on the NotifyWriteFile event instead.

Similarly, file size changes can occur frequently, and introducing delays is undesirable. Thus, the sample employs the NotifySetFileSize and NotifySetAllocationSize events.

In contrast, file and directory creation and deletion are tracked using the corresponding After* events. This approach is crucial because applications often need to monitor these events immediately to avoid missing a file that is created and subsequently deleted, or to catch the creation of a new file following the deletion of another. This capability is notably absent from the .NET implementation of FileSystemWatcher.

Changes to file times and attributes can be handled using both After* and Notify* events. The sample uses AfterSetFileAttributes to allow the application to respond to changes in file times as quickly as possible.

For renaming operations, the sample uses the NotifyRenameOrMoveFile event. If synchronous tracking is required, the implementation can be adjusted to use AfterRenameOrMoveFile instead.

By default, the components trigger After* and Notify* events only when the operation succeeds. This mode is appropriate for FileSystemWatcher, which must report actual filesystem updates. However, you may also want to capture failed requests. To enable event firing for failed operations, the application must set the ProcessFailedRequests property to true:

filter.ProcessFailedRequests = true;

When events for failed operations are enabled, the handler can check the Status parameter of the event to determine the outcome. A value of 0 or STATUS_SUCCESS indicates success. In some cases, other status values may also indicate successful completion, but this does not apply to the events discussed in this article.

Recursive and non-recursive tracking

When a rule is added using one of the Add*Rule methods, it is recursive by default, meaning that entries in subdirectories are also covered by the rule. For example, a rule added with:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", … )

will cover files like "c:\path\to\directory\child\grandchild\somefile.txt".

To exclude the contents of subdirectories from monitoring, an application can implement a passthrough rule. For instance:

filter.AddPassthroughRule("C:\\path\\to\\directory\\*\\?*.*", … );

This rule will cover the grandchildren of the specified directory, effectively excluding them from monitoring.

Tracked Operations

File creation

When a file or directory is created or opened, neither the caller nor the operating system can guarantee that the requested file or directory will exist when the request reaches the filesystem. To address this uncertainty, the OS provides a single CreateFile API function that includes a CreationDisposition parameter, which instructs the filesystem on how to handle cases where the file or directory may or may not exist.

CBFilter has two separate event groups: BeforeCreateFile, AfterCreateFile, NotifyCreateFile, and BeforeOpenFile, AfterOpenFile, NotifyOpenFile.

BeforeCreateFile and BeforeOpenFile indicate the caller's intent to create or open a file. BeforeCreateFile is only triggered when the value of the CreationDisposition in a call to CreateFile is CREATE_NEW (to create a new file and fail if it already exists) or FILE_SUPERSEDE (a kernel-mode flag that directs the filesystem to replace the file data). In the latter case, BeforeCreateFile is fired after BeforeOpenFile. AfterCreateFile and AfterOpenFile reflect the actual outcome of these operations.

The events in each pair (BeforeCreateFile and BeforeOpenFile, etc.) are typically handled similarly. However, when tracking the specific creation of a file or directory, there is a subtle distinction between handling AfterCreateFile and AfterOpenFile (or NotifyCreateFile and NotifyOpenFile).

Since AfterCreateFile and NotifyCreateFile are triggered only when the original CreationDisposition is CREATE_NEW, there's no need to check the operation's outcome (assuming its status is 0 or STATUS_SUCCESS).

In contrast, when handling AfterOpenFile or NotifyOpenFile, the handler must verify whether the file was actually created or if another event occurred. This is accomplished by checking the CreationStatus parameter of the event, which informs the handler whether the file existed prior to the event and, if so, what happened to the existing file.

To capture AfterCreateFile and AfterOpenFile events, a rule can be added using the AddFilterRule method:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_AFTER_CREATE | Constants.FS_CE_AFTER_OPEN, Constants.FS_NE_NONE); filter.OnAfterCreateFile += HandleCreateFile; // reference your handler method here filter.OnAfterOpenFile += HandleOpenFile; // reference your handler method here

File modification

The components can track all changes occurring to a file or directory, including changes to timestamps and attributes, security attributes, file data size, and allocation size, as well as operations involving extended attributes and reparse points. Additionally, they can monitor all operations related to Alternate Data Streams of a file or directory.

To replicate FileSystemWatcher, the sample focuses on a subset of updates when firing the Changed event. Specifically, it reports file writes and updates to timestamps, attributes, file size, and allocation size.

The decision to consider changes in timestamps and attributes or security attributes as file changes is left to the developers. For example, in backup scenarios, it may be sensible to track only actual data changes, such as size changes and data write operations. In contrast, a synchronization solution that needs to copy timestamps and attributes to a remote location will, of course, need to monitor these metadata changes as well.

To track changes in timestamps and attributes, the sample handles the AfterSetFileAttributes event. The corresponding rule can be added as follows:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_AFTER_SET_ATTRIBUTES, Constants.FS_NE_NONE); filter.OnAfterSetFileAttributes += HandleSetAttributes; // reference your handler method here

Alternatively, the NotifySetFileAttributes event can be used:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_NONE, Constants.FS_NE_SET_ATTRIBUTES); filter.OnNotifySetFileAttributes += HandleSetAttributes; // reference your handler method here

To track changes in file size and allocation size, the sample uses the NotifySetFileSize and NotifySetAllocationSize events, enabling them as follows:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_NONE, Constants.FS_NE_SET_SIZES); filter.OnNotifySetFileSize += HandleSetFileSize; // reference your handler method here filter.OnNotifySetAllocationSize += HandleSetFileSize; // reference your handler method here

To monitor changes in file data (i.e., file writing), the NotifyWriteFile event is utilized. This event detects the writing action but does not provide access to the data being written. If access to the data is needed, use the AfterWriteFile event instead.

The sample code for tracking file writing is:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_NONE, Constants.FS_NE_WRITE); filter.OnNotifyWriteFile += HandleWriteFile; // reference your handler method here

Changes to security attributes can be captured using the AfterSetFileSecurity or NotifySetFileSecurity events. In the sample, NotifySetFileSecurity is used to minimize the impact of synchronous event calls on system operations.

Here is a sample rule to track changes to security attributes using NotifySetFileSecurity:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_NONE, Constants.FS_NE_SET_SECURITY); filter.OnNotifySetFileSecurity += HandleSetFileSecurity; // reference your handler method here

File deletion

In Windows, deletion of a file or directory is not an atomic operation; no single request is sent to the filesystem for this purpose. Instead, a process opens the file and either specifies a special flag in the creation parameters (FILE_FLAG_DELETE_ON_CLOSE) or sets a disposition flag with a separate call to a Windows function. The file is then removed by the filesystem after all handles to the file and the file itself are closed.

CBFS Filter tracks the opening and closing of files and triggers the *DeleteFile events at the appropriate times.

BeforeDeleteFile is fired while the file is still present on the disk, whereas AfterDeleteFile is triggered once the file has been removed.

To handle the AfterDeleteFile event, an application can add the following rule:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_AFTER_DELETE, Constants.FS_NE_NONE); filter.OnAfterDeleteFile += HandleDeleteFile; // reference your handler method here

Note: The filter driver can track the number of open and close operations to determine the last closing event from the moment it is started by the OS and attached to the volume (disk). This is generally not an issue; however, if the driver has just been installed, it will not be aware of files that were opened and not closed prior to its installation. In very rare cases, this could lead to the driver reporting a deletion before it actually occurs.

Renaming and moving of files

In Windows, renaming and moving a file are treated as the same request; the OS sends the new fully qualified name of the file.

Both renaming and moving can be tracked by handling either the AfterRenameOrMoveFile or NotifyRenameOrMoveFile events. For example:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_AFTER_RENAME, Constants.FS_NE_NONE); filter.OnAfterRenameFile += HandleRenameFile; // reference your handler method here

The sample handles file renaming and moving differently. If a file is merely renamed (without changing its parent directory), a single Renamed event is fired. In contrast, when a file is moved, the sample triggers both Deleted and Created events. However, alternative approaches to handling these operations are certainly possible.

Rare operations with files

Some rare operations that can impact the accessibility of file data involve manipulating reparse points. The main operations to track are setting and deleting a reparse point. The presence of a reparse point may prevent data from being accessible or change the data available to a reader. Depending on the nature of the application, this may be considered a change to the file.

An application can detect these manipulations by handling the AfterSetReparsePoint and AfterDeleteReparsePoint events, or the NotifySetReparsePoint and NotifyDeleteReparsePoint events. The After* events include reparse point data, while the Notify* events do not.

To handle the After* operations, you can add the following rule:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_AFTER_SET_REPARSE_POINT | Constants.FS_CE_AFTER_DELETE_REPARSE_POINT, Constants.FS_NE_NONE); filter.OnAfterSetReparsePoint += HandleSetReparsePoint; // reference your handler method here filter.OnAfterDeleteReparsePoint += HandleDeleteReparsePoint; // reference your handler method here

Another relatively rare operation involves manipulating the extended attributes (EAs) of a file. The Win32 API does not provide direct means to read and write EAs, which is why they are seldom used by applications in Windows. However, Windows system components utilize them much more frequently. If an application is interested in tracking operations on EAs, it is possible to do so.

CBFilter includes the AfterSetEa event, while both CBFilter and CBMonitor provide access to the NotifySetEa event. To track the NotifySetEa event, use code similar to the following:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_NONE, Constants.FS_NE_SET_EA); filter.OnNotifySetEa += HandleSetEa; // reference your handler method here

Alternate Data Streams

Some filesystems, including NTFS and those created by CBFS Connect and CBFS Storage products, support Alternate Data Streams (for more information, see the MS-FSCC specification). These are named sequences of data within a file, with the file acting as a directory for these named streams. In NTFS, the main file data is also considered a data stream, but it does not have a name. For example, the main data stream of the file somefile.txt is represented as "somefile.txt::$DATA" (the semicolon serves as a separator, with the second part of the name indicating an unnamed stream).

Operations on streams are automatically included in the monitoring capabilities of CBMonitor and CBFilter.

If an application decides it is not interested in tracking stream operations, it should analyze the name of the file or directory associated with the event. If the name contains semicolons, it indicates that the operation is performed on a stream.

Alternatively, an application can add a passthrough rule with the same flags and a mask that excludes secondary streams:

filter.AddPassthroughRule("C:\\path\\to\\directory\\*.*:*", … ); // note the semicolon in the mask

Combining Rules

As observed in the file creation and opening examples above, it is possible to combine all relevant flags into a single rule (or two rules if a passthrough rule is used). The sample class demonstrates this by combining the flags and including them in one rule. To track the main operations described earlier, you can set up the rule with the following call:

filter.AddFilterRule("C:\\path\\to\\directory\\*.*", Constants.ACCESS_NONE, Constants.FS_CE_AFTER_CREATE | Constants.FS_CE_AFTER_DELETE | Constants.FS_CE_AFTER_OPEN | Constants.FS_CE_AFTER_SET_ATTRIBUTES, Constants.FS_NE_RENAME | Constants.FS_NE_SET_SECURITY | Constants.FS_NE_SET_SIZES | Constants.FS_NE_WRITE);

It makes no significant difference whether flags are added with one or several calls to AddFilterRule: the driver combines the flags related to the same mask into one record for processing.

Getting Started with CBFilter

You can find an evaluation version of the SDK for your platform and programming language in the Download Center. After downloading and installing the SDK, you will receive a library, sample code, and comprehensive documentation on your system. The documentation includes a "Getting Started" section with programming instructions. Additionally, free technical support is available during the evaluation phase.