Creating Virtual Drive in .NET

In this article, we describe advanced features such as move and delete operations, Microsoft Office and AutoCAD transactional save as well as items locking.

This article is about legacy version of the User File System. For the latest version refer to this article. For previous versions please refer to the articles in this section.

Move and Delete Operations

Unlike with other operations, the platform provides the means for the move/rename and delete operations cancellation. In the case, the operation fails, because of the lost connection, validation errors, or any errors in your remote storage, you can cancel it, keeping your files in sync.

Both move and delete provide two methods, that are being called before and after the operation. The Engine calls IFileSystemItem.MoveToAsync() and IFileSystemItem.DeleteAsync() before the operation and IFileSystemItem.MoveToCompletionAsync() and IFileSystemItem.DeleteCompletionAsync() - after. You can confirm the operation in the "before"-method, by calling the IConfirmationResultContext. ReturnConfirmationResult() method or cancel the opertion by calling IConfirmationResultContext.ReturnErrorResult().

If you do not call IConfirmationResultContext.ReturnConfirmationResult() or IConfirmationResultContext.ReturnErrorResult() inside your move or delete implementation, and the method completes without exceptions, the Engine will call ReturnConfirmationResult() automatically confirming the operation. If your code fails with an exception, the Engine will automatically call ReturnErrorResult() and the item will NOT be moved or deleted.

Note that if your application is not running, for example in case of the application failure, the user can still move files and folders on your virtual drive. It is expected that when your application starts, you will typically run full sync with your remote storage and all files being moved/renamed, or deleted in your user file system will be overwritten with changes from your remote storage.

Implementing Move

Move Operation Sequence

Below we will describe the sequence of the move operation for the move without overwriting.

In the case of a file, the platform opens the file handle and blocks it for reading and writing. The Engine calls IFile.OpenAsync() method.
The platform starts the move operation. The Engine calls IFileSystemItem.MoveToAsync() method.
The implementer calls IConfirmationResultContext.ReturnConfirmationResult() or IConfirmationResultContext.ReturnErrorResult() with error code.
- If the ReturnErrorResult() method is called, the platform displays an error message with Try Again and Cancel buttons. If the user cancels the operation the platform does NOT rename/move the item. In the case of a file, the platform closes the file handle and the Engine calls IFile.CloseAsync(). The Engine does NOT call IFileSystemItem. MoveToCompletionAsync() in this case and the move operation completes.
- If the ReturnConfirmationResult() method is called, the platform moves the item in the file system.
If the target is the offline folder, the platform lists the content of the target folder. The Engine calls IFolder. GetChildrenAsync() on the target folder.
The Engine calls IFileSystemItem.MoveToCompletionAsync() and the IFile.CloseAsync() method. These methods may be called in any sequence. Note that the MoveToCompletionAsync() is called for the item in the target path.

Note that after the move operation completes, the file or folder is left in the NOT In-Sync state.

Below is a move implementation in the WebDAV Drive sample:

public async Task MoveToAsync(
    string userFileSystemNewPath, 
    byte[] targetParentItemId, 
    IOperationContext operationContext = null, 
    IConfirmationResultContext resultContext = null)
{
    string remoteStorageOldPath = RemoteStoragePath;
    string remoteStorageNewPath = Mapping.MapPath(userFileSystemNewPath);

    await Program.DavClient.MoveToAsync(
        new Uri(remoteStorageOldPath), 
        new Uri(remoteStorageNewPath), true);
}

public async Task MoveToCompletionAsync(
    IMoveCompletionContext moveCompletionContext = null, 
    IResultContext resultContext = null)
{
    string userFileSystemNewPath = this.UserFileSystemPath;
    string userFileSystemOldPath = moveCompletionContext.SourcePath;
}

Synchronization After the Move Operation

As soon as the item is left in the not In-Sync state after the move operation, the Engine (or your synchronization service via IClientNotifications. UpdateAsync() call) will evenually try to sync the item to the remote storage, triggering the IFile.WriteAsync() or IFolder.WriteAsync() method call.

Before calling the IFile.WriteAsync() method, the Engine will verify if the file content was actually modified. It will pass the content stream parameter to WriteAsync() method only in case the file content has changed. In case only metadata has changed (as well as in the case of the move operation), the content parameter will be set to null. Inside your WriteAsync() method implementation will typically upload file content only in case the content is not null:

public async Task WriteAsync(
    IFileMetadata fileMetadata, 
    Stream content = null, 
    IOperationContext operationContext = null)
{
    MyRemoteStorageItem remoteStorageItem = ...

    if (content != null)
    {
        // Upload remote storage file content only if content was modified.
        await using (MyRsStream remoteStorageStream = remoteStorageItem.Open(...))
        {
            await content.CopyToAsync(remoteStorageStream);
            remoteStorageStream.SetLength(content.Length);
        }
    }

    // Update remote storage file metadata if needed.
    remoteStorageItem.Attributes = fileMetadata.Attributes;
    remoteStorageItem.CreationTimeUtc = fileMetadata.CreationTime.UtcDateTime;
    remoteStorageItem.LastWriteTimeUtc = fileMetadata.LastWriteTime.UtcDateTime;
    remoteStorageItem.LastAccessTimeUtc = fileMetadata.LastAccessTime.UtcDateTime;
    remoteStorageItem.LastWriteTimeUtc = fileMetadata.LastWriteTime.UtcDateTime;
}

Move with Overwrite on Folders

In the case the target folder exists, the platform requests confirmation from the user for the folder to overwrite. If the user confirms, the following sequence is executed:

The platform starts the source folder deletion. The Engine calls IFileSystemItem.DeleteAsync() for the source folder.
The platform moves each file from the source folder to the target folder. The Engine calls the sequence described in 'Move without Overwrite' section above - OpenAsync() -> MoveToAsync()-> ReturnConfirmationResult() -> MoveToCompletionAsync() -> CloseAsync() method for each file and folder.
The platform completes the source folder delete operation. The Engine calls IFileSystemItem.DeleteCompletionAsync().

Implementating Detele

Delete Operation Sequence

Below we will describe the sequence of the delete operation.

In the case of a file, the platform opens the file handle and blocks it for reading and writing. The Engine calls IFile.OpenAsync() method.
The platform starts the delete operation. The Engine calls IFileSystemItem.DeleteAsync() method.
The implementer calls IConfirmationResultContext.ReturnConfirmationResult() or IConfirmationResultContext.ReturnErrorResult() with error code.
- If the ReturnErrorResult() method is called, the platform displays an error message with Try Again and Cancel buttons. If the user cancels the operation the platform does NOT delete the item. In the case of a file, the platform closes the file handle and the Engine calls IFile.CloseAsync(). The Engine does NOT call IFileSystemItem.DeleteCompletionAsync() in this case and the delete operation completes.
- If the ReturnConfirmationResult() method is called, the platform deletes the item in the file system.
The Engine calls IFileSystemItem.DeleteCompletionAsync() and the IFile.CloseAsync() method. These methods may be called in any sequence. Note that the DeleteCompletionAsync() is called for the item that has already been deleted and does not exist in the user file system.

Below is a delete operation implementation in WebDAV Drive sample:

public async Task DeleteAsync(
    IOperationContext operationContext, 
    IConfirmationResultContext resultContext)
{
    await Program.DavClient.DeleteAsync(new Uri(RemoteStoragePath));
    Engine.ExternalDataManager(UserFileSystemPath, Logger).Delete();
}

public async Task DeleteCompletionAsync(
    IOperationContext operationContext, 
    IResultContext resultContext)
{
}

Transactional File Save Operation

Some applications, for example, Microsoft Office and AutoCAD, are saving documents in a certain sequence that minimizes chances of data loss in case the file save is failed. In addition to the transactional save, Microsoft Office and AutoCAD apps create a lock-file(s) intended to indicate that the file is opened for editing. This sequence is different for every application, however, the approximate sequence looks like the following:

The lock-file(s) are created: $file.ext.
The original file is renamed. In some cases, such a file is also marked with a hidden or temporary flag. document.ext -> document.bak
The new file is created: document.tmp and the new content is saved into it.
The new file is renamed to the original file name. document.tmp -> document.ext.
The backup file (document.bak) is deleted.
The lock-file ($file.ext) is deleted.

The above sequence is useful if a document is located on some network shared drive, but in our case, we want to avoid such files being uploaded to our remote storage.

Here are the typical tasks you would like to do while handling the transactional save operation:

Avoid temporary files, backup files as well as lock-files being uploaded to your remote storage.
Avoid renaming and deleting the original file in your remote storage.
Avoid the file content upload to your remote storage until the file is closed (optional).
Preserve custom data stored with the item, such as the eTag, item ID, etc, during the rename->delete->create sequence.
Automatically lock the file on open and unlock on close events. This is achieved by the locking mechanism provided by the User File System.

We will describe how to achieve the first three tasks in this article below. Custom data preservation requires storing such data outside of the provided storage inside a placeholder (as the placeholder will be deleted as part of the transactional save). The automatic locking is provided by the Engine in combination with IEngine.FilterAsync() implementation and is described in the Locking section in this article below.

Avoiding temporary files upload

To avoid temporary files, backup files as well as lock-files being uploaded to your remote storage you will implement the IEngine.FilterAsync() method:

public override async Task<bool> FilterAsync(
    OperationType operationType, 
    string userFileSystemPath, 
    string userFileSystemNewPath = null, 
    IOperationContext operationContext = null)
{
    switch(operationType)
    {
        case OperationType.Update:
        case OperationType.Unlock:
            // Prevents MS Office file upload and unlock untill the file is closed.
            return FilterHelper.AvoidSync(userFileSystemPath) 
                || FilterHelper.IsAppLocked(userFileSystemPath);

        case OperationType.Move:
        case OperationType.MoveCompletion:
            // When a hydrated file is deleted, it is moved to a Recycle Bin.
            return FilterHelper.IsRecycleBin(userFileSystemNewPath) 
                || FilterHelper.AvoidSync(userFileSystemNewPath);

        default:
            return FilterHelper.AvoidSync(userFileSystemPath);
    }
}

This method provides source and target item names, as well as the operation name (create, update, delete, move, open, close, lock, unlock). This method also provides information about the environment, from which you can extract the process name if needed. However, the environment info is not provided for create operation and operations called via the IClientNotifications interface.

As soon as you have filtered some files from being processed by the Engine and underlying platform, such files are NOT converted into placeholders by the Engine and remain in the user file system as regular files. The events are not fired for such items and they can not be tracked inside your IFile and IFolder interface s implementations. Including you can not track the move event for such files anymore.

The solution to this is to implement a file system watcher that will monitor rename events and convert regular files to placeholders:

public class FilteredDocsMonitor : IDisposable
{
    private readonly FileSystemWatcherQueued watcher = new FileSystemWatcherQueued();
    private readonly VirtualEngine engine;

    internal FilteredDocsMonitor(string userFileSystemRootPath, VirtualEngine engine)
    {
        this.engine = engine ?? throw new ArgumentNullException(nameof(engine));

        watcher.IncludeSubdirectories = true;
        watcher.Path = userFileSystemRootPath;
        watcher.NotifyFilter = NotifyFilters.FileName | NotifyFilters.Attributes;
        watcher.Changed += ChangedAsync;
        watcher.Renamed += RenamedAsync;
    }

    private async void ChangedAsync(object sender, FileSystemEventArgs e)
    {
        await CreateOrUpdateAsync(sender, e);
    }

    private async void RenamedAsync(object sender, RenamedEventArgs e)
    {
        // If the item was previously filtered by EngineWindows.FilterAsync(),
        // for example temp MS Office file was renamed SGE4274H -> file.xlsx,
        // we need to convert the file to a pleaceholder and upload it to the remote storage.

        await CreateOrUpdateAsync(sender, e);
    }

    /// <summary>
    /// Creates the item in the remote storate if the item is new. 
    /// Updates the item in the remote storage if the item in not new.
    /// </summary>
    private async Task CreateOrUpdateAsync(object sender, FileSystemEventArgs e)
    {
        string userFileSystemPath = e.FullPath;
        if (!FilterHelper.AvoidSync(userFileSystemPath))
        {
            if (engine.ExternalDataManager(userFileSystemPath).IsNew)
            {
                if (!PlaceholderItem.IsPlaceholder(userFileSystemPath))
                {
                    // New file/folder, creating new item in the remote storage.
                    await engine.ClientNotifications(userFileSystemPath).CreateAsync();
                }
            }
            else
            {
                // Update item.
                if (!PlaceholderItem.IsPlaceholder(userFileSystemPath))
                {
                    // The item was converted to regular file during MS Office or AutoCAD transactiona save,
                    // converting it back to placeholder and uploading to the remote storage.
                    PlaceholderItem.ConvertToPlaceholder(userFileSystemPath, null, null, false);
                    await engine.ClientNotifications(userFileSystemPath).UpdateAsync();
                }
                else if (!PlaceholderItem.GetItem(userFileSystemPath).GetInSync())
                {
                    // The item is modified in the user file system, uploading to the remote storage.
                    await engine.ClientNotifications(userFileSystemPath).UpdateAsync();
                }
            }
        }
    }

    public void Dispose()
    {
        watcher.Dispose();
    }
}

In addition to the move event, the above example will also track the attributes change event. It is required, because some applications, such as Notepad++ remove attributes from files during saving, making them fall out of the platform and the Engine control.

Locking

Locking Unlocking Items

To lock items, the IT Hit User File System provides the ILock interface. You will implement it on files and folders that support locking. This interface provides LockAsync(), UnlockAsync() and GetLockModeAsync() methods.

Inside your ILock.LockAsync() method implementation you will lock the file in your remote storage as well as you will save the LockMode, passed as a parameter, somewhere in your local storage, on the client machine:

public async Task LockAsync(LockMode lockMode, IOperationContext operationContext=null)
{
    Logger.LogMessage($"{nameof(ILock)}.{nameof(LockAsync)}()", UserFileSystemPath);

    ExternalDataManager cdm = Engine.CustomDataManager(UserFileSystemPath, Logger);
    LockManager lockManager = cdm.LockManager;
    if (!await lockManager.IsLockedAsync()
        && !Engine.CustomDataManager(UserFileSystemPath).IsNew)
    {
        // Set pending icon, so the user has feedback as lock operation may
        // take some time.
        await cdm.SetLockPendingIconAsync(true);

        // Call your remote storage here to lock the item.
        // Save the lock token and other lock info received from the remote 
        // storage on the client. Supply the lock-token as part of each remote 
        // storage update in the IFile.WriteAsync() method.

        LockInfo lockInfo = await Program.DavClient.LockAsync(
            new Uri(RemoteStoragePath), 
            LockScope.Exclusive, false, 
            null, 
            TimeSpan.MaxValue);
        ServerLockInfo serverLockInfo = new ServerLockInfo
            {
                LockToken = lockInfo.LockToken.LockToken,
                Exclusive = lockInfo.LockScope == LockScope.Exclusive,
                Owner = lockInfo.Owner,
                LockExpirationDateUtc = DateTimeOffset.Now.Add(lockInfo.TimeOut)
            };

        // Save lock-token and lock-mode.
        await lockManager.SetLockInfoAsync(serverLockInfo);
        await lockManager.SetLockModeAsync(lockMode);

        // Set lock icon and lock info in custom columns.
        await cdm.SetLockInfoAsync(serverLockInfo);

        Logger.LogMessage("Locked in remote storage succesefully.", UserFileSystemPath);
    }
}

In your ILock.UnlockAsync() method you will unlock the item in your remote storage:

public async Task UnlockAsync(IOperationContext operationContext=null)
{
    ExternalDataManager cdm = Engine.CustomDataManager(UserFileSystemPath, Logger);
    LockManager lockManager = cdm.LockManager;

    // Set pending icon, so the user has a feedback as unlock operation may 
    // take some time.
    await cdm.SetLockPendingIconAsync(true);

    // Read lock-token from lock-info file.
    string lockToken = (await lockManager.GetLockInfoAsync()).LockToken;
    LockUriTokenPair[] lockTokens = new LockUriTokenPair[] 
        { 
            new LockUriTokenPair(new Uri(RemoteStoragePath),
            lockToken)
        };

    // Unlock the item in the remote storage.
    try
    {
        await Program.DavClient.UnlockAsync(new Uri(RemoteStoragePath), lockTokens);
    }
    catch (ITHit.WebDAV.Client.Exceptions.ConflictException)
    {
        // The item is already unlocked.
    }

    // Delete lock-mode and lock-token info.
    lockManager.DeleteLock();

    // Remove lock icon and lock info in custom columns.
    await cdm.SetLockInfoAsync(null);

    Logger.LogMessage("Unlocked in the remote storage succesefully", UserFileSystemPath);
}

Inside your ILock.GetLockModeAsync() you will return your stored LockMode to the Engine:

public async Task<LockMode> GetLockModeAsync(IOperationContext operationContext=null)
{
    LockManager lm = Engine.CustomDataManager(UserFileSystemPath, Logger).LockManager;
    return await lm.GetLockModeAsync();
}

Storing the LockMode

During the ILock.LockAsync() call you must store the LockMode passed to this method on your local machine. When the file handle is closed the Engine will request the LockMode by calling your ILock.GetLockModeAsync() method implementation. You will typically return the LockMode passed into LockAsync() from this method. If your GetLockModeAsync() method returns LockMode.Auto, the Engine will unlock the file by calling the ILock. UnlockAsync() method.

Do NOT store the LockMode in your remote storage, as the ILock. GetLockModeAsync() method may be called from the Windows Explorer context menu and it expects a fast response in this case.

Manual Locking

To lock or unlock the document on the client machine, for example from your custom Windows Explorer lock context menu command, you will call methods of the IClientNotifications interface which is returned by the IEngine.ClientNotifications() call:

IClientNotifications cn = engine.ClientNotifications(@"C:\Users\User1\VD\myfile.ext");
cn.LockAsync(LockMode.Manual);

The IClientNotifications.LockAsync() method has a parameter that indicates if the document should be automatically unlocked when the file handle is closed. This parameter will be passed to your ILock.LockAsync() method implementation. Typically you will pass the LockMode.Manual parameter to IClientNotifications.LockAsync() call, which will indicate that you will unlock the document manually, by calling the IClientNotifications.UnlockAsync() method. If you pass the LockMode.Auto, the file will be unlocked by calling ILock.UnlockAsync() when the file handle is closed.

Automatic Locking

On the Windows platform, the IT Hit User File System supports automatic documents locking/unlocking. Typically automatic document locking is useful to automatically lock the document when it is opened by Microsoft Office and unlock on close.

To enable the automatic documents locking set the EngineWindows. AutoLock property to true when creating the Engine instance.

If the automatic documents locking is enabled, when a file handle is opened with the FILE_SHARE_READ or FILE_SHARE_DELETE or FILE_SHARE_NONE sharing mode, the Engine will call the ILock. LockAsync() method passing LockMode.Auto. When the file handle is closed it will call ILock.UnlockAsync() method.

Locking Events Design

Note that the locking event occurs after the file is being opened for writing. The failed locking event can NOT prevent the file from being opened for editing. This is because the Engine and underlying platform API do not provide any events that occur during the file open. The reason behind such a design is that file opening operation is a frequent event and must be instant, while any requests to the remote storage will slow down the file system performance. In addition to that, your file system may be in offline mode, with a server being unavailable and unable to lock the file.

In case your file failed to lock, either because it is being locked by another user, or because of any other reasons, you can mark the file as read-only, so it can NOT be saved. While the read-only attribute does NOT protect the file from modifications, still most applications, including Microsoft Office, will respect the read-only attribute and will NOT update the file.

Another option is to notify clients via web sockets (or another channel) and set the read-only flag on the file as well as render the lock icon in Windows Explorer on the client machines when the file is locked on the server.

Open and Close Events

The Engine provides the IFile.OpenAsync() and the IFile.CloseAsync() methods, that are called when the file handle is being opened and closed. Note that both events occur after the file is opened/closed and you can not prevent a file handle from being opened or closed.

For performance reasons, neither the Engine nor the underlying platform provides events that occur before the file handle open or close events or any events that can prevent the file from opening/losing.

Programming

Creating Virtual Drive in .NET

In this article

Creating Virtual Drive in .NET

Move and Delete Operations

Implementing Move

Move Operation Sequence

Synchronization After the Move Operation

Move with Overwrite on Folders

Implementating Detele

Delete Operation Sequence

Transactional File Save Operation

Avoiding temporary files upload

Locking

Locking Unlocking Items

Storing the LockMode

Manual Locking

Automatic Locking

Locking Events Design

Open and Close Events