User File System Engine for .NET Programming Guide

This article is about the legacy version of the User File System. For the latest version please refer to the articles in this section.

File System Registration

To let the platform know about the new file system you will register it using StorageProviderSyncRootManager class. You will typically register it during your application installation and will unregister on uninstall. Under the registered file system root you can create files and folders placeholders as well as each placeholder can store additional information, such as In-Sync status. Here is your typical sync-root registration code:

/// <summary>
/// Registers sync root.
/// </summary>
/// <param name="syncRootId">ID of the sync root.</param>
/// <param name="path">A root folder of your user file system. Your file system tree will be located under this folder.</param>
/// <param name="displayName">Human readable display name.</param>
/// <remarks>Call this method during application installation.</remarks>
public static async Task RegisterAsync(string syncRootId, string path, string displayName)
{
    StorageProviderSyncRootInfo storageInfo = new StorageProviderSyncRootInfo();
    storageInfo.Path = await StorageFolder.GetFolderFromPathAsync(path);
    storageInfo.Id = syncRootId;
    storageInfo.DisplayNameResource = displayName;
    storageInfo.IconResource = Path.Combine(Program.Settings.IconsFolderPath, "Drive.ico");
    storageInfo.Version = System.Reflection.Assembly.GetExecutingAssembly().GetName().Version.ToString();
    storageInfo.RecycleBinUri = new Uri("https://userfilesystem.com/recyclebin");
    storageInfo.Context = CryptographicBuffer.ConvertStringToBinary(path, BinaryStringEncoding.Utf8);

    storageInfo.HydrationPolicy = StorageProviderHydrationPolicy.Progressive;
    storageInfo.HydrationPolicyModifier = 
                StorageProviderHydrationPolicyModifier.AutoDehydrationAllowed | 
                StorageProviderHydrationPolicyModifier.ValidationRequired;

    // To implement folders on-demand placeholders population set 
    // the StorageProviderSyncRootInfo.PopulationPolicy to StorageProviderPopulationPolicy.Full.
    storageInfo.PopulationPolicy = StorageProviderPopulationPolicy.Full;

    // This configures when PlaceholderItem.GetInSync() should return true/false.
    storageInfo.InSyncPolicy = 
        StorageProviderInSyncPolicy.FileLastWriteTime           | StorageProviderInSyncPolicy.FileCreationTime |
        StorageProviderInSyncPolicy.FileHiddenAttribute         | StorageProviderInSyncPolicy.FileReadOnlyAttribute |
        StorageProviderInSyncPolicy.FileSystemAttribute         | StorageProviderInSyncPolicy.DirectoryLastWriteTime |
        StorageProviderInSyncPolicy.DirectoryCreationTime       | StorageProviderInSyncPolicy.DirectoryHiddenAttribute |
        StorageProviderInSyncPolicy.DirectoryReadOnlyAttribute  | StorageProviderInSyncPolicy.DirectorySystemAttribute;

    StorageProviderSyncRootManager.Register(storageInfo);
}

If you have any regular files and folders under your file system root during registration, they will NOT be converted into placeholders automatically. If needed, to convert them to placeholders, you can use the PlaceholderItem.ConvertToPlaceholder() overloaded methods.

Note that your file system location must be indexed by the indexing service.  

To unregister your sync root call StorageProviderSyncRootManager.Unregister() passing Sync Root ID specified during registration. When you unregister your file system all placeholders left under your file system root will be converted into regular files and folders and all data associated with each placeholder is lost. 

The Engine

When an application is reading a file, listing folder content, or performing other file system operations, the platform calls the Engine class to receive the data required to complete the operation, confirm the operation success, or to notify the Engine about the completed operations. In your program, you will derive your class from the EngineWindows class as well as you will implement IFile and IFolder interfaces to perform file system operations. You will create an instance of your Engine class on application start and dispose it on application exit.

To start processing calls from the file system call the Engine.StartAsync() method:

public class VfsEngine : EngineWindows
{
    public VfsEngine(string license, string path) : base(license, path)
    {
    }

    public override async Task<IFileSystemItem> GetFileSystemItemAsync(string path)
    {
        ...
    }
}

...

Console.Write("Press any key to exit.");
using(VfsEngine engine = new VfsEngine(licenseString, @"C:\Users\User1\VFS\"))
{
    await engine.StartAsync();
    Console.ReadKey();
}

Note that the platform does not map all file system operations directly into Engine calls. Applications working with the file system as well as OS expect a fast response from the file system and make a large number of reads and writes. Mapping all file open-read/write-close, and folder listing calls directly into the remote storage will make such a file system unresponsive as well as will make a high load on the remote storage. Instead, the platform provides a layer that reduces the number of file system calls and translates only calls required for data loading, some file system operations, such as move/rename and delete as well as additional notification calls that do not slow down the file system.

Engine Factory Method

To receive the data from your program, the Engine provides Engine.GetFileSystemItemAsync() virtual method, this is a faectory method that you must implement in your class and return file and folder items to the Engine. Your file and folder items must implement an IFile or IFolder interface to provide the data to the Engine. After the Engine.GetFileSystemItemAsync() call, the Engine calls methods of the IFile and IFolder interfaces on the returned item to get the data and complete operations. 

Your typical Engine.GetFileSystemItemAsync() method implementation will look like the following:

public override async Task<IFileSystemItem> GetFileSystemItemAsync(string path)
{
    if(File.Exists(path))
    {
        return new VfsFile(path);
    }
    if(Directory.Exists(path))
    {
        return new VfsFolder(path);
    }

    return null; 
}

Implementing Remote Storage to User File System Synchronization

Because the file system hierarchy, as well as the amount of data stored in files in your document management system, can be very large, the platform and the Engine provides on-demand loading mechanisms. The on-demand loading means that the folder content is created only when an application is listing folder content for the first time. Until then, the content of the folder does not exist on your disk. In a similar manner, the file content is loaded on the first application access. 

To support the on-demand loading, the Engine (as well as underlying Cloud Filter API) provides mechanisms for detecting initial folder listing and initial file content reading and provides means of separating it from subsequent updates. 

Initial folder content listing.

When any application is listing folder content, the Engine calls Engine.GetFileSystemItemAsync() factory method, requesting a folder item, and then calls IFolder.GetChildrenAsync() method on the returned item. Inside your IFolder.GetChildrenAsync() implementation you will list content of your remote storage and return information about each file and folder to the Engine: 

public async Task GetChildrenAsync(string pattern, IOperationContext operationContext, IFolderListingResultContext resultContext)
{
    // This method has a 60 sec timeout. 
    // To process longer requests and reset the timout timer call one of the following:
    // - resultContext.ReturnChildren() method.
    // - resultContext.ReportProgress() method.

    IEnumerable<FileSystemItemBasicInfo> children = await new UserFolder(UserFileSystemPath).EnumerateChildrenAsync(pattern);

    // To signal that the children enumeration is completed 
    // always call ReturnChildren(), even if the folder is empty.
    resultContext.ReturnChildren(children.ToArray(), children.Count());
}

To return the list of files and folders you will call the IFolderListingResultContext.ReturnChildren() method. The object implementing the IFolderListingResultContext is passed as the third parameter of IFolder.GetChildrenAsync() call.

Note that you can call the ReturnChildren() method multiple times inside the GetChildrenAsync() call, returning the list of files and folders in several turns. To specify the total amount of children, the ReturnChildren() method provides a second parameter, so the platform knows when children enumeration is completed.

The IFolder.GetChildrenAsync() method as well as IFile.TransferDataAsync(), IFileSystemItem.DeleteAsync() and IFileSystemItem.MoveToAsync() has a 60 sec timout. To extend the timeout and reset the timer, call the ReturnChildren() method. You can also call the ReportProgress() method, however it will only reset the timer, the ReportProgress() call has no impact on progress display inside the GetChildrenAsync() call. 

Subsequent folder content updates

After the first call to IFolder.GetChildrenAsync(), this method is never called again for this particular folder. Instead, you will update folder content using pooling or push technologies, such as Web Sockets or any other.

Creating new items. When you receive information about a file or folder is created, call the PlaceholderFolder.CreatePlaceholders() method, passing an array of objects implementing IFileBasicInfo and IFolderBasicInfo:

// Because of the on-demand population the file or folder placeholder may not exist in the user file system.
// Here we also check that the folder content was loaded into user file system (the folder is not offline).
if (Directory.Exists(userFileSystemParentPath)
    && !new DirectoryInfo(userFileSystemParentPath).Attributes.HasFlag(System.IO.FileAttributes.Offline))
{
    IFileSystemItemBasicInfo newItemInfo = Mapping.GetUserFileSysteItemBasicInfo(remoteStorageItem);
    uint created = new PlaceholderFolder(userFileSystemParentPath).CreatePlaceholders(new []{newItemsInfo});
}

Updating items. When a file or folder is being updated in your remote storage, to apply changes in the user file system, call the PlaceholderItem.SetItemInfo() method in your client application. This method will update file creation and modification dates, update file size, attributes, and custom data associated with the item. If the file is hydrated, it will also automatically update file content triggering IFile.TransferDataAsync() call. 

// Because of the on-demand population the file or folder placeholder may not exist in the user file system.
if (FsPath.Exists(userFileSystemPath))
{
    IFileSystemItemBasicInfo itemInfo = Mapping.GetUserFileSysteItemBasicInfo(remoteStorageItem);
    PlaceholderItem placeholderItem = PlaceholderItem.GetItem(userFileSystemPath);

    // Dehydrate/hydrate the file, update file size, custom data, creation date, modification date, attributes.
    placeholderItem.SetItemInfo(itemInfo);
}

Deleting items. When a file or folder is being deleted in the remote storage just delete the file or folder placeholder as you usually do with a regular file or folder, there is no any specific API exists for deletion:  

if (FsPath.IsFile(userFileSystemPath))
{
    File.Delete(userFileSystemPath);
}
else
{
    Directory.Delete(userFileSystemPath, true);
}

Moving/renaming items. The same thing is with the move and rename. You will simply move or rename a file in your user file system using a regular file API. However, after moving a file it will be marked as not in-sync. You can set the in-sync status using the PlaceholderItem.SetInSync() method call.

// Because of the on-demand population, the file or folder placeholder may not exist in the user file system.
if (FsPath.Exists(userFileSystemPath))
{
        Directory.Move(userFileSystemPath, userFileSystemNewPath);
        // The file is marked as not in sync after move/rename. Marking it as in-sync.
        PlaceholderItem.GetItem(userFileSystemNewPath).SetInSync(true);
}

 

Initial file content population (hydration)

Files created as a result of the IFolder.GetChildrenAsync() or PlaceholderFolder.CreatePlaceholders() calls initially do not have any content on disk (dehydrated). They are marked with offline attributes and have a cloud icon  in the Windows File Manager:

When an application is opening a file for reading or writing, the Engine calls IFile.TransferDataAsync() method. In your implementation you will load the file content from your remote storage and pass it to the Engine:

public async Task TransferDataAsync(long offset, long length, ITransferDataOperationContext operationContext, ITransferDataResultContext resultContext)
{
    // This method has a 60 sec timeout. 
    // To process longer requests and reset the timout timer call resultContext.ReportProgress() method.

    long optionalLength = length + operationContext.OptionalLength;
    byte[] buffer = await new UserFile(UserFileSystemPath).ReadAsync(offset, optionalLength);
    resultContext.ReturnData(buffer, offset, optionalLength);
}

The TransferDataAsync() method passes the offset in bytes in file content to start reading from and a length of the requested file content as the first two parameters. In the operationContext it also provides an optional bigger size of data that you can read from your remote storage and return it to the Engine, in case this is more convenient for you. In the above example, the bigger chunk of data is being read and returned to the Engine.

To return the data to the Engine, call the ReturnData() method of the ITransferDataResultContext interface passed as the resultContext parameter. Return the data along with the offset in the file content and length of data.

The TransferDataAsync() method has a 60 sec timeout. To extend the timeout and reset the timer, call the ReportProgress() method. In case the download takes more that 1 second, the ReportProgress() method call will also show the progress in Windows File manager in the Status column and in the Taskbar area:

When the download is completed the file icon turns into a checkbox on a white background , meaning the file is on disk and is in the in-sync state:

Note that such a file can be purged from the file system in case there is not enough space on the disk. To avoid this, the user can "pin" the file by calling the "Always keep on this device" menu in Windows File Manager.

To force the file hydration call the PlaceholderFile.Hydrate() method. 

Subsequent file content updates

When a file content is modified in your remote storage, you will notify the client about the change or will pull the remote storage to get the changes. The most simple way to update the file is to call the PlaceholderItem.SetItemInfo() as described above in the Implementing Remote Storage to User File System Synchronization section. If the item is hydrated it will automatically download the updated content:

IFileSystemItemBasicInfo itemInfo = Mapping.GetUserFileSysteItemBasicInfo(remoteStorageItem);
PlaceholderItem.GetItem(userFileSystemPath).SetItemInfo(itemInfo);

This call will also leave the item in the in-sync state (the file will be marked with ,  or ). However, the disadvantage of this approach is that the file will be blocked for the time of new content download.

Alternatively, you can download a file into a temporary location and then update the file in the user file system using a regular file API. This reduces the time of the file being blocked/unavailable but consumes extra space on disk for a temporary file. It also makes the update more complex, because, at the time when the content is ready to be updated, the file may be blocked for writing by the client application. Finally, when the update is completed, the file will be marked as not in-sync . To mark it as in-sync you will call the PlaceholderItem.SetInSync() method:

PlaceholderItem.GetItem(userFileSystemPath).SetInSync(true);

Implementing User File System to Remote Storage Synchronization

To process changes in the user file system and apply them in your remote storage you will combine detecting changes in IFile/IFolder interfaces with file system monitoring. File and folders move/rename and delete can be handled inside the IFileSystemItem.MoveToAsync() and IFileSystemItem.DeleteAsync() methods. File content updates can be handled inside the IFile.CloseAsync() or by monitoring the file system for changes. Files and folders creation, as well as folders updates, can be handled only using file system monitoring. 

Creating items. The initial files and folders creation in the user file system is not handled by the Engine or Cloud filter API. To find items that are being created in your user file system you will typically monitor the file system using FileSystemWatcher.

When a file or folder is created in the user file system it is created as a regular file or folder. When a file is being overwritten, it also being converted from a placeholder into a regular file. In most cases, you need to convert such files and folders into placeholders. To convert to placeholder call one of the  PlaceholderItem.ConvertToPlaceholder() overloaded methods. When converting you can specify if the file should be marked as in-sync or not in-sync, be dehydrated as well as pass some custom data to be stored with the placeholder. To detect if the item is a regular file/folder or a placeholder file/folder call the PlaceholderItem.IsPlaceholder() static method:

if (!PlaceholderItem.IsPlaceholder(userFileSystemPath))
{
    // Convert regular file/folder to placeholder. 
    // The file/folder was created or overwritten.
    PlaceholderItem.ConvertToPlaceholder(userFileSystemPath, false);
}

Updating items. 

When an application completes file modification and the file handle is closed, the Engine calls IFile.CloseAsync() method. This is the earliest time when you can get information about the file being modified.

To check if the file or folder is modified you can use the PlaceholderItem.GetInSync() method. This method returns false if file/folder attributes have changed, creation or modification date has changed or file content has changed. You can configure when registering the sync root using  StorageProviderSyncRootInfo.InSyncPolicy property.

To detect if the file content is modified you can use the PlaceholderFile.GetFileDataSizeInfo() method, which returns information about the number of bytes on disk, number of bytes in-sync with the remote storage, and not in-sync with the remote storage.

Important! The In-Sync status indicates if the file or folder is being modified on the client, in the user file system. It does NOT indicate if the file is modified on the server.

public async Task CloseAsync(IOperationContext operationContext, IResultContext context)
{
    // Here, if the file in the user file system is modified (not in-sync), you will send the file content, 
    // creation time, modification time and attributes to the remote storage.
    // We also send ETag, to make sure the changes on the server, if any, are not overwritten.

    string userFileSystemFilePath = UserFileSystemPath;

    // In case the file is moved it does not exist in user file system when CloseAsync() is called.
    if (FsPath.Exists(userFileSystemFilePath)
        && !FsPath.AvoidSync(userFileSystemFilePath))
    {

        // In case the file is overwritten it is converted to a regular file prior to CloseAsync().
        // we need to convert it back into file/folder placeholder.
        if (!PlaceholderItem.IsPlaceholder(userFileSystemFilePath))
        {
            PlaceholderItem.ConvertToPlaceholder(userFileSystemFilePath, false);
            Logger.LogMessage("Converted to placeholder", userFileSystemFilePath);
        }

        if (!PlaceholderItem.GetItem(userFileSystemFilePath).GetInSync())
        {
            Logger.LogMessage("Item modified", userFileSystemFilePath);

            try
            {
                await new RemoteStorageItem(userFileSystemFilePath).UpdateAsync();
                string remoteStorageFilePath = Mapping.MapPath(userFileSystemFilePath);
                Logger.LogMessage("Updated succesefully", remoteStorageFilePath);
            }
            catch (IOException ex)
            {
                // Either the file is already being synced in another thread or client or server file is blocked by concurrent process.
                // This is a normal behaviour.
                // The file must be synched by your synchronyzation service at a later time, when the file becomes available.
                Logger.LogMessage("Failed to upload file. Possibly in use by an application or blocked for synchronization in another thread:", ex.Message);
            }
        }
    }
}

Note that in many cases your file will be still blocked by the application and you will not be able to read its content. In this case, you must sent file content in some additional client-to-server synchronization service.

Deleting items. When a file or folder is deleted in the user file system the Engine calls IFileSystemItem.DeleteAsync() method:

public async Task DeleteAsync(IOperationContext operationContext, IConfirmationResultContext resultContext)
{
    Logger.LogMessage("IFileSystemItem.DeleteAsync()", this.UserFileSystemPath);

    string userFileSystemPath = this.UserFileSystemPath;
    string remoteStoragePath = null;
    try
    {
        remoteStoragePath = Mapping.MapPath(userFileSystemPath);

        if (!FsPath.AvoidSync(userFileSystemPath))
        {
            await new RemoteStorageItem(userFileSystemPath).DeleteAsync();
            Logger.LogMessage("Deleted succesefully", remoteStoragePath);
        }
    }
    catch (Exception ex)
    {
        // remove try-catch when error processing inside CloudProvider is fixed.
        Logger.LogError("Delete failed", remoteStoragePath, null, ex);
    }
    finally
    {
        resultContext.ReturnConfirmationResult();
    }
}

Moving/renaming items. When a file or folder is moved or renamed in the user file system the Engine calls IFileSystemItem.DeleteAsync() method:

public async Task MoveToAsync(string userFileSystemNewPath, IOperationContext operationContext, IConfirmationResultContext resultContext)
{
    string userFileSystemOldPath = this.UserFileSystemPath;
    Logger.LogMessage("IFileSystemItem.MoveToAsync()", userFileSystemOldPath, userFileSystemNewPath);

    try
    {
        if (FsPath.Exists(userFileSystemOldPath))
        {
            await new RemoteStorageItem(userFileSystemOldPath).MoveToAsync(userFileSystemNewPath, resultContext);                    
            string remoteStorageOldPath = Mapping.MapPath(userFileSystemOldPath);
            string remoteStorageNewPath = Mapping.MapPath(userFileSystemNewPath);
            Logger.LogMessage("Moved succesefully", remoteStorageOldPath, remoteStorageNewPath);
        }
    }
    finally
    {
        resultContext.ReturnConfirmationResult();
    }
}

 

Pinned and Unpinned Files and Folders