Sync ID Incoming Synchronization

In this article

Sync ID Incoming Synchronization

In this article we describe a cross-platform remote storage to user file system synchronization (incoming synchronization) approach based on Sync ID algorithm. 

After the initial on-demand population and hydration the folder listing and file content download never happens again (on Windows), or happens occasionally (on macOS). To apply changes made in your remote storage in the user file system, you will typically implement push notifications from your remote storage to clients machines, via web sockets or similar technologies. The Engine provides Sync ID algorithm support that allows you to request all changes that happened in the remote storage since provided sync token using a single request. In this article we describe how to request changes from your remote storage after you receive a change notification.

The functionality described in this section is supported in User File System v6.3 and later versions. 

You can find sample synchronization implementation in the following User File System samples:

Remote Storage Implementation

Requirements

To support Sync ID algorithm your remote storage must support the following functionality:  

  • Each item in your storage must have a remote storage ID associated with it, unique withing your file system. The ID should NOT change during lifetime of an item including during move operation.
  • Your remote storage must be able to get all changes that happened since provided sync-token. The Engine will request from your remote storage all items that changed and deleted (changed means created, updated or moved).
  • For each changed and deleted item your remote storage must provide a parent folder remote storage ID.
  • Your remote storage must be able to return item URI by remote storage item ID. This is required on macOS only.

Implementation Guides

You can find how to implement synchronization based on Sync ID in IT Hit WebDAV servers in the following articles: 

Demo and Testing Sites

The following demo sites provide Sync ID synchronization support, you can use them for demo and testing proposes. Note that you must navigate to the demo site in a web browser and in your code connect to the folder automatically created for you for testing purposes, which looks like https://demosite/User123456/  

Server Samples

The following sample WebDAV servers provide Sync ID synchronization support:

Synchronization Settings

In this mode you will set the IncomingSyncMode to IncomingSyncMode.SyncId:

engine.SyncService.IncomingSyncMode = IncomingSyncMode.SyncId;

Remote Storage Item ID Implementation

To support Sync ID algorithm you need a proper remote storage ID implementation. Here is how you implement the ID in your application:

Application Start Sequence

Here is your application startup sequence:

  1. The Engine initializes sync-token by sending a request to the remote storage. This step completes before file system calls processing is started.
  2. The Engine starts processing file system calls: listing folders and hydrating files.
  3. Your implementation starts receiving incoming notifications from remote storage and triggers synchronization by calling the IServerCollectionNotifications.ProcessChangesAsync() method.

Processing file system calls and receiving server notifications can run simultaneously and the same data may be received and processed in parallel, this is a normal synchronization lifecycle.

Detecting Sync-ID Algorithm Support

To detect if your implementation supports Sync ID algorithm, the Engine will first get your root folder by calling IEngine.GetFileSystemItemAsync() method and test the returned object  for ISynchronizationCollection interface support. If this interface is supported on your root folder it will start Sync-Token initialization.

Sync-Token initialization

To avoid any data loss during synchronization, the sync-token must be initialized before the Engine begins using any on-demand loading mechanisms, such as populating folders or hydrating files. To get the sync-token, the Engine calls the ISynchronizationCollection.GetChangesAsync() on the root folder passing 0 as a limit parameter. Setting the limit parameter to zero, indicates that the Engine does not need any changes from your remote storage, but instead requires a sync-token only.

Triggering Synchronization

You will typically start the remote storage to user file system synchronization after receiving a signal about changes from your remote storage via web sockets or similar communication technology. 

To start the synchronization you will call the IServerCollectionNotifications.ProcessChangesAsync() method. This will trigger the ISynchronizationCollection.GetChangesAsync() method call. 

The IServerCollectionNotifications interface is returned by the Engine from IEngine.ServerNotifications() method:

IServerCollectionNotifications n = await Engine.ServerNotifications(Engine.Path);
await n.ProcessChangesAsync();

In addition to that, on macOS, you may run web sockets inside your host application. As soon as host application runs in the separate process and does not have access to the Engine instance, the library on macOS provides ServerNotifications class that implements IServerCollectionNotifications interface that can be instantiated directly, without the Engine instance:

var fileProviderManager = NSFileProviderManager.FromDomain(domain);
IServerCollectionNotifications n = new ServerNotifications(fileProviderManager);
await n.ProcessChangesAsync();

Getting Changes from Remote Storage

To get changes from your remote storage, implement the ISynchronizationCollection interface on your root folder. This interface provides a single GetChangesAsync() method in which you will request all changes made in your remote storage since provided sync-token and return it to the Engine:

public class VirtualFolder : VirtualFileSystemItem, IFolder, ISynchronizationCollection
{
    ...
    public async Task<IChanges> GetChangesAsync(
        string syncToken, 
        bool deep, 
        long? limit, 
        CancellationToken ct)
    {
        Changes changes = new Changes();

        Client.PropertyName[] propNames = new Client.PropertyName[2];
        propNames[0] = new Client.PropertyName("resource-id", "DAV:");
        propNames[1] = new Client.PropertyName("parent-resource-id", "DAV:");

        // Get all changed and deleted items from the remote storage.
        Client.IChanges davChanges = await Session.GetChangesAsync(
            RemoteStorageID, 
            propNames, 
            syncToken, 
            deep, 
            limit);

        // A new sync token will be saved on the client machine.
        changes.NewSyncToken = davChanges.NewSyncToken;

        foreach (Client.IChangedItem remoteStorageItem in davChanges)
        {
            IFileSystemItemMetadata itemInfo = Mapping.GetUserFileSystemItemMetadata(
                remoteStorageItem);
            Change changeType;
            if (remoteStorageItem.ChangeType == Client.Change.Changed)
            {
                changeType = Change.Changed;
            }
            else
            {
                changeType = Change.Deleted;
            }
            ChangedItem changedItem = new ChangedItem(changeType, itemInfo);
            changes.Add(changedItem);
        }

        return changes;
    }
}

The Engine will process the list of changed and deleted items and apply it to the file system. It will also store the sync-token until the next GetChangesAsync() call.

In case the Engine fails to update any items returned by the GetChangesAsync() call, for example because they were blocked by applications or by the platform, they will be set to the conflict state and will be marked with a conflict icon.

How Changes are Applied in User File System

The Engine will go through the changes list returned from GetChangesAsync() method, one by one and will process each ChangedItem item based on the Change flag. Note that the Change.Changed flag indicates created, moved and updated items.

Deleted items

If the ChangedItem item is marked with the Change.Deleted flag the Engine will read the ChangedItem.Metadata.RemoteStorageItemId property and than find this item in user file system and delete it by calling IServerCollectionNotifications.DeleteAsync() method.

Created Items

If the item is marked with the Change.Changed the Engie will search for the item in user file system by ID specified in ChangedItem.Metadata.RemoteStorageItemId. If the item is not found, it will create an item by calling IServerCollectionNotifications.CreateAsync() method, passing the data provided in ChangedItem.Metadata property.

Moved Items

If the item is marked with with the Change.Changed and was found in user file system by by ChangedItem.Metadata.RemoteStorageItemId, the Engine will get parent folder in user file system and will read it's remote storage item ID. It will compare parent ID and the ID received in ChangedItem.Metadata.RemoteStorageParentItemId. If parent IDs do not match OR if the file names do not match, the item is considered moved. The Engine will move the item to the new parent or will rename the item by calling the IServerCollectionNotifications.MoveToAsync() method. Note that the name comparison is case insensitive. 

Updated Items

If the item is not deleted, created or moved, the Engine will perform an update by calling the IServerCollectionNotifications.UpdateAsync() method. During the update it will compare ChangedItem.Metadata.MetadataETag and ChangedItem.Metadata.ContentETag eTags with eTags stored with the item to detect if metadata should be updated and/or content should be re-downloaded. See Detecting Content & Metadata Changes article for more details.

 

See Also:

Next Article:

Folder Content Invalidation Mode