4. Support Hot Migration
Technical Story: [description | ticket/issue URL]
Migration is one of the most important topics of the oCIS story. We need to provide a concept how to migrate from oC10 to oCIS.
- Do not lose file blob or metadata.
- To prevent a sync surge from clients the etag for files should be migrated.
- To prevent internal links from breaking or pointing to wrong files the file id of existing files needs to be migrated.
- To prevent data loss trash and version blobs should be migrated.
- Existing shares like public links and federated shares must remain functional after the migration.
- To prevent internal shares the share type, permissions and expiry needs to be migrated.
- To prevent public links from breaking the url token, permissions, expiry and password needs to be migrated.
- What about federated shares?
- What about additional share permissions, e.g. comment on office files?
- Legacy clients need to keep working
- To keep existing clients working the
remote.php/webdavanddav/files/<username>webdav endpoints as well as the ocs API need to be available.
- To keep existing clients working the
- What about app passwords/tokens?
- Cold Migration: migrate data while systems are not online, so no user interaction happens in between.
- Hot Migration: one or both systems are online during migration.
Chosen option: “[option 1]”, because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)].
- [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …]
- …
- [e.g., compromising quality attribute, follow-up decisions required, …]
- …
The migration happens while the service is offline. File metadata, blobs and share data is exported from ownCloud 10 and imported in oCIS. This can happen user by user, where every user export would contain the file blobs, their metadata, trash, versions, shares and all metadata that belongs to the user’s storage. To prevent group shares from breaking, users in the same groups must be migrated in batch. Depending on the actual group shares in an instance this may effectively require a complete migration in a single batch.
- Good, because oCIS can be tested in a staging system without writing to the production system.
- Good, because file layout on disk can be changed to support new storage driver capabilities.
- Bad, because the export and import might require significant amounts of storage.
- Bad, because a rollback to the state before the migration might cause data loss of the changes that happened in between.
- Bad, because the cold migration can mean significant downtime.
The migration happens in subsequent stages while the service is online.
- Good, because the admin can migrate users from old to new backend in a controlled way.
- Good, because users and admins can learn to trust the new system.
- Good, because there can be preparations even long before the migrations happens in parallel on the oC10 codebase, i.e. addition of metadata that is needed while the system operates.
- Good, because the downtime of the system can be fairly small.
- Bad, because it is more complex and might drag on for a long time.
- Clarify responsibilities of share providers and storage providers · Issue #1377 · cs3org/reva (github.com) because the share manager for oCIS should store share information on the storage system. And storage provider should persist share creator · Issue #93 · cs3org/cs3apis (github.com) finally: eos: store share id in inherited xattr · Issue #543 · cs3org/reva (github.com)