> ## Documentation Index
> Fetch the complete documentation index at: https://ayakaleaf-pro.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# S3 migration

## S3 migration

<Info>
  These instructions are for v5.x and later. If you are following this guide for an earlier version please use `sharelatex` instead of `overleaf` in path names and `SHARELATEX_` prefix instead of `OVERLEAF_` for environment variables. For v6 and later, skip the legacy `user_files` commands.
</Info>

<Check>
  <strong>We'd love to hear from you!</strong> If you'd like to share with us how many files you migrated over, their overall volume, and how long the migration took email [`ayaka-notes@outlook.com`](mailto:support@overleaf.com) .
</Check>

This guide will walk you through the migration from on-disk storage to an S3-compatible object storage. It refers to sections of the introduction document on the [S3 Setup](/on-premises/configuration/overleaf-toolkit/s3).

### Requirements

* A S3 compatible object storage to talk to, see [#s3-setup](/on-premises/configuration/overleaf-toolkit/s3#s3-setup "mention") for options
* Free disk space for migrating existing data, about the current on disk size
* A maintenance window for doing the actual migration
* A full backup, including the config, to enable restoring from it

### Estimate the disk size needed for the migration

We can use `du` for calculating the current disk usage:

```shell theme={null}
docker exec sharelatex \
  du --human-readable --max-depth=0 /var/lib/overleaf/data/user_files

docker exec sharelatex \
  du --human-readable --max-depth=0 /var/lib/overleaf/data/template_files
```

In case you do not have sufficient disk space available on the current server, try attaching another disk to the server.

<Info>
  The history directories already have the correct layout. You can upload directly from the bind-mounted source folder, which does not require any additional disk space.
</Info>

### Migration steps

#### Step 0 shutdown the instance

We need to make sure that all user/template files will get migrated. It is best to shut down the instance to avoid missing newly uploaded files.

Please see our guide on performing a consistent a backup for the shutdown procedure.

#### Step 1 rewrite directory layout

We need to rewrite the directory layout of project files for uploading them to S3. The directory layout for local storage in filestore is `<project-id>_<file-id>` and the directory layout in S3 is `<project-id>/<file-id>`.

In the following, `/srv/overleaf-s3-migration` is used for storing the files in the new directory layout. Replace `/srv/overleaf-bind-mount` with the host directory mounted at `/var/lib/overleaf`. Run the copy commands on the host with permission to read and write these directories; the container remains stopped.

We can make use of `tar` for rewriting the layout:

```shell theme={null}
mkdir -p /srv/overleaf-s3-migration/user_files \
         /srv/overleaf-s3-migration/template_files
tar --create --directory /srv/overleaf-bind-mount/data/user_files . \
| tar --extract --directory /srv/overleaf-s3-migration/user_files \
  --transform=sx_x/x
tar --create --directory /srv/overleaf-bind-mount/data/template_files . \
| tar --extract --directory /srv/overleaf-s3-migration/template_files \
  --transform=sx_x/xg
```

#### Step 2 upload the files

Depending on your preference, you can use the minio mc S3 client or the aws cli for uploading the files to your S3 compatible object storage.

**aws cli**

<Info>
  * Here you should replace `overleaf-user-files`, `overleaf-template-files`, `overleaf-project-blobs` and `overleaf-chunks` with the names of your S3 buckets.
  * Also replace `/srv/overleaf-bind-mount` with the local path of the `/var/lib/overleaf` bind-mount. By default, this is `~/overleaf_data` in a docker-compose.yml deployment and `<toolkit-checkout>/data/overleaf` when using the Toolkit.
</Info>

```shell wrap theme={null}
aws s3 sync /srv/overleaf-s3-migration/user_files s3://overleaf-user-files
aws s3 sync /srv/overleaf-s3-migration/template_files s3://overleaf-template-files

aws s3 sync /srv/overleaf-bind-mount/data/history/overleaf-project-blobs s3://overleaf-project-blobs
aws s3 sync /srv/overleaf-bind-mount/data/history/overleaf-chunks s3://overleaf-chunks
```

**minio mc**

We are using the server alias "s3" here, you may have picked another name.

```shell wrap theme={null}
mc mirror /srv/overleaf-s3-migration/user_files s3/overleaf-user-files
mc mirror /srv/overleaf-s3-migration/template_files s3/overleaf-template-files

mc mirror /srv/overleaf-bind-mount/data/history/overleaf-project-blobs s3/overleaf-project-blobs
mc mirror /srv/overleaf-bind-mount/data/history/overleaf-chunks s3/overleaf-chunks
```

#### Step 3 start the instance pointing at S3

Add all the S3 related variables to your config, as detailed in the [Overview of variables](/on-premises/configuration/overleaf-toolkit/s3#overview-of-variables) section in the [S3](https://docs.overleaf.com/on-premises/configuration/overleaf-toolkit/s3) setup guide.

Keep the data-directory bind-mount: it may also contain Zotero or Mendeley encryption keys that are not migrated to S3.

<Check>
  Please keep the [bind-mount of a scratch disk for ephemeral files](/on-premises/support/troubleshooting#running-overleaf-with-an-nfs-filesystem) in place.
</Check>

You can now start the instance and validate the migration:

* can preview binary files in the editor
* can compile a PDF with images
* can upload new files

### Rolling back

You can roll back the migration gracefully in reversing the steps:

1. Shutdown the instance
2. Mirror back the files by flipping the sequence of source/destination
3. Write new files back into the local directory using an inverse `transform`
4. Restart the instance with the old configuration

```shell wrap theme={null}
# When using aws cli
aws s3 sync s3://overleaf-user-files /srv/overleaf-s3-migration/user_files
aws s3 sync s3://overleaf-template-files /srv/overleaf-s3-migration/template_files
aws s3 sync s3://overleaf-project-blobs /srv/overleaf-bind-mount/data/history/overleaf-project-blobs
aws s3 sync s3://overleaf-chunks /srv/overleaf-bind-mount/data/history/overleaf-chunks

# When using minio mc
mc mirror s3/overleaf-user-files /srv/overleaf-s3-migration/user_files
mc mirror s3/overleaf-template-files /srv/overleaf-s3-migration/template_files
mc mirror s3/overleaf-project-blobs /srv/overleaf-bind-mount/data/history/overleaf-project-blobs
mc mirror s3/overleaf-chunks /srv/overleaf-bind-mount/data/history/overleaf-chunks
```

```shell theme={null}
# Write files into local Server CE/Server Pro
tar --create --directory /srv/overleaf-s3-migration/user_files . \
| tar \
      --extract \
      --keep-old-files \
      --directory /srv/overleaf-bind-mount/data/user_files \
      --transform=sx./xx --transform=sx/x_x \
      --wildcards '*/*/*'

tar --create --directory /srv/overleaf-s3-migration/template_files . \
| tar \
      --extract \
      --keep-old-files \
      --directory /srv/overleaf-bind-mount/data/template_files \
      --transform=sx./xx --transform=sx/x_xg \
      --wildcards '*/*/*/*/pdf-converted-cache/*' \
      --wildcards '*/*/*/*/pdf' \
      --wildcards '*/*/*/*/zip'
```

<Info>
  The first transform removes the top level folder. The 2nd transform changes the directory layout to a flat one. The wildcards ensure that only files are extracted, not their parent (project) folders.
</Info>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.