How to download and upload all assets from a Beach instance

Download Flow resources from the instance's cloud storage for local development.

 
 Beach

Overview

This guide explains how you can download all photos, documents and other assets store in a Beach instance to your local machine, or upload these assets from your local installation to a Beach instance.

You will need the following before starting with this tutorial:

  • a running Beach instance
  • working SSH access to your instance
  • an installation of the same Neos / Flow project on your local machine, using a fresh database dump from your Beach instance
  • Local Beach

Background Information

Your Beach instance is using Google Cloud Storage for storing and providing the persistent resources of the Neos site or Flow application. This means that the Data/Persistent/Resources folder of your instance is empty and you need to download the assets from the Google Cloud Storage bucket instead.

The difference between private and public buckets

Flow stores all asset originals in a "storage" and this storage contains original and generated assets of your website. In Flow terms, we are talking about "resources", which is basically another name for "files".

Resources contained in the storage are usually automatically published to a "target". You could use two different Google Cloud Storage buckets – one for the Flow Storage and one for the Flow Target – but since all resources are public anyway, we optimized the configuration so that only one bucket is needed.

Note that in older Beach projects, you may still have a "storage" and a "target" bucket instead of "private" and "public". Even though they are named differently, they work exactly like the "private" and "public" buckets.

Therefore, if you want to upload or download resources from a Beach instance, you usually only deal with the "public.*" bucket.

Resource information stored in the database

Neos and Flow also keep track of all resources in certain database tables. This is where metadata such as filenames are stored. Your application will always need these two components, the binary data from your storage and the metadata in the database, in order to function. So please make sure that you have a database state which matches the current state of your resources. Tools like ./flow resource:clean can help you to figure out if the database matches the actual set of resource files.

Steps for downloading your assets

We use a tool called "Local Beach" for downloading the resources. There's a dedicated guide about how to set up Local Beach, so if you didn't do this already, please follow the few steps mentioned there.

Next, figure out the instance identifier of your instance. You'll find it right at the top in the instance view in the Beach user interface. The next thing you need to find is the namespace your project runs in in our cluster. This can be found on the project page in the "General" block as "namespace", it can be "beach" for example.

When you have that, get back to the command line and navigate to your project's root directory. Now you're ready to download all the resources from the cloud to your local project:

beach resource-download --instance instance-7018519e-xxxx-yyyy-zzzz-a01cbbab0fc6 --namespace beach

Finally, make sure that the database of your local project corresponds to the database of your Beach instance, so your local Neos or Flow actually knows about all the resources it now has in Data/Persistent/Resources/

Note on older Beach instances (a.k.a. older Flow/Neos setups)

There is something that needs to be taken into account for older Beach instances (respectively all those not using subdivied paths for asset storage for some reason).

If, after downloading your assets, Flow claims it cannot find those assets locally, the reason probably is that they were downloaded into hashed directories locally (as in Data/Persistent/Resources/a/b/abcde…) but Flow expects them to be all in the same folder. This is caused by the fact that the default value for subdivideHashPathSegment has been changed from false to true for Flow 6.0 (see PR #1689), and the download command assumes the current default.

This can be fixed in multiple ways:

  • You can move all the files from their subfolders to the top-level Resources folder
  • You can set the subdivideHashPathSegment option to true:
Neos:
  Flow:
    resource:
      targets:
        localWebDirectoryPersistentResourcesTarget:
          targetOptions:
            subdivideHashPathSegment: true

Steps for uploading your assets

Uploading works basically like downloading. Before you upload resources to your instance, please cross-check that you have copied the right instance identifier and project namespace. You don't want to mess up your production website by accident …

The upload command will copy the resources to your instance which meet both of the following criteria:

  • they exist in your local Data/Persistent/ directory
  • they have a corresponding entry in your resource database table of your Local Beach project

Resources are only added (copied), never deleted.

When you're ready, just start the upload as follows:

beach resource-upload --instance instance-7018519e-xxxx-yyyy-zzzz-a01cbbab0fc6 --namespace beach

The upload may set the MIME type of the uploaded assets to application/octet-stream, which could lead to unexpected results (e.g. PDF files being downloaded, instead of displayed in the browser.) To fix this, you can use the following command after uploading.

Make sure to run it from the Beach instance you uploaded to after having logged in to that instance via SSH!

./flow gcs:updateresourcemetadata