Add script to create new beamtime

@marc-olivier.andrez

Script added here create a new token and a folder structure for a new beamtime (name given from parameter).

The idea is to start container with mounter path to the token folder and data folder and given beamtime.

asapo_dir="$(mktemp -d)"
# asapo_dir=/home/karnem
# Start the Docker image
docker run -d --name asapo --network host  -e BEAMTIME=my_beamtime -v "$asapo_dir"/logs:/asapo/logs/ -v "$asapo_dir"/shared:/shared -v "$asapo_dir"/btdata:/data gitlab.desy.de:5555/asapo/asapo/asapo-standalone
cat "$asapo_dir"/logs/my_beamtime.token

If BEAMTIME variable is not defined, a default_beamtime is used. Script can be executed multiple times (for different beamtimes) after start of the docker.

# Create the token
docker exec asapo /create_token.sh my_beamtime2
cat "$asapo_dir"/logs/my_beamtime2.token

Script is included in the docker stating script and creates a beamtime ``

The script create following folder-structure for mounting:

• /data/offline/${beamtimeId} contains process data and raw sent with ingest mode: INGEST_MODE_WRITE_RAW_DATA_TO_OFFLINE_FS
• /data/online/${beamtimeId} contains raw data sent with default ingest mode
• /shared contains tokens for beamtimes

In additional to folder /asapo with structure:

• /asapo/logs/ - asapo logs
• /asapo/influxdb/ and /asapo/mongodb/ - influxdb and mongodb. Can be used to restart container with losses of asapo statistics and metadata.

There are some features to discuss:

A secret key is used in the docker is always the same. This can be changed by setting a secret key as parameter by running the container. This can be implemented by adding script that will be started after docker run. This script will create a file with secret key (given as parameter) and then start asapo, that will use this key.

Description of authentication approach would explain suggested script:

• ASAPO developed with respect to DESY data-storage infrastructure and offers two different passes to store data: "OnlinePath" and "OfflinePath". Names refers to the DESY infrastructure, but for ASAPO they are just strings. Potentially different passes may have different hardware implementation and therefore have different IO speed.
• ASAPO have two different data types raw and processed. I guess this is done to follow DESY infrastructure, but differences between handling of them is very small.
  • raw file path starts with raw, while processed with processed.
  • Writing to raw requires access write writeraw (described later)
  • ASAPO have two different mounts, "OnlinePath" one for raw data and "OfflinePath" to processed.
• ASAPO token contain expiration data, subject: beamtimeId or beamlineId and access rights: read, write, writeraw.
  • Authorization procedure is different for different subject.
  • Beamtimes token is valid to access data for given beamtime, while Beamline for all beamtimes of given beamline. Token created for Beamtime didn't have writeraw rights, that is needed to write to raw. Now I change it.
• In DESY data-storage infrastructure "OfflinePath" is different for all beamtimes, but "OnlinePath" is the same for all beamtimes of a given beamline. "OnlinePath" exists only for one "current" beamtime. Therefore data can be saved in "raw" only for one "current" beamtime of a given beamline. It reflected in several features of asapo authorisation:
  • If token have writeraw rights, asapo tries to find the current beamtime. It checks for beamtime_metadata json file and read information from this file. BeamtimeId should coincides with one in credentials.
  • "OnlinePath" is defined in asapo as CurrentBeamlinesFolder/beamline/folder, where CurrentBeamlinesFolder is given in authorizer config, beamline is beamlineId string and folder is commissioning is BeamtimeId starts with c and current otherwise.
• There are two types of tokens beamtime and beamline. It connected with following features:
  • If given beamtimeId!="auto" corresponding token subject should be beamtimeId, otherwise beamlineId
  • In case of beamtimeId!="auto", beamlineId can be auto and will be taken from expected file structure. RootBeamtimesFolder/*/gpfs/*/*/*/beamtime_id.