How to deploy

REQUISITES

To run the Orchestrator you need Docker and at least a MySQL Server instance (which may be local, remote, or in a docker container).

You can either run the image built accordingly to the previous chapter or run the image already pushed on Docker Hub indigodatacloud/orchestrator.

RUN THE CONTAINER

With MySQL dockerized on the same host

The Orchestrator can be run in 3 steps:

1. Run the MySQL deployments database with the command:

    docker run --name databaseOrchestrator -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=orchestrator -d mysql:5.7

2. Run the MySQL workflow database with the command:

    docker run --name databaseWorkflow -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=workflow -d mysql:5.7

3. Run the Orchestrator with the command:

    docker run --name orchestrator --link databaseWorkflow:databaseWorkflow \
        --link databaseOrchestrator:databaseOrchestrator -p 80:8080 -d indigodatacloud/orchestrator

With external databases

The Orchestrator can also be run using already deployed DBs; you just need to start it with the command

docker run --name orchestrator -e ORCHESTRATOR_DB_ENDPOINT=DOMAIN_NAME:PORT -e ORCHESTRATOR_DB_NAME=SCHEMA_NAME \
    -e ORCHESTRATOR_DB_USER=DB_USER -e ORCHESTRATOR_DB_PWD=DB_USER_PASSWORD  \
    -e WORKFLOW_DB_ENDPOINT=DOMAIN_NAME:PORT -e WORKFLOW_DB_NAME=SCHEMA_NAME -e WORKFLOW_DB_USER=DB_USER \
    -e WORKFLOW_DB_PWD=DB_USER_PASSWORD -p 80:8080 -d indigodatacloud/orchestrator

replacing the parameters DOMAIN_NAME, PORT, SCHEMA_NAME, DB_USER, DB_USER_PASSWORD with the correct values.

Required ports and network reachability

The Orchestrator Docker image exposes the 8080 TCP port; please remember to 1. Publish the exposed port to one of the host 2. Allow inbound connections to the published port of the host from all the IP addresses

CONFIGURATION

This is the list of additional parameters that allows to configure the orchestrator behavior.

• ORCHESTRATOR_URL

  • Description: Self reference to the Orchestrator REST interface

  • Format: http://{host}:{port}

  • Default value: http://localhost:8080

• IM_URL

  • Description: The Infrastructure Manager REST endpoint

  • Format: http://{host}:{port}

  • Default value: https://servproject.i3m.upv.es:8811

• CMDB_URL

  • Description: The CMDB REST endpoint

  • Format: http://{host}:{port}/cmdb

  • Default value: http://indigo.cloud.plgrid.pl/cmdb

• SLAM_URL

  • Description: The SLAM REST endpoint

  • Format: http://{host}:{port}/slam

  • Default value: http://indigo.cloud.plgrid.pl/slam

• CPR_URL

  • Description: The Cloud Provider Ranker endpoint

  • Format: https://{host}:{port}

  • Default value: https://indigo-paas.cloud.ba.infn.it/cpr

• MONITORING_URL

  • Description: The Zabbix Wrapper endpoint

  • Format: http://{host}:{port}

  • Default value: http://90.147.170.181

Import self-signed certificates (optional)

The Orchestrator supports connections to external services with self-signed certificates. In order to import these certificates you need to: 1. If you doesn't have them already, generate the PEM encoded public certificates from your private key certificates. 2. Copy the PEM encoded public certificates info a folder; this directory must contain only the certificates that you want to import 3. Run the orchestrator with the folder mounted as a volume; the mount point must be /orchestrator/trusted_certs

⚠️ The certificates are re-imported at every Orchestrator restart.

Configure IAM integration (optional)

By default the REST APIs are not authenticated; if you want to enable the IAM integration you must (for each IAM you want to associate):

1. Register the Orchestrator on IAM as protected resource server with

   1. HTTP Basic as Token Endpoint Authentication Method

   2. openid, profile and offline_access as scopes

   3. urn:ietf:params:oauth:grant-type:token-exchange as additional grant type

   4. Expiration time for the authorization token must be set to 3600 seconds

   5. Expiration time for the id token must be set to 1800 seconds

   6. Expiration info for the exchanged token must not be disabled

2. Retrieve the client id and the client secret

3. Retrieve the issuer value of the IAM from its WebFinger endpoint https://{iam-url}/.well-known/openid-configuration

4. Provide a file called application.yml and mount it (via docker bind-mounting) on /orchestrator/application.yml. Inside the file you need to provide the following configuration:

     oidc:
      enabled: true
      iam-properties:
        "[{issuer}]":
          orchestrator:
            client-id: '{client-id}'
            client-secret: '{client-secret}'
            scopes: [ "openid", "profile", "email", "offline_access" ]
          clues:
            client-id: '{client-id}'
            client-secret: '{client-secret}'
          audience: '{audience-uuid}'
          admingroup: '{admin-group}'

   with, as parameters

   • oidc.enabled

     • Description: Determines if the OAuth2 authentication and authorization is enabled

     • Format: true or false

     • Default value: false

   • {issuer}

     • Description: The issuer value of the IAM to which the orchestrator has been registered

     • Default value: https://iam-test.indigo-datacloud.eu/

   • orchestrator.client-id

     • Description: The Orchestrator OAuth2 client ID

   • orchestrator.client-secret

     • Description: The Orchestrator OAuth2 client secret

   • orchestrator.scopes (Optional)

     • Description: A list of OAuth2 scopes that will be used for token exchange.

     • Default value: [ "openid", "profile", "offline_access", "fts:submit-transfer" ]. Note: the scope fts:submit-transfer is required only for Rucio integration.

   • audience

     • Description: A freely generated string (e.g. a UUID) used to exchange tokens with Vault Server

   • admingroup

     • Description: The name of the group of users with administrator privileges

     • Default value: orchestrator-admins

5. If you have a Clues client registered in the IAM, you can configure the following parameters:

   • clues.client-id

     • Description: The CLUES OAuth2 client ID

   • clues.client-secret

     • Description: The CLUES OAuth2 client secret

Please make reference to the IAM guide for further information on how to register the Orchestrator as protected resource server.

⚠️ Even if the authentication is optional and disabled by default, you are highly encouraged to enable it, otherwise you will not be able to create deployments neither on OpenStack nor on OpenNebula.

Configure OneData (optional)

The Orchestrator, when the Chronos parameters are set, allows to exploit a OneData service space. This enables the users to execute tasks on Chronos that use temporary files hosted on a shared OneData space.

To enable this functionality you need to configure the following parameters:

• ONEDATA_ONEZONE_URL

  • Description: The endpoint of the default OneZone to use

  • Format: http://{host}:{port}

  • Default value: https://onezone-beta.cloud.ba.infn.it

• ONEDATA_SERVICE_SPACE_ONEZONE_URL

  • Description: (Optional) The OneZone to use when dealing with OneData service space

  • Format: {host}:{port}

  • Default value: The OneZone endpoint defined in $ONEDATA_ONEZONE_URL

• ONEDATA_SERVICE_SPACE_TOKEN

  • Description: The OneData service space token; you can retrieve it from the OneZone user interface

• ONEDATA_SERVICE_SPACE_NAME

  • Description: The name of the OneData space that you want to use as service space

  • Default value: INDIGO Service Space

• ONEDATA_SERVICE_SPACE_BASE_FOLDER_PATH

  • Description: The path (relative to the space one) to the folder that will host the files

  • Default value: /

Configure Vault (optional)

The Orchestrator allows, during the deployment of a Marathon service, to use a Vault server for storing and exchanging passwords and other sensitive data through the use of 'secrets'. This functionality is optional; to enable it, you have to configure the Vault server endpoint using the following parameter:

• VAULT_URL

  • Description: The Vault Service endpoint (if used)

  • Format: http://{host}:{port}

  • Default value: none

  • Example value: https://vault.cloud.ba.infn.it:8200

• VAULT_ROLE

  • Description: The jwt role (defined by the Vault admin) that the orchestrator will use to authenticate in Vault

  • Default value: none

The Vault server has to provide the following configurations:

• Key/Value Secrets Engine Version 1 enabled on path secret/;

• JWT auth backend enabled and configured with IAM;

• jwt role defined with the Orchestrator audience included in the bound_audiences;

• policy which grants write capabilities to the path secret/private.