How to deploy

REQUISITES

To run the Orchestrator you need Docker and at least a MySQL Server instance (which may be local, remote, or in a docker container).

You can either run the image built accordingly to the previous chapter or run the image already pushed on Docker Hub indigodatacloud/orchestrator.

RUN THE CONTAINER

With MySQL dockerized on the same host

The Orchestrator can be run in 3 steps:

1. Run the MySQL deployments database with the command:

    docker run --name databaseOrchestrator -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=orchestrator -d mysql:5.7

2. Run the MySQL workflow database with the command:

    docker run --name databaseWorkflow -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=workflow -d mysql:5.7

3. Run the Orchestrator with the command:

    docker run --name orchestrator --link databaseWorkflow:databaseWorkflow \
        --link databaseOrchestrator:databaseOrchestrator -p 80:8080 -d indigodatacloud/orchestrator

With external databases

The Orchestrator can also be run using already deployed DBs; you just need to start it with the command

docker run --name orchestrator -e ORCHESTRATOR_DB_ENDPOINT=DOMAIN_NAME:PORT -e ORCHESTRATOR_DB_NAME=SCHEMA_NAME \
    -e ORCHESTRATOR_DB_USER=DB_USER -e ORCHESTRATOR_DB_PWD=DB_USER_PASSWORD  \
    -e WORKFLOW_DB_ENDPOINT=DOMAIN_NAME:PORT -e WORKFLOW_DB_NAME=SCHEMA_NAME -e WORKFLOW_DB_USER=DB_USER \
    -e WORKFLOW_DB_PWD=DB_USER_PASSWORD -p 80:8080 -d indigodatacloud/orchestrator

replacing the parameters DOMAIN_NAME, PORT, SCHEMA_NAME, DB_USER, DB_USER_PASSWORD with the correct values.

Required ports and network reachability

The Orchestrator Docker image exposes the 8080 TCP port; please remember to 1. Publish the exposed port to one of the host 2. Allow inbound connections to the published port of the host from all the IP addresses

CONFIGURATION

This is the list of additional parameters that allows to configure the orchestrator behavior.

• ORCHESTRATOR_URL

  • Description: Self reference to the orchestrator REST interface

  • Format: http://{host}:{port}

  • Default value: http://localhost:8080

• IM_URL

  • Description: The Infrastructure Manager REST endpoint

  • Format: http://{host}:{port}

  • Default value: https://servproject.i3m.upv.es:8811

• CMDB_URL

  • Description: The CMDB REST endpoint

  • Format: http://{host}:{port}/cmdb

  • Default value: http://indigo.cloud.plgrid.pl/cmdb

• SLAM_URL

  • Description: The SLAM REST endpoint

  • Format: http://{host}:{port}/slam

  • Default value: http://indigo.cloud.plgrid.pl/slam

• CPR_URL

  • Description: The Cloud Provider Ranker endpoint

  • Format: https://{host}:{port}

  • Default value: https://indigo-paas.cloud.ba.infn.it/cpr

• MONITORING_URL

  • Description: The Zabbix Wrapper endpoint

  • Format: http://{host}:{port}

  • Default value: http://90.147.170.181

Configure IAM integration (optional)

By default the REST APIs are not authenticated; if you want to enable the IAM integration you must (for each IAM you want to associate):

1. Register the Orchestrator on IAM as protected resource server with

   1. HTTP Basic as Token Endpoint Authentication Method

   2. openid, profile and offline_access as scopes

   3. urn:ietf:params:oauth:grant-type:token-exchange as additional grant type

   4. Expiration time for the authorization token must be set to 3600 seconds

   5. Expiration time for the id token must be set to 1800 seconds

   6. Expiration info for the exchanged token must not be disabled

2. Retrieve the client id and the client secret

3. Retrieve the issuer value of the IAM from its WebFinger endpoint https://{iam-url}/.well-known/openid-configuration

4. Provide a file called application.yml and mount it (via docker bind-mounting) on /orchestrator/application.yml. Inside the file you need to provide the following configuration:

     oidc:
      enabled: true
      iam-properties:
        "[{issuer}]":
          orchestrator:
            client-id: '{client-id}'
            client-secret: '{client-secret}'
          clues:
            client-id: '{client-id}'
            client-secret: '{client-secret}'

   with, as parameters

   • oidc.enabled

     • Description: Determines if the OAuth2 authentication and authorization is enabled

     • Format: true or false

     • Default value: false

   • {issuer}

     • Description: The issuer value of the IAM to which the orchestrator has been registered

     • Default value: https://iam-test.indigo-datacloud.eu/

   • orchestrator.client-id

     • Description: The Orchestrator OAuth2 client ID

   • orchestrator.client-secret

     • Description: The Orchestrator OAuth2 client secret

5. Additionally, if you have a Clues client registered in the IAM, you can configure the following parameters:

   • clues.client-id

     • Description: The CLUES OAuth2 client ID

   • clues.client-secret

     • Description: The CLUES OAuth2 client secret

Please make reference to the IAM guide for further information on how to register the Orchestrator as protected resource server.

:warning: Even if the authentication is optional and disabled by default, you are highly encouraged to enable it, otherwise you will not be able to create deployments neither on OpenStack nor on OpenNebula.

Configure Chronos and Marathon (optional)

The orchestrator can both run jobs on Chronos and managed applications on Marathon; to do that you need to provide a file called application.yml and mount it (via docker bind-mounting) on /orchestrator/application.yml. Inside the file you need to provide the following configuration:

mesos:
  instances:
    "{cloud-provider-id}":
      chronos:
        url: '{chronos-endpoint}'
        username: '{chronos-username}'
        password: '{chronos-password}'
        local-volumes-host-base-path: '{base-path}'
      marathon:
        url: '{marathon-endpoint}'
        username: '{marathon-username}'
        password: '{marathon-password}'
        local-volumes-host-base-path: '{base-path}'
        load-balancer-ips: {marathon-lb-ips}

with, as parameters

• {cloud-provider-id}

  • Description: The Cloud Provider that hosts the Mesos cluster on which it runs Chronos; it should be the same id that's returned from the CMDB service

• chronos.url

  • Description: The Chronos REST endpoint

  • Format: http://{host}:{port}

• chronos.username

  • Description: The Chronos username

• chronos.password

  • Description: The Chronos password

• chronos.local-volumes-host-base-path

  • Description: (Optional) The host path on which the jobs' local volumes will be mounted. If not provided support for local volumes will be disabled. If provided, it must start with /

• marathon.url

  • Description: The Marathon REST endpoint

  • Format: http://{host}:{port}

• marathon.username

  • Description: The Marathon username

• marathon.password

  • Description: The Marathon password

• marathon.local-volumes-host-base-path

  • Description: (Optional) The host path on which the applications' local volumes will be mounted. If not provided support for local volumes will be disabled. If provided, it must start with /

• marathon.load-balancer-ips

  • Description: The list of Marathon LB IPs

Configure OneData (optional)

The Orchestrator, when the Chronos parameters are set, allows to exploit a OneData service space. This enables the users to execute tasks on Chronos that use temporary files hosted on a shared OneData space.

To enable this functionality you need to configure the following parameters:

• ONEDATA_ONEZONE_URL

  • Description: The endpoint of the default OneZone to which your OneData user is registered

  • Format: http://{host}:{port}

  • Default value: https://onezone.cloud.ba.infn.it:8443

• ONEDATA_SERVICE_SPACE_ONEPROVIDER_URL

  • Description: The OneData service space provider that hosts the service space

  • Format: {host}:{port}

  • Default value: cdmi-indigo.recas.ba.infn.it

• ONEDATA_SERVICE_SPACE_TOKEN

  • Description: The OneData service space token; you can retrieve it from the OneZone user interface

• ONEDATA_SERVICE_SPACE_NAME

  • Description: The name of the OneData space that you want to use as service space

  • Default value: INDIGO Service Space

• ONEDATA_SERVICE_SPACE_BASE_FOLDER_PATH

  • Description: The path (relative to the space one) to the folder that will host the files

  • Default value: Empty (files will be hosted directly in the space folder)