Usage
Last updated
Last updated
APIs are compliant with specification defined in: In particular this guide refers to specifications having version: v1.0.
Before submit any task, at least an application should be installed on the APIServer. Each application may have then one or several infrastructure defined where the application can execute. In APIServer terminology, the application may be a concrete running executable or an infrastructure. Below example shoes how to create a new task for the baseline example application having id 1 helloworld@localhost. THis application just generate an output file and does not use any input file.
curl -i -H "Content-Type: application/json" -X POST -d '{"application":"1","description":"helloworld@localhost test run","arguments": ["arg1","arg2","arg3"]}' http://localhost:8888/v1.0/tasks?user=brunor
answer: {"status": "WAITING", "application": 1, "date": "2015-10-08 16:22:51", "description": "helloworld@localhost test run", "output_files": ["hello.out", "hello.err"], "_links": [{"href": "/v1.0/tasks/1", "rel": "self"}, {"href": "/v1.0/tasks/1/input", "rel": "input"}], "user": "brunor", "input_files": ["hello.txt", "hello.sh"], "id": 1, "arguments": ["arg1", "arg2", "arg3"]}
Please notice the use of filter user=.... Any APIServer REST call can be performed on behalf of a given user. Users maybe or not the one registered on the APIServer, however the user executing the command must have the special role user_impersonate in order to behave as a different user.
The finalization, means that the task will be submitted; this happens as soon as all input_sandbox files have been sent
curl -i -F "file[]=@hello.txt" -F "file[]=@hello.sh" http://localhost:8888/v1.0/tasks/1/input?user=brunor
If there are no files to upload, the task will be submitted within the first curl command. Depending on the kind of action taken by the APIServer a different JSON output will be returned.
$ curl -H "Authorization: Bearer TEST_TOKEN" -F "file[]=@input.txt" http://localhost:8888/v1.0/tasks/31/input { "files": [ "input.txt" ], "message": "uploaded", "task": "31", "_links": [ { "href": "/v1.0/tasks/31", "rel": "task" } ], "gestatus": "triggered" }
In this case the task is ready for the daemon processing.
curl -H "Authorization: Bearer TEST_TOKEN" -F "file[]=@file.txt" http://localhost:8888/v1.0/tasks/32/input { "files": [ "file.txt" ], "message": "uploaded", "task": "32", "gestatus": "waiting" }
In this case the task is still waiting for necessary input files.
Uploaded files can be retrieved using API calls getting its URL from the GET action on tasks endpoint.
`
This second example uses another sample application pre-configured after the installation. This application behaves like helloworld@localhost it has id: 2 and makes use of input and output files.
curl -i -F "file[]=@sayhello.txt" -F "file[]=@sayhello.sh" http://localhost:8888/v1.0/tasks/14/input?user=brunor
In the case of the GridEngine executor interface, the original submission description JSON is translated in a new format recognized by the GridEngine.
Follows a JSON understandable by Grid and Cloud Engine:
Translated to:
JSON files above are located inside the task informative directory; see as_queue table field task_info in APIServer database.
Running test with GridEngine EI for accessing SSH
1) Insert the task request
curl -i -H "Content-Type: application/json" -X POST -d '{"application":"1","description":"helloworld@csgfsdk test run", "output_files": [ "stdout.txt","stderr.txt" ]}' http://localhost:8888/v1.0/tasks?user=brunor
3) 2) Finalize task sending POST call (no uploads)
curl -i -X POST http://localhost:8888/v1.0/tasks/1/input?user=brunor
This produced GridEngine understable JSON (application: 10000, means the APIServer app registered in UsersTracking database GridOperations table)
``{ 'infrastructure': { 'resourceManagers': 'ssh://90.147.74.95'}, 'commonName': 'helloworld@csgfsdk test run', 'application': '10000', 'jobDescription': { 'output': 'stdout.txt', 'executable': '/bin/hostname', 'arguments': '-f', 'error': 'stderr.txt'}, 'credentials': { 'username': 'jobtest', 'password': 'Xvf56jZ751f'}, 'identifier': u'task_id: 1' }```
The following commmand removes any reference to the task having id=2
curl -i -X DELETE http://localhost:8888/v1.0/tasks/2?user=brunor
Delete action is implemented by the APIServerDaemon which removes every task reference both from DB and file system except for the task table where the task entry will be kept and its status will be changed to 'PURGED'. Special DB cleaning procedures may delete PURGED records, avoiding the case that the last PURGED record is also the last record of the task table sorting it incrementally by task_id.
The system is able to associate run-time data to a running task this feature offers to store any kind of information related to the running task in the form of: (key-name, key-value) couple. Executor interfaces may use this field to store specific information related to the targeted infrastructure
Below functions related to application management
List all applications:
curl http://localhost:8888/v1.0/applications/
List the app having id 2:
curl http://localhost:8888/v1.0/applications/2
Creation of an application:
curl -i -H "Content-Type: application/json" -X POST -d '{"infrastructures": [1, {"description": "Tosca test at 90.147.170.168:32101","parameters": [{"name": "tosca_endpoint","value": "tosca://90.147.170.152:80/orchestrator/deployments"},{"name": "tosca_token","value": "11223344556677889900AABBCCDDEEFF"},{"name": "tosca_template","value": "tosca_template.yaml"},{"name": "tosca_parameters","value": "wait_ms=30000&max_waits=30"}],"enabled": true,"virtual": false,"name": "tosca_Test@BA"}],"parameters": [{"name": "target_executor","value": "SimpleTosca","description": ""},{"name": "jobdesc_executable","value": "tosca_test.sh","description": ""},{"name": "jobdesc_output","value": "stdout.txt","description": ""},{"name": "jobdesc_error","value": "stderr.txt","description": ""}],"enabled": true,"input_files": [{"override": false,"path": "/home/futuregateway/FutureGateway/fgAPIServer/apps/toscaTest","name": "tosca_template.yaml"},{"override": false,"path": "/home/futuregateway/FutureGateway/fgAPIServer/apps/toscaTest","name": "tosca_test.sh"}],"name": "hostname@tosca(1)","description": "hostname tester application on tosca by setup(1)"}' http://localhost:8888/v1.0/applications?user=brunor
Infrastructure JSON section, foresees both the possibility to use a generic infrastructure or an application level ifnrastructures. In the first case it is necessary to specify the id of the infrastructure Please be aware that application installation API does not take care of FutureGateway defined uers groups and values. In order to allow users to run this application it is necessary to allow at least one group to run this application, inserting a new record in table fg_group_apps: insert into fg_group_apps values ( \, \, now());.
Inside $FGLOCATION/fgAPIServer/apps directory, there are several applications that can be used as template to generate new applications. Once selected the most appropriate application to use, just make a full copy of the application direcotry, provide to the new directory the application name and go inside of it. In the application directory there are two files: setup_app.sh and stress_test.sh. Modify the first file to setup properly the new application, use the second file to execute the new application on the system. The second script can be executed only after the setup_app.sh script has been executed. The setup_app.sh script contains several configuration settings and it can be executed specifying as argument a session token (with no 'Authentication: Bearer' string).
After the applciation is created, it is important to assign this application to a group, so that their members can perform any allowed activity on it (See advanced operations).
Input files
The original API specifications did not have the concept of ‘application level’ files because not foreseen by the original API specification. The old fashioned way to specify default application files was the use of tag ‘input_files’ inside the application installation JSON. The ‘input_files’ can or cannot overridden by the user during task submission accordingly to a given flag. API specification have been modified in order to cover ‘application’ level files, considering a new JSON tag in application installation named ‘files’. Files accepts a list of filenames and then a new API endpoint named application/input/<app_id> with POST method upload these files. During the task submission these files will be included in the task file sandbox automatically and the user cannot modify/access them. In order to do not compromise the backward compatibility the fgAPIServer implements both tags ‘input_files’ and ‘files’.
Using input_files:
The override flag when true avoids to change the file file content during the task submission phase. The path have to point to a directory containing the file name specified in the name field. This method to specify input files only work with API server filesystem. This method is useful to write application installation scripts.
Using files:
Then upload each file content with:
View uploaded files:
curl -H "Content-Type: application/json" http://localhost:8888/v1.0/applications/input
This method is suggested when installing applications using external GUIs because files can be uploaded from anywhere. Specified input files cannot be changed by the user during the submission phase; they are threated as 'override == true' of the input_file case.
Delete
curl -i -H "Content-Type: application/json" -X DELETE
Change an application:
curl -H "Content-Type: application/json" -H "Authorization: Bearer TKN" -X PUT -d '{"files": ["tosca_template.yaml", "tosca_test.sh"],"name": "hostname@toscaIDC","parameters": [{"name": "target_executor","value": "ToscaIDC","description": ""}, {"name": "jobdesc_executable","value": "tosca_test.sh","description": "unused"},{"name": "jobdesc_output", "value": "stdout.txt","description": "unused"},{"name": "jobdesc_error", "value": "stderr.txt", "description": "unused"}], "outcome": "JOB","enabled": true,"id": "14","infrastructures": [4],"description": "hostname tester application on toscaIDC"}' http://localhost:8888/v1.0/applications/14
Application change supports both files and input_files. Using files the application will not change filenames already present in the application files list and removes files no more present in the new passed list. The use of the field input_files behaved in the same way of files except that uses path and override additional fields.
Infrastructures can be specified using registered Ids or using an explicit json declaration like in infrastructure creation. All specified infrastructure ids will be checked against the existing ids. New ids will be associated to the application while ids no more present in the passed json array will be removed. Specifiying explicitly an infrastrcucture description; a new infrastructure will be created and registered with the application; for instance:
curl -H "Content-Type: application/json" -H "Authorization: Bearer TKN" -X PUT -d '{"files": ["tosca_template.yaml", "tosca_test.sh"],"name": "hostname@toscaIDC","parameters": [{"name": "target_executor","value": "ToscaIDC","description": ""}, {"name": "jobdesc_executable","value": "tosca_test.sh","description": "unused"},{"name": "jobdesc_output", "value": "stdout.txt","description": "unused"},{"name": "jobdesc_error", "value": "stderr.txt", "description": "unused"}], "outcome": "JOB","enabled": true,"id": "14","infrastructures": [4, { "name": "New infra test", "parameters": [ { "name": "jobservice", "value": "ssh://localhost", "description": "local SSH host" }, { "name": "username", "value": "jobtest" }], "description": "New ssh infra", "enabled": true, "virtual": false }],"description": "hostname tester application on toscaIDC"}' http://localhost:8888/v1.0/applications/14
With the explicit infrastructure description in the application change, it is possible to specify application specific values to already existing infrastructures.
This is included in the json response after a GET: tasks/ inside 'runtime_data' json section specifying an input json in the form:
REST APIs can insert/change runtime values with PATCH: tasks/<task_id>
View task detail (watch runtime_data)
curl -i http://localhost:8888/v1.0/tasks/43?user=brunor
Insert a new data: http://localhost:8888/v1.0/applications/2
curl -i -H "Content-Type: application/json" -X PATCH -d '{"runtime_data": [ { "data_name": "test_data", "data_value": "test_value", "data_desc": "test description value"} ]}' http://localhost:8888/v1.0/tasks/43?user=brunor
The PATCH call on tasks endpoint has been enriched in order to support a task status change. Executor interfaces may recognize this change forcing the requested status of the task into the distributed infrastructure. In ToscaIDC executor interface, it is posstible to request status 'CANCELLED' thus will enforce to release the resource associtated to the task to be released by the TOSCA orchestrator. To specify a status change use:
curl -i -H "Content-Type: application/json" -X PATCH -d '{"status": "CANCELLED" }' http://localhost:8888/v1.0/tasks/43?user=brunor
curl -i -H "Content-Type: application/json" -X PATCH -d '{"runtime_data": [ { "data_name": "test_data", "data_value": "new_test_value"} ]}' http://localhost:8888/v1.0/tasks/43?user=brunor
This is the way futuregateway manages authorization and authentication as default. This method can be switched off, modified or substituded by other AuthN/Z systems
1) Obtain a login token (not really mandatory step; see step 2)
2 ) Obtain a session token using the logtoken via POST
2.a) It is possible to obtain a session tokens direcrtly from /auth call providing credentials as parameters (operation suggested only if the front is accessible via https.
PTV can be enabled by a configuration flag, in this case session tokens are verified against an existing portal which answers to a given endpoint. A username and password is necessary to contact the portal endpoint and an https connection should be adopted. The portal returns a flag telling if the user is allowed or not and optionally further information such the user name and its group in the portal so that this information can be used to map the portal user with a futuregateway user. While processing APIs using tokens related to users not yet registered in the API Server, the new user will be automatically registered using as reference the received subject field.
Once connected get from the browser shibboleth session and pass it as cookie; for instance: curl --cookie "_shibsession_64656661756c7468747470733a2f2f7367772e696e6469676f2d64617461636c6f75642e65752f73686962626f6c657468=_9dcb88c4f6b46005a1f2403079dd738f" -v https://sgw.indigo-datacloud.eu/apis/v1.0/tasks?user=brunor
Another option is:
Once logged place the section cookie in a 'cookie.txt' file
In curl commands specify option: -b cookie.txt
Any installed application on the APIServer may have defined one or more infrastructures where it can execute. The current version of the software can support both generic infrastructures shareable across different applications as well as application level infrastructures.
Infrastructure creation:
curl -i -H "Content-Type: application/json" -X POST -d '{ "name": “Infra test“, "parameters": [ { "name": "jobservice", "value": "ssh://localhost" }, { "name": "username", "value": "jobtest" }, { "name": "password", "value": "Xvf56jZ751f" } ], "date": "2017-01-10T16:59:10Z", "description": "ansshinfra", "enabled": true, "virtual": false }' http://localhost:8888/v1.0/infrastructures
View infrastructure:
curl -H "Authorization: Bearer TOKEN" http://localhost:8888/v1.0/infrastructures/2
It is still possible to install infrastructures when installing new applications specifying them in the application creation rest call instead to pass the infrastructure array of <id>s as reported in the specifications. In this case the user has to pass an array of defined infrastructures. This solution has been intentionally maintained because by previous experiences in most of the cases each application requires its own specific settings. However the infrastructure sharing among application is still possible using infrastructures API endpoints. It is still possible to define applications directly operating ad database level, for more information about this opportunity, please have a look on the 'Advanced operation' section.
Change infrastructure:
curl -H "Content-Type: application/json" -H "Authorization: Bearer TKN" -X PUT -d '{"name": "Infra test (changed)", "enabled": false, "id": 6, "virtual": true, "description": "ansshifnra (changed)", "parameters": [{"name": "jobservice", "value": "ssh://fgtest", "description": "fgtest ssh hots"} ] }' http://localhost:8888/v1.0/infrastructures/6
Executor interfaces (EI)s are the way the APIServer uses to reach any kind of distributed computing resource through the use of any possible middleware component. EIs may deal with very flexible system like JSAGA and its adaptors like in GridEngine and SimpleTosca executor interfaces. In other cases EIs may use APIs or even CLI commands to manage the distributed infrastructure like in the case of the ToscaIDC which deals directly with TOSCA orchestrator REST API calls. Below sections describe how to configure these interfaces, in particular how to setup properly infrastructure_parameters values while describing a new APIServer application.
The GridEngine executor interface target any distributed infrastructure that the Grid and Cloud component can address. The most popular and supported ones are: SSH, EMI/gLIte and rOCCI.
SSH Configuration . jobservice: ssh://\:\ . username: the ssh username . password: the ssh password (clear text) Omitting the password field or placing an empty string the GridEngine will attempt to establish the SSH connection using the .ssh private key file contained in the tomcat server home dir running the APIServerDaemon web application.
EMI/gLite Configuration . jobservice: wms endpoint . bdii: informatio provider endpoint . eToken_host eTokenServer hostnmae . eToken_port eTokenServer listening port . eToken_id: robot proxy identifier . voms: voms name . voms_role: the role associated to the voms . rfc_proxy: flag to request or not an RFC proxy
rOCCI Configuration . jobservice: rOCCI endpoint . os_tpl: OS template name . resource_tpl: resource template name . attributes_title: VM instance name . eToken_host eTokenServer hostnmae . eToken_port eTokenServer listening port . eToken_id: robot proxy identifier . voms: voms name . voms_role: the role associated to the voms . rfc_proxy: flag to request or not an RFC proxy
SimpleTosca is a deprecated EI that has been used in early phases of FG development. This EI has been designed to cover explicitly the Molecular Dynamics (MD) use case and it is no longer used. The description of this EI is provided just for completeness.
Configuration
. tosca_endpoint: The TOSCA endpoint
. tosca_token: Token for access TOSCA endpoint (unused)
. tosca_template: The yaml file name describing the TOSCA resource
. tosca_parameters: Parameters supported by jsaga-tosca-adaptor
The ToscaIDC executor interface deprecates the SimpleTosca and it may cover any possibility offered by the TOSCA orchestrator since it is possible to send any possible yaml file containing any possible yaml input parameter.
Configuration
. tosca_endpoint: The TOSCA endpoint
. tosca_template: The yaml file name describing the TOSCA resource
. tosca_parameters: TOSCA parameters (params=\<json file containing yaml input parameters\<)
Inside directory $FGLOCATION/apps there are several sample application making use of different EIs. Any user may use one of these tester applications to generate as a template its own specific application use case. For ToscaIDC there exists a sample application named: ToscaIDCTest. This application is configured to run as default withting the fgportal_ptv.py script which simulates the TOSCA orchestrator as well as a PTV endpoint.
The APIServer manages users, groups and roles however API specs do not take in account these entities. For instance when a new application is installed it is necessary to enable a group to grant execution rights to the new application. All these operations not foreseen by the API specifications must be performed managing directly the database. For this reason it is important for developers and administrators to know the APIServer database schema.
These tables are collecting the information related to APIServer installed applications
application: Store generic application information
application_file: Store input files required by the application
application_parameter:
infrastructure: This table should have a different name like application_infrastructure, since the primary key of this table consists of the couple (application id, infrastructure id). The infrormation about infrastructure is kept at application level in order to facilitate infrastructure customization for a specific application.
This table contains just infrastructure generic information and several flags like enabling the infrastructure or specify if the infrastructure is a virtual entity (something to be instantiated yet).
Please notice that infrastructure sharing among different applications can be defined playing around the primary key (application id, infrastructure id). However the use of infrastructure sharing is highly not recommended.
infrastructure_parameter: In the form of (key name, key value) couples this table contains application infrastructure specific settings.
infrastructure_parameter: In the form of (key name, key value) couples this table contains application infrastructure specific settings.
infrastructure_task: The functionality related to this table is not yet supported and it is related to the use of virtual infrastructures. Whenever a virtual infrastructure instance will be created by a task this table will link the task requiring the infrastructure with an infrastructure dinamically created by an 'infrastructure' application (see outcome field in applciation table).
task: Generic information about task activity is registered in this table.
task_arguments: Each task may spevify different arguments stored in the form o
task_input_file: Task specific input files
task_output_file: Task specific output files
runtime_data: This table is related to tasks as well, it stores in the form of (key name, key value) any information related to the task.
as_queue: The APIServer queue table keeps track of any command sent to the APIServer Daemon. This table normally stores activities to be performed on tasks: creation, deletion, etc. This table can be used also to extend the APIServer daemon functionalities (See for instance the executor interfaces). The queue is filled by the front-end (fgAPIServer) and consumed buy the queue polling daemon (APIServerDaemon).
The APIServer manages its own set of users, groups and roles. The baseline setup creates few users examples providing each a different set of privileges. Different sets of privileges may be assigned to groups and groups can be assigned to users.
fg_user: Stores generic information about the APIServer registered user.
fg_group: Stores generic information about the stored groups.
fg_user_group: Link available users with available groups
fg_group_apps: Associate an application to an existing group
fg_group_role: Store any role associated to the groups
fg_role: Store any possible role foreseen by the APIServer (Not all roles defined by the baseline setup are supported in current implementation).
Current defined roles are:
app_install: The user can install applications
app_change: The user chan perform changes to the applciations
app_delete: The user can delete applications
app_view: The user can see details of an application
app_run: The user can run the application
infra_add: The user can add an infrastructure
infra_change: The user chang make changes to an infrastructure
infra_delete: The user can delete an infrastructure
infra_view: The user can see details of an infrastructure
infra_attach: The user can attach an infrastructure to an application/task
infra_detach: The user can detach an infrastructure to an application/task
task_delete: The user can delete a task
task_view: The user can see task details
task_userdata: The user can access to task userdata
user_add: The user can add other users
user_del: The user can remove other users
user_change: The user can change details of another user
group_change: The user can change group settings
role_change: The user can change roles on groups
user_impersonate: The user can impersonate a different user
group_impersonate: The user can impersonate a different group
fg_token: Store associations between APIServer users and token received by the front-end (fgAPIServer).
db_patches: This table is used by the database schema updater script available under $FGLOCATION/fgAPIServer/db_patches directory. It stores all patches applied to the original baseline database schem installation.
Executor interface may have its own specific table or (as in the case of the Grid and Cloud engine) a whole database. The responsability to deal with these tables is in charge of the specific Executor Interface.
SimpleTosca/ToscaIDC: These Executor interfaces are using respectively the tables: simple_tosca and tosca_idc. Both tables have the same structure:
UsersTrackingDatabase: The UsersTrackingDatabase is used by the Grid and Cloud Engine component to keep track of each activity performed by this system. This database has been developed to fullfil the EGI Traceability policies defined for scientific gateways. The Grid and Cloud Engine was the core component of the Catania Science Gateway Framework CSGF from where the FutureGateway takes its orings. The GridEngine executor interface deals with the Grid and Cloud Engine component in order to target the following kind of distributed infrastructures: SSH, EMI/gLite, rOCCI. All of them are reached instructing properly the JSAGA component through the use of the corresponding JSAGA adaptor. This involves that applications targeting the GridEngine executor interface may run on all of those infrastructure just filling up the infrastructure and infrastructure_parameters_tables. This kind of configuration is visibile in the baseline setup with the sayhello application. When submitting a task linked to many infrastructures, only one of them will be selected using a random strategy. This behavior may be changed in the future with more sophisticated algorithms depending for instance on the loud charge of each targeted infrastructures.
APIServer software suite offers the fgTools repository where can be located any utility and helper tool. At the moment only two utilities are available:
pushProxy: This is an utility developed at the very ealry developments on TOSCA orchestrator. The utility has to be configured inside a cron job in order to timely send to a given endpoint a robotProxy extracted from the eTokenServer host.
updateCode: This utility helps FG developers to update source packages remotely. It allows to send local modified APIServerDaemon sources, recompiling and installing them on the remote service. It provides the same capability for other sources like adaptors and the front-end. It also provide options to send and receive files from/to the remote FG machine. Below the help message:
This script copies the specified source package into specified remote host using ssh connection, for this reason it is recommended to exchange the ssh keys between source and destination hosts By default the script stops the futuregateway service leaving it not running unless the the option -s is specified
`$ curl -H "Authorization: Bearer TEST_TOKEN" { "status": "SUBMIT", "description": "tester application run", "creation": "2017-09-19T08:04:22Z", "iosandbox": "/tmp/258e35ac-9d11-11e7-a33d-fa163e876b0e", "user": "00000000-0000-0000-0000-000000000000", ... "input_files": [ { "status": "READY", "url": "file?path=%2Ftmp%2F258e35ac-9d11-11e7-a33d-fa163e876b0e&name=input.txt", "name": "input.txt" } ], "last_change": "2017-09-19T08:08:00Z" } $ curl -H "Authorization: Bearer TEST_TOKEN" ""
`curl -i -H "Content-Type: application/json" -X POST -d '{"application":"2","description":"sayhello@csgfsdk test run", "arguments": ["\"I am saying hello!\""], "output_files": [{"name":"sayhello.data"}], "input_files": [{"name":"sayhello.sh"},{"name":"sayhello.txt"}]}'
1 curl -i -H "Content-Type: application/json" -X POST -d '{"appliDELETEcation":"2","description":"sayhello@csgfsdk test run", "arguments": ["\"I am saying hello!\""], "output_files": [{"name":"sayhello.data"}], "input_files": [{"name":"sayhello.sh"},{"name":"sayhello.txt"}]}'
To install a generic ifnrastructure, have been implemented the Infrastructure APIS in accordance to the Futuregateway specifications: