10. Tasks¶

A task is an arbitrary program which is started within Agora. They can take Agora objects as inputs (e.g. Datasets, Series, Exams etc.) or a user defined variable on startup (string, integer, float etc.). Tasks either either run directly on the Agora server, locally on the client machine or on any server with ssh access. A server task can run as a native process or within a docker container. If desired, the results of a task can be automatically imported into Agora after the task is finished.

Note

Anything that executes runs anywhere you have access.

10.1. Task Definition¶

Tasks are defined in the Agora settings in the “task” menu

Clicking on “New” opens the task definition page

10.1.1. Task Parameters¶

Task Name: The name of the task. A subfolder structure can be created with forward slashes. For example the task name:
```
Editors/Gyrotools/GTFlow
```
will generate the following task menu structure:
Run task on: Specifies where the task is run. Either on the local machine or on a server. When “host server” is selected the server can be chosen from the dropdown menu below.

Note

To see a server in the task definition it must first be added in the Host settings (see Host documentation)

Run task inside a docker container: Specifies if the task is run inside a docker container. When selected the name of the docker container can be entered in the text box below. Furthermore the volumes which are mounted inside the container can be specified.

Note

The docker container needs to be copied and built on the server before it can be used in Agora.

Inputs: The inputs for a task. The input can be any Agora object (e.g Dataset, Series, Exam etc.) or a user defined parameter (e.g. a string, float, integer etc) which is entered at runtime. Every input has a name and key which can be used as the placeholders for the commandline (see below). Inputs can be required or optional.
Command line: The commandline which is executed. A list of available placeholder is displayed on the right hand side. The available placeholders depend on the input type.

Note

In order to see the placeholder list that task needs to be created first. If you are creating the task for the first time, then press “Save and go back” and open the task again.

Success code: The exit code which marks a successful run. If the program returns a different exit code the task is marked as unsuccesful.
Parse output for error: A regular expression which indicates an erroneous run. If the regex matches anything in the output of the program then the task is marked as unsuccesful.
Outputs:: The outputs of the task. After the task is complete Agora searches the working directory (stored in the placeholder {{ working_dir }} ) for files which match the output type. All matching files are uploaded to Agora.
Add user or group:: Share the task with a user or group

10.2. Script Tasks¶

Agora tasks can also be defined using a script language. This allows for concise yet comprehensive task definitions which easy to read. Here is an example of a simple hello-world task running on a host server:

hello_world:
    host: host1
    script: echo "hello world"

A script task also can hold multiple jobs which run sequentially or in parallel depending on their “stage”. The stage keyword defines when the job is run. Jobs from the same stage run in parallel while jobs from different stages run sequentially. Different stages and their order are defined at the beginning of the task. Data can be shared between jobs via “artifacts”. An artifact defined in a job is available in all subsequent jobs. It can also be re-imported back into Agora via the “import” keyword. Finally each job can run on its own host.

The following example task defines 2 stages. The first stage creates a file on one host and defines it as artifact. The second stage runs on a different host, renames the file and re-imports it back to Agora

stages:
  - stage1
  - stage2

create_file:
  stage: stage1
  host: host_1
  script: |
    echo "created in stage1" > {{ output_dir }}/test.txt
  artifacts:
    output:
      path: "{{ output_dir }}/test.txt"

rename_file:
  stage: stage2
  host: host_2
  script: |
    mv {{ artifacts.output.path }} {{ output_dir }}/test_2.txt
  artifacts:
    output2:
      path: "{{ output_dir }}/test_2.txt"
      import: true
      import_path: "/My Agora/test"

For more examples checkout the predefined templates in the task definition.

To create a new script task select it in the dropdown on the Task toolbar:

10.3. Task Templates¶

A list of task templates is available on the task definition page. These templates can be used as starting point for your own task. You can select them from the dropdown at the top of the task definition:

The templates are available for both the UI- and Script-Tasks. Standby for additional templates in the future.

10.4. Docker and Podman Support¶

Agora tasks support both Docker and Podman container engines. Podman in particular allows to run containers without root privileges. The Podman syntax is very similar to the Docker syntax. Usually the “docker” command can simply be replaced with “podman”.

Select the Docker or Podman engine in the task definition, but note that the respective engine has to be installed on the selected task execution host:

10.5. Local tasks¶

Local tasks are executed on the local machine from which Agora is accessed from with a browser. In order to execute a local task the gtAgoraApp must be installed and running. If the gtAgoraApp is installed on multiple computers and logged in with the same credentials, the user has to select the target machine upon stating the task.

Note

A local task not necessarily has to run on the machine the user is currently sitting at.

10.5.1. Examples¶

Open a file in notepad:

In the first example we create a task which opens any file of type “Other” in the Windows notepad. To do so we create a local task with the following parameters:
Name: Viewers/Open in Notepad
Run task on: Local client computer
Inputs:
  Input Name: input
  Key: file
  Type: DataSet
  Required: Checked
  Type: Other

Command line: notepad {{ inputs.file.file.path }}
Outputs: None
To run the task select any file (e.g a logfile) and chose the notepad task from the menu

Run a Python script which is stored in Agora:

In this example we will run a Python script which is saved in Agora on a PAR/REC dataset.

We define a python script with 2 inputs:

The rec file
The output folder

The script will read the rec file and rotate the image by 90°:

import argparse
import os
import numpy as np
import math
from shutil import copyfile
parser = argparse.ArgumentParser()
parser.add_argument('file')
parser.add_argument('output_path')
args = parser.parse_args()
rec_size = os.path.getsize(args.file)
res = int(math.sqrt(rec_size/2))
with open(args.file, "rb") as recfile:
    data = np.fromfile(recfile, dtype=np.uint16, count=int(rec_size/2))
    data = np.reshape(data, (res, res))
    data = np.rot90(data)
    output_filename = 'rotated.rec'
    output_file = os.path.join(args.output_path, output_filename)
    data.astype('uint16').tofile(output_file)
    parfile = args.file.replace('.rec', '.par')
    output_parfile = os.path.join(args.output_path, 'rotated.par')
    copyfile(parfile, output_parfile)

Note

For simplicity the script ony works with rec files containing 1 image

To run the script within Agora we create the following local task:

Name: Python/Run Python Script
Run task on: Local client computer
Input1:
  Input Name: script
  Key: py
  Type: DataSet
  Required: Checked
  Type: Other
Input2:
  Input Name: datafile
  Key: df
  Type: DataSet
  Required: Checked
  Type: ParRec

Command line: python {{ inputs.py.file.path }} {{ inputs.df.rec.path }} {{ working_dir }}
Output:
  Output Name: output
  Output type: DataSet
  Regex:
  Dataset type: ParRec

../../../_images/execute_python_script.PNG

After defining the task, it can be started by selecting the python script and the PAR/REC file and selecting the task from the menu

The rotated PAR/REC image is automatically imported into Agora after the task is completed.

Run an MRecon reconstruction:

Our last example will demonstrate how to run a MRecon reconstruction in Matlab and automatically upload the result into Agora.

To do so create a task with the following parameter:
Name: Matlab/MRecon
Run task on: Local client computer
Input1:
  Input Name: datafile
  Key: raw_lab
  Type: DataSet
  Required: Checked
  Type: RAW Philips
  Command line: matlab -wait -nosplash -nodesktop -r "
                   r = MRecon('{{ inputs.raw_lab.raw.path }}');
                   r.Perform;
                   r.WritePar(['{{ working_dir }}', filesep, 'MRecon.par']);
                   r.WriteRec(['{{ working_dir }}', filesep, 'MRecon.rec']);
                   exit"
Output:
  Output Name: output
  Output type: DataSet
  Regex:
  Dataset type: ParRec
The command line first starts Matlab without user interface. Then a MRecon object is created from the RAW file selected as input. The reconstruction is Performed and a PAR/REC file is written into the working directory. Finally Matlab is closed.

The task is started by selecting a Philips RAW/LAB dataset and selcting the MRecon task from the menu:

10.6. Server tasks¶

In addition to local tasks, they can also be executed on any server with ssh access. On the server tasks can run natively or in a docker container (recommended). Apart from the server settings, the task definition is exactly the same as in a local task.

Note

To see a server in the task definition it must first be added in the Host settings (see Host documentation)

Docker containers:

One of the goals of modern software development is to keep applications on the same host or cluster isolated from one another so they don’t unduly interfere with each other’s operation or maintenance. This can be difficult, thanks to thepackages, libraries, and other software components required for them to run.

One solution to this problem are docker containers, which isolate applications’ execution environments from one another, but share the underlying OS kernel. They are typically measured in megabytes, and therefore use far fewer resources than a virtual machine for example. They start up almost immediately and can be packed densely on the same hardware and spun up and down en masse with little effort and overhead overhead. Furthermore docker includes its own versioning system which makes it easy to to create new, and rollback to old versions of a container.

Because Docker containers encapsulate everything an application needs to run (and only those things), they allow applications to be shuttled easily between environments. Any host with the Docker runtime installed - be it a developer’s laptop or a public cloud instance - can run a Docker container.

An online repository can be used for sharing containers. Docker Hub (https://hub.docker.com/) for example is a SaaS repository for sharing and managing containers. It contains many official Docker images from open-source projects and software vendors and unofficial images from the general public. You can download container images containing useful code, or upload your own, share them openly, or make them private instead. You can also create a local Docker registry if you prefer.

10.6.1. Examples¶

Run the rotate.py script from the example above on the server in a docker container:

In this example we will run the rotate.py python script from above in a docker container on a server. To do this we will create a docker configuration (Dockerfile) then copy the necessary files to the server and build the container. Finally we will create the task and run the container from Agora.

First create the python file rotate.py (see code above) and store it in a local folder. The script script requires the numpy library. Therefore we create a requirements.txt file add numpy and store it in the same direcory as our python script.

requirements.txt:
numpy
Afterwards we create the Dockerfile with the following content:

Dockerfile:
from python

COPY ./ /src
WORKDIR /src

RUN pip install -r requirements.txt

ENTRYPOINT ["python", "./rotate.py"]
As you can see our container is based on the official python container which includes the latest version of python. In the Dockerfile we copy our source files into the container set the working directory and install the required dependencies with pip. Finally we define our rotate script as entrypoint (will be called when we run the container).

We should now have a directory with the following content:

The next thing we need to do is copy the directory on the server and build the docker container:
# copy files
scp ./rotate/* user@host:/rotate/

# login to server
ssh user@host

# change the directory to the one containing the docker file
cd rotate

# build the container and call it rotate_py
docker build -t rotate_py ./
The container is now installed on the server and we are ready to define and run the task. In Agora we create a task with the following properties:
Name: Rotate
Run task on: Host Server
Select a host Server: Select the desired server (the server must have been configured in the Host settings)
Run task inside a docker container: checked
Name of the docker container: rotate_py
Volumes to be mounted: None
Input1:
  Input Name: datafile
  Key: par_rec
  Type: DataSet
  Required: Checked
  Type: ParRec
  Command line: {{ inputs.df.rec.path }} {{ working_dir }}
Output:
  Output Name: output
  Output type: DataSet
  Regex:
  Dataset type: ParRec
After defining the task, it can be started by selecting a PAR/REC file and choosing the Rotate task from the menu. The rotated PAR/REC image is automatically imported into Agora after the task is completed.