Tutorials#
Before going through these tutorials, make sure that your FastrPI installation is correctly set up, see Setting up FastrPI.
Installing and running a Network#
By using FastrPI existing Fastr Networks and Tools can be used, shared and re-used in a convenient way. In this tutorial we
go through the process of installing the addint Network which can add two integers together. We will first install
the Network and run it using Docker afterwards.
Seeing what’s out there#
Before we can install a Network, we first need to know which Networks are available. For this the fastrpi list
commands can be used. To see which Tools and Networks are installed, use fastrpi list installed:
user@machine: fastrpi list installed
...
Installed tools:
Name Version Package version
-------- --------- -----------------
Installed networks:
Name Package version
------------ -----------------
For this tutorial we start from a fresh install, so nothing is installed.
The command fastrpi list networks and fastrpi list tools will show you the Networks and Tools that are
available to be installed from the FastrPI repository. Running fastrpi list networks gives us:
user@machine: fastrpi list networks
...
Available networks:
Name Package version
-------------------- -----------------
addint 1.0.0
...
In this list we can see that Network addint, package version 1.0.0 is available. In the next section we will install
this Network.
Installing#
Installing a Network can be done by running
fastrpi install network addint -v 1.0.0
where we entered addint and 1.0.0 as the name and version of the Network respectively. This rather simple Network
uses only one Tool, also named addint, which also needs to be installed. When installing a Network, the FastrPI client
will do this automatically. You don’t have to, but you could install the Tool on its own by running
fastrpi install tool addint -v 1.0 -p 1.0
In this case addint is the name of the Tool, -v 1.0 signifies the version of the Tool, meaning the version of the
software which is wrapped by the Fastr Tool, and -p 1.0 is the package version, the version of the Tool package.
In the terminology of the Fastr system, the version, -v 1.0, corresponds to the ‘command version’ and the package
version, -p 1.0, corresponds to the ‘tool version’.
When installing a Tool, the associated container will also be retrieved. In the default configuration this is done using Docker.
Note
Bear in mind that if you would like to run a Network using either Docker or Singularity, that the Tools also have to be installed through FastrPI using the same container system.
After installing the Network and the Tool, we can run fastrpi list installed again to see the results:
user@machine: fastrpi list installed
...
Installed tools:
Name Version Package version
-------- --------- -----------------
addint 1.0 1.0
Installed networks:
Name Package version
------------ -----------------
addint 1.0.0
Running#
To run the Fastr Network, we call fastrpi run with some arguments. But before we can do that, we need to define
where the input data is and where we want the output data to end up. Within Fastr these are named Sources and Sinks,
respectively. For more information about this, see the
Fastr documentation. For our
network we need to create the file ./source_sink.py, with the contents:
def get_source_data():
source_data = {
'left_hand': {'s1': 4, 's2': 5, 's3': 6, 's4': 7},
'right_hand': {'s1': 1, 's2': 3, 's3': 3, 's4': 7},
}
return source_data
def get_sink_data():
sink_data = {'sum': 'vfs://output/fastr_result_{sample_id}.txt'}
return sink_data
The functions get_source_data() and get_sink_data() will be called by fastrpi run. The keys of the returned
dictionaries need to match the IDs of the Sources and Sinks in the Network itself. When using Fastr
Virtual File System (VFS) mount points in the Sources and Sink paths, in this case output, these mount
points have to be specified in the Fastr config. (If available, Source-Sink files can be re-used, see
this Note.) This means adding the following to your Fastr configuration:
mounts['output'] = "./folder/for/outputs"
After setting this up, it is now time to run the Network using the fastrpi run command mentioned above.
To run Network addint, version 1.0.0, after creating ./source_sink.py, we can call:
fastrpi run addint -v 1.0.0 --source_sink ./source_sink.py
This will show us the Fastr logs flashing by, resulting in a successful execution of the Network. The files that were produced can be found at the locations of the Sinks.
This concludes the tutorial on installing and running an existing Network. The next tutorial will be about creating or editing a Tool or Network yourselves and publishing it to the FastrPI repository.
Creating, editing and publishing a Tool package#
If you have used Fastr before, you might have some Fastr Tools and Networks to share with others. Maybe you have been working with Networks and Tools from FastrPI and want to improve or modify them. In this tutorial we will discuss the creating of a FastrPI Tool and Network package, the editing of a Tool or Network package and publishing them to the FastrPI repository.
Note
This tutorial assumes that you are familiar with the workings of Fastr. For more information on Fastr itself, see the Fastr documentation here.
Editing a package#
First we will look into editing the packages. If after using a Tool or Network we discover that there is an error
in the description, a bug in the code or that some functionality can be added, we can go and update the package. With
the fastrpi edit command the folder containing the Tool or Network package will be copied into the current folder,
so the files can be edited with your favorite text editor. To do this, run the command:
fastrpi edit tool addint -v 1.0 -p 1.0
After completion the nested folders tools/addint/1.0/1.0 will have appeared, which will contain
tools/addint/1.0/1.0/
├── addint.yaml
├── bin
│ └── addint.py
├── Dockerfile
├── LICENSE
├── LICENSE_addint
└── manifest.fastrpi_yaml
If the change is a simple change in the metadata of the FastrPI package or the Fastr Tool definition addint.yaml,
a change in the package can be signified by incrementing the tool version in the Fastr Tool definition and the package
version in the package manifest. Within a package all the Fastr Tool definitions should have the same tool versions.
When any changes are made to the software, in this case the Python script addint.py, these changes should be reflected as
an increment of the command version in the Fastr Tool definition and the version in the FastrPI Tool package.
Similarly we can edit a Network package. For this we use
fastrpi edit network addint -v 1.0.0
which will again create a folder addint-v1.0.0. Editing a Network happens in the same manner as editing a Tool package.
One difference is that any change to the singular version that Network packages have. The package version in the
manifest needs to match the version indicated in the Fastr Network definition, i.e., ‘network.py’.
After we are done editing our Network or Tool packages we can test it and publish it. This will be explained in the sections below.
Note
If a Network package contains a file source_sink.py or source_sink_data.py, this file can be copied using the
fastrpi edit function. In the case of addint-v1.0.0 calling the following command will result in the folder
addint-v1.0.0 containing the file source_sink.py:
fastrpi edit network -s addint -v 1.0.0
Creating a package#
Let’s say that we had our fun with adding integers, but that we now also
would like to multiply two numbers. For this we will have to add a Tool which performs the multiplication and
a Network to use the Tool. Luckily the new Tool and Network will look very similar to the addint Tool and
Network, so we can again use fastrpi edit to retrieve the files for the addint Tool package:
fastrpi edit tool addint -v 1.0 -p 1.0
to get the folders tools/addint/1.0/1.0:
tools/addint/1.0/1.0/
├── addint.yaml
├── bin
│ └── addint.py
├── Dockerfile
├── LICENSE
├── LICENSE_addint
└── manifest.fastrpi_yaml
To implement the multiplication functionality we write a Python script multiply.py, a Dockerfile
to containerize this Python script and a Fastr Tool defintion multiply.yaml. We will use the same license
for multiply.py as for addint.py, so we will leave the LICENSE file in place. The files
manifest.fastrpi_yaml and LICENSE_addint we will delete; appropriate versions for the multiply Tool
package will be created during the package creation and publishing process. To keep things clear, we rename the folder
addint to multiply. The folder of files now contains the following files:
tools/multiply/1.0/1.0/
├── multiply.yaml
├── bin
│ └── multiply.py
├── Dockerfile
└── LICENSE
From these files we would like to create a Tool package, so we call fastrpi create, specifying the
package name multiply and the Fastr Tool definitions, i.e., multiply.yaml:
fastrpi create tool --name multiply ./multiply.yaml
By following the prompts a manifest.fastrpi_yaml file will be created. It is important to select your
scripts in the last stage of the fastrpi create prompt. If these files are not defined in the manifest, they
will not be sent to be published. Before publishing the package it is a good practice to first check
if everything is correct, see the documentation on Manifests for the required
items in the manifest.
Testing a package#
After editing or creating a package we should see if it works.
When editing a Tool package, the a nested folder strucutre is created. For example for Tool addint-v1.0-p1.0,
the nested folders are ./tools/addint/1.0/1.0/, for a tool multiply-v1.0-p1.1 the structure would be
./tools/multiply/1.0/1.1. By adding the path
./tools to the Fastr configuration, these Tools can then be discovered and used by Fastr. To do so, you can add
this line to your Fastr configuration:
tools_path += ['path/to/tools']
The Tools AddInt from Tool package addint-v1.0-p1.0 and Multiply from Tool package multiply-v1.0-p1.1 can then
be used in a Fastr Network by calling
network.create_node('addint/1.0/AddInt:1.0,...) and network.create_node('multiply/1.0/1.1/Multiply:1.0,...)
respectively. The nested folder structure created by fastrpi edit is the same as the folder structure when
the Tools are installed, so the references in the Networks can be made in the same way.
Do not overlook testing the Docker container associated to the Tool your developing or editing.
A quick test is to see if the script or binary you want to use can be reached by calling the help function,
e.g., by running ./program --help.
When editing a Network, for example to change one of its Tools, the network.py file can be run directly using
fastrpi runlocal network.py --source_sink source_sink.py
Publishing a package#
In the previous sections we created new Tool packages which we would like to publish to the FastrPI repository. If you have a manifest file, publishing a Tool package is as simple as calling:
fastrpi publish tool ./manifest.fastrpi_yaml
After executing fastrpi publish the Tool package will first be checked locally, after which it is pushed
to the FastrPI repository where a CI/CD pipeline will run the same tests again. See Package Checks to see
an overview of the checks.
When the package has passed all the checks it will be available to install from the FastrPI repository.