Deploying Syrup

To deploy Syrup on your system you first have to download the latest distribution found at sourceforge. The distribution contains documentation, jar files and some example Workflows. Please read the release notes for additional information.

Installing the binary

Unpack syrup.zip in a directory of choice. The binary is relocatable at any location and on any filesystem that can handle long file names.

After installing the following directories/files should be there:

  1. ~/syrup/RELEASE_NOTES.txt - contains the latest release notes.
  2. ~/syrup/LICENSE.txt - contains the license notes.
  3. ~/syrup/bin - contains .sh and .bat scripts to run Syrup.
  4. ~/syrup/lib - contains (additional) jar files.
  5. ~/syrup/jdni - contains the default serialized deployment.
  6. ~/syrup/test - contains the sample Workflows.
  7. ~/syrup/src - contains the source code of Syrup.
  8. ~/syrup/xsl - contains utility XSLT stylesheets.

Dependencies

JRE 1.4.2

The current version of Syrup is extensively tested with JRE 1.4.2 and should work without any problems. It is however possible to run Syrup with JRE 1.3.1 or lower versions. To switch to a specific version of the JRE you need set the environment variable JAVA_HOME to point to the version of choice and add the directory containing the java executable in the PATH environment variable.

UNIX

All the .sh file found in ~/syrup/bin are fully self contained and can be run without any setup. The shell scripts are nothing more than convenient wrappers to set environment variables like CLASSPATH and finally to run the java executable. You can use the shell scripts as a basis to build your own Syrup scripts or to learn how Syrup works.

Windows

When running ~/syrup/bin/syrup.bat or any of the packaged .bat files, you'll need to set the SYRUP_HOME environment variable to ~/syrup. When using Windows, please replace *.sh into *.bat for the script examples found in this document.

Additional jars

Together with the syrup.jar the following additional jars are also found in the distribution:

  1. xalan.jar - provides correct XML serialization.
  2. mysql.jar - used for the default JDBC backing store implementation of Syrup.
  3. fscontext.jar - filesystem implementation of JDNI.
  4. providerutil.jar - JDNI utility functions.

These jars are added by default to the CLASSPATH when syrup.sh or syrup.bat is run. If there is a need for more jars (because of runtime dependencies) they can be added to the CLASSPATH environment variable.

The runtime architecture

Note that 'running Syrup' actually means running Workers that use the Syrup implementation provided by the distribution. Packaged with Syrup is the org.syrup.workers.DefaultWorker. When this Worker is run, it first finds a concrete Syrup implementation. When found, the Worker uses the WorkSpace API to query and execute Tasks.

Choosing an implementation

The choice of what implementation will be used is deferred to the deployer, the one that manages the IT infrastructure. The org.syrup.workers.DefaultWorker uses JDNI (Java Directory Naming Interface) to fetch an implementation from the directory services provided by JDNI. For an in-depth discussion of JDNI, see (URL).

JDNI

Normally, Workers will be running inside an J2EE container and the JDNI service will be available by default.

When Workers are run in standalone mode (from the command line) they will use the filesystem implementation of JDNI. In this mode, the deployer needs to setup a concrete Syrup implementation and serialize this with the aid of a filesystem, located at ~/syrup/jdni/. Conveniently, the distribution contains the necessary scripts to do just that, so setting up Syrup is just a matter of minutes.

JDBC

Syrup is enabled to work on top of a standard JDBC implementation. To deploy Syrup in this configuration you'll need to run the following command:

  1. ~/syrup/bin/deploy.sh
    Deploys the default MySQL implementation of Syrup using localhost and database test.
  2. ~/syrup/bin/deploy.sh jdbc:mysql://host/test
    Deploys the default MySQL implementation connecting to host using database test. 
  3. ~/syrup/bin/deploy.sh jdbc:mysql://host/test username password
    Deploys the default MySQL implementation connecting to host using database test with username and password.

Note that all commands are based on the filesystem implementation of JDNI. If you want to deploy Syrup using a different JDNI setup you have to change the -Djdni options accordingly. See ~/syrup/bin/syrup.sh for ideas of how to pass other JDNI implementations.

Running Syrup

After deploying Syrup, it needs to be set to its embryonic state before anything else.

~/syrup/bin/syrup.sh reset

The initial WorkSpace

The WorkSpace is now ready and set to receive any external events from the 'outside' world. The initial WorkFlow contains two Tasks. The first one represents the 'outer world'. The second Task is the Launch Task. From this, the Syrup system can be fully bootstrapped - it is ready to receive its first WorkFlow.

The first Workflow

While you are running WorkFlows, you can always check the state of the WorkSpace using the command: 

~/syrup/bin/syrup.sh get

This commands sends its output to stdout and dumps the full WorkSpace with all its Tasks and Links, serialized into XML. Running this command when the WorkSpace is in its initial state should get you the following: 

<get>
 <ptask-context key="2" parentKey="0" name="root"
  functionClass="org.syrup.functions.Launch" description="This is the bootstrapping task"
  orType="true" executable="false" done="false" creationTime="19700101-010000"
  modificationTime="19700101-010000" modifications="0" worker="" isParent="false">
   <in-1 lkey="1-1" lparentKey="0" lname="external" />
   <in-2 lkey="1-2" lparentKey="0" lname="external" />
   <out-1 lkey="1-1" lparentKey="0" lname="external" />
   <out-2 lkey="1-2" lparentKey="0" lname="external" />
 </ptask-context>
 <ptask-context key="1" parentKey="0" name="external"
  functionClass="" description="External world representation" orType="false"
  executable="false" done="false" creationTime="19700101-010000"
  modificationTime="19700101-010000" modifications="0" worker="" isParent="false">
   <in-1 lkey="2-1" lparentKey="0" lname="root" />
   <in-2 lkey="2-2" lparentKey="0" lname="root" />
   <out-1 lkey="2-1" lparentKey="0" lname="root" />
   <out-2 lkey="2-2" lparentKey="0" lname="root" />
  </ptask-context>
</get>

Let's start the helloworld.xml example by feeding it to the WorkSpace via its first 'external' input. 

~/syrup/bin/syrup.sh in1 < ~/syrup/test/helloworld.xml

Now the Launch Task is ready to expand the WorkFlow into new Tasks and for this to happen the Launch Task needs to be executed.

Executing Tasks in a single step.

To execute all Tasks contained in a WorkSpace you have to run the following command: 

~/syrup/bin/syrup.sh step -executable=equal true -worker=equal ""

This command queries for all executable Tasks in the WorkSpace that are not being executed. Then the Worker executes them one by one and finally terminates. Because the Launch Task is the only Task that is executable right now, it will be immediately executed by the Worker.

After the Worker has terminated, the Launch Task has expanded the helloworld WorkFlow and added new Tasks to the WorkSpace. You need to run another round of execution to execute the newly added helloworld Tasks. 

~/syrup/bin/syrup.sh step -executable=equal true -worker=equal ""

Now a result is ready on the external output (check this with get). The following command writes the final result to out.txt containing the message 'Hello World!'. 

~/syrup/bin/syrup.sh out1 > out.txt

Running a daemon Worker

You can also run the org.syrup.workers.DefaultWorker in daemon mode. In this mode, the Worker will run forever until it is killed or if the WorkSpace cannot be reached by the Worker. This daemon service is helpful when there is no cron available. However, the alternative way of running Workers in 'daemon' mode is by using cron, for instance by using the following crontab entry: 

* * * * *  ~/syrup/bin/syrup.sh step -executable=equal true -worker=equal ""

The cron daemon will now execute syrup.sh every single minute. If there are no Tasks to be executed the Worker will terminate immediatly. But if there are lots of Tasks to be executed then every minute an additional Worker will be started by the cron. And each additional Worker will concurrently be running next to previously launched Workers.

If this cascading behaviour is not desirable, you can decide to run the org.syrup.workers.DefaultWorker in daemon mode. Use the following command to do just that: 

~/syrup/bin/syrup.sh execute -executable=equal true -worker=equal ""

Running multiple daemon Workers

Running multiple Workers is easy. Just start additional Workers in daemon mode. Syrup makes sure that Workers don't execute the same Tasks and that they are efficiently scheduled among multiple Workers.

Querying the WorkSpace

While multiple daemon Workers are running you can query the WorkSpace to see what is going on. The following commands show some of the possible queries. 

  1. ~/syrup/bin/syrup.sh get -executable=equal true -worker=equal ""
    Gets all executable Tasks.
  2. ~/syrup/bin/syrup.sh get -executable=equal -worker=not ""
    Gets all executable Tasks that are executed by Workers.
  3. ~/syrup/bin/syrup.sh get -functionClass=like "%Launch%"
    Gets all Tasks which functionClass matches *Launch*.
  4. ~/syrup/bin/syrup.sh get -modificationTime=greater 20050103-190000
    Gets all Tasks that are modified after 2005-01-03 - 19:00:00.
  5. ~/syrup/bin/syrup.sh get -isParent=equal false
    Gets all Tasks that are not parent.
  6. ~/syrup/bin/syrup.sh get -parentKey=34334233 -name=equal sh1
    Gets the Task with parentKey '34334233' and name 'sh1'.

You can build any query to match certain attributes of the Tasks held by the WorkSpace. Just take a look at the XML attributes generated by the get command to see what Task attributes can be queried. Note that you can also use the same attribute qualifiers to execute certain Tasks, not only to get their state. For instance, you may want to execute Tasks that have been modified before a certain time! 

~/syrup/bin/syrup.sh step -executable=equal true -worker=equal "" -modificationTime=less 20050103-190000

Platform dependencies

Syrup is platform independent as far as Java is platform independent. If one wants to run Shell Tasks however, platform dependencies cannot be avoided or need to be carefully circumvented. If you decide not to use Shells (which is good) you may want to add extra Function implementations that can replace them. To do this, you should implement the org.syrup.Function interface and add these implementations to the CLASSPATH when running Workers. Please look in the ~/syrup/src/org/syrup/functions directory for examples of how to build your own Functions.

Hints and Tips

Sample Workflows

The ~/syrup/test directory contains a number of examples. You can try these on by one, following a scheme similar to running the helloworld.xml example.

Visualization with yed

You can download the latest version of yed. Load the yed preferences file found at ~/syrup/xsl/yed.ypf. To visualize the WorkSpace you first have to dump the state of the WorkSpace with: 

~/syrup/bin/syrup.sh get > st.xml

Then transform st.xml into st.graphml using the build-in XLST processor of java. 

java org.apache.xalan.xslt.Process -IN st.xml -XSL ~/syrup/xsl/state_to_yed.xsl > st.graphml

Load st.graphml with yed and use yed's hierarchical layout tool for creating nice diagrams.

Multi-platform WorkFlows

Running Workers executing multi-platform WorkFlows may seem impossible at first but can be achieved relatively straight-forward by using environments.

Each Task can be assigned an environment indicating that it should be only run by Workers that match it. A Task with environment 'windows' should only be executed by Workers that are run on the Windows platform. Alternatively, Tasks with environment 'unix' should only be executed by Workers that are run on the UNIX platform.

The following steps shows how you can execute distributed, multi-platform WorkFlows.

  1. ~/syrup/bin/deploy.bat jdbc:mysql://host/test username password
    (On windows). Deploy the Syrup implementation.
  2. ~/syrup/bin/deploy.sh jdbc:mysql://host/test username password
    (On unix). Deploy the Syrup implementation (same database as the previous step).
  3. ~/syrup/bin/syrup.sh reset
    (On unix). Resets the WorkSpace to the initial phase.
  4. ~/syrup/bin/syrup.bat execute -executable=equal true -worker=equal "" -environment=equal windows
    (On windows). Selects and executes all Tasks in environment 'windows'.
  5. ~/syrup/bin/syrup.sh execute -executable=equal true -worker=equal "" -environment=equal unix
    (On unix). Selects and executes all Tasks in environment 'unix'.
  6. ~/syrup/bin/syrup.sh execute -executable=equal true -worker=equal "" -environment=equal ""
    (On unix). Selects and executes Tasks agnostic to the environment.
  7. ~/syrup/bin/syrup.bat execute -executable=equal true -worker=equal "" -environment=equal ""
    (On windows). Selects and executes Tasks agnostic to the environment.
  8. ~/syrup/bin/syrup.bat in1 < ~/syrup/test/multiplatform.xml
    (On windows). Starts the multi platform WorkFlow by feeding it to the Launch Task.

Troubleshooting

Crashed Workers

There is always a chance that Workers will crash or fail for some unknown reason, at least, unknown to Syrup. In this rare case, a Worker may still be registered as executing a Task but is actually not doing anything. One way to find out that a Worker has crashed is to send it a request, using its registered address. If there is no reply, such Worker can be considered to be not functioning.

Use the following command to stop Workers that are not functioning. 

~/syrup/bin/syrup.sh stop -executable=equal true -worker=not ""

This command will send requests to Workers that have a registered address. The stop command frees all the Tasks that don't receive a reply from the associated Worker. After that, other Workers may re-execute the re-enabled Tasks.

Firewalls

When Workers execute Tasks behind firewalls it is possible that Workers cannot 'see' each other when requests are sent to them. In such case, a Worker may unjustifiable be freed by another. You have to make sure that Workers can always connect to each other so that they are able to check for activity. However, the chance of Workers not seeing each other is low because the URL addresses take ports that are in the high range (for example: http://10.231.22.120:2342) and are normally not blocked by firewalls.

Impure Tasks

Running Tasks that are impure, like removing files or changing state outside the realm of Syrup, may result in serious problems. Specifically, running multiple daemons that execute impure Tasks, without carefully setting them up, is not advised.

Still, if impure Tasks cannot be avoided you may want consider to put impure Tasks in a separate environment 'stateful' and use a seperate Worker daemon to execute them. This is similar to the approach of running multi-platform WorkFlows.

~/syrup/bin/syrup.sh execute -executable=equal true -worker=equal "" -environment=equal stateful

This command will serialize all stateful executions by a single Worker. For the remaining Tasks that are pure, you can run multiple Workers with the additional qualifier:

~/syrup/bin/syrup.sh execute -executable=equal true -worker=equal "" -environment=not stateful

_