The last time Hackerfall tried to access this page, it returned a not found error. A cached version of the page is below, or clickhereto continue anyway

Ketrew: Home

Ketrew: Keep Track of Experimental Workflows

Ketrew is:

This is the 2.0.0 version of Ketrew, serialized content is backwards compatible with the 1.x.x series but the protocol is not; the EDSL API requires minor changes (all caught by the OCaml compiler). See also the documentation for the master branch.

Build & Install

Ketrew requires at least OCaml 4.02.2 and should be able to build & work on any Unix platform.

From Opam

If you have opam up and running:

opam install ketrew

Then you need at runtime ssh in the $PATH.

This gets you the ketrew executable and the ketrew_pure and ketrew libraries.

Without Opam

See the development documentation to find out how to build Ketrew (and its dependencies) from the sources.

Getting Started

Ketrew is very flexible and hence may seem difficult to understand, let's get a very minimalistic workflow running.

The first time you use Ketrew, you need to configure it, simplest by calling ketrew init, and please choose an authentication token:

rm -fr ~/.ketrew/ # if you had a previous configuration there
ketrew init --with-token my-not-so-secret-token

by default this will configure Ketrew in $HOME/.ketrew/ with a client/server mode not using TLS on port 8756 (see ketrew init --help you can even ask it to generate self-signed TLS certificates). See also the documentation on the configuration file learn how to tweak it.

You can check that the client or the server are configured:

ketrew print-configuration
ketrew print-configuration --configuration-profile server
ketrew print-configuration -P daemon

You may then start a server:

KETREW_PROFILE=daemon ketrew start-server

and then open the GUI:

ketrew gui

which is just trying to load http://127.0.0.1:8756/gui?token=my-not-so-secret-token

You can always stop the server or check its status:

ketrew stop -P daemon
ketrew status -P daemon

The ketrew submit sub-command can create tiny workflows:

ketrew submit --wet-run --tag 1st-workflow --tag command-line --daemonize /tmp/KT,"du -sh $HOME"

The job will appear on the WebUI and you can inspect/restart/kill it.

If you don't like Web UI's you can use the text-based UI:

ketrew interact

The EDSL: Defining Workflows

The previous section uses ketrew submit to launch an extremely simple workflow, to go further we need the EDSL.

Overview

The EDSL is an OCaml library where all the functions are used to build a workflow data-structure. Then, one function: Ketrew.Client.submit is used to submit workflows to the engine.

A workflow is a Graph of targets.

There are 3 kinds of links between targets:

Any OCaml program can use the EDSL (script, compiled, or even inside the toplevel), see the documentation of the EDSL API.

Example

This example is a single-target workflow that runs an arbitrary shell command on an LSF-based cluster:

#use "topfind"
#thread
#require "ketrew"
let run_command_with_lsf cmd =
  let module KEDSL = Ketrew.EDSL in
  let host =
    
    KEDSL.Host.parse
      "ssh://user42@MyLSFCluster/home/user42/ketrew-playground/?shell=bash"
    
  in
  let program =
    
    KEDSL.Program.sh cmd in
  let lsf_build_process =
    
    KEDSL.lsf
      ~queue:"normal-people" ~wall_limit:"1:30"
      ~processors:(`Min_max (1,1)) ~host program
  in
  
  KEDSL.target
     "run_command_with_lsf"
     ~make:lsf_build_process

let () =
  let workflow =
     
     run_command_with_lsf Sys.argv.(1) in
  
  Ketrew.Client.submit workflow
  

If you actually have access to an LSF cluster and want to try this workflow, put it in a file my_second_workflow.ml, and simply:

ocaml my_second_workflow.ml 'du -sh $HOME'

To learn more about the EDSL, you can also explore examples of more and more complicated workflows (work-in-progress).

Where to Go Next

From here:

License

It's Apache 2.0.

Continue reading on seb.mondet.org