Getting started

This page describes how to setup a working Job-Runner environment, including the Job-Runner WebSocket Server and the Job-Runner Worker. After a successful installation, you’ll have four processes running:

  1. ./manage.py runserver, webserver for admin, REST api and dashboard (part of Job-Runner)
  2. ./manage.py broadcast_queue, queue broadcaster (part of Job-Runner)
  3. ./scripts/job_runner_ws_server, WebSocket server (part of Job-Runner WebSocket Server)
  4. ./scripts/job_runner_worker --config-path job_runner_worker.ini, the worker executing our jobs

For an explanation of the role of each component, see Overview.

Install all components

  1. Create three virtualenv environments called job-runner, job-runner-ws-server and job-runner-worker. If you are on Ubuntu, you might want to install the virtualenvwrapper package first:

    # if you need to install virtualenvwrapper first
    $ sudo apt-get install virtualenvwrapper
    
    $ mkvirtualenv job-runner && deactivate
    $ mkvirtualenv job-runner-ws-server && deactivate
    $ mkvirtualenv job-runner-worker && deactivate

    See also

    http://virtualenvwrapper.readthedocs.org/en/latest/ if you are not familiar with virtualenvwrapper and virtualenv

  2. Install the Job-Runner in the job-runner virtualenv. To activate this virtualenv, execute workon job-runner. See Setup Job-Runner for installation details.

  3. Install the Job-Runner WebSocket Server in the job-runner-ws-server virtualenv. See the documentation in the job-runner-ws-server repository for installation instructions.

  4. Install the Job-Runner Worker in the job-runner-worker virtualenv. See the documentation in the job-runner-worker repository for installation instructions.

Get all components up and running

Job-Runner

  1. Open two console tabs and execute in the first one:

    $ workon job-runner
    $ manage.py runserver

    Execute in the second tab:

    $ workon job-runner
    $ manage.py broadcast_queue

    Note

    The first process is responsible for serving all HTTP requests (dashboard, admin and RESTful API). The second process is responsible for broadcasting scheduled runs to the subscribed workers.

  2. Open a browser and point it to http://localhost:8000/admin/. This will open the admin interface of the Job-Runner. While executing manage.py syncdb you were asked to create superuser credentials, use these to login. If you did not create any, open a new console and execute:

    $ workon job-runner
    $ manage.py createsuperuser
  3. In the admin interface, assign yourself to a group:

    1. Go to Auth - Users
    2. Click your username
    3. Under Permissions - Groups click the + icon to add yourself to a new group (call it Test Group for now)
    4. Save the user

    Note

    You need to be assigned to at least one group, so in the following steps, you can give yourself access to a project.

  4. Create a worker:

    1. Go to Job Runner - Workers
    2. Click on the Add Worker button
      • Title: Test Worker
      • API key: testworker
      • Secret: verysecret
    3. Click the save button

    Note

    A worker is responsible for the actual execution of the job and will communicate back with the Job-Runner over the RESTful API with the credentials you’ve just setup.

  5. Create a worker-pool:

    1. Go to Job Runner - Worker pools
    2. Click on the Add worker pool button
      • Title: Test Worker Pool
      • Workers: Select Test Worker
    3. Click the save button

    Note

    A worker-pool contains one or multiple workers. The advantage of having multiple workers is that jobs using this pool will be loadbalanced. For now however we keep it simple and assign only one worker.

  6. Create a project:

    1. Go to Job-Runner - Projects
    2. Click the Add project button
      • Title: Test Project
      • Viewers (groups): select Test Group
      • Project admins (groups): select Test Group
      • Worker pools: select Test Worker Pool
    3. Click the save button

    Note

    You’ve just created a project which is visible for users in Test Group, as well users in Test Group will be able to start had-hoc job-runs and are able to enable / disable the enqueue of a job. Test Worker Pool will be available when you create a new job within this project.

  7. You now have created a group, project and a worker-pool with one worker :) Leave both processes you started in the first step running!

Job-Runner WebSocket Server

  1. Open a new console tab and execute:

    $ workon job-runner-ws-server
    $ job_runner_ws_server

    Note

    The WebSocket server is responsible for sending events (sent by the workers) to the dashboard. This makes it possible to show the realtime state of job-runs.

  2. That’s it! Leave this process running :)

Job-Runner Worker

  1. Open a new console tab and execute:

    $ workon job-runner-worker
  2. Create a file named job-runner-worker.ini with the following content:

    [job_runner_worker]
    api_base_url=http://localhost:8000/
    api_key=testworker
    secret=verysecret
    concurrent_jobs=4
    log_level=debug
    script_temp_path=/tmp
    ws_server_hostname=localhost
    ws_server_port=5555
    broadcaster_server_hostname=localhost
    broadcaster_server_port=5556

    Please refer to the documentation in the job-runner-worker repository for the meaning of these variables.

  3. Now start the worker by executing:

    $ job_runner_worker --config-path job-runner-worker.ini

Congratulations! You now have all components up and running. If you point your browser to http://localhost:8000/, you will see an empty dashboard, with top-right a label Dashboard is live, meaning that the dashboard is connected to the WebSocket server. If this is red with a warning, please make sure the job_runner_ws_server process is still running! An other reason could be that your browser can’t connect to the port the WebSocket server is listening on (eg: when you are running the components in a VM and port 5000 is not forwarded).

Your first job

In this part, you’ll setup and schedule your first job! This will be a simple Python script, printing “Hello world!” and then sleeping between 3 - 15 sec. This script will be re-scheduled every 1 minute after the schedule dts of the previous run.

Note

There are two ways of rescheduling:

  1. re-scheduling by schedule dts
  2. re-scheduling by complete dts

If you have a job-run that is scheduled to run every minute (starting 00:00:00) and would take 25 seconds to complete, it would run at the following times for each case:

  1. 00:00:00 - 00:01:00 (00:00:00 + 1 min) - 00:02:00 (00:01:00 + 1 min)
  2. 00:00:00 - 00:01:25 (00:00:25 + 1 min) - 00:02:50 (00:01:50 + 1 min)
  1. Point your browser to http://localhost:8000/admin/

  2. First create a Python template which will form the base for all future Python jobs:

    1. Go to Job-Runner - Job templates

    2. Click the Add job template button

    3. Enter the following:

      • Title: Python

      • Body:

        #!/usr/bin/env python
        
        {{ content|safe }}
        
      • Project: Select Test Project

    4. Click the save button

  3. Now create the actual job:

    1. Go to Job-Runner - Jobs

    2. Click the Add job button

    3. Enter the following:

      • Title: Hello world!

      • Job template: Select Python

      • Worker pool: Select Test Worker Pool

      • Script content:

        import random
        import time
        
        print "Hello world!"
        time.sleep(random.randint(3, 15))
        
      • Reschedule interval: 1

      • Reschedule interval type: select Every x minutes

    4. Under Runs, select the current date and type by clicking on the date-picker and time-picker icons.

    5. Save the job.

  4. Now go to http://localhost:8000/. If all components are set-up correctly, you should see the job you just created moving from scheduled > in queue > started > completed!