The last time Hackerfall tried to access this page, it returned a not found error. A cached version of the page is below, or click here to continue anyway

Announcing phoenix | SDSLabs Blog

As part of the recently conducted BackdoorCTF 15, we developed a redis based queue taskrunner for phantomjs tasks called phoenix that we are open-sourcing today. This blog post is about the problem we faced and the solution we developed and how it can help you.

problem

As part of two challenges in the CTF (MEDUSA, JSHUNT), we wanted a browser solution that could automatically open the webpages for the challenge in a safe environment after each submission. These challenges are web challenges and require techniques such as XSS, which means that they can only be exploited in a real browser.

Unfortunately, running a full browser such as Chrome/Firefox is not feasible in such sitations. Instead, we decided to use the most popular headless browser phantomjs for our task. Using phantom is hard, though. It has its own API, which uses JS, but is completely different in its own manner (too many synchronous methods, for one).

As part of the CTF, we had to build a queue system as well, which would do the following:

Since a lot of this is common to both the tasks, we decided to create a small tool that helps us run these jobs.

phoenix

phoenix handles the following for you:

This is all done via a mix of redis, phantomjs and nodejs.

configuration

phoenix is configured via a config.yml file, which is expected to be present wherever phoenix is run. You can see the sample config file for a list of configuration options that phoenix supports. These include things like user agent support, custom headers, request body (in json or post format), basic authentication support.

Another thing we support is the ability to run custom javascript before the page visit starts. This allows you to set default variables (such as tokens and secrets) that will be available in the JS context once the page visit is made.

page visit

We have a lot of sensible defaults for phantom, which include a maximum timeout of 10 seconds, and extended timeouts for whenever a web request (such as image/script/ajax) is made.

logging

All jobs that start get their own id, and create their own directory with three files: config.json, page.log, and browser.log. The first holds the complete configuration object that is passed to phantom. The second is the console log of the web page (which means any console.log statements made in the web page context). The third is a higher level log containing browser events such as redirects, web requests and timeout extensions.

We plan to make the logging format configurable, but for now it is file system based.

job config

It is assumed that (generally) each job will have its own url, slightly differing from the tasks standard url. This is handled in two ways:

queueing

To push a job to a queue, you do the following:

log in redis

For each job, the job id (generated randomly) is stored in the channel:log:id key in redis. This can be polled to check whether the job has finished or not.

As you can see, phoenix is a very robust mechanism to handle a variety of job queues. It is available on npm today as the phantom-phoenix package, which sets up a binary called phoenix. Further instructions on usage etc can be found on the repository.

We hope it will be useful to people looking to setup a queue system based around phantomjs instances. Since it uses redis as the queue mechanism, it can be easily scaled to multiple machines as well.

Continue reading on blog.sdslabs.co