Varnish in non-compiler environments

Last weekend I’ve visited the Loadays conference where I sat in the presentation of Thijs Feryn’s “Varnish in action”. Even though most of the talk was pretty familiar for me personally, a real interesting question was raised from the audience: is it possible to run varnish in an environment where there is no compiler available. It looks that I’ve just found the answer..

The problem

Varnish, for the people who don’t know what it is, is a HTTP accelerator, or more precisely, a caching proxy and load balancer. The main thing with varnish is that is uses it’s own configuration language called VCL (varnish configuration language). This language is compiled at run-time by a c-compiler into a binary for mostly performance reasons. Nice thing about it that it’s easy to add your own varnish functionality with the help of your own C code. For an example, see the blogpost of one of my colleagues about this.

So since varnish will compile the configuration, a C compiler must be present on your web-server. This however is not possible to comply to some environments. For instance,  when servers handles credit card processing, the gaming industry (servers connecting to betting sites etc). Rules state that no compilers or make functionality must be present on servers directly connected to the internet. This means that Varnish is not a viable option.

The solution

Varnish supports the “cc_command” functionality where you can actually change the compiler flags that Varnish will use for compiling the configuration. You use this when you want to link against shared objects you need when you use your own defined functionality (again, see the blogpost about mobile detection with Varnish). This “hook” can be misused by not actually calling a compiler, but doing some other stuff.

Varnish normal cc_command is :

exec cc -fpic -shared -Wl,-x -o %o %s

where %o will be the name of the shared object that is generated, and %s is the source. Most likely, these files will be something like ./vcl.1P9zoqAU.so and ./vcl.1P9zoqAU.c. What Varnish cares about is that there should be a .so delivered that has the same name as %o. So we use another cc_command:

exec cp /usr/local/lib/varnish/config.so %o

What this will do is copy a binary shared object to the actual shared object that is used by varnish. If this shared object is already a binary created by varnish, it will run this as the configuration. It’s that easy actually :)

You can add this command to your standard varnish startup script:

varnishd -s malloc,32M -a 0.0.0.0:81 -p 'cc_command=exec cp /usr/local/lib/varnish/config.so %o'

This will start varnish with a 32MB memory cache, running on port 81. The -p option will add our custom “compiler” command.

Now the only problem we have is creating the actually /usr/local/lib/varnish/config.so. But this is easy as well. On our “development” system (a staging outside the protected infrastructure that is actually able to compile), we have another varnish instance which main purpose is to create a shared object configuration:

varnishd -s malloc,32M -p 'cc_command=exec sh /usr/local/bin/genconfigso.sh %s %o'

Varnish will call the /usr/local/bin/genconfigso.sh script that does the following:

#!/bin/sh 
cc -fpic -shared -Wl,-x -o $2 $1 
cp $2 config.so

After “starting” varnish, there will be config.so which you then can copy to your production varnish system.

Conclusion

This is just a proof of concept since it’s not tested in production environments. There can be problems that I haven’t tested yet, like reloading the configuration for instance. But at least it shows that it’s not needed to have a compiler on your system in order to run varnish.