OS Zoomstack - A mini adventure

by Dave Barter on

Ordnance Survey recently released OS Zoomstack a new mapping dataset that is currently available to anyone on a trial basis. Zoomstack presents map data as vector tiles. I've written about these before here. Vector tiles are the way forward for those of us rendering maps for a number of reasons:-

  • the client does the vast majority of rendering work reducing server requirements
  • they're compact and download faster
  • they provide new options for dynamic client side styling of maps emerge

I've been interested in vector tiles for a long time and within Nautoguide we've been working towards replacing all of our WMS base map services with these tiles. We've cracked the workflows for generating these tiles but have been held back by Openlayers, the mapping library we've built our client-side libraries upon.

We love Openlayers, it's easy to extend and we've invested heavily in controls and libraries that create a rich mapping experience. We also like the fact that it is purely open source and has no links with the Mapbox ecosystem. Mapbox are lovely I'm sure, but they are a large VC funded body. We have no idea what will and will not be free in the medium to long term.

Unfortunately for us, Ordnance Survey have targeted Zoomstack at Mapbox users and infrastructure. This makes sense as Mapbox led the charge with vector tiles and have the richest libraries and tools for styling and rendering them.

Hmmmmmm.

I decided to see if we could get Openlayers to support the OS styles natively. It would be great for us if we could simply ingest the Zoomstack styles into our own codebase. I managed it and here is a quick overview of "how".

A style parser

Openlayers and mapbox implement client side styling in completely different manners.

Openlayers requires a style function for a layer that has access to the entire styling API. This function returns an array of style objects that are then applied to the features concerned. This is hugely flexible, you can write massive swathes of logic within these styles and cope with a myriad of feature property combinations. The downside is you have to write javascript to achieve this.

Mapbox uses a style file constructed in json. This is relatively easy to understand and has the advantage of working well with graphical design tools such as Mapbox Studio.

As you can see there is not a lot of compatibility going on here. So I sat down to see if I could write some code to parse a style file within an Openlayers style function

It turned out to be relatively straightforward. I needed to do the following things:-

  1. write a simple function to parse the Mapbox json format
  2. write colour converters for hsl/hex to rgb
  3. write an interpolation function to interpolate colours/widths based on zoom level
  4. write a filter function to cope with filter conditions in the OS Zoomstack style

It took about a day to do, the code is by no means perfect but I've got a lovely render of the OS Zoomstack lite style running in Openlayers with no Mapbox code required at all.

Screen-Shot-2018-09-12-at-11.57.34
Dave I can see your house from here

Along the way I had to solve a number of issues. The hardest of which was label density. OS have labeled every section of road in Zoomstack which is fine for single streets but causes issues like this with main roads

Screen-Shot-2018-09-12-at-12.00.49

I managed to reduce this density by keeping a temporary array of road names and using this to decide whether or not to render one. It works but is not ideal.

Openlayers performed reasonably well with these tiles but nowhere near as well as Mapbox. This is down to WebGL. WebGL uses graphics cards to speed the rendering within web browsers for vector based objects. Mapbox uses this natively, unfortunately Openlayers doesn't. WebGL is available within Openlayers but not for vector tile rendering.

If you are interested all of the code is here

(ps. I am well aware of the Boundless style function here but it doesn't support interpolation which Zoomstack styles use extensively)

Integrating mapbox-gl within Openlayers

The problem with mapbox-gl is that it is focused upon the display of maps rather than the features on top of them. We have a set of drawing, placement and interaction controls within Openlayers that we'd have to write ground up to use with mapbox-gl. The ideal scenario would be to use mapbox-gl for display and Openlayers for interaction, could this be done?

To Google and we find this. A stroke of genius and I wish I'd thought of it. The approach is simple:-

  • implement a Mapbox layer and stop it listening to any map events
  • bind the Openlayers map events to this layer via a view
  • job done

I excitedly asked Dan our web developer to see if this would work in our framework. Five minutes later he'd created this.

It works perfectly and we have the best of both worlds.

Going fully serverless

Our mission within Nautoguide is to rule the world. And ruling the world requires that you can scale. It's not easy to dominate from a single 4-core PC sweating in the corner of the office. To this end we've made the bold move into an almost entirely serverless architecture. All of our APIs reside in Amazon Lambda as micro-services and we host our database within Amazon RDS, eagerly awaiting the Aurora serverless PostgreSQL offering.

But Zoomstack is provided in an mbtiles format and as such needs a tile server to push it out to clients. This is pretty easy to set up if you use docker, one line will get you up and running:-

docker run --rm -it -v "<path to OS-Zoomstack.mbtiles directory>":/data -p 8080:80 klokantech/tileserver-gl

However, we don't want any servers at all. Most modern web clients will pull tiles from a static web location using zxy notation. Where z represents the current zoom level and x/y represent the co-ordinates of the required tile. For example in Openlayers this is:-

url: 'http://localhost:8080/data/OS-Open-Zoomstack/{z}/{x}/{y}.pbf'

If we can get the tiles loaded into Amazon S3 in a directory structure that reflects our URL then we do not require a server at all and are not worried about the volume of tile requests as that's Amazon's problem (and our accountant's when he gets our hosting bill).

This is relatively straightforward to do. First we need to unpack the tiles from the mbtiles file. This can be done using a utility that comes with Mapbox Tippecanoe called "tile-join".

tile-join --output-to-directory=os-zoomstack OS-Open-Zoomstack.mbtiles

Having run this command we're left with the directory os-zoomstack which contains a sub directory for every zoom level. We now need to upload this to Amazon S3. This would be unbelievably tedious if done by hand as you will quickly find there are many thousands of sub directories. So let's use another utility s3-parallel-put to help us get them up there:-

Create the file load_levels.sh as

for ZOOM_LEVEL in "$@"
do
        /usr/bin/s3-parallel-put --header=Content-Type:application/x-protobuf \
        --header=Content-Encoding:gzip \
        --put=stupid --bucket=tiles.url.com \
        --prefix=zoomstack/$ZOOM_LEVEL \
        --insecure --host=s3.amazonaws.com \
        --bucket_region=eu-west-1 $ZOOM_LEVEL/
done

Make sure you've added your S3 credentials into environment variables. Then run:-

./load_levels.sh 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Apologies for the unix bias on this bit. I'm sure you Windows users can figure something out using Powershell. But at the end of all of this you'll have Zoomstack tiles in S3 and no need for any servers whatsoever.

The end bit

We should all salute Ordnance Survey for releasing this data set. It's a brilliant step forward and removes a whole raft of hassle involved in creating your own vector tiles. I think they've got the layering just about right. There are a few minor niggles about labeling and attributes that we'll be feeding back but on the whole we think they've done a great job. Be aware that this is currently a trial and we don't know what Ordnance Survey are planning next. I'd urge them to keep this out there as open data.

As a result of this release we're now converting all of our basemaps over to vector tiles. The user experience is definitely enhanced and now that we've found a path to integrating mapbox-gl with Openlayers we feel we've got the best of both worlds.

OS Zoomstack - A mini adventure

Contact us