Beeble GeoJSON

In today’s interconnected world, moving data is a given – especially when applications rely on network-based communication. Of course this is nothing new. Since almost the birth of computers we’ve had the need to transport data from one place to another.

The rise of the Internet has only exponentially accelerated this need and, today, one of the most widely used open data-interchange standards is JSON (JavaScript Object Notation). It’s text-based, human-readable, enjoys wide support and maps easily to JavaScript objects. So if you’re looking to share Smallworld GIS data, you could do far worse than JSON.

But JSON is a high-level format for describing name/value pairs and ordered lists of values. It says nothing about what those pairs and values should contain (just as XML says nothing about what elements should be defined for specific applications). To meaningfully transfer GIS data, we need a particular grammar (or vocabulary), much like SOAP back in the heyday of XML.

GeoJSON

And that’s where GeoJSON enters the picture. But while SOAP was a heavyweight protocol, with quite a bit of associated WS* baggage, and not specifically created for spatial applications, GeoJSON is light and airy. It’s flexible but still has the required structure to keep the wheels turning and the spatial data flowing.

As the name suggests, GeoJSON is built on JSON and is able to describe geographical features and their associated non-geographical attributes. So it is an ideal format for serializing data from Smallworld. If we can serialize GIS data to GeoJSON, these data become accessible to a host of applications around the world.

But it’s even better than that.

There are myriad libraries and applications that natively understand GeoJSON, so moving data out of the proprietary VMDS and into an open, widely-understood format results in massive benefits (as an example, MongoDB understands and stores GeoJSON).

But how can we do this? As it turns out, it’s quite easy. The GeoJSON format is relatively simple and, using Magik’s inheritance mechanisms, we can implement GeoJSON serialization rather efficiently.

I had written a JSON module in Magik but, fortunately, decided to do a quick Google search before extending it to GeoJSON. And lo and behold, someone had already done it. Acefeal (a Spanish blog) created a nice little Magik GeoJSON module.

It does most serialization well but it doesn’t handle join fields. I added support for joins and also added checks to ensure circular joins are handled, as well as adding a depth parameter so you can specify how deep the joins are followed. I’ve also refactored the code.

You can find the refactored beebleGeoJSON code on GitHub. Go ahead and download it to your favourite folder.

Now that you’ve got your hands on the code, let’s do a little GeoJSON Magic (see what I just did there?).

The first thing you’ll want to do is load the module into a session (or image if you’re still using version 4.x). Do that by typing the following command at the magik prompt (or use whichever method you’re comfortable with), of course change the path to the folder in which you downloaded beebleGeoJSON.

Magik> load_file_list("D:\Services\Smallworld521\cambridge_db\beeble\beeble_geojson")

Now you can serialize objects. Let’s do a hotel from the Cambridge database.

Magik> v << gis_program_manager.cached_dataset(:gis)
Magik> t << v.collections[:hotel]
Magik> e << t.an_element()
Magik> e.as_geojson()

This should produce the following output.

Magik> e.as_geojson()
{"type":"Feature","properties":{"Record ID":"hotel132:(Acorn Guest House)","Name":"Acorn Guest House","Address 1":"154 Chesterton Road","Address 2":"Cambridge","Type":"Guest House","Annotation":"unset"},"geometry":{"type":"GeometryCollection","geometries":[{"type":"Point","coordinates":[0.129869073,52.215243686]}]}}
Magik>

You can copy the output to an online GeoJSON viewer (I like the one at http://geojson.io) to see it prettified and on a map.

You can also serialize the entire hotel table like this…

Magik> t.as_geojson()

There are only 15 hotels in the table, so it finishes quickly, but keep in mind it will take far more resources and time if you do this for a table with a million records, so ensure you know your tables before GeoJSONing them willy-nilly.

Saving GeoJSON Output

How about persisting your output? Would you like to do that?

Yeah… me too.

You can serialize a few tables and write them to files using the code below. It outputs GeoJSON for towns, hotels and footpaths to individual files.

_global serialize_objs <<
_proc(p_folder) 
    
    _local l_tables << {:town, :hotel, :footpath}

    _local l_v << gis_program_manager.cached_dataset(:gis)
    
    _for l_tn _over l_tables.fast_elements()
    _loop 

        _local l_fn << write_string(p_folder, "\", l_tn, ".geojson")
        _local l_etos << external_text_output_stream.new(l_fn)
        _local l_t << v.collections[l_tn]

        write("serializing:", l_tn, " to ", l_fn)  
        
        l_t.as_geojson(l_etos)

        l_etos.close()

    _endloop 

_endproc 

Notice line 17, the as_geojson() method can take a stream as an optional argument, so we can pass it a file stream argument (of course we could also pass it a socket stream if we wanted to communicate across a network — such as the Internet).

Compile the code and then execute it using the following command (ensure you change the output path, given in the only argument, to one suitable for your environment).

Magik> serialize_objs("D:\Trillian\docs\GeoJSON")

Once it completes, the GeoJSON files should be in the folder you specified.

You can even store your newly minted GeoJSON in MongoDB – because Mongo understands GeoJSON. Mongo also supports basic spatial querying, so it’s not a giant leap to see how you can create applications independent of Smallworld and serve GIS data to the world in a massively scalable and efficient manner. Of course Mongo isn’t a full spatial database and doesn’t have a network model or built-in connectivity, among other things, so your mileage may vary depending on your applications’ needs.

Into the Future

If you’ve been following along so far, you can probably see where this is going. Once we have the ability to serialize to GeoJSON, it opens new opportunities to spread Smallworld GIS data to other applications and even access GIS data via a browser.

We’ll eventually get to implementing a microservices server that communicates with NodeJS on the backend, is load-balanced by NGINX and runs a ReactJS frontend with all the usual frontend candy (such as Bootstrap, CSS and such). We’ll even get around to running each group of related microservices in its own Docker container and then deploying, managing and orchestrating it all with Kubernetes… but it all starts with GeoJSON.

Enjoy!