Architecture

The GISCollective platform is made of a few simple components. Each component has its own specific purpose and they are designed to easily scale based on your needs. The relation between those components can be seen here:

The platform is open source, and the source code of all the components can be found on GitLab. We also strive to use for the development of the platform open source programming languages, and frameworks that are supported by open source communities. Our preferred programming language is DLang, but we also use Node.js, Kotlin and Swift.

Web Frontend

Our web frontend is written using ember.js. Our setup supports server-side rendering using fastboot, which means that the content hosted on GISCollective can be easily indexed by search engines. To have this feature enabled, the frontend servers also connect to the REST API servers. Of course, after the initial render in the browser, the ember app is initialized and it will use the API server directly, without needing to render the pages on the server.

If needed, the web frontend can be scaled horizontally. That’s why we recommend adding a load balancer between the users and those servers. You can find the source code in this repository: https://gitlab.com/GISCollective/frontend/web-frontend.

Public REST API server

This server provides a REST API interface for interacting with the data. The server can be scaled up and down based on your needs. It’s written in DLang and it performs very well on setups with limited resources.

We try to have a clean project structure, so in the source folder you can find a folder dedicated to each model, where you can also find the middlewares applied to the routes generated by the crate library.

The source code of this project can be found here: https://gitlab.com/GISCollective/backend/ogm-server.

HMQ

The HMQ is a simple message queue that works with the HTTP and web socket protocols. It’s used as a load balancer for the tasks services. We use it with the HTTP protocol, so making a POST request to a route means adding a new message to that channel, which will be forwarded to the appropriate task instance. It also supports message de duplication based on an id passed to the message object. This is the only server which does not need scaling because it is fast enough for handling traffic even for large projects.

The source code of this project can be found here: https://gitlab.com/GISCollective/backend/hmq

Task server

Our main task server is written in DLang and it uses the same models library like the REST API server. This helps with having a consistent model definition within the projects. This server watches for changes in the stored models and triggers the appropriate tasks like generating stats or converting file formats. It also has tasks that are triggered by other tasks or user actions like generating geopackage files or sending emails. This server is designed for scaling, and you can setup as many instances as you need to be able to handle all the work in time.

The source code of this project can be found here: https://gitlab.com/GISCollective/backend/tasks

Node Tasks server

The node tasks package is a server similar to the Tasks Server written in DLang, but its main programming language is JavaScript. This helps us to take advantage of the node.js packages environment, and we use it to write tasks when we want to use libraries for existing file formats or standards.

The source code of this project can be found in our node monorepo: https://gitlab.com/GISCollective/backend/node

Batch runner

The batch runner runs every hour and it’s a short living process which scans the database and it fixes any inconsistencies that are detected. This project is also responsible for cleaning the garbage files that have no links to any document from the database.

The source code of this project can be found in our node monorepo: https://gitlab.com/GISCollective/backend/node

Fonts

The fonts server is a simple web server that serves font assets for vector tiles styles.

The source code of this project can be found in our node monorepo: https://gitlab.com/GISCollective/backend/node