Skip to content
/ lacuna Public

A docker operator that manages Google Cloud Pub/Sub topics and subscriptions.

Notifications You must be signed in to change notification settings

aplr/lacuna

Repository files navigation

Lacuna - a Pub/Sub Docker Operator

Test Docker

Lacuna is a Kubernetes-style operator that runs locally on your machine and manages Google Cloud Pub/Sub topics and push subscriptions for your docker containers. It is designed to run alongside a local Pub/Sub emulator and manages topics and subscriptions automatically by observing the docker containers running on your machine.

Overview

While testing locally, having a local Pub/Sub emulator helps to replicate the behavior of the production environment. However, it can be tedious to manually create topics and subscriptions for each service that needs to interact with Pub/Sub. This is especially true for Pub/Sub because topics and subscriptions can not be created using the gcloud CLI, but must be created using a proper Pub/Sub API client, which causes maintainance overhead for push subscriptions. Lacuna aims to solve this by creating topics and subscriptions for each container that needs to interact with Pub/Sub just by using docker labels.

Limitations

Currently, Lacuna only supports Google Cloud Pub/Sub, but it can be extended to support other messaging systems. Also, only push subscriptions are supported, as pull subscriptions have to be implemented in the consuming service anyways. However, Lacuna can still be used to create the topics pull subscriptions can subscribe to.

Usage

Use Lacuna by running it as a docker container alongside your Pub/Sub emulator. Lacuna will automatically create topics and push subscriptions for each container that has the lacuna.enabled label set to true. Lacuna will also take care of deleting topics and subscriptions when containers are stopped.

Make sure to mount the docker socket into the Lacuna container so it can observe container events.

Example

version: "3.9"

services:
    json-server:
        image: codfish/json-server:latest
        restart: unless-stopped
        ports:
            - "8080:80"
        labels:
            lacuna.enabled: true
            lacuna.subscription.test.topic: test
            lacuna.subscription.test.endpoint: http://json-server/messages
            lacuna.subscription.test.ack-deadline: "30s"
    pubsub:
        image: gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators
        command: gcloud beta emulators pubsub start --host-port=0.0.0.0:8085 --project=pubsub
        restart: unless-stopped
        ports:
            - "8085:8085"
    lacuna:
        build: .
        restart: unless-stopped
        environment:
            LACUNA_PUBSUB_PROJECT_ID: pubsub
            # Configure the pubsub client to use the emulator
            # See https://cloud.google.com/pubsub/docs/emulator#env
            PUBSUB_EMULATOR_HOST: pubsub:8085 
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock

Configuration

Lacuna is configured using docker labels. The following labels are supported:

Label Description Required
lacuna.enabled Enables Lacuna for the container. Yes
lacuna.subscription.<name>.topic The name of the topic to subscribe to. Yes
lacuna.subscription.<name>.endpoint The endpoint to send messages to. Yes
lacuna.subscription.<name>.<option> See options below. No

Subscription Options

For each subscription, the following options can be set. For a detailed description of each option, see the Pub/Sub API documentation.

Option Description
ack-deadline The number of seconds the subscriber has to acknowledge a message.
retain-acked-messages Whether to retain acknowledged messages.
retention-duration The number of seconds to retain acknowledged messages.
enable-ordering Whether to enable message ordering.
expiration-ttl The number of seconds a message can be retained.
filter A filter expression.
deliver-exactly-once Whether to deliver messages exactly once.
dead-letter-topic The name of the dead letter topic.
max-dead-letter-delivery-attempts The maximum number of delivery attempts for a message.
retry-minimum-backoff The minimum backoff time for retrying a message.
retry-maximum-backoff The maximum backoff time for retrying a message.

Acknowledgements

Lacuna's label-based configuration is inspired by Ofelia, a job scheduler for docker containers.

About

A docker operator that manages Google Cloud Pub/Sub topics and subscriptions.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages