Skip to content

nixys/nxs-data-anonymizer

BranchesTags

Repository files navigation

photo_2023-07-28_15-28-52

nxs-data-anonymizer

Introduction

nxs-data-anonymizer is a tool for anonymizing PostgreSQL and MySQL/MariaDB/Percona databases' dump.

Features

  • Supported databases and versions:
    • PostgreSQL (9/10/11/12/13/14/15/all versions)
    • MySQL/MariaDB/Percona (5.7/8.0/8.1/all versions)
  • Flexible data faking based on:
    • Go templates and Sprig template’s library like Helm. You may also use values of other columns for same row to build more flexible rules
    • External commands you may execute to create table field values
    • Security enforcement rules
    • Link cells across the database to generate the same values
  • Stream data processing. It means that you can a use the tool through a pipe in command line and redirect dump from source DB directly to the destination DB with required transformations
  • Easy to integrate into your CI/CD

Who can use the tool

Development and project teams which are dealing with production and test/dev/stage or dynamic namespaces with databases and need to ensure security and prevent data leaks.

Quickstart

Inspect your database structure and set up the nxs-data-anonymizer config in accordance with the sensitive data you need to anonymize.

You are able to use this tool in any way you want. Three most common ways are described below.

Console

In order to operate with your database anonymization via console you need to go through the following steps:

  • Download and untar the nxs-data-anonymizer binary
  • Run the nxs-data-anonymizer through the command line with arguments you want to use

For example, use the following command if you need to anonymize your PostgreSQL database from production to dev on fly (PostgreSQL Client need to be installed):

export PGPASSWORD=password; pg_dump -U postgres prod | /path/to/nxs-data-anonymizer -t pgsql -c /path/to/nxs-data-anonymizer.conf | psql -U postgres dev

GitLab CI

This section describes how to integrate nxs-data-anonymizer into your GitLab CI. You may add jobs presented below into your .gitlab-ci.yml and adjust it for yourself.

Job: anonymize prod

Job described in this section is able to perform the following tasks:

  • Run when special tag for main branch is set
  • Create a production database dump, anonymize and upload it into s3 bucket

Job sample:

anonymize:
  stage: anonymize
  image: nixyslab/nxs-data-anonymizer:latest
  variables:
    GIT_STRATEGY: none
    PG_HOST: ${PG_HOST_PROD}
    PG_USER: ${PG_USER_PROD}
    PGPASSWORD: ${PG_PASS_PROD}
  before_script: 
  - echo "${S3CMD_CFG}" > ~/.s3cmd
  - echo "${NXS_DA_CFG}" > /nxs-data-anonymizer.conf
  script:
  - pg_dump -h ${PG_HOST} -U ${PG_USER} --schema=${PG_SCHEMA} ${PG_DATABASE} | /nxs-data-anonymizer -t pgsql -c /nxs-data-anonymizer.conf | gzip | s3cmd put - s3://bucket/anondump.sql.gz
  only:
  - /^v.*$/
  except:
  - branches
  - merge_requests
Job: update stage

Job described in this section deals with the following:

  • Manual job for stage branch
  • Download the anonymized dump from s3 bucket and load into stage database

Job sample:

restore-stage:
  stage: restore
  image: nixyslab/nxs-data-anonymizer:latest
  variables:
    GIT_STRATEGY: none
    PG_HOST: ${PG_HOST_STAGE}
    PG_USER: ${PG_USER_STAGE}
    PGPASSWORD: ${PG_PASS_STAGE}
  before_script: 
  - echo "${S3CMD_CFG}" > ~/.s3cmd
  script:
  - s3cmd --no-progress --quiet get s3://bucket/anondump.sql.gz - | gunzip | psql -h ${PG_HOST} -U ${PG_USER} --schema=${PG_SCHEMA} ${PG_DATABASE}
  only:
  - stage
  when: manual
CI/CD variables

This section contains a description of CI/CD variables used in GitLab CI job samples above.

General
Variable Description
S3CMD_CFG S3 storage config
PG_SCHEMA PgSQL schema
PG_DATABASE PgSQL database name
Production
Variable Description
NXS_DA_CFG nxs-data-anonymizer config
PG_HOST_PROD PgSQL host
PG_USER_PROD PgSQL user
PG_PASS_PROD PgSQL password
Stage
Variable Description
PG_HOST_STAGE PgSQL host
PG_USER_STAGE PgSQL user
PG_PASS_STAGE PgSQL password

GitHub Actions

This section describes how to integrate nxs-data-anonymizer into your GitHub Actions. You may add jobs presented below into your .github workflows and adjust it for yourself.

Job: anonymize prod

Job described in this section is able to perform the following tasks:

  • Run when special tag is set
  • Create a production database dump, anonymize and upload it into s3 bucket
on:
  push:
    tags:
    - v*.*

jobs:
  anonymize:
    runs-on: ubuntu-latest
    container:
      image: nixyslab/nxs-data-anonymizer:latest
      env:
        PG_HOST: ${{ secrets.PG_HOST_PROD }}
        PG_USER: ${{ secrets.PG_USER_PROD }}
        PGPASSWORD: ${{ secrets.PG_PASS_PROD }}
        PG_SCHEMA: ${{ secrets.PG_SCHEMA }}
        PG_DATABASE: ${{ secrets.PG_DATABASE }}
    steps:
    - name: Create services configs
      run: |
        echo "${{ secrets.S3CMD_CFG }}" > ~/.s3cmd
        echo "${{ secrets.NXS_DA_CFG }}" > /nxs-data-anonymizer.conf
    - name: Anonymize
      run: |
        pg_dump -h ${PG_HOST} -U ${PG_USER} --schema=${PG_SCHEMA} ${PG_DATABASE} | /nxs-data-anonymizer -t pgsql -c /nxs-data-anonymizer.conf | gzip | s3cmd put - s3://bucket/anondump.sql.gz
Job: update stage

Job described in this section deals with the following:

  • Manual job
  • Download the anonymized dump from s3 bucket and load into stage database
on: workflow_dispatch

jobs:
  restore-stage:
    runs-on: ubuntu-latest
    container:
      image: nixyslab/nxs-data-anonymizer:latest
      env:
        PG_HOST: ${{ secrets.PG_HOST_STAGE }}
        PG_USER: ${{ secrets.PG_USER_STAGE }}
        PGPASSWORD: ${{ secrets.PG_PASS_STAGE }}
        PG_SCHEMA: ${{ secrets.PG_SCHEMA }}
        PG_DATABASE: ${{ secrets.PG_DATABASE }}
    steps:
    - name: Create services configs
      run: |
        echo "${{ secrets.S3CMD_CFG }}" > ~/.s3cmd
    - name: Restore
      run: |
        s3cmd --no-progress --quiet get s3://bucket/anondump.sql.gz - | gunzip | psql -h ${PG_HOST} -U ${PG_USER} --schema=${PG_SCHEMA} ${PG_DATABASE}
GitHub Actions secrets

This section contains a description of secrets used in GitHub Actions job samples above.

General
Variable Description
S3CMD_CFG S3 storage config
PG_SCHEMA PgSQL schema
PG_DATABASE PgSQL database name
Production
Variable Description
NXS_DA_CFG nxs-data-anonymizer config
PG_HOST_PROD PgSQL host
PG_USER_PROD PgSQL user
PG_PASS_PROD PgSQL password
Stage
Variable Description
PG_HOST_STAGE PgSQL host
PG_USER_STAGE PgSQL user
PG_PASS_STAGE PgSQL password

Settings

Default configuration file path: /nxs-data-anonymizer.conf. The file is represented in yaml.

Command line arguments

Argument Short Required Having value Default value Description
--help -h No No - Show program help message
--version -v No No - Show program version
--conf -c No Yes /nxs-data-anonymizer.conf Configuration file path
--input -i No Yes - File to read data from. If not specified stdin will be used
--log-format -l No Yes json Log file format. You are available to use either json or plain value
--output -o No Yes - File to write data to. If not specified stdout will be used
--type -t Yes Yes - Database dump file type. Available values: pgsql, mysql

General settings

Option Type Required Default value Description
logfile String No stderr Log file path. You may also use stdout and stderr
loglevel String No info Log level. Available values: debug, warn, error and info
progress Progress No - Anonymization progress logging
variables Map of Variables (key: variable name) No - Global variables to be used in a filters. Variables are set at the init of application and remain unchanged during the runtime
link Slice of Link No - Rules to link specified columns across the database
filters Map of Filters (key: table name) No - Filters set for specified tables (key as a table name). Note: for PgSQL you also need to specify a scheme (e.g. public.tablename)
security Security No - Security enforcement for anonymizer
Progress settings
Option Type Required Default value Description
rhythm String No 0s Frequency write into the log a read bytes count. Progress will be written to the log only when this option is specified and has none-zero value. You may use a human-readable values (e.g. 30s, 5m, etc)
humanize Bool No false Set this option to true if you need to write into the log a read bytes count in a human-readable format. On false raw bytes count will be written to the log
Variables settings
Option Type Required Default value Description
type String No template Type of field value: template and command are available
value String Yes - The value to be used as global variable value within the filters. In accordance with the type this value may be either Go template or command. See below for details
Link settings

Link is used to create the same data with specified rules for different cells across the database.

Each link element has following properties:

  • Able to contain multiple tables and columns for each table
  • All specified cells with the same data before anonymization will have same data after
  • One common rule to generate new values
Option Type Required Default value Description
type String No template Type of field value: template and command are available
value String Yes - The value to be used to replace at every cell in specified column. In accordance with the type this value may be either Go template or command. See below for details
unique Bool No false If true checks the generated value for cell is unique whole an all columns specified for link element
Filters settings

Filters description for specified table.

Option Type Required Default value Description
columns Map of Columns (key: column name) No - Filter rules for specified columns of table (key as a column name)
Columns settings
Option Type Required Default value Description
type String No template Type of field value: template and command are available
value String Yes - The value to be used to replace at every cell in specified column. In accordance with the type this value may be either Go template or command. See below for details
unique Bool No false If true checks the generated value for cell is unique whole the column

Go template

To anonymize a database fields you may use a Go template with the Sprig template library's functions.

Additional filter functions:

  • null: set a field value to NULL
  • isNull: compare a field value with NULL

You may also use the following data in a templates:

  • Current table name. Statement: {{ .TableName }}
  • Current column name. Statement: {{ .CurColumnName }}
  • Values of other columns in the rules for same row (with values before substitutions). Statement: {{ .Values.COLUMN_NAME }} (e.g.: {{ .Values.username }})
  • Global variables. Statement: {{ .Variables.VARIABLE_NAME }} (e.g.: {{ .Variables.password }})
  • Raw column data type. Statement: {{ .ColumnTypeRaw }}
  • Regex's capturing groups for the column data type. This variable has array type so you need to use range or index to access specific element. Statement: {{ index .ColumnTypeGroups 0 0 }}. See Types for details

Command

To anonymize a database fields you may use a commands (scripts or binaries) with any logic you need. The command's concept has following properties:

  • The command's stdout will be used as a new value for the anonymized field
  • Command must return zero exit code, otherwise nxs-data-anonymizer will falls with error (in this case stderr will be used as an error text)
  • Environment variables with the row data are available within the command:
    • ENVVARTABLE: contains a name of the current table
    • ENVVARCURCOLUMN: contains the current column name
    • ENVVARCOLUMN_{COLUMN_NAME}: contains values (before substitutions) for all columns for the current row
    • ENVVARGLOBAL_{VARIABLE_NAME}: contains value for specified global variable
    • ENVVARCOLUMNTYPERAW: contains raw column data type
    • ENVVARCOLUMNTYPEGROUP_{GROUP_NUM}_{SUBGROUPNUM}: contains regex's capturing groups for the column data type. See Types for details
Security settings
Option Type Required Default value Description
policy Policy No - Security policy for entities
exceptions Exceptions No - Exceptions for entities
defaults Defaults No - Default filters for entities
Policy settings
Option Type Required Default value Description
tables String No pass Security policy for tables. If value skip is used all undescribed tables in config will be skipped while anonymization
columns String No pass Security policy for columns. If value randomize is used all undescribed columns in config will be randomized (with default rules in accordance to types) while anonymization

Values to masquerade a columns in accordance with the types see below.

PgSQL:

Type Value to masquerade
smallint 0
integer 0
bigint 0
smallserial 0
serial 0
bigserial 0
decimal 0.0
numeric 0.0
real 0.0
double 0.0
character randomized character data"
bpchar randomized bpchar data
text randomized text data

MySQL:

Type Value to masquerade
bit 0
bool 0
boolean 0
tinyint 0
smallint 0
mediumint 0
int 0
integer 0
bigint 0
float 0.0
double 0.0
double precision 0.0
decimal 0.0
dec 0.0
char randomized char (String will be truncated to "COLUMN_SIZE" length.)
varchar randomized varchar (String will be truncated to "COLUMN_SIZE" length.)
tinytext randomized tinytext
text randomized text
mediumtext randomized mediumtext
longtext randomized longtext
enum Last value from enum
set Last value from set
date 2024-01-01
datetime 2024-01-01 00:00:00
timestamp 2024-01-01 00:00:00
time 00:00:00
year 2024
json {"randomized": "json_data"}
binary cmFuZG9taXplZCBiaW5hcnkgZGF0YQo=
varbinary cmFuZG9taXplZCBiaW5hcnkgZGF0YQo=
tinyblob cmFuZG9taXplZCBiaW5hcnkgZGF0YQo=
blob cmFuZG9taXplZCBiaW5hcnkgZGF0YQo=
mediumblob cmFuZG9taXplZCBiaW5hcnkgZGF0YQo=
longblob cmFuZG9taXplZCBiaW5hcnkgZGF0YQo=
Exceptions settings
Option Type Required Default value Description
tables Slice of strings No - Table names without filters which are not be skipped while anonymization if option security.policy.tables set to skip
columns Slice of strings No - Column names (in any table) without filters which are not be randomized while anonymization if option security.policy.columns set to randomize
Defaults settings
Option Type Required Default value Description
columns Map of Filters No - Default filter for columns (in any table). That filters will be applied for columns with this names without described filters
types Slice of Types No - Custom filters for types (in any table). With this filter rules you may override default filters for types
Types settings
Option Type Required Default value Description
regex String Yes - Regular expression. Will be checked for match for column data type (in CREATE TABLE section). Able to use capturing groups within the regex that available as an additional variable data in the filters (see Columns for details). This ability helps to create more flexible rules to generate the cells value in accordance with data type features
rule Columns Yes - Rule will be applied columns with data types matched for specified regular expression

Example

Imagine you have a simple table users in your production PgSQL like this:

id username password api_key
1 admin ZjCX6wUxtXIMtip epezyj0cj5rqrdtxklnzxr3f333uibtz6avek7926141t1c918
2 alice tuhjLkgwwetiwf8 2od4vfsx2irj98hgjaoi6n7wjr02dg79cvqnmet4kyuhol877z
3 bob AjRzvRp3DWo6VbA owp7hob5s3o083d5hmursxgcv9wc4foyl20cbxbrr73egj6jkx

You need to get a dump with fake values:

  • For admin: preset fixed value for a password and API key to avoid the need to change an app settings in your dev/test/stage or local environment after downloading the dump
  • For others: usernames in format user_N (where N it is a user ID) and unique random passwords and API keys

In accordance with these conditions, the nxs-data-anonymizer config may look like this:

variables:
  adminPassword:
    type: template
    value: "preset_admin_password"
  adminAPIKey:
    value: "preset_admin_api_key"

filters:
  public.users:
    columns:
      username:
        value: "{{ if eq .Values.username \"admin\" }}{{ .Values.username }}{{ else }}user_{{ .Values.id }}{{ end }}"
      password:
        type: command
        value: /path/to/script.sh
        unique: true
      api_key:
        value: "{{ if eq .Values.username \"admin\" }}{{ .Variables.adminAPIKey }}{{ else }}{{- randAlphaNum 50 | nospace | lower -}}{{ end }}"
        unique: true

The /path/to/script.sh script content is following:

#!/bin/bash

# Print preset password if current user is admin
if [ "$ENVVARCOLUMN_username" == "admin" ];
then
    echo -n "$ENVVARGLOBAL_adminPassword"
    exit 0
fi

# Generate password for other users
p=$(pwgen -s 5 1 2>&1) 
if [ ! $? -eq 0 ];
then

    # On error print message to stderr and exit with non zero code

    echo -n "$p" >&2
    exit 1
fi

# Print generated password
echo $p | tr -d '\n'

exit 0

Now you may execute the following command in order to load anonymized data into your dev DB:

pg_dump ... | ./nxs-data-anonymizer -c filters.conf | psql -h localhost -U user example

As a result:

id username password api_key
1 admin preset_admin_password preset_admin_api_key
2 user_2 Pp4HY dhx4mccxyd8ux5uf1khpbqsws8qqeqs4efex1vhfltzhtjcwcu
3 user_3 vu5TW lgkkq3csskuyew8fr52vfjjenjzudokmiidg3cohl2bertc93x

It's easy.

Roadmap

Following features are already in backlog for our development team and will be released soon:

  • Global variables with the templated values you may use through the filters for all tables and columns
  • Ability to delete tables and rows from faked dump
  • Ability to output into log a custom messages. It’s quite useful it order to obtain some generated data like admin passwords, etc
  • Support of a big variety of databases

Feedback

For support and feedback please contact me:

For news and discussions subscribe the channels:

License

nxs-data-anonymizer is released under the Apache License 2.0.

About

A tool for anonymizing PostgreSQL and MySQL databases' dump

Resources

Readme

License

Apache-2.0 license
Activity
Custom properties

Stars

224 stars

Watchers

13 watching

Forks

10 forks
Report repository

Releases 20

v1.11.0 Latest
Sep 18, 2024
+ 19 releases

Packages

No packages published

Contributors 5

Languages