Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audit logs #5646

Merged
merged 81 commits into from
Feb 18, 2023
Merged
Show file tree
Hide file tree
Changes from 76 commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
69fc71d
added initail version of clickhouse based analytics
azhavoro Jan 31, 2023
085d111
update image
azhavoro Jan 31, 2023
a683ecc
revert change
azhavoro Jan 31, 2023
b48011e
Merge remote-tracking branch 'origin/develop' into az/audit_logs
azhavoro Jan 31, 2023
5d303a6
linter
azhavoro Jan 31, 2023
a6bdde7
linter
azhavoro Jan 31, 2023
de61804
removed Grafana dockerfile
azhavoro Jan 31, 2023
7c39b96
fix csv downloading
azhavoro Jan 31, 2023
e3900aa
used UUID for query ID
azhavoro Feb 1, 2023
6996263
added more dashboards
azhavoro Feb 2, 2023
201c867
added user activity table
azhavoro Feb 2, 2023
76464b0
updated rego rules
azhavoro Feb 2, 2023
069b3b6
delete JobCommit
azhavoro Feb 2, 2023
1d439e2
updated grafana dashboards
azhavoro Feb 3, 2023
ecbe706
deprecate old services
azhavoro Feb 3, 2023
e766e02
fix docker compose
azhavoro Feb 3, 2023
fb57f44
update documentation
azhavoro Feb 3, 2023
5733bbf
minor refactoring
azhavoro Feb 3, 2023
9240bf5
update rules
azhavoro Feb 3, 2023
227fb48
logs -> events
azhavoro Feb 3, 2023
ec12b71
fix
azhavoro Feb 3, 2023
5522b04
fixed import
azhavoro Feb 3, 2023
ec5e62d
Merge remote-tracking branch 'develop' into az/audit_logs
azhavoro Feb 3, 2023
f34a622
t
azhavoro Feb 3, 2023
a16795f
t
azhavoro Feb 3, 2023
bc72b22
fix model
azhavoro Feb 3, 2023
c133070
linter
azhavoro Feb 3, 2023
15e5fd7
fix remark
azhavoro Feb 3, 2023
66ba85f
t
azhavoro Feb 3, 2023
b718845
try to build images without cache
sizov-kirill Feb 7, 2023
4427bbe
update OPA csv file
azhavoro Feb 7, 2023
e619f78
Merge branch 'az/audit_logs' of github.com:opencv/cvat into az/audit_…
azhavoro Feb 7, 2023
aa555b2
fix unit tests
azhavoro Feb 7, 2023
04d4b00
update vector config
azhavoro Feb 7, 2023
92cf339
fixed comments
azhavoro Feb 8, 2023
625b318
Merge remote-tracking branch 'develop' into az/audit_logs
azhavoro Feb 8, 2023
6b8b737
update worker_import compose config
azhavoro Feb 9, 2023
7798ace
fixed worker logs
azhavoro Feb 9, 2023
397a861
added ant-btn logging logic
klakhov Feb 14, 2023
1c6e554
wip
azhavoro Feb 14, 2023
f137741
code refactoring
azhavoro Feb 14, 2023
1a8a69f
Merge remote-tracking branch 'develop' into az/audit_logs
azhavoro Feb 14, 2023
a8b7e16
apply comments part 1
azhavoro Feb 14, 2023
0a5d1df
apply comments part 2
azhavoro Feb 14, 2023
56327c1
remove clogger
azhavoro Feb 14, 2023
ff473bd
apply comments part 3
azhavoro Feb 14, 2023
da42cca
apply comments part 4
azhavoro Feb 15, 2023
865922c
added log class
klakhov Feb 15, 2023
177e876
apply comments part 5
azhavoro Feb 15, 2023
7987e5e
fix
azhavoro Feb 15, 2023
4850aac
fix
azhavoro Feb 15, 2023
372f8dc
add filtering through the iam for the events endpoint
sizov-kirill Feb 15, 2023
3793b53
remove rules for events from server.rego
sizov-kirill Feb 15, 2023
ffdf143
remove rules for events from server.csv
sizov-kirill Feb 15, 2023
cebdc39
add newline
sizov-kirill Feb 15, 2023
9eb3111
Merge remote-tracking branch 'origin/az/audit_logs' into az/audit_logs
azhavoro Feb 15, 2023
b7cf15d
return 403 when non-admin user try to get events of other user in san…
sizov-kirill Feb 15, 2023
118173a
fix typo
sizov-kirill Feb 15, 2023
8e0c956
change minimal allowed privilege from USER to WORKER
sizov-kirill Feb 15, 2023
647b2e5
add events.csv and opa test scipt generator for events
sizov-kirill Feb 15, 2023
51e9459
Merge remote-tracking branch 'origin/sk/add-opa-filters-for-events' i…
azhavoro Feb 15, 2023
e1a2027
rename
azhavoro Feb 15, 2023
21c4f61
fix
azhavoro Feb 15, 2023
8c3ce8e
Merge remote-tracking branch 'origin/develop' into az/audit_logs
azhavoro Feb 15, 2023
3174af1
fix perm
azhavoro Feb 15, 2023
d41bfd5
fix
azhavoro Feb 15, 2023
ba1cdd8
fix test data, minor improvements
azhavoro Feb 15, 2023
0fc276d
remove SEND_EXCEPTION from server.rego
sizov-kirill Feb 16, 2023
a103778
added Active users panel, disable debug messages
azhavoro Feb 16, 2023
1b017b0
fix duration
azhavoro Feb 16, 2023
e324432
Merge branch 'az/audit_logs' of github.com:opencv/cvat into az/audit_…
azhavoro Feb 16, 2023
8a86d17
minor code changes
bsekachev Feb 16, 2023
346fb49
remove analytics/deprecated
azhavoro Feb 17, 2023
f014890
added user_name, user_email, org_slug fields
azhavoro Feb 17, 2023
0ca4195
Merge branch 'az/audit_logs' of github.com:opencv/cvat into az/audit_…
azhavoro Feb 17, 2023
6a07b98
fix timestamp for server update event
azhavoro Feb 17, 2023
6ddfbf2
comments
azhavoro Feb 17, 2023
97f1971
fix queue
azhavoro Feb 18, 2023
28485c8
updated changelog
azhavoro Feb 18, 2023
30aee83
v8.2.0
azhavoro Feb 18, 2023
5f24fe3
v1.49.0
azhavoro Feb 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 0 additions & 42 deletions .github/workflows/cache.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,24 +35,6 @@ jobs:
${{ runner.os }}-build-ui-${{ needs.get-sha.outputs.sha }}
${{ runner.os }}-build-ui-

- uses: actions/cache@v3
id: elasticsearch-cache-action
with:
path: /tmp/cvat_cache_elasticsearch
key: ${{ runner.os }}-build-elasticsearch-${{ github.sha }}
restore-keys: |
${{ runner.os }}-build-elasticsearch-${{ needs.get-sha.outputs.sha }}
${{ runner.os }}-build-elasticsearch-

- uses: actions/cache@v3
id: logstash-cache-action
with:
path: /tmp/cvat_cache_logstash
key: ${{ runner.os }}-build-logstash-${{ github.sha }}
restore-keys: |
${{ runner.os }}-build-logstash-${{ needs.get-sha.outputs.sha }}
${{ runner.os }}-build-logstash-

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

Expand All @@ -72,34 +54,10 @@ jobs:
cache-from: type=local,src=/tmp/cvat_cache_ui
cache-to: type=local,dest=/tmp/cvat_cache_ui-new

- name: Caching CVAT Elasticsearch
uses: docker/build-push-action@v2
with:
context: ./components/analytics/elasticsearch/
file: ./components/analytics/elasticsearch/Dockerfile
cache-from: type=local,src=/tmp/cvat_cache_elasticsearch
cache-to: type=local,dest=/tmp/cvat_cache_elasticsearch-new
build-args: ELK_VERSION=6.8.23

- name: Caching CVAT Logstash
uses: docker/build-push-action@v2
with:
context: ./components/analytics/logstash/
file: ./components/analytics/logstash/Dockerfile
cache-from: type=local,src=/tmp/cvat_cache_logstash
cache-to: type=local,dest=/tmp/cvat_cache_logstash-new
build-args: ELK_VERSION=6.8.23

- name: Moving cache
run: |
rm -rf /tmp/cvat_cache_server
mv /tmp/cvat_cache_server-new /tmp/cvat_cache_server

rm -rf /tmp/cvat_cache_ui
mv /tmp/cvat_cache_ui-new /tmp/cvat_cache_ui

rm -rf /tmp/cvat_cache_elasticsearch
mv /tmp/cvat_cache_elasticsearch-new /tmp/cvat_cache_elasticsearch

rm -rf /tmp/cvat_cache_logstash
mv /tmp/cvat_cache_logstash-new /tmp/cvat_cache_logstash
32 changes: 0 additions & 32 deletions .github/workflows/full.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,38 +115,6 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@master

- name: Getting CVAT Elasticsearch cache from the default branch
uses: actions/cache@v3
with:
path: /tmp/cvat_cache_elasticsearch
key: ${{ runner.os }}-build-elasticsearch-${{ needs.search_cache.outputs.sha }}

- name: Getting CVAT Logstash cache from the default branch
uses: actions/cache@v3
with:
path: /tmp/cvat_cache_logstash
key: ${{ runner.os }}-build-logstash-${{ needs.search_cache.outputs.sha }}

- name: Building CVAT Elasticsearch
uses: docker/build-push-action@v2
with:
context: ./components/analytics/elasticsearch/
file: ./components/analytics/elasticsearch/Dockerfile
cache-from: type=local,src=/tmp/cvat_cache_elasticsearch
tags: cvat_elasticsearch:latest
load: true
build-args: ELK_VERSION=6.8.23

- name: Building CVAT Logstash
uses: docker/build-push-action@v2
with:
context: ./components/analytics/logstash/
file: ./components/analytics/logstash/Dockerfile
cache-from: type=local,src=/tmp/cvat_cache_logstash
tags: cvat_logstash:latest
load: true
build-args: ELK_VERSION=6.8.23

- name: Download CVAT server image
uses: actions/download-artifact@v3
with:
Expand Down
32 changes: 0 additions & 32 deletions .github/workflows/schedule.yml
Original file line number Diff line number Diff line change
Expand Up @@ -129,18 +129,6 @@ jobs:
path: /tmp/cvat_cache_ui
key: ${{ runner.os }}-build-ui-${{ needs.search_cache.outputs.sha }}

- name: Getting CVAT Logstash cache from the default branch
uses: actions/cache@v3
with:
path: /tmp/cvat_cache_logstash
key: ${{ runner.os }}-build-logstash-${{ needs.search_cache.outputs.sha }}

- name: Getting CVAT Elasticsearch cache from the default branch
uses: actions/cache@v3
with:
path: /tmp/cvat_cache_elasticsearch
key: ${{ runner.os }}-build-elasticsearch-${{ needs.search_cache.outputs.sha }}

- name: Building CVAT UI image
uses: docker/build-push-action@v2
with:
Expand All @@ -150,26 +138,6 @@ jobs:
tags: cvat/ui:latest
load: true

- name: Building CVAT Logstash image
uses: docker/build-push-action@v2
with:
context: ./components/analytics/logstash/
file: ./components/analytics/logstash/Dockerfile
cache-from: type=local,src=/tmp/cvat_cache_logstash
build-args: ELK_VERSION=6.8.23
tags: cvat_logstash
load: true

- name: Building CVAT Elasticsearch image
uses: docker/build-push-action@v2
with:
context: ./components/analytics/elasticsearch/
file: ./components/analytics/elasticsearch/Dockerfile
cache-from: type=local,src=/tmp/cvat_cache_elasticsearch
build-args: ELK_VERSION=6.8.23
tags: cvat_elasticsearch
load: true

- name: CVAT server. Extract metadata (tags, labels) for Docker
id: meta-server
uses: docker/metadata-action@master
Expand Down
16 changes: 12 additions & 4 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,9 @@
"env": {
"CVAT_SERVERLESS": "1",
"ALLOWED_HOSTS": "*",
"IAM_OPA_BUNDLE": "1"
"IAM_OPA_BUNDLE": "1",
"DJANGO_LOG_SERVER_HOST": "localhost",
"DJANGO_LOG_SERVER_PORT": "8282"
},
"args": [
"runserver",
Expand Down Expand Up @@ -144,11 +146,14 @@
"rqworker",
"import",
"--worker-class",
"cvat.rqworker.SimpleWorker",
"cvat.rqworker.SimpleWorker"
],
"django": true,
"cwd": "${workspaceFolder}",
"env": {},
"env": {
"DJANGO_LOG_SERVER_HOST": "localhost",
"DJANGO_LOG_SERVER_PORT": "8282"
},
"console": "internalConsole"
},
{
Expand All @@ -167,7 +172,10 @@
],
"django": true,
"cwd": "${workspaceFolder}",
"env": {},
"env": {
"DJANGO_LOG_SERVER_HOST": "localhost",
"DJANGO_LOG_SERVER_PORT": "8282"
},
"console": "internalConsole"
},
{
Expand Down
50 changes: 50 additions & 0 deletions components/analytics/clickhouse/init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/bin/bash

CLICKHOUSE_DB="${CLICKHOUSE_DB:-cvat}";
CLICKHOUSE_USER="${CLICKHOUSE_USER:-user}";
CLICKHOUSE_PASSWORD="${CLICKHOUSE_PASSWORD:-password}";

cat <<EOT > /etc/clickhouse-server/users.d/user.xml
<yandex>
<!-- Docs: <https://clickhouse.tech/docs/en/operations/settings/settings_users/> -->
<users>
<${CLICKHOUSE_USER}>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably it is necessary to check that CLICKHOUSE_USER value without spaces and special symbols.

<profile>default</profile>
<networks>
<ip>::/0</ip>
</networks>
<password>${CLICKHOUSE_PASSWORD}</password>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, do we need to check the value? What if contains </password> or just <

<quota>default</quota>
</${CLICKHOUSE_USER}>
</users>
</yandex>
EOT

clickhouse-client --query "CREATE DATABASE IF NOT EXISTS ${CLICKHOUSE_DB}";

echo "
CREATE TABLE IF NOT EXISTS ${CLICKHOUSE_DB}.events
(
\`scope\` String NOT NULL,
\`obj_name\` String NULL,
\`obj_id\` UInt64 NULL,
\`obj_val\` String NULL,
\`source\` String NOT NULL,
\`timestamp\` DateTime64(3, 'Etc/UTC') NOT NULL,
\`count\` UInt16 NULL,
\`duration\` UInt32 DEFAULT toUInt32(0),
\`project_id\` UInt64 NULL,
\`task_id\` UInt64 NULL,
\`job_id\` UInt64 NULL,
\`user_id\` UInt64 NULL,
\`user_name\` String NULL,
\`user_email\` String NULL,
\`org_id\` UInt64 NULL,
\`org_slug\` String NULL,
\`payload\` String NULL
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(timestamp)
ORDER BY (timestamp)
SETTINGS index_granularity = 8192
;" | clickhouse-client
91 changes: 0 additions & 91 deletions components/analytics/docker-compose.analytics.yml

This file was deleted.

4 changes: 0 additions & 4 deletions components/analytics/elasticsearch/Dockerfile

This file was deleted.

3 changes: 0 additions & 3 deletions components/analytics/elasticsearch/elasticsearch.yml

This file was deleted.

28 changes: 28 additions & 0 deletions components/analytics/grafana/dashboards/Untitled-1.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@

nmanovic marked this conversation as resolved.
Show resolved Hide resolved
SELECT
time,
uniqExact(user_id) Users
FROM
(
SELECT
user_id,
toStartOfInterval(timestamp, INTERVAL 5 minute) as time
FROM cvat.events
WHERE
user_id IS NOT NULL AND
time >= now() - INTERVAL 5 DAY
GROUP BY
user_id,
time
ORDER BY time ASC WITH FILL STEP toIntervalMinute(5)
)
GROUP BY time
ORDER BY time

SELECT
uniqExact(user_id) as "Active users (now)"
FROM
cvat.events
WHERE
user_id IS NOT NULL AND
timestamp >= now() - INTERVAL 5 DAY
Loading