Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot connect to the CVAT server #6202

Closed
shintaro-matsui opened this issue May 26, 2023 · 20 comments
Closed

Cannot connect to the CVAT server #6202

shintaro-matsui opened this issue May 26, 2023 · 20 comments

Comments

@shintaro-matsui
Copy link

My actions before raising this issue

Searched for previous issues (#5591)

Expected Behaviour

I was expecting CVAT to run in browser at http://localhost:8080, as described in the guidelines.

Current Behaviour

After running docker compose up -d and then accessing http://localhost:8080, the following pop-up error appears

Cannot connect to the server
Make sure the CVAT backend and all necessary services (Database, Redis and Open Policy Agent) are running and available. 0 or >earlier, manual actions may be needed, see the Upgrade Guide.

Possible Solution

I started my PC and run CVAT, and now I get the above error.
I don't remember doing anything special the last time I did this.

Your Environment

  • Ubuntu 18.04 LTS
  • Docker version 23.0.3, build 3e7cbfd

I am not familiar with Docker and I have looked around but could not find a solution.
What should I check first?

@azhavoro
Copy link
Contributor

Did you follow the recommendation and check the container logs?

@ganindu7
Copy link

I have the same issue (but I installed with a helm chart)

in the backend server pod I ran health checks

python manage.py health_check
Cache backend: default   ... working
Cache backend: media     ... working
DatabaseBackend          ... working
DiskUsage                ... working
MemoryUsage              ... working
MigrationsHealthCheck    ... working
OPAHealthCheck           ... working
$ python manage.py testserver
Creating test database for alias 'default'...
Getting flipped tasks...
Conversion started...

Start schema migration...
INFO:0024_auto_20191023_1025:
Start schema migration...

Schema migration is finished...
INFO:0024_auto_20191023_1025:
Schema migration is finished...

Start data migration...
INFO:0024_auto_20191023_1025:
Start data migration...
The data migration has been started for creating manifest`s files
The data migration has been started for creating manifest`s files
INFO:0038_manifest:The data migration has been started for creating manifest`s files
Need to update 0 data objects
Need to update 0 data objects
INFO:0038_manifest:Need to update 0 data objects
Migration has been started. Need to create 0 directories.
Migration has been started. Need to create 0 directories.
INFO:0055_jobs_directories:Migration has been started. Need to create 0 directories.
Migration has been finished successfully.
Migration has been finished successfully.
INFO:0055_jobs_directories:Migration has been finished successfully.
Migration has been started. Need to create 0 previews.
Migration has been started. Need to create 0 previews.
INFO:0056_jobs_previews:Migration has been started. Need to create 0 previews.

Deleting Data previews...
INFO:0062_delete_previews:
Deleting Data previews...

Deleting Job previews...
INFO:0062_delete_previews:
Deleting Job previews...

Deleting CloudStorage previews...
INFO:0062_delete_previews:
Deleting CloudStorage previews...

Deleting skeleton Labels without skeletons...
INFO:0064_delete_or_rename_wrong_labels:
Deleting skeleton Labels without skeletons...

Deleting duplicate skeleton sublabels and renaming duplicate Labels...
INFO:0064_delete_or_rename_wrong_labels:
Deleting duplicate skeleton sublabels and renaming duplicate Labels...
CommandError: Error: No database fixture specified. Please provide the path of at least one fixture in the command line.

the admin panel login works fine

http://cvat.gsrv.lan:32608/admin/
Screenshot from 2023-05-26 19-48-13

But this happends when I try to log in to the gui.
Screenshot from 2023-05-26 19-50-46

this is the full backend server pod log

g@gsrv:~$ kubectl logs  -n cvat-gsrv  cvat-backend-server-6887d99fbd-t8wph -f
2023-05-26 18:28:05,045 INFO RPC interface 'supervisor' initialized
2023-05-26 18:28:05,045 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-05-26 18:28:05,045 INFO supervisord started with pid 1
2023-05-26 18:28:06,049 INFO spawned: 'ssh-agent' with pid 7
2023-05-26 18:28:06,052 INFO spawned: 'clamav_update' with pid 8
2023-05-26 18:28:06,071 INFO spawned: 'runserver' with pid 9
2023-05-26 18:28:06,071 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139821883798432 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stdout)>
2023-05-26 18:28:06,071 DEBG fd 14 closed, stopped monitoring <POutputDispatcher at 139821883798288 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stderr)>
2023-05-26 18:28:06,071 INFO exited: clamav_update (exit status 0; not expected)
2023-05-26 18:28:06,071 DEBG received SIGCHLD indicating a child quit
2023-05-26 18:28:06,074 DEBG 'ssh-agent' stdout output:
SSH_AUTH_SOCK=/tmp/ssh-agent.sock; export SSH_AUTH_SOCK;
echo Agent pid 7;

2023-05-26 18:28:06,074 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = SOCKET
debug2: fd 3 setting O_NONBLOCK

2023-05-26 18:28:06,083 DEBG 'runserver' stderr output:
wait-for-it.sh: waiting for cvat-postgresql:5432 without a timeout

2023-05-26 18:28:06,089 DEBG 'runserver' stderr output:
wait-for-it.sh: cvat-postgresql:5432 is available after 0 seconds

2023-05-26 18:28:06,090 DEBG 'runserver' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2023-05-26 18:28:06,326 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:28:06,330 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug2: process_request_identities: replying with 0 allowed of 0 available keys

2023-05-26 18:28:06,339 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:28:06,340 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-26 18:28:06,342 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:EBbB2b6j9z65OMIXjLKm/Tm2zxhjNm3CQwE13KlLXM8 "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-26 18:28:06,342 DEBG 'runserver' stderr output:
Identity added: /home/django/.ssh/id_rsa (django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn)

2023-05-26 18:28:07,343 INFO success: ssh-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-26 18:28:07,347 INFO spawned: 'clamav_update' with pid 91
2023-05-26 18:28:07,347 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-26 18:28:07,372 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139821883798384 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stdout)>
2023-05-26 18:28:07,372 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 139821883798480 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stderr)>
2023-05-26 18:28:07,372 INFO exited: clamav_update (exit status 0; not expected)
2023-05-26 18:28:07,372 DEBG received SIGCHLD indicating a child quit
2023-05-26 18:28:09,385 INFO spawned: 'clamav_update' with pid 284
2023-05-26 18:28:09,389 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139821883798432 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stdout)>
2023-05-26 18:28:09,389 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 139821883787920 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stderr)>
2023-05-26 18:28:09,389 INFO exited: clamav_update (exit status 0; not expected)
2023-05-26 18:28:09,389 DEBG received SIGCHLD indicating a child quit
2023-05-26 18:28:12,401 INFO spawned: 'clamav_update' with pid 285
2023-05-26 18:28:12,405 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139821883798384 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stdout)>
2023-05-26 18:28:12,405 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 139821883798576 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stderr)>
2023-05-26 18:28:12,405 INFO exited: clamav_update (exit status 0; not expected)
2023-05-26 18:28:12,405 DEBG received SIGCHLD indicating a child quit
2023-05-26 18:28:13,408 INFO gave up: clamav_update entered FATAL state, too many start retries too quickly
2023-05-26 18:28:36,921 DEBG 'runserver' stdout output:
SSH keys were not found
Copying them from keys volume

2023-05-26 18:28:36,922 DEBG 'runserver' stderr output:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/opt/venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "cvat-postgresql" (10.106.161.41), port 5432 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/django/manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 117, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/loader.py", line 58, in __init__
    self.build_graph()
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/loader.py", line 235, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/recorder.py", line 81, in applied_migrations
    if self.has_table():
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/recorder.py", line 57, in has_table
    with self.connection.cursor() as cursor:
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection

2023-05-26 18:28:36,922 DEBG 'runserver' stderr output:
    with self.wrap_database_errors:
  File "/opt/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/opt/venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "cvat-postgresql" (10.106.161.41), port 5432 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?


2023-05-26 18:28:38,014 DEBG fd 15 closed, stopped monitoring <POutputDispatcher at 139821883798768 for <Subprocess at 139821883788064 with name runserver in state RUNNING> (stdout)>
2023-05-26 18:28:38,014 DEBG fd 19 closed, stopped monitoring <POutputDispatcher at 139821883798816 for <Subprocess at 139821883788064 with name runserver in state RUNNING> (stderr)>
2023-05-26 18:28:38,014 INFO exited: runserver (exit status 1; not expected)
2023-05-26 18:28:38,014 DEBG received SIGCHLD indicating a child quit
2023-05-26 18:28:39,040 INFO spawned: 'runserver' with pid 286
2023-05-26 18:28:39,051 DEBG 'runserver' stderr output:
wait-for-it.sh: waiting for cvat-postgresql:5432 without a timeout

2023-05-26 18:28:40,052 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-26 18:28:43,080 DEBG 'runserver' stderr output:
wait-for-it.sh: cvat-postgresql:5432 is available after 4 seconds

2023-05-26 18:28:43,080 DEBG 'runserver' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2023-05-26 18:28:43,302 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:28:43,302 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-26 18:28:43,304 DEBG 'ssh-agent' stderr output:
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:EBbB2b6j9z65OMIXjLKm/Tm2zxhjNm3CQwE13KlLXM8 "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-26 18:28:43,307 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-26 18:29:29,912 DEBG 'runserver' stdout output:
Operations to perform:
  Apply all migrations: account, admin, auth, authtoken, contenttypes, dataset_repo, db, engine, organizations, sessions, sites, socialaccount, webhooks
Running migrations:
  No migrations to apply.

2023-05-26 18:29:31,826 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:29:31,826 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:EBbB2b6j9z65OMIXjLKm/Tm2zxhjNm3CQwE13KlLXM8 "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-26 18:29:31,828 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-26 18:29:32,791 DEBG 'runserver' stdout output:

165 static files copied to '/home/django/static'.

2023-05-26 18:29:33,339 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:29:33,340 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-26 18:29:33,342 DEBG 'ssh-agent' stderr output:
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:EBbB2b6j9z65OMIXjLKm/Tm2zxhjNm3CQwE13KlLXM8 "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-26 18:29:33,345 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-26 18:30:48,562 DEBG 'runserver' stdout output:
Successfully ran command.
Server URL         : http://localhost:8080/
Server Root        : /tmp/cvat-server
Server Conf        : /tmp/cvat-server/httpd.conf
Error Log File     : /dev/stderr (INFO)
Operating Mode     : daemon
Request Capacity   : 5 (1 process * 5 threads)
Request Timeout    : 60 (seconds)
Startup Timeout    : 15 (seconds)
Queue Backlog      : 100 (connections)
Queue Timeout      : 45 (seconds)
Server Capacity    : 20 (event/worker), 20 (prefork)
Server Backlog     : 500 (connections)
Locale Setting     : C.UTF-8

2023-05-26 18:30:48,679 DEBG 'runserver' stderr output:
[Fri May 26 18:30:48.678678 2023] [mpm_event:notice] [pid 286:tid 140099398150016] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.4 Python/3.10 configured -- resuming normal operations
[Fri May 26 18:30:48.678746 2023] [mpm_event:info] [pid 286:tid 140099398150016] AH00490: Server built: 2023-03-01T22:43:55
[Fri May 26 18:30:48.678749 2023] [core:notice] [pid 286:tid 140099398150016] AH00094: Command line: 'apache2 (mod_wsgi-express) -f /tmp/cvat-server/httpd.conf -D MOD_WSGI_KEEP_ALIVE -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'

2023-05-26 18:30:48,679 DEBG 'runserver' stderr output:
[Fri May 26 18:30:48.678697 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164): Starting process 'localhost:8080' with threads=5.
[Fri May 26 18:30:48.679233 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164): Python home /opt/venv.
[Fri May 26 18:30:48.679280 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164): Initializing Python.

2023-05-26 18:30:48,699 DEBG 'runserver' stderr output:
[Fri May 26 18:30:48.699323 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164): Attach interpreter ''.

2023-05-26 18:30:48,703 DEBG 'runserver' stderr output:
[Fri May 26 18:30:48.703031 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164): Imported 'mod_wsgi'.
[Fri May 26 18:30:48.703070 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164, process='localhost:8080', application=''): Loading Python script file '/tmp/cvat-server/handler.wsgi'.

2023-05-26 18:30:49,026 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:30:49,026 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:EBbB2b6j9z65OMIXjLKm/Tm2zxhjNm3CQwE13KlLXM8 "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-26 18:30:49,029 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-26 18:31:01,333 DEBG 'runserver' stderr output:
[Fri May 26 18:31:01.333018 2023] [wsgi:info] [pid 1164:tid 140096294712896] mod_wsgi (pid=1164): Application startup timer triggered 'localhost:8080'.

2023-05-26 18:31:01,333 DEBG 'runserver' stderr output:
[Fri May 26 18:31:01.333154 2023] [wsgi:info] [pid 1164:tid 140096294712896] mod_wsgi (pid=1164): Application startup timer cancelled 'localhost:8080'.

2023-05-26 18:31:22,809 DEBG 'runserver' stderr output:
[Fri May 26 18:31:22.808466 2023] [wsgi:error] [pid 1164:tid 140096294712896] [remote 192.168.251.158:44468] [2023-05-26 18:31:22,808] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:31:22,809 DEBG 'runserver' stderr output:
[Fri May 26 18:31:22.808565 2023] [wsgi:error] [pid 1164:tid 140096294712896] [remote 192.168.251.158:44468] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:32:12,839 DEBG 'runserver' stderr output:
[Fri May 26 18:32:12.838749 2023] [wsgi:error] [pid 1164:tid 140096261142080] [remote 192.168.251.158:45660] [2023-05-26 18:32:12,838] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:32:12,841 DEBG 'runserver' stderr output:
[Fri May 26 18:32:12.841036 2023] [wsgi:error] [pid 1164:tid 140096261142080] [remote 192.168.251.158:45660] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:32:52,799 DEBG 'runserver' stderr output:
[Fri May 26 18:32:52.799619 2023] [wsgi:error] [pid 1164:tid 140096261142080] [remote 192.168.251.158:52798] [2023-05-26 18:32:52,799] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:32:52,800 DEBG 'runserver' stderr output:
[Fri May 26 18:32:52.800489 2023] [wsgi:error] [pid 1164:tid 140096261142080] [remote 192.168.251.158:52798] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:33:42,838 DEBG 'runserver' stderr output:
[Fri May 26 18:33:42.838814 2023] [wsgi:error] [pid 1164:tid 140096294712896] [remote 192.168.251.158:52110] [2023-05-26 18:33:42,838] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:33:42,840 DEBG 'runserver' stderr output:
[Fri May 26 18:33:42.840722 2023] [wsgi:error] [pid 1164:tid 140096294712896] [remote 192.168.251.158:52110] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:34:22,800 DEBG 'runserver' stderr output:
[Fri May 26 18:34:22.800785 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:46566] [2023-05-26 18:34:22,800] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:34:22,802 DEBG 'runserver' stderr output:
[Fri May 26 18:34:22.802621 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:46566] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:35:49,116 DEBG 'runserver' stderr output:
[Fri May 26 18:35:49.112634 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:55496] [2023-05-26 18:35:49,112] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:35:49,125 DEBG 'runserver' stderr output:
[Fri May 26 18:35:49.124806 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:55496] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:35:53,017 DEBG 'runserver' stderr output:
[Fri May 26 18:35:53.017364 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:58484] [2023-05-26 18:35:53,017] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:35:53,022 DEBG 'runserver' stderr output:
[Fri May 26 18:35:53.022368 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:58484] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:36:59,839 DEBG 'runserver' stderr output:
[Fri May 26 18:36:59.839811 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:49144] [2023-05-26 18:36:59,839] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:36:59,840 DEBG 'runserver' stderr output:
[Fri May 26 18:36:59.839910 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:49144] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:37:22,801 DEBG 'runserver' stderr output:
[Fri May 26 18:37:22.801042 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:45464] [2023-05-26 18:37:22,800] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:37:22,801 DEBG 'runserver' stderr output:
[Fri May 26 18:37:22.801919 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:45464] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:38:29,841 DEBG 'runserver' stderr output:
[Fri May 26 18:38:29.840507 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:57048] [2023-05-26 18:38:29,840] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:38:29,842 DEBG 'runserver' stderr output:
[Fri May 26 18:38:29.842033 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:57048] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:38:52,801 DEBG 'runserver' stderr output:
[Fri May 26 18:38:52.801036 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:40356] [2023-05-26 18:38:52,800] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:38:52,801 DEBG 'runserver' stderr output:
[Fri May 26 18:38:52.801146 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:40356] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:39:59,840 DEBG 'runserver' stderr output:
[Fri May 26 18:39:59.839951 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:45170] [2023-05-26 18:39:59,839] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:39:59,842 DEBG 'runserver' stderr output:
[Fri May 26 18:39:59.842491 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:45170] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:40:22,801 DEBG 'runserver' stderr output:
[Fri May 26 18:40:22.801096 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:54500] [2023-05-26 18:40:22,801] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:40:22,802 DEBG 'runserver' stderr output:
[Fri May 26 18:40:22.802407 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:54500] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:41:29,839 DEBG 'runserver' stderr output:
[Fri May 26 18:41:29.839216 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:47696] [2023-05-26 18:41:29,839] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:41:29,841 DEBG 'runserver' stderr output:
[Fri May 26 18:41:29.841389 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:47696] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:41:52,801 DEBG 'runserver' stderr output:
[Fri May 26 18:41:52.801148 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:57128] [2023-05-26 18:41:52,801] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:41:52,802 DEBG 'runserver' stderr output:
[Fri May 26 18:41:52.802091 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:57128] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:42:59,837 DEBG 'runserver' stderr output:
[Fri May 26 18:42:59.837380 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:40368] [2023-05-26 18:42:59,837] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:42:59,839 DEBG 'runserver' stderr output:
[Fri May 26 18:42:59.839210 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:40368] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:43:22,800 DEBG 'runserver' stderr output:
[Fri May 26 18:43:22.800607 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:54246] [2023-05-26 18:43:22,800] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:43:22,801 DEBG 'runserver' stderr output:
[Fri May 26 18:43:22.801906 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:54246] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:44:52,801 DEBG 'runserver' stderr output:
[Fri May 26 18:44:52.800717 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:35104] [2023-05-26 18:44:52,800] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:44:52,803 DEBG 'runserver' stderr output:
[Fri May 26 18:44:52.803067 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:35104] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:46:22,803 DEBG 'runserver' stderr output:
[Fri May 26 18:46:22.801908 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:48514] [2023-05-26 18:46:22,801] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:46:22,804 DEBG 'runserver' stderr output:
[Fri May 26 18:46:22.803849 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:48514] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:47:52,802 DEBG 'runserver' stderr output:
[Fri May 26 18:47:52.801545 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:46980] [2023-05-26 18:47:52,801] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:47:52,803 DEBG 'runserver' stderr output:
[Fri May 26 18:47:52.803562 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:46980] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:49:22,810 DEBG 'runserver' stderr output:
[Fri May 26 18:49:22.803981 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:60084] [2023-05-26 18:49:22,803] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:49:22,811 DEBG 'runserver' stderr output:
[Fri May 26 18:49:22.811722 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:60084] WARNING:django.request:Unauthorized: /api/events


the ingress seem to be working as well

Name:             cvat
Labels:           app.kubernetes.io/instance=cvat
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=cvat
                  app.kubernetes.io/version=latest
                  helm.sh/chart=cvat
Namespace:        cvat-gsrv
Address:          
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host           Path  Backends
  ----           ----  --------
  cvat.gsrv.lan  
                 /api         cvat-backend-service:8080 (192.168.251.152:8080)
                 /admin       cvat-backend-service:8080 (192.168.251.152:8080)
                 /static      cvat-backend-service:8080 (192.168.251.152:8080)
                 /django-rq   cvat-backend-service:8080 (192.168.251.152:8080)
                 /git         cvat-backend-service:8080 (192.168.251.152:8080)
                 /opencv      cvat-backend-service:8080 (192.168.251.152:8080)
                 /profiler    cvat-backend-service:8080 (192.168.251.152:8080)
                 /            cvat-frontend-service:80 (192.168.251.148:80)
Annotations:     kubernetes.io/ingress.class: traefik
                 meta.helm.sh/release-name: cvat
                 meta.helm.sh/release-namespace: cvat-gsrv
                 traefik.ingress.kubernetes.io/router.entrypoints: web
Events:          <none>

@ganindu7
Copy link

Here is a backend server log again (this time I was using the latest release branch)

kubectl logs --namespace cvat-gsrv cvat-backend-server-d64969648-5zxg8
Defaulted container "cvat-app-backend-server-container" out of: cvat-app-backend-server-container, user-data-permission-fix (init)
2023-05-27 17:45:11,642 INFO RPC interface 'supervisor' initialized
2023-05-27 17:45:11,642 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-05-27 17:45:11,642 INFO supervisord started with pid 1
2023-05-27 17:45:12,648 INFO spawned: 'ssh-agent' with pid 7
2023-05-27 17:45:12,652 INFO spawned: 'clamav_update' with pid 8
2023-05-27 17:45:12,673 INFO spawned: 'runserver' with pid 9
2023-05-27 17:45:12,674 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140061064098720 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stdout)>
2023-05-27 17:45:12,674 DEBG fd 14 closed, stopped monitoring <POutputDispatcher at 140061064098576 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stderr)>
2023-05-27 17:45:12,674 INFO exited: clamav_update (exit status 0; not expected)
2023-05-27 17:45:12,676 DEBG received SIGCHLD indicating a child quit
2023-05-27 17:45:12,677 DEBG 'ssh-agent' stdout output:
SSH_AUTH_SOCK=/tmp/ssh-agent.sock; export SSH_AUTH_SOCK;
echo Agent pid 7;

2023-05-27 17:45:12,677 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = SOCKET
debug2: fd 3 setting O_NONBLOCK

2023-05-27 17:45:12,682 DEBG 'runserver' stderr output:
wait-for-it.sh: waiting for cvat-postgresql:5432 without a timeout

2023-05-27 17:45:13,683 INFO success: ssh-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-27 17:45:13,687 INFO spawned: 'clamav_update' with pid 16
2023-05-27 17:45:13,687 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-27 17:45:13,692 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140061064098672 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stdout)>
2023-05-27 17:45:13,692 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140061064098768 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stderr)>
2023-05-27 17:45:13,693 INFO exited: clamav_update (exit status 0; not expected)
2023-05-27 17:45:13,693 DEBG received SIGCHLD indicating a child quit
2023-05-27 17:45:15,700 INFO spawned: 'clamav_update' with pid 19
2023-05-27 17:45:15,705 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140061064098720 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stdout)>
2023-05-27 17:45:15,705 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140061064088208 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stderr)>
2023-05-27 17:45:15,705 INFO exited: clamav_update (exit status 0; not expected)
2023-05-27 17:45:15,705 DEBG received SIGCHLD indicating a child quit
2023-05-27 17:45:18,715 INFO spawned: 'clamav_update' with pid 23
2023-05-27 17:45:18,719 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140061064098672 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stdout)>
2023-05-27 17:45:18,719 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140061064098864 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stderr)>
2023-05-27 17:45:18,719 INFO exited: clamav_update (exit status 0; not expected)
2023-05-27 17:45:18,719 DEBG received SIGCHLD indicating a child quit
2023-05-27 17:45:19,722 INFO gave up: clamav_update entered FATAL state, too many start retries too quickly
2023-05-27 17:45:48,985 DEBG 'runserver' stderr output:
wait-for-it.sh: cvat-postgresql:5432 is available after 36 seconds

2023-05-27 17:45:48,985 DEBG 'runserver' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2023-05-27 17:45:49,200 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:45:49,203 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug2: process_request_identities: replying with 0 allowed of 0 available keys

2023-05-27 17:45:49,221 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:45:49,222 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-27 17:45:49,224 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:kx1H/Kp+1xknufZlSVxut3W0wwaC+QBVtwyLgOOLQbs "django@cvat-backend-worker-export-57b7c75c79-679cz" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-27 17:45:49,224 DEBG 'runserver' stderr output:
Identity added: /home/django/.ssh/id_rsa (django@cvat-backend-worker-export-57b7c75c79-679cz)

2023-05-27 17:47:23,876 DEBG 'runserver' stdout output:
SSH keys were not found
Copying them from keys volume
Operations to perform:
Apply all migrations: account, admin, auth, authtoken, contenttypes, dataset_repo, db, engine, organizations, sessions, sites, socialaccount, webhooks
Running migrations:
Applying contenttypes.0001_initial...
2023-05-27 17:47:23,974 DEBG 'runserver' stdout output:
OK
Applying auth.0001_initial...
2023-05-27 17:47:24,874 DEBG 'runserver' stdout output:
OK
Applying account.0001_initial...
2023-05-27 17:47:25,219 DEBG 'runserver' stdout output:
OK
Applying account.0002_email_max_length...
2023-05-27 17:47:25,244 DEBG 'runserver' stdout output:
OK
Applying admin.0001_initial...
2023-05-27 17:47:25,413 DEBG 'runserver' stdout output:
OK
Applying admin.0002_logentry_remove_auto_add...
2023-05-27 17:47:25,423 DEBG 'runserver' stdout output:
OK
Applying admin.0003_logentry_add_action_flag_choices...
2023-05-27 17:47:25,432 DEBG 'runserver' stdout output:
OK
Applying contenttypes.0002_remove_content_type_name...
2023-05-27 17:47:25,447 DEBG 'runserver' stdout output:
OK
Applying auth.0002_alter_permission_name_max_length...
2023-05-27 17:47:25,455 DEBG 'runserver' stdout output:
OK
Applying auth.0003_alter_user_email_max_length...
2023-05-27 17:47:25,466 DEBG 'runserver' stdout output:
OK
Applying auth.0004_alter_user_username_opts...
2023-05-27 17:47:25,473 DEBG 'runserver' stdout output:
OK
Applying auth.0005_alter_user_last_login_null...
2023-05-27 17:47:25,482 DEBG 'runserver' stdout output:
OK
Applying auth.0006_require_contenttypes_0002...
2023-05-27 17:47:25,488 DEBG 'runserver' stdout output:
OK
Applying auth.0007_alter_validators_add_error_messages...
2023-05-27 17:47:25,498 DEBG 'runserver' stdout output:
OK
Applying auth.0008_alter_user_username_max_length...
2023-05-27 17:47:25,556 DEBG 'runserver' stdout output:
OK
Applying auth.0009_alter_user_last_name_max_length...
2023-05-27 17:47:25,563 DEBG 'runserver' stdout output:
OK
Applying auth.0010_alter_group_name_max_length...
2023-05-27 17:47:25,579 DEBG 'runserver' stdout output:
OK
Applying auth.0011_update_proxy_permissions...
2023-05-27 17:47:25,585 DEBG 'runserver' stdout output:
OK
Applying auth.0012_alter_user_first_name_max_length...
2023-05-27 17:47:25,597 DEBG 'runserver' stdout output:
OK
Applying authtoken.0001_initial...
2023-05-27 17:47:25,743 DEBG 'runserver' stdout output:
OK
Applying authtoken.0002_auto_20160226_1747...
2023-05-27 17:47:25,764 DEBG 'runserver' stdout output:
OK
Applying authtoken.0003_tokenproxy...
2023-05-27 17:47:25,772 DEBG 'runserver' stdout output:
OK
Applying engine.0001_release_v0_1_0...
2023-05-27 17:47:27,543 DEBG 'runserver' stdout output:
OK
Applying engine.0002_labeledpoints_labeledpointsattributeval_labeledpolygon_labeledpolygonattributeval_labeledpolyline_la...
2023-05-27 17:47:29,562 DEBG 'runserver' stdout output:
OK
Applying engine.0003_objectpath_shapes...
2023-05-27 17:47:29,583 DEBG 'runserver' stdout output:
OK
Applying engine.0004_task_z_order...
2023-05-27 17:47:29,598 DEBG 'runserver' stdout output:
OK
Applying engine.0005_auto_20180609_1512...
2023-05-27 17:47:29,806 DEBG 'runserver' stdout output:
OK
Applying engine.0006_auto_20180629_1501...
2023-05-27 17:47:29,865 DEBG 'runserver' stdout output:
OK
Applying engine.0007_task_flipped...
2023-05-27 17:47:29,875 DEBG 'runserver' stdout output:
OK
Applying engine.0008_auto_20180917_1424...
2023-05-27 17:47:29,891 DEBG 'runserver' stdout output:
OK
Applying engine.0009_auto_20180917_1424...
2023-05-27 17:47:35,154 DEBG 'runserver' stdout output:
OK
Applying engine.0010_auto_20181011_1517...
2023-05-27 17:47:35,705 DEBG 'runserver' stdout output:
OK
Applying engine.0011_add_task_source_and_safecharfield...
2023-05-27 17:47:35,779 DEBG 'runserver' stdout output:
OK
Applying engine.0012_auto_20181025_1618...
2023-05-27 17:47:35,797 DEBG 'runserver' stdout output:
OK
Applying engine.0013_auth_no_default_permissions...
2023-05-27 17:47:36,065 DEBG 'runserver' stdout output:
OK
Applying engine.0014_job_max_shape_id...
2023-05-27 17:47:36,266 DEBG 'runserver' stdout output:
OK
Applying dataset_repo.0001_initial...
2023-05-27 17:47:36,356 DEBG 'runserver' stdout output:
OK
Applying dataset_repo.0002_auto_20190123_1305...
2023-05-27 17:47:36,365 DEBG 'runserver' stdout output:
OK
Applying dataset_repo.0003_gitdata_lfs...
2023-05-27 17:47:36,372 DEBG 'runserver' stdout output:
OK
Applying dataset_repo.0004_rename...
2023-05-27 17:47:36,408 DEBG 'runserver' stdout output:
OK
Applying dataset_repo.0005_auto_20201019_1100...
2023-05-27 17:47:36,416 DEBG 'runserver' stdout output:
OK
Applying dataset_repo.0006_gitdata_format...
2023-05-27 17:47:36,443 DEBG 'runserver' stdout output:
OK
Applying db.0001_initial...
2023-05-27 17:47:36,490 DEBG 'runserver' stdout output:
OK
Applying organizations.0001_initial...
2023-05-27 17:47:36,978 DEBG 'runserver' stdout output:
OK
Applying engine.0015_db_redesign_20190217...
2023-05-27 17:47:38,085 DEBG 'runserver' stdout output:
OK
Applying engine.0016_attribute_spec_20190217...
2023-05-27 17:47:38,178 DEBG 'runserver' stdout output:
OK
Applying engine.0017_db_redesign_20190221...
2023-05-27 17:47:41,246 DEBG 'runserver' stdout output:
OK
Applying engine.0018_jobcommit...
2023-05-27 17:47:41,396 DEBG 'runserver' stdout output:
OK
Applying engine.0019_frame_selection...
2023-05-27 17:47:41,465 DEBG 'runserver' stdout output:
OK
Applying engine.0020_remove_task_flipped...
2023-05-27 17:47:41,496 DEBG 'runserver' stdout output:
Getting flipped tasks...
Conversion started...
OK
Applying engine.0021_auto_20190826_1827...
2023-05-27 17:47:41,508 DEBG 'runserver' stdout output:
OK
Applying engine.0022_auto_20191004_0817...
2023-05-27 17:47:41,698 DEBG 'runserver' stdout output:
OK
Applying engine.0023_auto_20200113_1323...
2023-05-27 17:47:41,843 DEBG 'runserver' stdout output:
OK
Applying engine.0024_auto_20191023_1025...
2023-05-27 17:47:42,254 DEBG 'runserver' stdout output:

Start schema migration...

2023-05-27 17:47:42,258 DEBG 'runserver' stderr output:
INFO:0024_auto_20191023_1025:
Start schema migration...

2023-05-27 17:47:42,258 DEBG 'runserver' stdout output:

Schema migration is finished...

Start data migration...

2023-05-27 17:47:42,258 DEBG 'runserver' stderr output:
INFO:0024_auto_20191023_1025:
Schema migration is finished...
INFO:0024_auto_20191023_1025:
Start data migration...

2023-05-27 17:47:42,629 DEBG 'runserver' stdout output:
OK
Applying engine.0025_auto_20200324_1222...
2023-05-27 17:47:42,636 DEBG 'runserver' stdout output:
OK
Applying engine.0026_auto_20200719_1511...
2023-05-27 17:47:42,649 DEBG 'runserver' stdout output:
OK
Applying engine.0027_auto_20200719_1552...
2023-05-27 17:47:42,673 DEBG 'runserver' stdout output:
OK
Applying engine.0028_labelcolor...
2023-05-27 17:47:42,706 DEBG 'runserver' stdout output:
OK
Applying engine.0029_data_storage_method...
2023-05-27 17:47:42,732 DEBG 'runserver' stdout output:
OK
Applying engine.0030_auto_20200914_1331...
2023-05-27 17:47:42,854 DEBG 'runserver' stdout output:
OK
Applying engine.0031_auto_20201011_0220...
2023-05-27 17:47:42,870 DEBG 'runserver' stdout output:
OK
Applying engine.0032_remove_task_z_order...
2023-05-27 17:47:42,886 DEBG 'runserver' stdout output:
OK
Applying engine.0033_projects_adjastment...
2023-05-27 17:47:42,964 DEBG 'runserver' stdout output:
OK
Applying engine.0034_auto_20201125_1426...
2023-05-27 17:47:43,587 DEBG 'runserver' stdout output:
OK
Applying engine.0035_data_storage...
2023-05-27 17:47:43,595 DEBG 'runserver' stdout output:
OK
Applying engine.0036_auto_20201216_0943...
2023-05-27 17:47:43,706 DEBG 'runserver' stdout output:
OK
Applying engine.0037_task_subset...
2023-05-27 17:47:43,723 DEBG 'runserver' stdout output:
OK
Applying engine.0038_manifest...
2023-05-27 17:47:43,741 DEBG 'runserver' stdout output:
The data migration has been started for creating manifest`s files

2023-05-27 17:47:43,741 DEBG 'runserver' stderr output:
The data migration has been started for creating manifests files INFO:0038_manifest:The data migration has been started for creating manifests files

2023-05-27 17:47:43,743 DEBG 'runserver' stdout output:
Need to update 0 data objects

2023-05-27 17:47:43,743 DEBG 'runserver' stderr output:
Need to update 0 data objects
INFO:0038_manifest:Need to update 0 data objects

2023-05-27 17:47:43,746 DEBG 'runserver' stdout output:
OK
Applying engine.0039_auto_training...
2023-05-27 17:47:44,168 DEBG 'runserver' stdout output:
OK
Applying engine.0040_cloud_storage...
2023-05-27 17:47:44,288 DEBG 'runserver' stdout output:
OK
Applying engine.0041_auto_20210827_0258...
2023-05-27 17:47:44,312 DEBG 'runserver' stdout output:
OK
Applying engine.0042_auto_20210830_1056...
2023-05-27 17:47:44,401 DEBG 'runserver' stdout output:
OK
Applying engine.0043_auto_20211027_0718...
2023-05-27 17:47:44,415 DEBG 'runserver' stdout output:
OK
Applying engine.0044_auto_20211115_0858...
2023-05-27 17:47:44,431 DEBG 'runserver' stdout output:
OK
Applying engine.0045_auto_20211123_0824...
2023-05-27 17:47:44,459 DEBG 'runserver' stdout output:
OK
Applying engine.0046_data_sorting_method...
2023-05-27 17:47:44,469 DEBG 'runserver' stdout output:
OK
Applying engine.0047_auto_20211110_1938...
2023-05-27 17:47:44,669 DEBG 'runserver' stdout output:
OK
Applying engine.0048_auto_20211112_1918...
2023-05-27 17:47:44,848 DEBG 'runserver' stdout output:
OK
Applying engine.0049_auto_20220202_0710...
2023-05-27 17:47:44,863 DEBG 'runserver' stdout output:
OK
Applying engine.0050_auto_20220211_1425...
2023-05-27 17:47:45,001 DEBG 'runserver' stdout output:
OK
Applying engine.0051_auto_20220220_1824...
2023-05-27 17:47:45,248 DEBG 'runserver' stdout output:
OK
Applying engine.0052_alter_cloudstorage_specific_attributes...
2023-05-27 17:47:45,262 DEBG 'runserver' stdout output:
OK
Applying engine.0053_data_deleted_frames...
2023-05-27 17:47:45,286 DEBG 'runserver' stdout output:
OK
Applying engine.0054_auto_20220610_1829...
2023-05-27 17:47:45,474 DEBG 'runserver' stdout output:
OK
Applying engine.0055_jobs_directories...
2023-05-27 17:47:45,495 DEBG 'runserver' stdout output:
Migration has been started. Need to create 0 directories.

2023-05-27 17:47:45,495 DEBG 'runserver' stderr output:
Migration has been started. Need to create 0 directories.
INFO:0055_jobs_directories:Migration has been started. Need to create 0 directories.

2023-05-27 17:47:45,495 DEBG 'runserver' stdout output:
Migration has been finished successfully.

2023-05-27 17:47:45,496 DEBG 'runserver' stderr output:
Migration has been finished successfully.
INFO:0055_jobs_directories:Migration has been finished successfully.

2023-05-27 17:47:45,498 DEBG 'runserver' stdout output:
OK
Applying engine.0056_jobs_previews...
2023-05-27 17:47:45,517 DEBG 'runserver' stdout output:
Migration has been started. Need to create 0 previews.

2023-05-27 17:47:45,517 DEBG 'runserver' stderr output:
Migration has been started. Need to create 0 previews.
INFO:0056_jobs_previews:Migration has been started. Need to create 0 previews.

2023-05-27 17:47:45,521 DEBG 'runserver' stdout output:
OK
Applying engine.0057_auto_20220726_0926...
2023-05-27 17:47:46,075 DEBG 'runserver' stdout output:
OK
Applying engine.0058_auto_20220809_1236...
2023-05-27 17:47:46,632 DEBG 'runserver' stdout output:
OK
Applying engine.0059_labeledshape_outside...
2023-05-27 17:47:46,647 DEBG 'runserver' stdout output:
OK
Applying engine.0060_alter_label_parent...
2023-05-27 17:47:46,669 DEBG 'runserver' stdout output:
OK
Applying engine.0061_auto_20221130_0844...
2023-05-27 17:47:46,685 DEBG 'runserver' stdout output:
OK
Applying engine.0062_delete_previews...
2023-05-27 17:47:46,704 DEBG 'runserver' stdout output:

Deleting Data previews...

2023-05-27 17:47:46,704 DEBG 'runserver' stderr output:
INFO:0062_delete_previews:
Deleting Data previews...

2023-05-27 17:47:46,708 DEBG 'runserver' stdout output:

Deleting Job previews...

2023-05-27 17:47:46,708 DEBG 'runserver' stderr output:
INFO:0062_delete_previews:
Deleting Job previews...

2023-05-27 17:47:46,708 DEBG 'runserver' stdout output:

Deleting CloudStorage previews...

2023-05-27 17:47:46,708 DEBG 'runserver' stderr output:
INFO:0062_delete_previews:
Deleting CloudStorage previews...

2023-05-27 17:47:46,717 DEBG 'runserver' stdout output:
OK
Applying engine.0063_delete_jobcommit...
2023-05-27 17:47:46,758 DEBG 'runserver' stdout output:
OK
Applying engine.0064_delete_or_rename_wrong_labels...
2023-05-27 17:47:46,777 DEBG 'runserver' stdout output:

Deleting skeleton Labels without skeletons...

2023-05-27 17:47:46,777 DEBG 'runserver' stderr output:
INFO:0064_delete_or_rename_wrong_labels:
Deleting skeleton Labels without skeletons...

2023-05-27 17:47:46,780 DEBG 'runserver' stdout output:

Deleting duplicate skeleton sublabels and renaming duplicate Labels...

2023-05-27 17:47:46,780 DEBG 'runserver' stderr output:
INFO:0064_delete_or_rename_wrong_labels:
Deleting duplicate skeleton sublabels and renaming duplicate Labels...

2023-05-27 17:47:46,785 DEBG 'runserver' stdout output:
OK
Applying engine.0065_auto_20230221_0931...
2023-05-27 17:47:46,884 DEBG 'runserver' stdout output:
OK
Applying engine.0066_auto_20230319_1252...
2023-05-27 17:47:46,918 DEBG 'runserver' stdout output:
OK
Applying engine.0067_alter_cloudstorage_credentials_type...
2023-05-27 17:47:46,931 DEBG 'runserver' stdout output:
OK
Applying sessions.0001_initial...
2023-05-27 17:47:46,984 DEBG 'runserver' stdout output:
OK
Applying sites.0001_initial...
2023-05-27 17:47:47,006 DEBG 'runserver' stdout output:
OK
Applying sites.0002_alter_domain_unique...
2023-05-27 17:47:47,032 DEBG 'runserver' stdout output:
OK
Applying socialaccount.0001_initial...
2023-05-27 17:47:47,327 DEBG 'runserver' stdout output:
OK
Applying socialaccount.0002_token_max_lengths...
2023-05-27 17:47:47,363 DEBG 'runserver' stdout output:
OK
Applying socialaccount.0003_extra_data_default_dict...
2023-05-27 17:47:47,377 DEBG 'runserver' stdout output:
OK
Applying webhooks.0001_initial...
2023-05-27 17:47:47,588 DEBG 'runserver' stdout output:
OK
Applying webhooks.0002_alter_webhookdelivery_status_code...
2023-05-27 17:47:47,663 DEBG 'runserver' stdout output:
OK
Applying webhooks.0003_alter_webhookdelivery_status_code...
2023-05-27 17:47:47,673 DEBG 'runserver' stdout output:
OK
Applying webhooks.0004_alter_webhook_target_url...
2023-05-27 17:47:47,878 DEBG 'runserver' stdout output:
OK

2023-05-27 17:47:49,114 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:47:49,115 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-27 17:47:49,116 DEBG 'ssh-agent' stderr output:
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints

2023-05-27 17:47:49,116 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:kx1H/Kp+1xknufZlSVxut3W0wwaC+QBVtwyLgOOLQbs "django@cvat-backend-worker-export-57b7c75c79-679cz" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-27 17:47:49,119 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-27 17:47:50,046 DEBG 'runserver' stdout output:

166 static files copied to '/home/django/static'.

2023-05-27 17:47:50,566 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:47:50,567 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-27 17:47:50,568 DEBG 'ssh-agent' stderr output:
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:kx1H/Kp+1xknufZlSVxut3W0wwaC+QBVtwyLgOOLQbs "django@cvat-backend-worker-export-57b7c75c79-679cz" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-27 17:47:50,572 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-27 17:48:16,264 DEBG 'runserver' stdout output:
Successfully ran command.
Server URL : http://localhost:8080/
Server Root : /tmp/cvat-server
Server Conf : /tmp/cvat-server/httpd.conf
Error Log File : /dev/stderr (INFO)
Operating Mode : daemon
Request Capacity : 5 (1 process * 5 threads)
Request Timeout : 60 (seconds)
Startup Timeout : 15 (seconds)
Queue Backlog : 100 (connections)
Queue Timeout : 45 (seconds)
Server Capacity : 20 (event/worker), 20 (prefork)
Server Backlog : 500 (connections)
Locale Setting : C.UTF-8

2023-05-27 17:48:16,351 DEBG 'runserver' stderr output:
[Sat May 27 17:48:16.350446 2023] [mpm_event:notice] [pid 9:tid 139633266231168] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.4 Python/3.10 configured -- resuming normal operations
[Sat May 27 17:48:16.350500 2023] [mpm_event:info] [pid 9:tid 139633266231168] AH00490: Server built: 2023-03-01T22:43:55
[Sat May 27 17:48:16.350504 2023] [core:notice] [pid 9:tid 139633266231168] AH00094: Command line: 'apache2 (mod_wsgi-express) -f /tmp/cvat-server/httpd.conf -D MOD_WSGI_KEEP_ALIVE -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'

2023-05-27 17:48:16,351 DEBG 'runserver' stderr output:
[Sat May 27 17:48:16.350453 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Starting process 'localhost:8080' with threads=5.
[Sat May 27 17:48:16.350980 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Python home /opt/venv.
[Sat May 27 17:48:16.351027 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Initializing Python.

2023-05-27 17:48:16,371 DEBG 'runserver' stderr output:
[Sat May 27 17:48:16.371743 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Attach interpreter ''.

2023-05-27 17:48:16,375 DEBG 'runserver' stderr output:
[Sat May 27 17:48:16.375441 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Imported 'mod_wsgi'.
[Sat May 27 17:48:16.375480 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662, process='localhost:8080', application=''): Loading Python script file '/tmp/cvat-server/handler.wsgi'.

2023-05-27 17:48:16,689 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:48:16,690 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-27 17:48:16,691 DEBG 'ssh-agent' stderr output:
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:kx1H/Kp+1xknufZlSVxut3W0wwaC+QBVtwyLgOOLQbs "django@cvat-backend-worker-export-57b7c75c79-679cz" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-27 17:48:16,694 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-27 17:48:17,700 DEBG 'runserver' stderr output:
[Sat May 27 17:48:17.700799 2023] [wsgi:info] [pid 662:tid 139630162835008] mod_wsgi (pid=662): Application startup timer triggered 'localhost:8080'.

2023-05-27 17:48:17,701 DEBG 'runserver' stderr output:
[Sat May 27 17:48:17.700943 2023] [wsgi:info] [pid 662:tid 139630162835008] mod_wsgi (pid=662): Application startup timer cancelled 'localhost:8080'.

2023-05-27 17:48:27,043 DEBG 'runserver' stderr output:
[Sat May 27 17:48:27.043573 2023] [wsgi:error] [pid 662:tid 139630154442304] [remote 192.168.251.158:35490] [2023-05-27 17:48:27,043] WARNING django.request: Unauthorized: /api/events

2023-05-27 17:48:27,043 DEBG 'runserver' stderr output:
[Sat May 27 17:48:27.043677 2023] [wsgi:error] [pid 662:tid 139630154442304] [remote 192.168.251.158:35490] WARNING:django.request:Unauthorized: /api/events

2023-05-27 17:49:45,445 DEBG 'runserver' stderr output:
[Sat May 27 17:49:45.443724 2023] [mpm_event:error] [pid 9:tid 139633266231168] AH10159: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting

2023-05-27 17:49:47,571 DEBG 'runserver' stderr output:
[Sat May 27 17:49:47.571790 2023] [wsgi:error] [pid 744:tid 139633258665536] [client 192.168.251.158:47226] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:49:55,627 DEBG 'runserver' stderr output:
[Sat May 27 17:49:55.626886 2023] [wsgi:error] [pid 744:tid 139633253938752] [client 192.168.251.158:36670] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:50:03,707 DEBG 'runserver' stderr output:
[Sat May 27 17:50:03.706884 2023] [wsgi:error] [pid 663:tid 139633253406272] [client 192.168.251.158:60206] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:50:03,804 DEBG 'runserver' stderr output:
[Sat May 27 17:50:03.804297 2023] [wsgi:info] [pid 662:tid 139630179620416] mod_wsgi (pid=662): Daemon process request time limit exceeded, stopping process 'localhost:8080'.
[Sat May 27 17:50:03.804366 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Shutdown requested 'localhost:8080'.

2023-05-27 17:50:08,804 DEBG 'runserver' stderr output:
[Sat May 27 17:50:08.804525 2023] [wsgi:info] [pid 662:tid 139625561585216] mod_wsgi (pid=662): Aborting process 'localhost:8080'.
[Sat May 27 17:50:08.804548 2023] [wsgi:info] [pid 662:tid 139625561585216] mod_wsgi (pid=662): Exiting process 'localhost:8080'.

2023-05-27 17:50:11,772 DEBG 'runserver' stderr output:
[Sat May 27 17:50:11.772011 2023] [wsgi:error] [pid 663:tid 139633252607552] [client 192.168.251.158:38672] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:50:19,843 DEBG 'runserver' stderr output:
[Sat May 27 17:50:19.843528 2023] [wsgi:error] [pid 744:tid 139633252873792] [client 192.168.251.158:38686] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:50:24,371 DEBG 'runserver' stderr output:
[Sat May 27 17:50:24.371529 2023] [wsgi:error] [pid 663:tid 139633258665536] [client 192.168.251.152:32782] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api

2023-05-27 17:50:27,971 DEBG 'runserver' stderr output:
[Sat May 27 17:50:27.970867 2023] [wsgi:error] [pid 744:tid 139633252607552] [client 192.168.251.158:42802] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:50:32,264 DEBG 'runserver' stderr output:
[Sat May 27 17:50:32.264410 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Dumping stack trace for active Python threads.
[Sat May 27 17:50:32.264429 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624345232960 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264433 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264435 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264437 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264438 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624353625664 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264440 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264441 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264443 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264444 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624907249216 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264446 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264447 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264449 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264451 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624915641920 executing file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 666, in _execute_with_retry
[Sat May 27 17:50:32.264454 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 499, in init,
[Sat May 27 17:50:32.264456 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 43, in ,
[Sat May 27 17:50:32.264458 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 42, in init,
[Sat May 27 17:50:32.264460 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/djangocache.py", line 31, in init,
[Sat May 27 17:50:32.264463 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/cache/init.py", line 44, in create_connection,
[Sat May 27 17:50:32.264465 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/connection.py", line 62, in getitem,
[Sat May 27 17:50:32.264467 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/health/backends.py", line 42, in check_status,
[Sat May 27 17:50:32.264469 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check,
[Sat May 27 17:50:32.264473 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 35, in _run,
[Sat May 27 17:50:32.264475 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run,
[Sat May 27 17:50:32.264476 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker,
[Sat May 27 17:50:32.264478 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264479 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264480 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264482 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139630146049600 executing file "/usr/lib/python3.10/threading.py", line 320, in wait
[Sat May 27 17:50:32.264483 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 453, in result,
[Sat May 27 17:50:32.264485 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel,
[Sat May 27 17:50:32.264487 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator,
[Sat May 27 17:50:32.264489 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 44, in run_check,
[Sat May 27 17:50:32.264490 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 16, in errors,
[Sat May 27 17:50:32.264492 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/views.py", line 91, in get,
[Sat May 27 17:50:32.264493 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func,
[Sat May 27 17:50:32.264496 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper,
[Sat May 27 17:50:32.264497 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch,
[Sat May 27 17:50:32.264499 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view,
[Sat May 27 17:50:32.264501 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response,
[Sat May 27 17:50:32.264503 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264504 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/iam/views.py", line 91, in call,
[Sat May 27 17:50:32.264506 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264509 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,

2023-05-27 17:50:32,264 DEBG 'runserver' stderr output:
[Sat May 27 17:50:32.264510 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264521 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264523 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264524 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264526 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264527 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/crum/init.py", line 97, in call,
[Sat May 27 17:50:32.264529 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264531 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/engine/middleware.py", line 17, in call,
[Sat May 27 17:50:32.264532 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264533 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264535 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264536 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264537 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264539 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264540 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264541 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264543 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264544 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264545 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264547 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264549 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264551 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 130, in get_response,
[Sat May 27 17:50:32.264552 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 133, in call,
[Sat May 27 17:50:32.264554 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/mod_wsgi/server/init.py", line 1568, in handle_request.
[Sat May 27 17:50:32.264556 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624924034624 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264557 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264559 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264560 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264561 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624932427328 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264562 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264564 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264565 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264566 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624940820032 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264567 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264569 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264570 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264571 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624949212736 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264572 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264574 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264575 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264576 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624957605440 executing file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 666, in _execute_with_retry
[Sat May 27 17:50:32.264578 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 499, in init,
[Sat May 27 17:50:32.264580 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 43, in ,
[Sat May 27 17:50:32.264581 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 42, in init,
[Sat May 27 17:50:32.264582 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/djangocache.py", line 31, in init,
[Sat May 27 17:50:32.264584 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/cache/init.py", line 44, in create_connection,
[Sat May 27 17:50:32.264585 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/connection.py", line 62, in getitem,
[Sat May 27 17:50:32.264586 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/health/backends.py", line 42, in check_status,
[Sat May 27 17:50:32.264588 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check,
[Sat May 27 17:50:32.264589 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 35, in _run,
[Sat May 27 17:50:32.264590 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run,
[Sat May 27 17:50:32.264592 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker,
[Sat May 27 17:50:32.264593 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264594 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264595 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264596 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139625511228992 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264598 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264599 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264600 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264601 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139625519621696 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264603 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264604 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264606 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264607 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139625528014400 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264608 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264610 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264611 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264612 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139625536407104 executing file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 721, in _transact
[Sat May 27 17:50:32.264614 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/contextlib.py", line 135, in enter,
[Sat May 27 17:50:32.264616 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 796, in set,
[Sat May 27 17:50:32.264618 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 123, in set,
[Sat May 27 17:50:32.264619 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/djangocache.py", line 165, in set,
[Sat May 27 17:50:32.264621 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/health/backends.py", line 44, in check_status,
[Sat May 27 17:50:32.264622 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check,
[Sat May 27 17:50:32.264623 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 35, in _run,
[Sat May 27 17:50:32.264624 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run,
[Sat May 27 17:50:32.264626 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker,
[Sat May 27 17:50:32.264627 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264628 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264629 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264630 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139630137656896 executing file "/usr/lib/python3.10/threading.py", line 320, in wait
[Sat May 27 17:50:32.264632 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 453, in result,
[Sat May 27 17:50:32.264633 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel,
[Sat May 27 17:50:32.264635 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator,
[Sat May 27 17:50:32.264636 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 44, in run_check,
[Sat May 27 17:50:32.264638 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 16, in errors,
[Sat May 27 17:50:32.264639 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/views.py", line 91, in get,
[Sat May 27 17:50:32.264640 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func,
[Sat May 27 17:50:32.264641 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper,
[Sat May 27 17:50:32.264643 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch,
[Sat May 27 17:50:32.264644 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view,
[Sat May 27 17:50:32.264645 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response,
[Sat May 27 17:50:32.264647 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264648 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/iam/views.py", line 91, in call,
[Sat May 27 17:50:32.264649 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264651 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264652 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264653 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264655 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264656 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264657 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264658 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/crum/init.py", line 97, in call,
[Sat May 27 17:50:32.264660 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264662 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/engine/middleware.py", line 17, in call,
[Sat May 27 17:50:32.264663 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264664 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264666 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264667 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264668 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264670 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264671 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264672 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264674 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264675 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264676 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264678 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264679 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264680 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 130, in get_response,
[Sat May 27 17:50:32.264682 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 133, in call,
[Sat May 27 17:50:32.264683 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/mod_wsgi/server/init.py", line 1568, in handle_request.
[Sat May 27 17:50:32.264684 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139630129264192 executing file "/usr/lib/python3.10/threading.py", line 320, in wait
[Sat May 27 17:50:32.264686 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 453, in result,
[Sat May 27 17:50:32.264687 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel,
[Sat May 27 17:50:32.264689 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator,
[Sat May 27 17:50:32.264690 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 44, in run_check,
[Sat May 27 17:50:32.264691 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 16, in errors,
[Sat May 27 17:50:32.264693 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/views.py", line 91, in get,
[Sat May 27 17:50:32.264694 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func,
[Sat May 27 17:50:32.264695 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper,
[Sat May 27 17:50:32.264697 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch,
[Sat May 27 17:50:32.264698 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view,
[Sat May 27 17:50:32.264699 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response,
[Sat May 27 17:50:32.264701 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264702 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/iam/views.py", line 91, in call,
[Sat May 27 17:50:32.264703 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264704 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264706 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264707 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264708 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264710 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264711 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264712 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/crum/init.py", line 97, in call,
[Sat May 27 17:50:32.264713 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264715 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/engine/middleware.py", line 17, in call,
[Sat May 27 17:50:32.264717 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264718 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264719 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264720 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264722 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264723 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264724 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264726 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264727 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264728 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264730 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264731 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264732 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264734 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 130, in get_response,
[Sat May 27 17:50:32.264735 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 133, in call,
[Sat May 27 17:50:32.264736 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/mod_wsgi/server/init.py", line 1568, in handle_request.
[Sat May 27 17:50:32.264737 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139626062329408 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264739 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264740 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264743 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264744 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139626070722112 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264745 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264747 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264748 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264749 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139626079114816 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
[Sat May 27 17:50:32.264751 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264752 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264753 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264755 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139626087507520 executing file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 744, in _transact
[Sat May 27 17:50:32.264757 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/contextlib.py", line 142, in exit,
[Sat May 27 17:50:32.264758 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 796, in set,
[Sat May 27 17:50:32.264759 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 123, in set,
[Sat May 27 17:50:32.264761 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/djangocache.py", line 165, in set,
[Sat May 27 17:50:32.264762 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/health/backends.py", line 44, in check_status,
[Sat May 27 17:50:32.264763 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check,
[Sat May 27 17:50:32.264765 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 35, in _run,
[Sat May 27 17:50:32.264766 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run,
[Sat May 27 17:50:32.264767 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker,
[Sat May 27 17:50:32.264768 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run,
[Sat May 27 17:50:32.264770 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner,
[Sat May 27 17:50:32.264771 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap.
[Sat May 27 17:50:32.264773 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139630154442304 executing file "/usr/lib/python3.10/threading.py", line 320, in wait
[Sat May 27 17:50:32.264774 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 453, in result,
[Sat May 27 17:50:32.264776 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel,
[Sat May 27 17:50:32.264777 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator,
[Sat May 27 17:50:32.264778 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 44, in run_check,
[Sat May 27 17:50:32.264780 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 16, in errors,
[Sat May 27 17:50:32.264781 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/views.py", line 91, in get,
[Sat May 27 17:50:32.264782 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func,
[Sat May 27 17:50:32.264784 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper,
[Sat May 27 17:50:32.264785 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch,
[Sat May 27 17:50:32.264786 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view,
[Sat May 27 17:50:32.264788 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response,
[Sat May 27 17:50:32.264789 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264790 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/iam/views.py", line 91, in call,
[Sat May 27 17:50:32.264792 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264793 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264794 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264795 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264797 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264798 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264800 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264802 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/crum/init.py", line 97, in call,
[Sat May 27 17:50:32.264803 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264804 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/engine/middleware.py", line 17, in call,
[Sat May 27 17:50:32.264805 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264807 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264808 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264809 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264811 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264812 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264813 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264814 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264816 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264817 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264818 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264820 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264821 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,

2023-05-27 17:50:32,265 DEBG 'runserver' stderr output:
[Sat May 27 17:50:32.264822 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 130, in get_response,
[Sat May 27 17:50:32.264833 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 133, in call,
[Sat May 27 17:50:32.264834 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/mod_wsgi/server/init.py", line 1568, in handle_request.
[Sat May 27 17:50:32.264836 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139630162835008 executing file "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
[Sat May 27 17:50:32.264838 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1096, in join,
[Sat May 27 17:50:32.264839 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 235, in shutdown,
[Sat May 27 17:50:32.264841 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 649, in exit,
[Sat May 27 17:50:32.264842 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 43, in run_check,
[Sat May 27 17:50:32.264844 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 16, in errors,
[Sat May 27 17:50:32.264845 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/views.py", line 91, in get,
[Sat May 27 17:50:32.264847 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func,
[Sat May 27 17:50:32.264848 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper,
[Sat May 27 17:50:32.264849 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch,
[Sat May 27 17:50:32.264851 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view,
[Sat May 27 17:50:32.264852 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response,
[Sat May 27 17:50:32.264853 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264854 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/iam/views.py", line 91, in call,
[Sat May 27 17:50:32.264856 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264857 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264858 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264860 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264862 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264863 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264865 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264866 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/crum/init.py", line 97, in call,
[Sat May 27 17:50:32.264868 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264869 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/engine/middleware.py", line 17, in call,
[Sat May 27 17:50:32.264870 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264871 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264873 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264874 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264875 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264877 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264878 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264879 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264881 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264882 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264883 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264884 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,
[Sat May 27 17:50:32.264886 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,
[Sat May 27 17:50:32.264887 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 130, in get_response,
[Sat May 27 17:50:32.264889 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 133, in call,
[Sat May 27 17:50:32.264890 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/mod_wsgi/server/init.py", line 1568, in handle_request.

2023-05-27 17:50:32,493 DEBG 'runserver' stderr output:
[Sat May 27 17:50:32.493121 2023] [wsgi:info] [pid 9:tid 139633266231168] mod_wsgi (pid=662): Process 'localhost:8080' has died, deregister and restart it.
[Sat May 27 17:50:32.493148 2023] [wsgi:info] [pid 9:tid 139633266231168] mod_wsgi (pid=662): Process 'localhost:8080' has been deregistered and will no longer be monitored.

2023-05-27 17:50:32,493 DEBG 'runserver' stderr output:
[Sat May 27 17:50:32.493678 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985): Starting process 'localhost:8080' with threads=5.

2023-05-27 17:50:32,494 DEBG 'runserver' stderr output:
[Sat May 27 17:50:32.494175 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985): Python home /opt/venv.

2023-05-27 17:50:32,494 DEBG 'runserver' stderr output:
[Sat May 27 17:50:32.494230 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985): Initializing Python.

2023-05-27 17:50:32,515 DEBG 'runserver' stderr output:
[Sat May 27 17:50:32.515504 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985): Attach interpreter ''.

2023-05-27 17:50:32,519 DEBG 'runserver' stderr output:
[Sat May 27 17:50:32.519334 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985): Imported 'mod_wsgi'.
[Sat May 27 17:50:32.519374 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985, process='localhost:8080', application=''): Loading Python script file '/tmp/cvat-server/handler.wsgi'.

2023-05-27 17:50:32,833 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:50:32,834 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-27 17:50:32,835 DEBG 'ssh-agent' stderr output:
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:kx1H/Kp+1xknufZlSVxut3W0wwaC+QBVtwyLgOOLQbs "django@cvat-backend-worker-export-57b7c75c79-679cz" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-27 17:50:32,839 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-27 17:50:33,827 DEBG 'runserver' stderr output:
[Sat May 27 17:50:33.827240 2023] [wsgi:error] [pid 985:tid 139630162835008] [remote 192.168.251.152:32782] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'.
[Sat May 27 17:50:33.827330 2023] [wsgi:error] [pid 985:tid 139630154442304] [remote 192.168.251.158:42802] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'.

2023-05-27 17:50:33,827 DEBG 'runserver' stderr output:
[Sat May 27 17:50:33.827380 2023] [wsgi:error] [pid 985:tid 139630162835008] [remote 192.168.251.158:46290] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'.
[Sat May 27 17:50:33.827391 2023] [wsgi:error] [pid 985:tid 139630146049600] [remote 192.168.251.152:58362] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'.
[Sat May 27 17:50:33.827418 2023] [wsgi:error] [pid 985:tid 139630129264192] [remote 192.168.251.158:41920] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'.
[Sat May 27 17:50:33.827516 2023] [wsgi:info] [pid 985:tid 139630154442304] mod_wsgi (pid=985): Application startup timer triggered 'localhost:8080'.
[Sat May 27 17:50:33.827517 2023] [wsgi:error] [pid 985:tid 139630137656896] [remote 192.168.251.152:56128] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'.

2023-05-27 17:50:33,827 DEBG 'runserver' stderr output:
[Sat May 27 17:50:33.827621 2023] [wsgi:info] [pid 985:tid 139630154442304] mod_wsgi (pid=985): Application startup timer cancelled 'localhost:8080'.

2023-05-27 17:50:37,891 DEBG 'runserver' stderr output:
[Sat May 27 17:50:37.891695 2023] [wsgi:error] [pid 985:tid 139630129264192] [remote 192.168.251.158:37642] [2023-05-27 17:50:37,891] WARNING django.request: Unauthorized: /api/events

2023-05-27 17:50:37,894 DEBG 'runserver' stderr output:
[Sat May 27 17:50:37.893960 2023] [wsgi:error] [pid 985:tid 139630129264192] [remote 192.168.251.158:37642] WARNING:django.request:Unauthorized: /api/events

2023-05-27 17:51:33,887 DEBG 'runserver' stderr output:
[Sat May 27 17:51:33.887711 2023] [wsgi:error] [pid 744:tid 139633253672512] [client 192.168.251.158:49386] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:51:33,888 DEBG 'runserver' stderr output:
[Sat May 27 17:51:33.888291 2023] [wsgi:error] [pid 744:tid 139633253140032] [client 192.168.251.158:49396] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:51:33,891 DEBG 'runserver' stderr output:
[Sat May 27 17:51:33.891583 2023] [wsgi:error] [pid 663:tid 139633253140032] [client 192.168.251.158:46128] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

@ganindu7
Copy link

Screenshot from 2023-05-27 18-54-55

@ganindu7
Copy link

Control panel is working fine

Screenshot from 2023-05-27 18-58-58

my values.override.yaml

  secret:
    password: password***
    postgres_password: password***
    replication_password: password***


analytics:
  enabled: false

Apart from that everything else is stock, (used a NFS storage provisoner)

@ganindu7
Copy link

I Think the cause may be because of
Screenshot from 2023-05-27 19-40-20

@shintaro-matsui
Copy link
Author

@azhavoro

Is it the following steps?
My CVAT root directory seems to have no .env.

Open a new terminal window.
Run VScode from the virtual environment (run the following command from CVAT root directory):
source .env/bin/activate && code
Inside VScode, Open CVAT root dir
Select server: debug configuration and run it (F5) to run REST server and its workers

Results of docker ps

CONTAINER ID   IMAGE                                      COMMAND                   CREATED          STATUS          PORTS                                                                                          NAMES
1e063d97e199   cvat/ui:dev                                "/docker-entrypoint.…"   11 minutes ago   Up 11 minutes   80/tcp                                                                                         cvat_ui
6931beed6959   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_worker_annotation
5a02e6ed0edc   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_worker_import
0484b4928b93   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_worker_export
216f7056cc9a   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_server
9444ef2509f5   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_utils
ad1bb01b1524   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_worker_webhooks
06fabbcbaac1   timberio/vector:0.26.0-alpine              "/usr/local/bin/vect…"   11 minutes ago   Up 11 minutes                                                                                                  cvat_vector
a8b39d7e884f   clickhouse/clickhouse-server:22.3-alpine   "/entrypoint.sh"          11 minutes ago   Up 11 minutes   8123/tcp, 9000/tcp, 9009/tcp                                                                   cvat_clickhouse
32c1ba705e9e   traefik:v2.9                               "/entrypoint.sh --pr…"   11 minutes ago   Up 11 minutes   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 80/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp   traefik
e2a8a54fc330   postgres:15-alpine                         "docker-entrypoint.s…"   11 minutes ago   Up 11 minutes   5432/tcp                                                                                       cvat_db
03df36df1f3f   grafana/grafana-oss:9.3.6                  "sh -euc 'mkdir -p /…"   11 minutes ago   Up 11 minutes   3000/tcp                                                                                       cvat_grafana
d4b97986f685   redis:4.0-alpine                           "docker-entrypoint.s…"   11 minutes ago   Up 11 minutes   6379/tcp                                                                                       cvat_redis
97ef4c98312c   openpolicyagent/opa:0.45.0-rootless        "/opa run --server -…"   11 minutes ago   Up 11 minutes                                                                                                  cvat_opa

Results of docker logs cvat_opa

{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:52Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:52Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:53Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:55Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:58Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:09:02Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:09:08Z"}

@shintaro-matsui
Copy link
Author

I checked #5922 and confirmed that there is no problem with the memory limit.

@pepijnob
Copy link

pepijnob commented Jun 1, 2023

I have the same error. Tried installing it on a new PC, it was working in the beginning, then after deploying Segment anything on Nuclio I got the same error on the new PC as well. When I delete the function in the nuclio dashboard, it is still not working.

@AdimasBarelang
Copy link

I got the same problems, at first it will works fine but then. The error appears to be said the server is error like posr above.

@trandinh
Copy link

trandinh commented Jun 26, 2023

I got the same problems and had already checked and confirmed that there is no problem with the memory.
For me I found out that the problems may related to cvat_db -> there are no table created ( no List of relations found in cvat_db)
but still not yet found out where the problems come from

cvat_db log:


fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
sh: locale: not found
2023-06-26 10:04:26.717 UTC [31] WARNING:  no usable system locales were found
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....2023-06-26 10:05:10.292 UTC [37] LOG:  starting PostgreSQL 15.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit
2023-06-26 10:05:10.584 UTC [37] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
.2023-06-26 10:05:10.960 UTC [40] LOG:  database system was shut down at 2023-06-26 10:04:31 UTC
2023-06-26 10:05:11.312 UTC [37] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down....2023-06-26 10:05:12.280 UTC [37] LOG:  received fast shutdown request
2023-06-26 10:05:12.485 UTC [37] LOG:  aborting any active transactions
2023-06-26 10:05:12.489 UTC [37] LOG:  background worker "logical replication launcher" (PID 43) exited with exit code 1
2023-06-26 10:05:12.489 UTC [38] LOG:  shutting down
2023-06-26 10:05:12.850 UTC [38] LOG:  checkpoint starting: shutdown immediate
...........2023-06-26 10:05:23.427 UTC [38] LOG:  checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.792 s, sync=8.863 s, total=10.938 s; sync files=250, longest=0.350 s, average=0.036 s; distance=4222 kB, estimate=4222 kB
2023-06-26 10:05:23.435 UTC [37] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2023-06-26 10:05:23.842 UTC [1] LOG:  starting PostgreSQL 15.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit
2023-06-26 10:05:23.842 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-06-26 10:05:23.842 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-06-26 10:05:24.307 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-06-26 10:05:24.567 UTC [54] LOG:  incomplete startup packet
2023-06-26 10:05:24.567 UTC [53] LOG:  database system was shut down at 2023-06-26 10:05:23 UTC
2023-06-26 10:05:24.881 UTC [1] LOG:  database system is ready to accept connections
2023-06-26 10:06:28.728 UTC [61] ERROR:  relation "authtoken_token" does not exist at character 363
2023-06-26 10:06:28.728 UTC [61] STATEMENT:  SELECT "authtoken_token"."key", "authtoken_token"."user_id", "authtoken_token"."created", "auth_user"."id", "auth_user"."password", "auth_user"."last_login", "auth_user"."is_superuser", "auth_user"."username", "auth_user"."first_name", "auth_user"."last_name", "auth_user"."email", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."date_joined" FROM "authtoken_token" INNER JOIN "auth_user" ON ("authtoken_token"."user_id" = "auth_user"."id") WHERE "authtoken_token"."key" = '607c2de23bd84928f8a397d3ca8a96a2836351a7' LIMIT 21

look like problems start from CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down....2023-06-26 10:05:12.280 UTC [37] LOG:  received fast shutdown request
2023-06-26 10:05:12.485 UTC [37] LOG:  aborting any active transactions
2023-06-26 10:05:12.489 UTC [37] LOG:  background worker "logical replication launcher" (PID 43) exited with exit code 1
2023-06-26 10:05:12.489 UTC [38] LOG:  shutting down
2023-06-26 10:05:12.850 UTC [38] LOG:  checkpoint starting: shutdown immediate
...........2023-06-26 10:05:23.427 UTC [38] LOG:  checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.792 s, sync=8.863 s, total=10.938 s; sync files=250, longest=0.350 s, average=0.036 s; distance=4222 kB, estimate=4222 kB
2023-06-26 10:05:23.435 UTC [37] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.```

@SjoerdTimovanVliet
Copy link

SjoerdTimovanVliet commented Aug 12, 2023

@shintaro-matsui did you solve it? I have the same issue
I am using ubuntu 22.04 LTS and cvat version v2.6.0 and also had this issue with v2.5.0

@may109
Copy link

may109 commented Aug 20, 2023

same problem here, with v246 and v260. It works at the beginning for few hours and starts to fail, with a slightly different error.

cvat_db                    FATAL:  password authentication failed for user "root"
cvat_db                  DETAIL:  User "root" has no password assigned.
cvat_db                   |     Connection matched pg_hba.conf line 102: "host all all all md5"
cvat_server               | Traceback (most recent call last):
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
cvat_server               |     self.connect()
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
cvat_server               |     return func(*args, **kwargs)
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
cvat_server               |     self.connection = self.get_new_connection(conn_params)
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
cvat_server               |     return func(*args, **kwargs)
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
cvat_server               |     connection = self.Database.connect(**conn_params)
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
cvat_server               |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
cvat_server               | psycopg2.OperationalError: connection to server at "cvat_db" (192.168.16.3), port 5432 failed: fe_sendauth: no password supplied

@SjoerdTimovanVliet
Copy link

Any workarounds found?

@shintaro-matsui
Copy link
Author

Hi.
Unfortunately, this issue has not been resolved.
I am not working on this issue at this time as I reinstalled the OS once due to a number of other issues.

I am sorry that I could not be of help.

@Petros626
Copy link

Petros626 commented Sep 24, 2023

isnt it just docker compose down and docker compose up -d which can help to avoid this. I've read this in some issues here

@bartmch
Copy link

bartmch commented Nov 29, 2023

I exactly have the same issue as the OP. Unfortunately docker compose down/up did not work.

When I ran CVAT as per OP's description for the first time, it ran fine and was able to annotate an image. Later, when I wanted to start with the next image I got the same "Cannot connect to server" error. I deleted everything to start from scratch, but it was giving me the "Cannot connect to server" from the start.

@leotor4
Copy link

leotor4 commented Jan 9, 2024

cvat_server takes time to initialize. It's normal to see the following error message sometimes during initialization:

{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}

Wait for a few minutes, and then check if you can access your application.

*Remember to persist your CVAT_HOST environment variable.

@jtziebarth
Copy link

@bsekachev Just jumping in to say that I have this issue too running v2.16.2. Currently the only solution I know is to wipe docker clean and rebuild all over again, which is very inconvenient. Is there any other solution that exists?

@jpfleischer
Copy link

cvat_server takes time to initialize. It's normal to see the following error message sometimes during initialization:

{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}

Wait for a few minutes, and then check if you can access your application.

*Remember to persist your CVAT_HOST environment variable.

Thanks, the solution was to export CVAT_HOST=mysite.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests